[Build]: [GPU] Max single allocation size exceeded when compiling OpenVINO model on Intel GPU

[jason11489]

OpenVINO Version

2026.0.0

Operating System

WSL2 for Windows

Hardware Architecture

x86 (64 bits)

Target Platform

• OS: Windows (WSL2 with Ubuntu 24.04)
• Device: Intel Integrated GPU (iGPU)
• OpenVINO version: 2026.0.0

Build issue description

I am trying to run a PyTorch model using OpenVINO by converting it to IR format and compiling it on an Intel GPU.

However, I encounter a single allocation memory limit issue during compilation.

Build script or step-by-step to reproduce

config = {"GPU_ENABLE_LARGE_ALLOCATIONS": "YES"}
compiled = core.compile_model(ov_model_for_compile, "GPU", config)

Expected Behavior

The 1GB single memory allocation limit should be removed, so that allocations larger than 1GB (e.g., 1.2GB) can be handled and compiled without issues.
Using the GPU_ENABLE_LARGE_ALLOCATIONS environment variable or config option should allow large models to be compiled and run on Intel GPU without errors.

Current Behavior

When allocating more than 1GB, the following error occurs and compilation fails.
The same issue occurs even when using the GPU_ENABLE_LARGE_ALLOCATIONS environment variable or config option.

Relevant log output

Exception from src/inference/src/dev/plugin.cpp:53:
Check '!exceed_allocatable_mem_size' failed at src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cpp:112:
[GPU] Exceeded max size of memory object allocation: requested 1244659712 bytes, but max alloc size supported by device is 1073741824 bytes.Please try to reduce batch size or use lower precision.

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.

[Sahilbhatane]

I was able to reproduce this issue locally on my setup. The allocation failure still happens even when GPU_ENABLE_LARGE_ALLOCATIONS is provided, which matches the behavior described here.

From debugging, it looks like the flag either isn’t being interpreted correctly (especially when passed as a string like "YES"), or it’s not propagating properly down to the point where check_allocatable enforces the device max allocation size.

My current plan to fix this is:

• First verify the config handling path (especially bool vs string handling in Python and C++ APIs)

• Confirm whether enable_large_allocations is actually set in ExecutionConfig during model compilation

• Based on that:

  • If it’s a parsing issue → fix config handling / normalize accepted values
  • If it’s propagation → fix the property flow into the GPU runtime
  • If behavior is intentional → align docs to reflect actual enforcement logic

@mlukasze or @openvino-dev-samples can I work on this now? i have reproduced it and planning above way to fix it.

[Sahilbhatane]

• If it’s a parsing issue → fix config handling / normalize accepted values
• If it’s propagation → fix the property flow into the GPU runtime
• If behavior is intentional → align docs to reflect actual enforcement logic

From above 3 option first 2 of them are proper and working as intended, as such {"GPU_ENABLE_LARGE_ALLOCATIONS": "YES"} is actually handled correctly by ov::Any (string → bool conversion works as expected), so the issue isn’t with config parsing, so the propagation issue.

That only made me to change docs to properly guide user to this if anyone gets same failure with this three small changes:

• Improved the assertion message: fixed a minor formatting issue and added GPU_ENABLE_LARGE_ALLOCATIONS as a suggested workaround so users immediately know what to try
• Updated the internal option description (options.inl) from “>4gb” to “exceed the device max allocation size” to reflect actual behavior
• Clarified the public API doc (properties.hpp) to explicitly mention that this flag bypasses the CL_DEVICE_MAX_MEM_ALLOC_SIZE check, not just the >4GB USM path

With these changes, the behavior stays the same but the guidance is much clearer when users hit this limit.

I’ll open a focused PR with these updates NOW.

[Sahilbhatane]

@jason11489 , do check the PR, it will help you solve the issue for now, after merge you will be able to see docs guide and error guide changes properly when max single allocation size error hits.

[jason11489]

@Sahilbhatane Thank you for your kind response. However, I think I didn't explain my situation in enough detail. I want to remove the 1GB limit on single memory allocations and be able to allocate more than 1GB. For this, I have tried using GPU_ENABLE_LARGE_ALLOCATIONS = "YES".

What I expect:
Allocating more than 1GB works without any issues.

Current situation:
When allocating more than 1GB, I get the error: [GPU] Exceeded max size of memory object allocation

[Sahilbhatane]

Thanks for the details @jason11489 . The [GPU] Exceeded max size of memory object allocation message comes from a check in the GPU plugin that should be skipped when large allocations are enabled on the engine handling that allocation.

So if you’re still seeing this exact error, it usually means large allocations weren’t actually enabled for that particular compile/run. This can happen if the option wasn’t applied at compile_model, you’re using a different OpenVINO build, or a cached compiled model was created without the hint.

To help narrow it down, could you share:

1. The exact OpenVINO version (or commit if you built from source)
2. A minimal snippet showing how you’re setting the option (Python or C++)
3. Whether compiled model caching is enabled - if yes, it’s worth trying a clean compile after setting the property to rule out stale cache

If the property is definitely set at compile time and the issue still reproduces on a recent build, then it likely points to a propagation issue in that path, and it would be worth opening a separate issue with a small reproducer.

If it is in fact the propagation issue then let the current PR get merge for better docs Guide (won't hurt to get better error logs), and will it be ok if I create new issue referring this issue and comment above for propagation path issue?

[jason11489]

Thanks for the detailed response! @Sahilbhatane

Here's my setup and findings:

OpenVINO version: 2026.0.0

Test case 1 – GPU_ENABLE_LARGE_ALLOCATIONS does NOT work

The following code converts a PyTorch model with a large parameter (~1GB) to OpenVINO IR and compiles it with GPU_ENABLE_LARGE_ALLOCATIONS: YES. Despite setting the property and clearing the compiled model cache, I still get the [GPU] Exceeded max size of memory object allocation error.

import torch
import torch.nn as nn
import numpy as np
import openvino as ov
 

class LargeTensorAdder(nn.Module):
    def __init__(self, num_elements: int = 256 * 1024 * 1024, dtype=torch.float32):
        super().__init__()
        bytes_per_element = torch.tensor([], dtype=dtype).element_size()
        total_bytes = num_elements * bytes_per_element
        print(f"[LargeTensorAdder] Large tensor size: {total_bytes / (1024**3):.2f} GB "
              f"({num_elements:,} elements, {bytes_per_element} bytes/element)")
 
        self.large_param = nn.Parameter(torch.zeros(num_elements, dtype=dtype))
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        num_input_elements = x.numel()
        param_slice = self.large_param[:num_input_elements].view(x.shape)
        return x + param_slice
 
 

def export_and_compile(
    num_elements: int = 256 * 1024 * 1024,
    input_shape: list = [1, 1024],
    device: str = "GPU",         # "CPU" / "GPU" / "NPU"
):

    model = LargeTensorAdder(num_elements=num_elements, dtype=torch.float32)
    model.eval()
 
    dummy_input = torch.ones(input_shape, dtype=torch.float32)
 
    try:
        core = ov.Core()
        ov_model = ov.convert_model(
            model,
            example_input=dummy_input, 
        )
        print("  [OK] ov.convert_model success")
    except Exception as e:
        print(f"  [FAIL] ov.convert_model fail: {e}")
        raise
 
 
 
    try:
        compile_config= {"GPU_ENABLE_LARGE_ALLOCATIONS": "YES"} 
        compiled_model = core.compile_model(ov_model,"GPU",compile_config)
        print(f"  [OK] {device} compile success")
    except Exception as e:
        print(f"  [FAIL] {device} compile fail: {e}")
        raise
 
    return compiled_model, core
 
 def run_inference(compiled_model, input_shape: list = [1, 1024]):
    print("\n[Step 6] Inference Test...")
 
    infer_request = compiled_model.create_infer_request()
 
    input_data = np.ones(input_shape, dtype=np.float32)
 
    infer_request.infer({0: input_data})
    output = infer_request.get_output_tensor(0).data
 
    print(f"  Input  shape : {input_data.shape}")
    print(f"  Output shape : {output.shape}")
    print(f"  Output[:5]   : {output.flatten()[:5]}")

    return output
 

if __name__ == "__main__":
    INPUT_SHAPE   = [1, 1024]

    TARGET_GB = 3
    NUM_ELEMENTS  = int(TARGET_GB * (1024**3) / 4)
    DEVICE        = "GPU"    # "CPU" / "GPU" / "NPU"
 

    compiled_model, core = export_and_compile(
        num_elements=NUM_ELEMENTS,
        input_shape=INPUT_SHAPE,
        device=DEVICE,
    )
 

    output = run_inference(compiled_model, input_shape=INPUT_SHAPE)

Test case 2 – GPU_ENABLE_LARGE_ALLOCATIONS WORKS

When building the model directly using OpenVINO opsets (instead of converting from PyTorch), the property works as expected and compilation succeeds without the error.

import openvino as ov
import numpy as np
import os
 
def main():
    core = ov.Core()
   
    num_elements = 300_000_000
    shape = [1, num_elements]
   
    print(f"Generating dummy data of size: {num_elements * 4 / (1024**3):.2f} GB")
    data = np.random.rand(*shape).astype(np.float32)
 
    param = ov.opset13.parameter(shape, np.float32, name="input")
    const = ov.opset13.constant(data, np.float32, name="large_constant")
    add = ov.opset13.add(param, const)
    model = ov.Model([add], [param], "LargeMemoryModel")
 
    print("Compiling model to GPU... (This will trigger GPU memory allocation)")
    config = {"GPU_ENABLE_LARGE_ALLOCATIONS": "YES"}
   
    compiled_model = core.compile_model(model, "GPU")
    print("Model compilation finished. Check GPU memory usage (e.g., using 'intel_gpu_top' or 'nvidia-smi').")
 
if __name__ == "__main__":
    main()

The key difference seems to be the model source — the property appears to be properly propagated when using native OpenVINO opsets, but not when the model is converted from PyTorch via ov.convert_model().

Could you create a new issue referencing this one and the comment above to track it separately?

[Sahilbhatane]

Appreciate you putting together a clean repro and especially the comparison between PyTorch convert_model and a native OpenVINO opset model - it made the root cause a lot easier to isolate.

What’s actually happening here after looking in In ProgramBuilder::ProgramBuilder(..., model, engine, config, ...) (src/plugins/intel_gpu/src/plugin/program_builder.cpp) is that GPU_ENABLE_LARGE_ALLOCATIONS is getting applied too late in the flow. The large Constant tensors are pushed to the GPU during ProgramBuilder, which runs before the GPU program object is initialized. Right now, the “large allocations” flag only kicks in during program construction, so those early allocations still go through the default limits. That lines up with the failure you’re seeing, even when the property is set and the cache is cleared.

The fix on our side is to enable enable_large_allocations on the engine right at the start of ProgramBuilder, before any constant allocate_memory calls happen. That way, the early uploads follow the same allocation policy as the rest of the pipeline. This is independent of the error message and documentation improvements, which we’ll also clean up.

I'm tracking this as a runtime compile-path issue (linked / to be filed). so ill create one now referring this issue and comments in that as well.

And yeah, the minimal scripts you shared were exactly what I needed here- made debugging a lot more straightforward thanks for that @jason11489 .

till that time If you want local fix I can provide you that here if you want?

[jason11489]

Thank you for your kind explanation and for creating a new issue! @Sahilbhatane

I'm curious about the solution. Is the way to solve this problem to change

src/plugins/intel_gpu/include/intel_gpu/runtime/options.inl

OV_CONFIG_RELEASE_OPTION(ov::intel_gpu::hint, enable_large_allocations, false, "Allow buffer allocations that exceed the device max allocation size. Enabling this option may lead to performance degradation")

to true and then build OpenVINO from source? Or is there a simpler solution?

[jason11489]

@Sahilbhatane I understood what you explained, and I first tried running a case with GPU_ENABLE_LARGE_ALLOCATIONS = "YES" applied. After that, I ran the torch model, and it worked without any issues. It seems like the problem you mentioned was correct.

import torch
import torch.nn as nn
import numpy as np
import openvino as ov
import gc


core = ov.Core()
core.set_property({"GPU_ENABLE_LARGE_ALLOCATIONS": "YES"})

shape = [1, 4]
param = ov.opset13.parameter(shape, np.float32, name="input")
const = ov.opset13.constant(np.ones(shape, dtype=np.float32), np.float32, name="const")
add = ov.opset13.add(param, const)
model = ov.Model([add], [param], "SimpleAddModel")

compiled_model = core.compile_model(model, "GPU")
print("Model compiled successfully.")

class LargeTensorAdder(nn.Module):
    def __init__(self, num_elements: int = 256 * 1024 * 1024, dtype=torch.float32):
        super().__init__()
        bytes_per_element = torch.tensor([], dtype=dtype).element_size()
        total_bytes = num_elements * bytes_per_element
        print(f"[LargeTensorAdder] Large tensor size: {total_bytes / (1024**3):.2f} GB "
              f"({num_elements:,} elements, {bytes_per_element} bytes/element)")
 
        self.large_param = nn.Parameter(torch.zeros(num_elements, dtype=dtype))
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        num_input_elements = x.numel()
        param_slice = self.large_param[:num_input_elements].view(x.shape)
        return x + param_slice
 
 

def export_and_compile(
    num_elements: int = 256 * 1024 * 1024,
    input_shape: list = [1, 1024],
    device: str = "GPU",         # "CPU" / "GPU" / "NPU"
):

    model = LargeTensorAdder(num_elements=num_elements, dtype=torch.float32)
    model.eval()
 
    dummy_input = torch.ones(input_shape, dtype=torch.float32)
 
    try:
        
        ov_model = ov.convert_model(
            model,
            example_input=dummy_input, 
        )
        print("  [OK] ov.convert_model success")
    except Exception as e:
        print(f"  [FAIL] ov.convert_model fail: {e}")
        raise
 
 
 
    try:
        compile_config= {"GPU_ENABLE_LARGE_ALLOCATIONS": "YES"} 
        compiled_model = core.compile_model(ov_model,"GPU",compile_config)
        print(f"  [OK] {device} compile success")
    except Exception as e:
        print(f"  [FAIL] {device} compile fail: {e}")
        raise
 
    return compiled_model, core
 
def run_inference(compiled_model, input_shape: list = [1, 1024]):
    print("\n[Step 6] Inference Test...")
 
    infer_request = compiled_model.create_infer_request()
 
    input_data = np.ones(input_shape, dtype=np.float32)
 
    infer_request.infer({0: input_data})
    output = infer_request.get_output_tensor(0).data
 
    print(f"  Input  shape : {input_data.shape}")
    print(f"  Output shape : {output.shape}")
    print(f"  Output[:5]   : {output.flatten()[:5]}")

    return output
 

if __name__ == "__main__":
    INPUT_SHAPE   = [1, 1024]

    TARGET_GB = 3
    NUM_ELEMENTS  = int(TARGET_GB * (1024**3) / 4)
    DEVICE        = "GPU"    # "CPU" / "GPU" / "NPU"
 

    compiled_model, core = export_and_compile(
        num_elements=NUM_ELEMENTS,
        input_shape=INPUT_SHAPE,
        device=DEVICE,
    )
 

    output = run_inference(compiled_model, input_shape=INPUT_SHAPE)