API_OPENAI VLM engine is not Azure OpenAI GPT-5 compatible due to hardcoded generation params

[janosbabik]

Bug

When using Docling picture description through the OpenAI-compatible API VLM engine against Azure OpenAI GPT-5 chat deployments, Docling sends request parameters that Azure rejects.

Observed issues:

• max_tokens is sent, but Azure requires max_completion_tokens for this model
• temperature=0.0 is sent, but this deployment only accepts the default temperature value

This makes picture description fail unless downstream users monkeypatch Docling internals.

Relevant areas appear to be:

• docling/models/inference_engines/vlm/api_openai_compatible_engine.py
• docling/models/stages/picture_description/picture_description_vlm_engine_model.py

The failure path appears to be:

• picture description input is created with temperature=0.0 and max_new_tokens=200
• OpenAI-compatible API engine maps max_new_tokens to max_tokens
• Azure rejects the request

Typical Azure errors:

• Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.
• Unsupported value: 'temperature' does not support 0.0 with this model. Only the default (1) value is supported.

Steps to reproduce

1. Configure Docling picture description with PictureDescriptionVlmEngineOptions.from_preset(...)
2. Use ApiVlmEngineOptions(engine_type=VlmEngineType.API_OPENAI, ...)
3. Point the URL to an Azure OpenAI chat completions endpoint, for example:
   https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=<version>
4. Use a GPT-5 chat deployment that supports image input
5. Run document conversion with picture description enabled
6. Observe request failure from Azure OpenAI because of unsupported generation parameters

Docling version

2.78.0

Python version

3.12

[dosubot]

Hi @janosbabik! I'm Dosu and I’m helping the docling team.

This is a confirmed issue. I found that Issue #2523 tracks a related bug where the temperature parameter cannot be overridden in the VLM pipeline.

The root causes are:

1. Hardcoded parameters in picture description: In picture_description_vlm_engine_model.py (lines 135-137), temperature=0.0 and max_new_tokens=200 are hardcoded, ignoring the existing generation_config option infrastructure.

2. Parameter merge order bug: Even when users try to override via ApiVlmEngineOptions.params, the default temperature overwrites user values due to incorrect merge order.

3. No Azure parameter mapping: The api_openai_compatible_engine.py translates max_new_tokens → max_tokens, but Azure GPT-5 requires max_completion_tokens.

For Azure reasoning models specifically, the temperature must be completely omitted (not just set to 1.0), which the current implementation doesn't support.

No merged PRs address this yet. You may want to link this issue to #2523 since they share the temperature override problem, and both would benefit from a fix that makes generation parameters configurable and supports Azure-specific parameter naming.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[majiayu000]

Hi, I've investigated this and the root cause is in ApiVlmEngine._process_single_input() where user/model params are applied before engine defaults, so temperature from VlmEngineInput (hardcoded 0.0) always overrides user params, and max_tokens is always used regardless of user preferences.

Fix: Reverse the param merge order so user/model spec params take precedence over engine defaults. Additionally, when max_completion_tokens is present in user params, the conflicting max_tokens is automatically removed.

This allows Azure OpenAI users to configure:

ApiVlmEngineOptions(
    params={"model": "gpt-5", "max_completion_tokens": 4096, "temperature": 1.0}
)

PR: #3116