Skip to content

fix(azure): omit temperature for unrecognised deployment names - #3018

Open
Aftabbs wants to merge 4 commits into
confident-ai:mainfrom
Aftabbs:fix/azure-temperature-unknown-model
Open

fix(azure): omit temperature for unrecognised deployment names#3018
Aftabbs wants to merge 4 commits into
confident-ai:mainfrom
Aftabbs:fix/azure-temperature-unknown-model

Conversation

@Aftabbs

@Aftabbs Aftabbs commented Aug 8, 2026

Copy link
Copy Markdown

Summary

Fixes #3015.

Azure users commonly pass a deployment name (e.g. gpt5-prod-eu, my-gpt4-deployment) as the model argument instead of the canonical model name. When that string is absent from the OPENAI_MODELS_DATA catalogue, ModelDataRegistry.get() falls back to DeepEvalModelData() whose supports_temperature defaults to True, so temperature=0.0 is forwarded to the API. Newer Azure-hosted models (GPT-5 family and any future model) reject the temperature parameter and return a 400 error.

Root cause

# azure_model.py (before)
if self.model_data and self.model_data.supports_temperature is False:
    temperature = None

The guard only fires when supports_temperature is explicitly False (i.e. a known reasoning model). An unrecognised deployment name returns the default DeepEvalModelData() with supports_temperature=True, so the guard never fires and temperature is sent regardless.

Fix

# azure_model.py (after)
if self.model_data.supports_temperature is False or model not in OPENAI_MODELS_DATA:
    temperature = None

Two cases now correctly suppress temperature:

  1. Known reasoning modelssupports_temperature=False explicitly (o-series).
  2. Unknown deployment names – not present in the catalogue; we cannot confirm support, so we omit temperature defensively.

Known standard models (gpt-4o, gpt-4-turbo, etc.) remain unaffected because they are in the catalogue.

Test plan

  • TestAzureModelTemperature::test_unknown_deployment_name_temperature_is_none – unrecognised model name → temperature is None by default.
  • TestAzureModelTemperature::test_unknown_deployment_name_explicit_temperature_also_none – user-supplied temperature still dropped for unrecognised names.
  • All 4 pre-existing temperature tests continue to pass (no regression for gpt-4o or o3-mini).
tests/test_core/test_models/test_azure_model.py::TestAzureModelTemperature - 6 passed
tests/test_core/test_models/test_azure_model.py - 32 passed

AzureOpenAI users frequently pass a deployment name (e.g. "gpt5-prod-eu")
as the `model` argument.  When that string is absent from the model
catalogue, `ModelDataRegistry.get()` returns a default `DeepEvalModelData`
whose `supports_temperature` is `True`, so temperature was forwarded to the
API — causing 400 errors for newer models that do not accept it.

Extend the existing temperature guard to also cover models that are not
found in the catalogue: we cannot confirm temperature support for unknown
deployment names, so we omit it defensively.

Closes confident-ai#3015
@vercel

vercel Bot commented Aug 8, 2026

Copy link
Copy Markdown

@Aftabbs is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

Aftabbs added 3 commits August 9, 2026 20:18
Reformat the temperature-suppression condition so black passes: wrap the
two-clause `or` onto separate lines inside parentheses (line was 91 chars,
black limit is 88).  Also move the `model not in OPENAI_MODELS_DATA` check
first to make the short-circuit intent clearer — when the deployment name
is absent from the catalogue we suppress temperature immediately without
inspecting model_data.

Logic is unchanged from the previous commit: unknown Azure deployment names
(not in OPENAI_MODELS_DATA) and known reasoning models (supports_temperature
is False) both get temperature=None.

Signed-off-by: Aftabbs <aftabbs.wwe@gmail.com>
Pre-existing lint failure: .scripts/release.py was not black-compliant,
causing CI to fail on all open PRs. Format-only change, no logic change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AzureOpenAIModel defaults temperature to 0.0, causing 400 errors with GPT-5 models

1 participant