When trying to use json_schema in response_format, wrapper field(schema) cannot be used.

[khe0613]

Open AI's Create chat completion API documentation explicitly states that when using json_schema in response_format, a wrapper field(schema) should be used as follows:

"response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "type": "object",
        "properties": {
          "content": {"type": "string"}
        }
      }
    }
  }

 

However, currently in TensorRT-LLM, when using json_schema in response_format, you must define the json schema directly instead of using wrapper field(schema) . (See #8934)

"response_format": {
    "type": "json_schema",
    "json_schema": {
      "type": "object",
      "properties": {
        "content": {"type": "string"}
      }
    }
  }

 

I have two questions regarding this situation.

1. Is there a reason why TensorRT-LLM is implemented not to use wrapper field(schema)?
2. Do you have any plans to modify it to comply with the Open AI spec?

Thank you 😊

[karljang]

@JunyiXu-nv, could you please review this question? I would greatly appreciate it. Thanks!

[JunyiXu-nv]

Thanks for your report! @khe0613

I filed a PR to fix it: #11497. Could you have a try?