tito: vLLM backend kind — token-exact turns via render/generate/derender

[Meirtz]

Why

The TITO gateway's token-recording chat flow was hardwired to sglang's dialect (input_ids request extension + meta_info.output_token_logprobs). vLLM's chat-completions endpoint silently ignores unknown request fields (extra="allow"), so pointing the gateway at vLLM would "work" while quietly re-tokenizing messages server-side — exactly the retokenization drift TITO exists to prevent.

What

A backend-kind adapter layer, selected by backend_kind (TITOGatewayConfig / --backend-kind {sglang,vllm}, default sglang):

• engine/upstream.py — UpstreamAdapter protocol (validate_request + chat_turn). The sglang flow moved here verbatim; session_app.py now runs prepare → adapter.chat_turn → checkpoint and knows no backend-specific fields.
• vllm kind (requires vLLM ≥ 0.24.0, the first release with the derender endpoints) — the token-native three-call chain per turn:
  1. POST /v1/chat/completions/render — the agent's chat request → GenerateRequest, resolving sampling params server-side; its from-scratch token ids are discarded,
  2. POST /inference/v1/generate — runs on the session's exact accumulated prompt ids; the response carries the exact completion ids (stop/EOS included),
  3. POST /v1/chat/completions/derender — token output → chat completion via the server-side tool/reasoning parsers (chat_request + model + prompt_tokens forwarded).
• Forcing mirrors the sglang path: non-streaming, logprobs on (per-token cross-check + recorded rollout logprobs), an explicit top_logprobs: null coerced to 0. finish_reason is rewritten to "tool_calls" for parity with vLLM's own chat endpoint (derender passes it through verbatim). Upstream errors pass through verbatim; malformed 200 bodies surface as clean 502s.
• Adapter request preconditions (vllm's required model) run before the session lock — phase 1 can commit a rollback, and a rejected retry must leave no side effects.
• Reasoning dialect: message_matches treats reasoning_content (sglang/DeepSeek) and reasoning (vLLM) as one logical key and tolerates absence on either side (clients drop the field on echo); the vllm adapter mirrors reasoning into reasoning_content on the stored trajectory copy so templates and the mismatch audit can read it.
• VLLM_BASE_URL joins the discovery env vars; README documents the backend kinds and the vLLM server parser flags.

All wire shapes were verified line-by-line against the vLLM v0.24.0 release tag (notably: the released GenerateResponse has no usage field — the gateway counts tokens from choices[].token_ids and supplies prompt_tokens to derender itself; render copies stream from the chat request — re-forced off before generate).

Tests

25 new tests: an HTTP-level suite (test_gateway_vllm_http.py) driving the real FastAPI app against a v0.24.0-faithful fake replica (render/generate/derender, including the no-usage response shape and the logprobs↔top_logprobs mapping), plus config/CLI/discovery coverage. Existing sglang tests unchanged and green.

🤖 Generated with Claude Code