Add support for OpenAI Chat Completions API compatible third party providers & local models

[nathan-t4]

Add support for local models on CADAM (e.g. hosted by LMStudio, llama.cpp, etc.), as mentioned in #120.

To add a new local model, the user adds local-models.json to the base repo. Detailed instructions are in the README.md

Features

1. Does not break compatibility (without the new local-models.json, everything works as normal)
2. Not all model providers have multimodal tool results for VLMs (https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling). In fact, all OpenAI chat completions compatible third-party model providers (e.g. LMStudio, llama.cpp, vllm, ollama, etc...) do not support multimodal tool results. Thus, the tool result message format is modified from the original hydratedMessages:

return {
  type: 'content' as const,
  value: [
    { type: 'text' as const, text },
    {
      type: 'image-data' as const,
      data: downloaded.base64,
      mediaType: downloaded.mediaType,
    },
  ],
};

to a text-only tool output APPENDED WITH a text and image user messages that includes the multi-view rendering.

result.push({
  role: 'user',
  parts: [
    {
      type: 'text',
      text: 'Multi-view inspection render from the tool result above.',
    },
    {
      type: 'file',
      mediaType: downloaded.mediaType,
      url: `data:${downloaded.mediaType};base64,${downloaded.base64}`,
    },
  ],
});

For more details, see changes in src/server/aiChat.ts.

3. Conversation suggestions and titles can be generated with third-party models that have "useForAux": true, instead of just the default claude haiku model.

Tested with DeepSeek v4 Flash (hosted by DeepSeek) -- a pure text LLM! :

---

Summary by cubic

Adds first‑class support for OpenAI Chat Completions–compatible local/third‑party models (LM Studio, ollama, llama.cpp, vllm) via local-models.json, surfaced in the picker through /api/local-models. Also fixes strict local VLM tool‑image handling by appending the inspection render as a follow‑up user message, routes titles/suggestions to a local aux model when available, caps model picker height, and makes local/third‑party runs free and skip token gates.

• New Features

  • Catalog (local-models.json): entries support id, name, description, baseUrl, optional apiKey (env var name), supportsTools, supportsThinking, supportsVision, supportsForcedToolChoice, useForAux; only entries with baseUrl are active; file is gitignored.
  • API + UI: new /api/local-models exposes picker‑safe configs (no secrets); client hook useParametricModels merges local entries with cloud; useIsLocalModel removes UI token gating for local runs; UI vision checks use the dynamic list; model pickers are scrollable with a capped height.
  • Routing: anthropic/*, google/* stay direct; catalog ids with a baseUrl route to a per‑target @openrouter/ai-sdk-provider client with compatibility: "compatible" and normalized /v1; others go via OpenRouter; per‑model API keys are resolved server‑side from .env.local.
  • Tool results: cloud models with vision receive inline inspection images; local/strict servers send text‑only tool output and the latest inspection image is appended as a user message (storage‑checked), gated by supportsVision.
  • Behavior: forced tool_choice is disabled for Claude 5 and catalog models with supportsForcedToolChoice: false (fallback: disable when supportsThinking: true); local/third‑party models are billed at 0, skip preflight token checks, and do not consume tokens; selecting a catalog id without a baseUrl returns a clear error.
  • Aux tasks: conversation titles/suggestions use a catalog model marked useForAux when available; else fall back to Anthropic when configured.
  • Tests: added coverage for catalog parsing/routing, supportsForcedToolChoice, and latest‑build detection.

• Migration

  • Optional: copy local-models.json.example to local-models.json, set each model’s id, baseUrl, and flags.
  • If a provider needs auth, set apiKey to the env var name (e.g., DEEPSEEK_API_KEY) and add that variable to .env.local.
  • Restart the dev server; models with a baseUrl appear in the picker. Cloud routes remain unchanged.

^{Written for commit cbffde4. Summary will update on new commits.}

[vercel]

@nathan-t4 is attempting to deploy a commit to the Adam Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

Adds first-class support for OpenAI Chat Completions-compatible local/third-party models (LM Studio, ollama, vllm, DeepSeek, etc.) via an optional local-models.json catalog. The feature is fully backward-compatible and ships with thorough unit tests, a clean server/client separation (API keys stay server-side), and correctly resolved all issues from the previous review round.

• New catalog + routing: shared/localModels.ts handles pure routing logic; src/server/localChatConfig.ts wires disk I/O and env-var key resolution; a new /api/local-models endpoint serves picker-safe configs to the client.
• Tool result hydration: local VLM vision is delivered by appending the inspection render as a follow-up role: user message in buildHydratedMessages, bypassing the strict OpenAI-compatible tool-message format; findLatestBuildToolCallId ensures only the newest build render is re-downloaded per request.
• Billing + token gating: local model runs return 0 billing tokens, skip the preflight balance check, and disable the "limit reached" UI gate; aux tasks (title, suggestions) route through a catalog entry marked useForAux when available.

Confidence Score: 4/5

Safe to merge after adding error logging to the loadCatalogFromDisk catch block.

The one defect that breaks expected behavior is in localChatConfig.ts: when local-models.json contains invalid JSON the catch block returns [] and caches it for the server lifetime with no log output. A developer with a minor syntax error will see all local models disappear silently with no diagnostic path. All previously raised issues have been correctly addressed in this revision.

src/server/localChatConfig.ts - the catch block in loadCatalogFromDisk needs a log call before returning the empty fallback.

Important Files Changed

Filename	Overview
shared/localModels.ts	New module: pure routing logic for local model catalog - parsing, provider routing, URL normalization, picker config mapping. Well-tested, no sensitive data exposed.
src/server/localChatConfig.ts	New server module: reads and caches local-models.json, resolves API keys from env. Silent catch on JSON parse error caches [] with no logging, making misconfigured files impossible to diagnose.
src/server/aiChat.ts	Adds local model routing via createOpenRouter with compatibility mode, buildHydratedMessages for injecting inspection images as follow-up user messages, and getAuxLanguageModel. Dead initial assignment in toModelOutput else branch; otherwise logic is sound.
src/routes/api/local-models.ts	New unauthenticated GET endpoint returning picker-safe catalog configs; no secrets exposed but endpoint lacks auth unlike all other API routes.
src/hooks/useLocalModels.ts	New hooks: useParametricModels merges cloud + local models, useIsLocalModel checks model identity via the /api/local-models endpoint. Correct staleTime, proper schema validation.
shared/localModels.test.ts	New test file with thorough coverage of all exported functions including getLocalModels blank-URL filtering, supportsForcedToolChoice parsing, and picker config security (apiKey/baseUrl not exposed).
src/server/messageUtils.ts	Adds findLatestBuildToolCallId - walks messages newest-to-oldest to find the last completed build tool call. Clean implementation with good test coverage.
src/lib/utils.ts	Renames PARAMETRIC_MODELS to CLOUD_PARAMETRIC_MODELS internally, re-exports as spread copy, and adds optional models parameter to parametricModelSupportsVision. Clean refactor.
local-models.json.example	New example catalog file with two well-documented entries; valid JSON (no trailing comma), includes supportsForcedToolChoice: false example for the DeepSeek entry.
shared/imageRefs.ts	Adds inspectionPreviewStoragePath helper, replacing the inline string template that was previously duplicated between aiChat.ts and ChatSession.tsx.

_{Reviews (10): Last reviewed commit: "Add supportsForcedToolChoice for third p..." | Re-trigger Greptile}

[cubic-dev-ai]

5 issues found across 9 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>