Support for Local LLM Endpoints (LM Studio, Ollama, etc.)

[mattiapintus]

Summary

Add support for custom API endpoints to enable Crystal to work with local LLM servers like LM Studio, Ollama, and other OpenAI-compatible endpoints.

Motivation

Crystal is an excellent tool for managing multiple AI coding sessions in parallel. However, it currently only supports Claude Code (Anthropic) and Codex (OpenAI). Many developers prefer or need to use local LLMs for various reasons:

• Privacy & Security: Keep code and proprietary information completely local
• Cost: Avoid API usage fees, especially for experimentation and testing
• Offline Development: Work without internet connectivity
• Custom Models: Use fine-tuned or specialized models for specific domains
• Compliance: Meet organizational requirements that prohibit sending code to external services

Proposed Solution

Configuration Options

Add a new assistant type (e.g., "Custom OpenAI-Compatible") with configurable settings:

- Base URL (e.g., http://localhost:1234/v1)
- Model name/identifier
- API key (optional, for servers that require it)
- Custom headers (optional)

Compatible Services

This would enable Crystal to work with popular local LLM servers:

• LM Studio: Desktop app for running local LLMs with OpenAI-compatible API
• Ollama: CLI tool for running local models
• text-generation-webui: Web interface for various LLM backends
• LocalAI: Drop-in replacement for OpenAI API
• llama.cpp server: Direct llama.cpp HTTP server
• Any other OpenAI-compatible endpoint

UI Mockup

In the session creation flow, add an assistant type selector:

• Claude Code (Anthropic)
• Codex (OpenAI)
• Custom Endpoint ← NEW
  • When selected, show fields for base URL, model, etc.

Implementation Considerations

Minimal Change Approach

Since LM Studio and similar tools provide OpenAI-compatible APIs, the implementation could leverage existing OpenAI integration with minimal changes:

1. Add a new assistant type enum value
2. Add configuration fields for custom endpoints
3. Modify API client initialization to use custom base URL when appropriate
4. Update UI to show custom endpoint configuration

Feature Parity

Some features might not be available with all local models:

• Model capabilities vary (context length, tool use, etc.)
• Some endpoints might not support all OpenAI API features
• Consider adding optional feature detection or warnings

Testing

Local models can be unpredictable, so consider:

• Connection testing before session creation
• Clear error messages for unsupported features
• Graceful degradation when certain capabilities aren't available

Benefits

1. Broader User Base: Attract developers who must or prefer to use local models
2. Development & Testing: Allow Crystal development without API costs
3. Flexibility: Users can choose the best model for their specific use case
4. Educational Use: Students and learners can use Crystal without API costs
5. Compliance: Enable use in environments with strict data policies

Alternative Approaches

1. Plugin System: Instead of built-in support, create a plugin architecture for custom assistants
2. Proxy Mode: Allow Crystal to proxy through a custom middleware that translates to local LLMs
3. Configuration File: Support advanced configuration through a JSON/YAML file

Additional Context

• LM Studio API documentation: https://lmstudio.ai/docs/local-server
• OpenAI API compatibility is a common standard in the local LLM ecosystem
• Similar tools (Cursor, Continue, etc.) already support custom endpoints

Questions for Maintainers

• Would you prefer this as a built-in feature or a plugin system?
• Are there specific local LLM providers you'd like to prioritize?
• Would you accept a PR implementing this feature?

---

Thank you for considering this feature request! Crystal is already an amazing tool, and local LLM support would make it accessible to an even wider community of developers.

[YehyunLee]

This is awesome @btcleet
thanks for making ticket on this topic!

[richardchen874-sys]

The part that stands out to me is "Support for Local LLM Endpoints (LM Studio, Ollama, etc.)". That is usually where agent workflow routing starts to matter in practice, because the integration works at small scale but gets harder once real users, retries, and longer sessions show up.

I would make provider/model choice explicit per agent step, then log usage and fallback reason for each tool loop. That makes it easier to move simple steps to cheaper routes while keeping stronger models for planning or final synthesis.

I am testing an OpenAI-compatible multi-model API layer around official Chinese models, so this kind of agent workflow routing is directly relevant. For crystal, would routing be configured per agent role, per tool call, or dynamically from runtime signals?