rewrite docs

Author: isaacbmiller

README.md

@@ -1,68 +1,55 @@
 # dspy-cli
 
-A command-line interface tool for creating and serving DSPy projects, inspired by Ruby on Rails.
+Deploy DSPy programs as HTTP APIs with standard containerization, routing, and OpenAPI specifications. Setup time: 3-5 minutes.
 
-## Installation
+**For:** Developers embedding AI features in existing applications (Chrome extensions, Notion automations, web apps) who need HTTP endpoints without manual infrastructure work.
 
-```bash
-uv add dspy-cli
-```
-
-### Installing for Development/Testing
-
-If you're testing or developing dspy-cli itself:
+**Problem:** Prototype DSPy modules work locally but require Docker configs, API boilerplate, and route management to deploy. This creates friction between development and production.
 
-```bash
-# Clone or navigate to the dspy-cli repository
-cd /path/to/dspy-cli
-
-# Sync dependencies
-uv sync --extra dev
-
-# Now the dspy-cli command is available
-dspy-cli --help
-```
+**Solution:** Convention-based project structure with auto-discovery. Define signatures, implement modules, deploy. Infrastructure handled automatically.
 
 ## Quick Start
 
-### Create a new DSPy project
-
 ```bash
-dspy-cli new my-project
-cd my-project
-```
+# Install
+uv tool install dspy-cli
 
-### Create a project with a custom program name
+# Create project
+dspy-cli new blog-tagger -s "post -> tags: list[str]"
+cd blog-tagger
 
-```bash
-dspy-cli new my-project -p custom_program
+# Serve locally
+dspy-cli serve
 ```
 
-### Create a project with a custom signature
+Test the endpoint:
 
 ```bash
-dspy-cli new blog-tagger -s "post -> tags: list[str]"
+curl -X POST http://localhost:8000/BlogTaggerPredict \
+  -H "Content-Type: application/json" \
+  -d '{"post": "How to build Chrome extensions with AI..."}'
 ```
 
-### Serve your DSPy programs as an API
+Response:
 
-```bash
-dspy-cli serve --port 8000 --host 0.0.0.0
+```json
+{
+  "tags": ["chrome-extensions", "ai", "development", "javascript"]
+}
 ```
 
 ## Features
 
-- **Project scaffolding**: Generate a complete DSPy project structure with boilerplate code
-- **Code generation**: Quickly scaffold new DSPy programs with signatures and modules using Rails-style generators
-- **Convention over configuration**: Organized directory structure for modules, signatures, optimizers, and metrics
-- **HTTP API server**: Automatically serve your DSPy programs as REST endpoints
-- **Flexible configuration**: YAML-based model configuration with environment variable support
-- **Logging**: Request logging to both STDOUT and per-module log files
+- **Auto-discovery** - Modules in `src/*/modules/` automatically exposed as HTTP endpoints
+- **Standard containers** - Docker image generation with FastAPI server included
+- **OpenAPI specs** - Auto-generated from DSPy signatures for integration with tools and clients
+- **Hot reload** - Edit modules without restarting the development server
+- **Model configuration** - Switch LLMs via config file without code changes
+- **MCP tool integration** - Served programs work as MCP tools for AI assistants
+- **Convention-based structure** - Organized directories for modules, signatures, optimizers, metrics
 
 ## Project Structure
 
-When you create a new project, dspy-cli generates the following structure:
-
 ```
 my-project/
 ├── pyproject.toml
@@ -84,85 +71,51 @@ my-project/
 
 ## Commands
 
-### `new`
-
-Create a new DSPy project with boilerplate structure.
+### Create Project
 
 ```bash
-dspy-cli new [PROJECT_NAME] [OPTIONS]
+dspy-cli new PROJECT_NAME [OPTIONS]
 ```
 
-**Options:**
-- `-p, --program-name TEXT`: Name of the initial program (default: converts project name)
-- `-s, --signature TEXT`: Inline signature string (e.g., `"question -> answer"` or `"post -> tags: list[str]"`)
+Options:
+- `-s, --signature TEXT` - Inline signature (e.g., `"question -> answer"`)
+- `-p, --program-name TEXT` - Initial program name
 
-**Examples:**
-
-```bash
-# Basic project
-dspy-cli new my-project
-
-# With custom program name
-dspy-cli new my-project -p custom_program
-
-# With custom signature
-dspy-cli new blog-tagger -s "post -> tags: list[str]"
-
-# With both program name and signature
-dspy-cli new analyzer -p text_analyzer -s "text, context: list[str] -> summary, sentiment: bool"
-```
-
-### `generate` (alias: `g`)
-
-Generate new components in an existing DSPy project.
+### Generate Components
 
 ```bash
 dspy-cli generate scaffold PROGRAM_NAME [OPTIONS]
-dspy-cli g scaffold PROGRAM_NAME [OPTIONS]
 ```
 
-**Options:**
-- `-m, --module TEXT`: DSPy module type to use (default: Predict)
-  - Available: `Predict`, `ChainOfThought` (or `CoT`), `ProgramOfThought` (or `PoT`), `ReAct`, `MultiChainComparison`, `Refine`
-- `-s, --signature TEXT`: Inline signature string (e.g., `"question -> answer"`)
+Options:
+- `-m, --module TEXT` - Module type (`Predict`, `ChainOfThought`, `ReAct`)
+- `-s, --signature TEXT` - Inline signature (e.g. "question -> answer")
 
-**Examples:**
+### Serve Locally
 
 ```bash
-# Basic scaffold with default Predict module
-dspy-cli g scaffold categorizer
-
-# Scaffold with ChainOfThought
-dspy-cli g scaffold categorizer -m CoT
-
-# Scaffold with custom signature
-dspy-cli g scaffold qa -m CoT -s "question -> answer"
-
-# Complex signature with types
-dspy-cli g scaffold search -s "query, context: list[str] -> answer, confidence: float"
+dspy-cli serve [--port PORT] [--host HOST]
 ```
 
-### `serve`
+Auto-generated endpoints:
+- `GET /programs` - List all programs with schemas
+- `POST /{program}` - Execute program with JSON payload
 
-Start an HTTP API server that exposes your DSPy programs.
+### Deploy
 
 ```bash
-dspy-cli serve [OPTIONS]
+flyctl launch
+flyctl secrets set OPENAI_API_KEY=your-key-here
+flyctl deploy
 ```
 
-**Options:**
-- `--port INTEGER`: Port to run the server on (default: 8000)
-- `--host TEXT`: Host to bind to (default: 0.0.0.0)
-
-**Endpoints:**
-- `GET /programs`: List all discovered programs with their schemas
-- `POST /{program}`: Execute a program with JSON payload
+See [Deployment Guide](docs/deployment.md) for detailed instructions and other platforms.
 
 ## Configuration
 
-### dspy.config.yaml
+### Model Settings
 
-Configure your language models and routing:
+`dspy.config.yaml`:
 
 ```yaml
 models:
@@ -174,25 +127,91 @@ models:
       max_tokens: 16000
       temperature: 1.0
       model_type: chat
-    anthropic:sonnet-4-5:
-      model: anthropic/claude-sonnet-4-5
-      env: ANTHROPIC_API_KEY
-      model_type: chat
 
-# Optional: per-program model overrides
+# Per-program overrides
 program_models:
   MySpecialProgram: anthropic:sonnet-4-5
 ```
 
-### .env
+### Environment Variables
 
-Store your API keys and secrets:
+`.env`:
 
 ```
 OPENAI_API_KEY=sk-...
 ANTHROPIC_API_KEY=sk-ant-...
 ```
 
+See [Configuration Guide](docs/configuration.md) for complete settings reference.
+
+## Integration Examples
+
+### Chrome Extension
+
+```javascript
+async function summarizePage(content) {
+  const response = await fetch('https://your-app.fly.dev/summarizer', {
+    method: 'POST',
+    body: JSON.stringify({ content }),
+    headers: { 'Content-Type': 'application/json' }
+  });
+  return response.json();
+}
+```
+
+### Python Client
+
+```python
+import requests
+
+def tag_content(page_content):
+    response = requests.post(
+        'https://your-app.fly.dev/BlogTaggerPredict',
+        json={'post': page_content}
+    )
+    return response.json()['tags']
+```
+
+### JavaScript/TypeScript
+
+```javascript
+fetch('https://your-app.fly.dev/analyzer', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({ 
+    text: document.body.innerText,
+    context: ['documentation', 'technical'] 
+  })
+})
+.then(res => res.json())
+.then(data => console.log(data.summary, data.sentiment));
+```
+
+## Development Installation
+
+For contributing or testing dspy-cli itself:
+
+```bash
+cd /path/to/dspy-cli
+uv sync --extra dev
+dspy-cli --help
+```
+
+Run tests:
+
+```bash
+cd dspy-cli
+uv run pytest
+```
+
+## Next Steps
+
+- [Architecture Overview](docs/architecture.md) - Project structure and design decisions
+- [CLI Reference](docs/cli-reference.md) - Complete command documentation
+- [Configuration Guide](docs/configuration.md) - Model settings and environment variables
+- [Deployment Guide](docs/deployment.md) - Production deployment workflows
+- [MCP Integration](docs/mcp-integration.md) - Using with AI assistants
+
 ## License
 
 MIT