Agent-Specific CLI Generation with Tool-to-Command Mapping

[Ash-Blanc]

Feature Summary

Enable automatic generation of dedicated CLI tools for each agent project created with better-agents. When creating an agent (e.g., "customer-support-agent"), the framework should provide an option to generate a corresponding CLI (e.g., customer-support) that:

• Exposes all agent tools as individual CLI commands
• Includes universal/fundamental commands common to all better-agents projects
• Provides a production-ready interface for agent interaction outside of development environments

Problem/Use Case

Current State:
Agents built with better-agents are excellent for development and testing but lack a streamlined way to be deployed as production-ready CLI tools. Developers must manually create CLIs or API wrappers to expose agent functionality to end users or integrate with existing workflows.

Real-World Scenarios:

1. DevOps Agent: An agent with tools for deploy, rollback, check-health, view-logs should be usable as devops-agent deploy --env production rather than through code or API calls

2. Data Analysis Agent: Tools like analyze_dataset, generate_report, visualize become CLI commands: data-agent analyze --file data.csv --output report.pdf

3. CI/CD Integration: Agents with code review capabilities should be callable as code-reviewer review --pr 123 --repo org/repo in GitHub Actions or GitLab CI

4. System Administration: An agent with backup, restore, monitor tools becomes a first-class system utility

5. Multi-Agent Orchestration: Different agents (research, writer, editor) with individual CLIs can be chained in bash scripts or workflows

Proposed Solution

Core Implementation

During project initialization:

better-agents init my-customer-agent --with-cli

# Or add to existing project
better-agents add cli --name customer-support

Generated CLI structure:

my-customer-agent/
├── app/                          # Agent code
│   └── agent.py
├── cli/                          # Generated CLI
│   ├── __init__.py
│   ├── commands/
│   │   ├── agent_tools.py       # Auto-generated from agent tools
│   │   └── universal.py         # Common better-agents commands
│   ├── main.py                  # CLI entry point
│   └── config.py                # CLI configuration
├── pyproject.toml               # Includes CLI entry point
└── README.md

Agent Tools → CLI Commands Mapping

Automatic Tool Detection:

The framework uses AST parsing to extract tool definitions and type hints from agent code, automatically generating Click/Typer commands that map to agent tools.

CLI Usage:

# Install CLI
cd my-customer-agent
pip install -e .  # or uv pip install -e .

# Use agent-specific commands
customer-support search-knowledge-base "how to reset password" --limit 5
customer-support create-ticket "Login issue" "User cannot access account" --priority high

# Interactive mode
customer-support chat --interactive
customer-support chat --input "Help me troubleshoot email delivery"

Universal/Fundamental Commands

Common to ALL better-agents project CLIs:

Project Management:

<agent-cli> info                    # Show agent configuration, tools, version
<agent-cli> test                    # Run agent test scenarios
<agent-cli> validate                # Validate agent configuration and prompts
<agent-cli> lint                    # Check agent code quality

Development:

<agent-cli> dev                     # Start development server with hot-reload
<agent-cli> shell                   # Interactive Python shell with agent loaded
<agent-cli> debug --trace           # Run with detailed execution tracing

Prompts:

<agent-cli> prompts list            # List all prompts
<agent-cli> prompts push <name>     # Push prompt to LangWatch
<agent-cli> prompts pull <name>     # Pull latest prompt version
<agent-cli> prompts diff <name>     # Compare local vs remote

Scenarios:

<agent-cli> scenarios run           # Run all scenario tests
<agent-cli> scenarios run <name>    # Run specific scenario
<agent-cli> scenarios record        # Record new scenario from interaction

Evaluations:

<agent-cli> eval list               # List evaluation notebooks
<agent-cli> eval run <notebook>     # Execute evaluation notebook

Deployment:

<agent-cli> build                   # Build for production
<agent-cli> export --format api     # Export as REST API spec
<agent-cli> export --format docker  # Generate Dockerfile
<agent-cli> export --format lambda  # Package for AWS Lambda

Observability:

<agent-cli> logs --tail             # Stream agent execution logs
<agent-cli> metrics                 # Show performance metrics
<agent-cli> trace <execution-id>    # View detailed trace

CLI Configuration

# cli/config.yaml (auto-generated)
cli:
  name: customer-support
  version: 0.1.0
  
agent:
  path: app/agent.py
  class: CustomerSupportAgent
  
commands:
  tool_mapping:
    enabled: true
    prefix: ""  # Optional prefix for tool commands
    
  universal:
    enabled: true
    groups:
      - project
      - prompts  
      - scenarios
      - evaluations
      
  custom:
    - name: deploy
      script: scripts/deploy.sh
      description: Deploy agent to production
      
output:
  format: json  # json | text | yaml
  color: true
  verbose: false

CLI Frameworks Support

Support multiple CLI framework backends:

better-agents init my-agent --with-cli --cli-framework click
better-agents init my-agent --with-cli --cli-framework typer
better-agents init my-agent --with-cli --cli-framework argparse

Framework Characteristics:

• Click: Rich features, widely adopted, excellent documentation
• Typer: Modern, type-hints based, great for Python 3.7+
• Argparse: Stdlib, no dependencies, simple deployments

Implementation Roadmap

Phase 1: MVP (v0.1.0)

• Basic CLI scaffolding with --with-cli flag
• Auto-generate commands from agent @tool decorators
• Universal commands: info, test, dev
• Click framework support
• Simple JSON/text output formats

Phase 2: Enhanced Features (v0.2.0)

• Typer and argparse framework support
• Full universal commands suite (prompts, scenarios, evaluations)
• Interactive chat mode
• Configuration file support (cli/config.yaml)
• Custom command registration

Phase 3: Production Ready (v0.3.0)

• Export to Docker/Lambda/API
• Authentication and authorization
• Rate limiting and error handling
• Telemetry and observability
• Multi-agent CLI orchestration

Phase 4: Advanced (v0.4.0+)

• Auto-completion (bash/zsh/fish)
• Plugin system for custom commands
• CLI marketplace/registry
• Cross-platform binaries (PyInstaller/cx_Freeze)

Benefits

1. Production Deployment: Agents become first-class CLI tools usable in any environment
2. CI/CD Integration: Easy integration into automated workflows
3. Developer Experience: Consistent interface across all better-agents projects
4. Discoverability: --help automatically documents all agent capabilities
5. Automation: Enable bash scripting and workflow orchestration
6. Testing: CLI commands are testable with standard testing frameworks
7. Standardization: Universal commands ensure consistent patterns

Alternative Solutions

1. Manual CLI Creation: Users create CLIs themselves - inconsistent patterns, more work
2. API-Only Approach: Expose agents only via HTTP APIs - requires additional infrastructure
3. Jupyter Notebooks: Use notebooks for interaction - not suitable for automation
4. Generic Agent CLI: One CLI for all agents - loses agent-specific customization

Technical Considerations

• Use AST parsing to extract tool definitions and type hints
• Support for Pydantic models as CLI arguments with automatic validation
• Error handling with user-friendly messages
• Logging integration with better-agents observability
• Environment variable support for API keys and configuration

Related Work

This feature builds on proven patterns from:

• AWS CLI: aws s3 cp maps to S3 service operations
• Kubectl: kubectl get pods maps to Kubernetes API operations
• GitHub CLI: gh pr create maps to GitHub API operations
• Stripe CLI: stripe customers create maps to Stripe API operations

[Ash-Blanc]

Implementation Plan - Agent-Specific CLI Generation (MVP Phase)

Based on the feature requirements and architectural analysis, here's the comprehensive implementation plan:

Architecture Overview

The feature introduces a new better_agents.cli_gen submodule with 6 core components:

1. cli_gen/parser.py - AST-based tool extractor
2. cli_gen/generator.py - Jinja2-based CLI script generator
3. cli_gen/models.py - Pydantic CLI argument models
4. cli_gen/commands.py - Universal command definitions
5. cli_gen/integrator.py - Framework integration hook
6. cli_gen/templates/cli_template.j2 - Click CLI skeleton

Core Module Responsibilities

parser.py: AST-based tool extraction from agent code. Identifies @tool decorators, extracts function signatures, type hints, defaults.

generator.py: Jinja2 template rendering. Converts tool metadata to CLI code. Supports Click (MVP), Typer and Argparse (Phase 2).

models.py: Pydantic schemas. ToolSpec for tool metadata, CliConfig for generation settings. Auto-generates Pydantic models from function sigs.

commands.py: Universal Click commands - info(), test(), dev() in MVP phase.

integrator.py: Main pipeline orchestrator. generate_cli(agent, lib='click', output_dir=None) function.

templates/cli_template.j2: Click CLI skeleton with tool command loops and universal command group.

Integration Points

Integration Points

• Agent class: Add with_cli parameter to Agent.build(with_cli=False)
• Builder: Call generate_cli() during build if flag is True
• Entry point: Auto-add to pyproject.toml scripts section
• No breaking changes: Feature is opt-in via flag

Testing Strategy

• Unit tests for each module independently
• Integration tests for full pipeline
• Verify generated CLI is executable
• Test all universal commands (info, test, dev)

Estimated Effort

• Total LOC: ~500-600 for MVP (excluding tests)
• Time estimate: 6-8 hours for implementation
• Dependencies: jinja2, click, pydantic (already in better-agents)

Next Steps

Ready to implement MVP with detailed architecture from Grok analysis. Will create feature branch with complete implementation, tests, and documentation. Phase 2 (Typer/Argparse support + full universal commands) to follow after MVP review.

Recommendation: Approve architecture and proceed with implementation.

[Ash-Blanc]

✅ MVP Code Generated by Grok - Ready for PR

Status: All 6 modules + Jinja2 template generated with complete production-ready code.

Files to Create (All code generated and verified):

1. src/better_agents/cli_gen/init.py - Package init
2. src/better_agents/cli_gen/models.py - Pydantic schemas (ToolSpec, CliConfig)
3. src/better_agents/cli_gen/parser.py - AST-based tool extraction from @tool decorators
4. src/better_agents/cli_gen/commands.py - Universal Click commands (info, test, dev)
5. src/better_agents/cli_gen/generator.py - Jinja2 CLI code generator
6. src/better_agents/cli_gen/integrator.py - Main orchestrator pipeline
7. src/better_agents/cli_gen/templates/cli_template.j2 - Click CLI skeleton template

Implementation Pipeline:

Agent (@tool) -> Parser (AST) -> Models (Pydantic) -> Commands (Click) -> Generator (Jinja2) -> CLI Script

Features Implemented:

✅ AST-based automatic @tool decorator detection
✅ Pydantic auto-model generation from function signatures
✅ Universal commands: info (metadata), test (run), dev (interactive shell)
✅ Jinja2 template rendering with Click subcommands
✅ Full error handling and type annotations
✅ Standalone executable CLI generation
✅ pyproject.toml integration

Code Quality:

• Production-ready (full implementations, not examples)
• Type-hinted throughout
• Comprehensive docstrings
• Error handling for edge cases
• Follows better-agents conventions

To Create PR:

All code is ready in Grok conversation. Can now create feature branch feat/agent-cli-generation and implement these 7 files directly into the repository.

[Ash-Blanc]

🚀 QUICK IMPLEMENTATION GUIDE - Create & Push PR in 5 minutes

FASTEST APPROACH: Copy from Grok conversation

1. All production-ready code is in Grok conversation:

   • https://grok.com/c/5cc58ad6-6a3a-4059-be29-930946fb2134?rid=58fae2ad-3f6c-4b2f-b79d-2a33d731c4de

2. Create 7 files in your fork:

   • src/better_agents/cli_gen/__init__.py
   • src/better_agents/cli_gen/models.py
   • src/better_agents/cli_gen/parser.py
   • src/better_agents/cli_gen/commands.py
   • src/better_agents/cli_gen/generator.py
   • src/better_agents/cli_gen/integrator.py
   • src/better_agents/cli_gen/templates/cli_template.j2

3. Create PR:

   • From your fork branch to langwatch/better-agents:main
   • Title: feat: Agent-Specific CLI Generation with Tool-to-Command Mapping
   • Description: Closes #71
   • Add body from original issue description

All code sections available in:

• Grok Response: Complete implementations for all 6 Python modules + 1 Jinja2 template
• This Issue: Feature spec + architecture + testing guide
• Branch created: feat/agent-cli-generation ready on Ash-Blanc fork

Status:

✅ Feature Specification Complete
✅ Architecture Finalized
✅ Production Code Generated
✅ Branch Ready
⏳ Waiting for file creation and PR submission