Add Dynamic Tool System with Lua Integration

CHANGELOG.md

@@ -23,6 +23,29 @@ All notable changes to this project will be documented in this file.
 - **Multi-Transport Support**: Server now supports both stdio (default) and
   HTTP server transport modes for different integration scenarios
 
+### Experimental Features (Unstable)
+
+⚠️ **Warning**: The following features are experimental and unstable. They may
+change significantly or be removed in future versions without prior notice.
+Use at your own risk.
+
+- **Dynamic Tool System**: Sophisticated dynamic tool registration system through
+  `HybridToolRouter` enabling extensible tool functionality without code changes
+- **Connection-Scoped Tools**: Tools automatically registered/unregistered with
+  connection lifecycle for enhanced modularity
+- **Tool Visibility**: Enhanced tool visibility through new resource system
+- **Tool Registration Resources**: New `nvim-tools://` URI scheme for monitoring
+  tool availability and connection mappings
+- **HybridToolRouter**: Combines static tools (from `#[tool_router]` macro) with
+  dynamic tools using lock-free concurrent data structures
+- **Conflict Resolution**: Prevents naming conflicts between static and dynamic tools
+- **Dynamic Routing**: Enhanced modular architecture with dynamic routing
+  capabilities
+- **Tool Registration API**: Improved extensibility through dynamic tool
+  registration API
+
+### Architecture Improvements
+
 ### New CLI Options
 
 - `--connect <MODE>` - Connection mode: 'manual' (default), 'auto', or specific

CLAUDE.md

@@ -92,21 +92,40 @@ The codebase follows a modular architecture with clear separation of concerns:
 
 - **`src/server/core.rs`**: Core infrastructure and server foundation
   - Contains `NeovimMcpServer` struct and core methods
+  - Integrates `HybridToolRouter` for static and dynamic tool management ⚠️ **(Experimental)**
   - Manages multiple concurrent connections via
     `Arc<DashMap<String, Box<dyn NeovimClientTrait + Send>>>`
   - Handles multi-connection lifecycle with deterministic connection IDs
   - Provides utility functions (BLAKE3 hashing, socket discovery, etc.)
   - Error conversion between `NeovimError` and `McpError`
+  - Exposes dynamic tool registration API for connection-scoped tools ⚠️ **(Experimental)**
 
 - **`src/server/tools.rs`**: MCP tool implementations
   - Implements 23 MCP tools using the `#[tool]` attribute
   - Contains parameter structs for tool requests
   - Focuses purely on MCP tool logic and protocol implementation
   - Clean separation from core infrastructure
 
+- **`src/server/hybrid_router.rs`** ⚠️ **(Experimental)**: Dynamic tool routing system
+  - Implements `HybridToolRouter` that combines static and dynamic tools
+  - Supports connection-scoped dynamic tool registration
+  - Provides automatic tool lifecycle management with connection cleanup
+  - Uses lock-free concurrent data structures for high performance
+  - Enables extensibility while maintaining backwards compatibility
+
+- **`src/server/lua_tools.rs`** ⚠️ **(Experimental)**: Lua custom tool
+  integration system
+  - Implements `LuaToolConfig` structure for Lua-defined tool configuration
+  - Provides `discover_lua_tools()` function for automatic tool discovery from Neovim
+  - Implements `LuaToolValidator` for JSON Schema parameter validation
+  - Handles conversion between Neovim msgpack values and JSON for seamless integration
+  - Provides `discover_and_register_lua_tools()` for automatic registration
+    during connection setup
+  - Implements error handling and response conversion for Lua MCP responses
+
 - **`src/server/resources.rs`**: MCP resource handlers
   - Implements `ServerHandler` trait for MCP capabilities
-  - Provides server metadata, tool discovery, and resource handling
+  - Uses `HybridToolRouter` for dynamic tool discovery and execution
   - Supports `nvim-diagnostics://` URI scheme for diagnostic resources
   - Handles resource listing and reading operations
 
@@ -125,7 +144,12 @@ The codebase follows a modular architecture with clear separation of concerns:
 This modular architecture provides several advantages:
 
 - **Clear Separation of Concerns**: Core infrastructure, MCP tools, and
-  resource handlers are cleanly separated
+  resource handlers are cleanly separated (dynamic routing ⚠️
+  **experimental**)
+- **Extensibility**: `HybridToolRouter` enables dynamic tool registration
+  without code changes ⚠️ **(Experimental)**
+- **Performance**: Lock-free concurrent data structures for high-throughput
+  tool routing ⚠️ **(Experimental)**
 - **Easier Maintenance**: Each file has a single, well-defined responsibility
 - **Better Testing**: Components can be tested independently with focused unit tests
 - **Improved Readability**: Developers can quickly find relevant code based on functionality
@@ -137,10 +161,12 @@ This modular architecture provides several advantages:
 ### Data Flow
 
 1. **MCP Communication**: stdio/HTTP transport ↔ MCP client ↔ `NeovimMcpServer`
-2. **Neovim Integration**: `NeovimMcpServer` → `NeovimClientTrait` → `nvim-rs` →
+2. **Tool Routing** ⚠️ **(Experimental)**: MCP tool request →
+   `HybridToolRouter` → static/dynamic tool execution
+3. **Neovim Integration**: `NeovimMcpServer` → `NeovimClientTrait` → `nvim-rs` →
    TCP/Unix socket → Neovim instance
-3. **Tool Execution**: MCP tool request → async Neovim API call → response
-4. **Resource Access**: MCP resource request → diagnostic data retrieval →
+4. **Tool Execution**: Routed tool call → async Neovim API call → response
+5. **Resource Access**: MCP resource request → diagnostic data retrieval →
    structured JSON response
 
 ### Connection Management
@@ -150,6 +176,8 @@ This modular architecture provides several advantages:
 - **Deterministic connection IDs** generated using BLAKE3 hash of target string
 - **Connection isolation**: Each connection operates independently with
   proper session isolation
+- **Dynamic tool lifecycle** ⚠️ **(Experimental)**: Connection-scoped tools
+  automatically cleaned up on disconnect
 - **Proper cleanup** of TCP connections and background tasks on disconnect
 - **Connection validation** before tool execution using connection ID lookup
 
@@ -203,11 +231,11 @@ The server provides these 23 tools (implemented with `#[tool]` attribute):
 6. **`lsp_code_actions`**: Get LSP code actions with universal document
    identification (supports buffer IDs, project-relative paths, and absolute paths)
 7. **`lsp_hover`**: Get LSP hover information with universal document
-    identification (supports buffer IDs, project-relative paths, and absolute paths)
+   identification (supports buffer IDs, project-relative paths, and absolute paths)
 8. **`lsp_document_symbols`**: Get document symbols with universal document
-    identification (supports buffer IDs, project-relative paths, and absolute paths)
+   identification (supports buffer IDs, project-relative paths, and absolute paths)
 9. **`lsp_references`**: Get LSP references with universal document
-    identification (supports buffer IDs, project-relative paths, and absolute paths)
+   identification (supports buffer IDs, project-relative paths, and absolute paths)
 10. **`lsp_resolve_code_action`**: Resolve code actions that may have
     incomplete data
 11. **`lsp_apply_edit`**: Apply workspace edits using Neovim's LSP utility
@@ -253,6 +281,16 @@ The server provides connection-aware resources via multiple URI schemes:
 - **`nvim-connections://`**: Lists all active Neovim connections with
   their IDs and targets
 
+**Tool Registration Overview** ⚠️ **(Experimental)**:
+
+- **`nvim-tools://`**: Overview of all tools and their connection mappings,
+  showing static tools (available to all connections) and dynamic tools
+  (connection-specific)
+- **`nvim-tools://{connection_id}`**: List of tools available for a specific
+  connection, including both static and connection-specific dynamic tools
+
+*Note: Tool registration resources are experimental and may change in future versions.*
+
 **Connection-Scoped Diagnostics** via `nvim-diagnostics://` URI scheme:
 
 - **`nvim-diagnostics://{connection_id}/workspace`**: All diagnostic
@@ -264,6 +302,160 @@ Resources return structured JSON with diagnostic information including severity,
 messages, file paths, and line/column positions. Connection IDs are deterministic
 BLAKE3 hashes of the target string for consistent identification.
 
+## Dynamic Tool System ⚠️ **(Experimental)**
+
+⚠️ **Warning**: The dynamic tool system is experimental and unstable.
+It may change significantly or be removed in future versions without
+prior notice. Use at your own risk in production environments.
+
+The server includes a sophisticated dynamic tool registration system through
+`HybridToolRouter` with comprehensive Lua integration for user-extensible
+custom tools:
+
+### HybridToolRouter Architecture
+
+**Core Design:**
+
+- **Dual Tool Support**: Combines static tools (from `#[tool_router]` macro)
+  with dynamic tools
+- **Connection-Scoped Tools**: Tools that are automatically
+  registered/unregistered with connection lifecycle
+- **Conflict Resolution**: Prevents naming conflicts between static and dynamic tools
+- **Performance Optimized**: Lock-free concurrent access using `Arc<DashMap>`
+- **Clean Tool Names**: Connection-specific tools maintain clean names without prefixes
+- **Tool Visibility**: New resource system provides insight into
+  tool-connection mappings
+
+**Key Components:**
+
+```rust
+pub struct HybridToolRouter {
+    /// Static tools from #[tool_router] macro
+    static_router: ToolRouter<NeovimMcpServer>,
+
+    /// Dynamic tools using nested structure:
+    /// tool_name -> connection_id -> tool
+    dynamic_tools: Arc<DashMap<String, DashMap<String, DynamicToolBox>>>,
+
+    /// Connection-specific tool mapping: connection_id -> tool_names
+    connection_tools: Arc<DashMap<String, HashSet<String>>>,
+}
+```
+
+### Lua Dynamic Tools System
+
+**User-Extensible Tool Registration:**
+
+The system now supports custom tool registration through Neovim
+configuration using Lua functions:
+
+```lua
+-- In your Neovim configuration (init.lua or plugin config)
+require("nvim-mcp").setup({
+    custom_tools = {
+        save_buffer = {
+            description = "Save a specific buffer by ID",
+            parameters = {
+                type = "object",
+                properties = {
+                    buffer_id = {
+                        type = "integer",
+                        description = "The buffer ID to save",
+                        minimum = 1,
+                    },
+                },
+                required = { "buffer_id" },
+            },
+            handler = function(params)
+                -- Tool implementation using MCP helpers
+                return MCP.success({
+                    buffer_id = params.buffer_id,
+                    message = "Buffer saved successfully",
+                })
+            end,
+        },
+    },
+})
+```
+
+**MCP Helper Functions:**
+
+The Lua plugin provides helper functions for creating compatible MCP responses:
+
+- `MCP.success(data)`: Create successful tool response
+- `MCP.error(code, message, data)`: Create error response
+- `MCP.text(text)`: Create plain text response
+- `MCP.json(data)`: Create JSON response
+
+### Dynamic Tool Registration
+
+**Connection-Scoped Tools:**
+
+- Tools registered with format: `{connection_id}_{tool_name}`
+- Automatically cleaned up when connection disconnects
+- Isolated per connection to prevent interference
+- Perfect for connection-specific functionality
+
+**Tool Lifecycle Management:**
+
+1. **Configuration**: Users define tools in Neovim setup with Lua
+   handlers
+2. **Discovery**: MCP server queries Neovim for registered tools via
+   `get_registered_tools()`
+3. **Registration**: Tools are registered in `HybridToolRouter` during
+   connection setup
+4. **Validation**: JSON Schema validation using `jsonschema` crate for
+   parameter checking
+5. **Execution**: Routed through `HybridToolRouter` with priority:
+   dynamic → static
+6. **Cleanup**: Connection-scoped tools automatically removed on
+   disconnect
+
+### Integration with Core Server
+
+**NeovimMcpServer Integration:**
+
+```rust
+impl NeovimMcpServer {
+    /// Register a connection-specific tool with clean name
+    pub fn register_dynamic_tool(
+        &self,
+        connection_id: &str,
+        tool: DynamicToolBox,
+    ) -> Result<(), McpError>
+
+    /// Remove all dynamic tools for a connection
+    pub fn unregister_dynamic_tools(&self, connection_id: &str)
+}
+```
+
+**Lua Tool Integration:**
+
+The server automatically discovers and registers Lua-defined tools through
+the new `lua_tools.rs` module:
+
+- **`LuaToolConfig`**: Structure for Lua tool configuration and execution
+- **`discover_lua_tools()`**: Queries Neovim for configured custom tools
+- **`discover_and_register_lua_tools()`**: Automatic registration during
+  connection setup
+- **JSON Schema Validation**: Parameter validation using `jsonschema` crate
+- **Response Conversion**: Seamless conversion between Lua MCP responses
+  and Rust `CallToolResult`
+
+**Benefits:**
+
+- **User Extensibility**: Users can define custom tools without modifying
+  server code
+- **Project-Specific Workflows**: Different Neovim instances can have
+  different tool sets
+- **Automatic Discovery**: Tools are discovered and registered during
+  connection setup
+- **Parameter Validation**: Robust input validation using JSON Schema
+- **Error Handling**: Structured error reporting with safe Lua execution
+- **Connection Isolation**: Tools are scoped to specific connections
+- **Performance**: Efficient tool routing with minimal overhead
+- **Reliability**: Automatic cleanup prevents resource leaks
+
 ## Key Dependencies
 
 - **`rmcp`**: MCP protocol implementation with stdio transport, streamable
@@ -280,6 +472,13 @@ BLAKE3 hashes of the target string for consistent identification.
 - **`regex`**: Pattern matching for connection-scoped resource URI parsing
 - **`blake3`**: Fast, deterministic hashing for connection ID generation
 
+**Dynamic Tool System Dependencies** ⚠️ **(Experimental)**:
+
+- **`jsonschema`**: JSON Schema validation for Lua custom tool parameters
+- **`serde_json`**: JSON serialization/deserialization with enhanced
+  support for Lua integration
+- **`async-trait`**: Async trait support for dynamic tool execution
+
 **HTTP Server Transport Dependencies:**
 
 - **`hyper`**: High-performance HTTP library for HTTP server transport
@@ -289,10 +488,10 @@ BLAKE3 hashes of the target string for consistent identification.
 **Testing and Development Dependencies:**
 
 - **`tempfile`**: Temporary file and directory management for integration tests
-- **`serde_json`**: JSON serialization/deserialization with enhanced Claude
-  Code compatibility
 - **Enhanced deserialization**: Support for both string and struct formats
   in CodeAction and WorkspaceEdit types
+- **Lua tool testing** ⚠️ **(Experimental)**: Integration tests for custom tool registration
+  and execution
 
 ## Testing Architecture
 
@@ -335,7 +534,7 @@ To add a new connection-aware tool to the server:
 5. **Testing**: Update integration tests in `src/server/integration_tests.rs`
 
 6. **Registration**: The tool is automatically registered by the
-   `#[tool_router]` macro
+   `#[tool_router]` macro and handled through `HybridToolRouter` ⚠️ **(Experimental)**
 
 **New Tool Parameter Structures:**
 
@@ -418,12 +617,37 @@ standard rustfmt conventions.
 
 ## Neovim Lua Plugin
 
-The project includes a Neovim Lua plugin at `lua/nvim-mcp/init.lua` that:
+The project includes a comprehensive Neovim Lua plugin at
+`lua/nvim-mcp/init.lua` that:
+
+**Core RPC Functionality:**
 
 - Automatically starts a Neovim RPC server on a Unix socket/named pipe
 - Generates unique pipe paths based on git root and process ID
 - Provides a `setup()` function for initialization
 - Enables seamless MCP server connection without manual TCP setup
 
+**Custom Tool Registration** ⚠️ **(Experimental)**:
+
+- Supports user-defined custom tools through `custom_tools` configuration
+- Maintains a global tool registry (`_tool_registry`) for configured tools
+- Provides `get_registered_tools()` API for MCP server tool discovery
+- Implements `execute_tool()` API for safe tool execution with error handling
+
+**MCP Helper Functions:**
+
+- `MCP.success(data)`: Create successful tool responses
+- `MCP.error(code, message, data)`: Create structured error responses
+- `MCP.text(text)`: Create plain text responses
+- `MCP.json(data)`: Create JSON responses
+
+**Tool Management Features:**
+
+- Automatic tool validation during setup
+- Safe tool execution using `pcall()` for error handling
+- Structured error reporting with MCP-compatible format
+- Connection-scoped tool isolation
+
 This eliminates the need to manually start Neovim with `--listen` for MCP
-server connections.
+server connections and enables users to define project-specific custom tools
+through their Neovim configuration.