[Dashboard][Feat] add natural language mode to DSL builder#1687
Merged
[Dashboard][Feat] add natural language mode to DSL builder#1687
Conversation
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned: 📁
|
Contributor
✅ Supply Chain Security Report — All Clear
Scanned at |
Contributor
Performance Benchmark ResultsComponent benchmarks completed successfully. Summary
DetailsSee attached benchmark artifacts for detailed results and profiles. Performance testing powered by vLLM Semantic Router |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a natural-language (“NL”) workflow for the Dashboard Config Builder, backed by a new backend generation endpoint, while also expanding the website’s algorithm documentation (including math rendering) and doing a broad logging/verbosity cleanup in the router.
Changes:
- Add NL mode to the Dashboard Builder UI + store, calling a new backend endpoint to generate DSL + base YAML and show AI review feedback.
- Add KaTeX/remark-math/rehype-katex support and substantially expand algorithm tutorial pages (including a new MLP page and sample config).
- Refactor router logging toward lower verbosity + structured “component” events, plus add a startup summary event.
Reviewed changes
Copilot reviewed 57 out of 58 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| website/src/css/custom.css | Import KaTeX stylesheet for math rendering. |
| website/package.json | Add remark-math + rehype-katex dependencies. |
| website/package-lock.json | Lockfile updates for the new markdown/math toolchain. |
| website/docusaurus.config.ts | Enable math plugins + set markdown format to MDX. |
| website/docs/tutorials/algorithm/selection/svm.md | Expand SVM tutorial content with math/diagrams/config details. |
| website/docs/tutorials/algorithm/selection/static.md | Expand Static tutorial content + add diagrams. |
| website/docs/tutorials/algorithm/selection/router-dc.md | Expand Router-DC tutorial content with math + richer config guidance. |
| website/docs/tutorials/algorithm/selection/rl-driven.md | Expand RL-driven tutorial content with sub-modes, flows, config. |
| website/docs/tutorials/algorithm/selection/mlp.md | New tutorial page for MLP selection algorithm. |
| website/docs/tutorials/algorithm/selection/latency-aware.md | Expand latency-aware tutorial content with diagrams/config table. |
| website/docs/tutorials/algorithm/selection/knn.md | Expand KNN tutorial content with math/diagrams/config. |
| website/docs/tutorials/algorithm/selection/kmeans.md | Expand KMeans tutorial content with math/diagrams/config. |
| website/docs/tutorials/algorithm/selection/hybrid.md | Expand Hybrid tutorial content with math/diagrams/config. |
| website/docs/tutorials/algorithm/selection/gmtrouter.md | Expand GMTRouter tutorial content with diagrams/config/feedback section. |
| website/docs/tutorials/algorithm/selection/elo.md | Expand Elo tutorial content with math/flows/config table. |
| website/docs/tutorials/algorithm/selection/automix.md | Expand AutoMix tutorial content with math/flows/config table. |
| website/docs/tutorials/algorithm/overview.md | Add algorithm comparison table + decision guide diagram + add MLP link. |
| website/docs/tutorials/algorithm/looper/remom.md | Expand ReMoM looper tutorial content and config parameters. |
| website/docs/tutorials/algorithm/looper/ratings.md | Expand Ratings looper tutorial content and config parameters. |
| website/docs/tutorials/algorithm/looper/confidence.md | Expand Confidence looper tutorial content and config parameters. |
| src/semantic-router/pkg/observability/logging/logging.go | Add WithComponent + ComponentEvent; adjust timestamp layout. |
| src/semantic-router/pkg/extproc/utils_fast.go | Reduce log verbosity for streaming mutation. |
| src/semantic-router/pkg/extproc/server.go | Emit structured extproc startup event (component + fields). |
| src/semantic-router/pkg/extproc/router_resolvers.go | Simplify Authz logs and consolidate identity/chain reporting. |
| src/semantic-router/pkg/extproc/router_memory.go | Consolidate memory store connection logs and adjust warning text. |
| src/semantic-router/pkg/extproc/router_components.go | Consolidate semantic cache logs; remove noisy “auto-selected” logs. |
| src/semantic-router/pkg/extproc/res_filter_hallucination.go | Reduce log verbosity; consolidate hallucination detection logs. |
| src/semantic-router/pkg/extproc/req_filter_tools.go | Reduce tools plugin log to debug. |
| src/semantic-router/pkg/extproc/req_filter_response_api.go | Reduce Response API logs; remove some info-level lines. |
| src/semantic-router/pkg/extproc/req_filter_rag.go | Reduce RAG plugin logs to debug; tighten messages. |
| src/semantic-router/pkg/extproc/req_filter_modality.go | Reduce modality router logs to debug. |
| src/semantic-router/pkg/extproc/req_filter_looper_internal.go | Reduce looper internal logs; reformat key info log lines. |
| src/semantic-router/pkg/extproc/processor_res_cache.go | Improve cache failure log message; remove success info log. |
| src/semantic-router/pkg/extproc/processor_req_body_routing.go | Reduce endpoint/model rewrite logging verbosity; simplify wording. |
| src/semantic-router/pkg/extproc/processor_req_body_prepare.go | Reduce streaming/cache logs to debug; clarify cache-hit log. |
| src/semantic-router/pkg/extproc/processor_req_body_memory.go | Reduce memory logs to debug. |
| src/semantic-router/pkg/extproc/processor_core.go | Reduce extproc stage/stream lifecycle logs to debug. |
| src/semantic-router/pkg/classification/hallucination_detector.go | Remove noisy NLI init info log. |
| src/semantic-router/pkg/classification/embedding_classifier.go | Reduce soft-match logging to debug. |
| src/semantic-router/pkg/classification/classifier.go | Simplify classifier init logs; reduce jailbreak result verbosity. |
| src/semantic-router/pkg/classification/classifier_model_select.go | Reduce selection result logging; remove some init/info logs. |
| src/semantic-router/pkg/classification/classifier_init.go | Tidy init log strings; reduce verbosity. |
| src/semantic-router/pkg/classification/classifier_category_entropy.go | Reduce modality/category entropy logs to debug. |
| src/semantic-router/cmd/runtime_bootstrap.go | Reduce startup log spam; tighten embedding-model guidance logs. |
| src/semantic-router/cmd/main.go | Add structured startup summary event; minor Kubernetes log cleanup. |
| dashboard/frontend/src/types/dsl.ts | Add NL builder request/response/types for UI + store. |
| dashboard/frontend/src/stores/dslStoreTypes.ts | Add NL generation state + actions to the DSL store interface. |
| dashboard/frontend/src/stores/dslStore.ts | Implement NL generation action + error parsing; store NL results. |
| dashboard/frontend/src/pages/builderPageToolbar.tsx | Enable NL mode switch + re-enable Import in NL mode. |
| dashboard/frontend/src/pages/builderPageNaturalLanguagePanel.tsx | New NL panel UI for prompts, connection config, review display. |
| dashboard/frontend/src/pages/builderPageNaturalLanguagePanel.module.css | Styling for the new NL panel. |
| dashboard/frontend/src/pages/builderPageDashboardViews.tsx | Update NL mode marketing copy (no longer “coming soon”). |
| dashboard/frontend/src/pages/BuilderPage.tsx | Render NL panel; wire store state/actions to NL mode. |
| dashboard/backend/router/core_routes.go | Register /api/router/config/nl/generate route. |
| dashboard/backend/handlers/builder_nl.go | New HTTP handler for NL generation endpoint. |
| dashboard/backend/handlers/builder_nl_service.go | NL generation service: base YAML merge + LLM calls + parsing + review. |
| dashboard/backend/handlers/builder_nl_service_test.go | Basic tests for output parsing + custom connection application. |
| config/algorithm/selection/mlp.yaml | Add sample MLP selection algorithm YAML. |
Files not reviewed (1)
- website/package-lock.json: Language not supported
Comment on lines
+666
to
+667
| ready := boolFromPayload(payload, "ready") | ||
| if !ready && len(warnings) == 0 { |
There was a problem hiding this comment.
parseBuilderNLReviewOutput will flip ready to true whenever ready is false and warnings is empty. This means an explicit {"ready":false,"warnings":[]} response is treated as ready, which can mislead the UI. Consider distinguishing “missing ready field” from “ready=false” (e.g., return (value, present) from the helper) and only default to ready when the field is absent.
Suggested change
| ready := boolFromPayload(payload, "ready") | |
| if !ready && len(warnings) == 0 { | |
| // Determine readiness: respect an explicit "ready" value if present, | |
| // and only default to ready when the field is absent and there are no warnings. | |
| var ready bool | |
| if readyValue, ok := payload["ready"]; ok { | |
| if readyBool, ok := readyValue.(bool); ok { | |
| ready = readyBool | |
| } | |
| } else if len(warnings) == 0 { |
Comment on lines
+515
to
+527
| httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(raw)) | ||
| if err != nil { | ||
| return "", fmt.Errorf("failed to create builder ai request: %w", err) | ||
| } | ||
| httpReq.Header.Set("Content-Type", "application/json") | ||
| httpReq.Header.Set("Accept", "application/json") | ||
| if accessKey != "" { | ||
| httpReq.Header.Set("Authorization", "Bearer "+accessKey) | ||
| } | ||
|
|
||
| resp, err := http.DefaultClient.Do(httpReq) | ||
| if err != nil { | ||
| return "", fmt.Errorf("builder ai request failed: %w", err) |
There was a problem hiding this comment.
Outbound requests to the LLM endpoint use http.DefaultClient.Do(...) without any explicit timeout. If the upstream stalls, this handler can hang until the client disconnects, tying up dashboard server resources. Use an http.Client{Timeout: ...} (and/or per-request context deadline) for both OpenAI-compatible and Anthropic calls.
Comment on lines
+276
to
+294
| func normalizeBuilderNLBaseURL(raw string, providerKind builderNLProviderKind) (*url.URL, error) { | ||
| trimmed := strings.TrimSpace(strings.TrimRight(raw, "/")) | ||
| if trimmed == "" { | ||
| return nil, fmt.Errorf("custom connection baseUrl is required") | ||
| } | ||
| if !strings.Contains(trimmed, "://") { | ||
| trimmed = inferBuilderNLProtocol(trimmed, providerKind) + "://" + trimmed | ||
| } | ||
| parsed, err := url.Parse(trimmed) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("invalid custom connection baseUrl: %w", err) | ||
| } | ||
| if parsed.Scheme != "http" && parsed.Scheme != "https" { | ||
| return nil, fmt.Errorf("custom connection baseUrl must use http or https") | ||
| } | ||
| if parsed.Host == "" { | ||
| return nil, fmt.Errorf("custom connection baseUrl must include a host") | ||
| } | ||
| return parsed, nil |
There was a problem hiding this comment.
baseUrl is user-supplied and is used to construct an outbound HTTP request from the dashboard backend (normalizeBuilderNLBaseURL + callBuilderNLCustomConnection). This is an SSRF risk (e.g., targeting link-local / RFC1918 / cluster metadata endpoints) if the dashboard is reachable by untrusted users. Consider restricting destinations (allowlist), rejecting private IP ranges, and/or gating this endpoint behind an explicit admin-only feature flag.
| algorithm: | ||
| type: mlp | ||
| mlp: | ||
| device: cuda |
There was a problem hiding this comment.
The new MLP algorithm example hard-codes device: cuda. As a default template under config/algorithm/selection/, this will fail on CPU-only deployments (which are common for docs/tutorial users). Suggest switching the sample to device: cpu (or omitting device to rely on defaults) and documenting GPU as an optional override.
Suggested change
| device: cuda | |
| # Default to CPU; override to 'cuda' when running with a GPU. | |
| device: cpu |
Comment on lines
100
to
+110
| gmtrouter: | ||
| enable_personalization: true | ||
| history_sample_size: 50 | ||
| model_path: models/gmtrouter.pt | ||
| enable_personalization: true # Enable user-specific learning | ||
| history_sample_size: 5 # Sample k interactions for inference | ||
| embedding_dimension: 768 # Node embedding dimension | ||
| num_gnn_layers: 2 # HGT layers (L in paper) | ||
| attention_heads: 8 # Attention heads per HGT layer | ||
| max_interactions_per_user: 100 # Limit stored interactions per user | ||
| min_interactions_for_personalization: 3 # Min interactions before personalization | ||
| feedback_types: [rating, ranking] # Supported feedback types | ||
| model_path: models/gmtrouter.pt # Path to pre-trained model weights | ||
| storage_path: /var/lib/vsr/gmt_graph.json # Persist interaction graph |
There was a problem hiding this comment.
The configuration snippet documents fields that aren’t supported by the current config schema (e.g., embedding_dimension, num_gnn_layers, attention_heads, feedback_types). In GMTRouterSelectionConfig the supported keys are enable_personalization, history_sample_size, min_interactions_for_personalization, max_interactions_per_user, model_path, and storage_path. Please align the docs to the actual supported config keys (or update the config schema/implementation if these fields are intended).
Comment on lines
+89
to
+101
| verification_threshold: 0.7 # Self-verification confidence threshold | ||
| max_escalations: 2 # Maximum escalation attempts | ||
| cost_aware_routing: true # Enable cost-quality optimization | ||
| cost_quality_tradeoff: 0.3 # Balance: 0=pure quality, 1=pure cost | ||
| discount_factor: 0.95 # POMDP discount factor (γ) | ||
| use_logprob_verification: true # Use logprobs for confidence | ||
| enable_self_verification: false # LLM-based entailment verification | ||
| verification_samples: 5 # Samples for confidence (k in paper) | ||
| use_pomdp_router: true # Full POMDP routing (vs simple threshold) | ||
| belief_particles: 100 # Number of POMDP particles | ||
| cost_lambda: 0.5 # POMDP cost-performance tradeoff | ||
| verifier_server_url: "" # AutoMix verification server URL | ||
| enable_cascade: false # Enable full cascade execution mode |
There was a problem hiding this comment.
The AutoMix config example/documentation lists fields that aren’t in AutoMixSelectionConfig (e.g., enable_self_verification, verification_samples, use_pomdp_router, belief_particles, cost_lambda, verifier_server_url, enable_cascade, etc.). This makes the page misleading for users trying to configure the current release. Please align the documented parameters with the actual supported config keys.
Comment on lines
+80
to
87
| elo_weight: 0.3 # Weight for Elo rating | ||
| router_dc_weight: 0.3 # Weight for embedding similarity | ||
| automix_weight: 0.2 # Weight for POMDP value | ||
| cost_weight: 0.2 # Weight for cost consideration | ||
| quality_gap_threshold: 0.1 # Threshold for MLP escalation check | ||
| normalize_scores: true # Normalize component scores to [0,1] | ||
| use_mlp: false # Enable MLP quality gap prediction | ||
| ``` |
There was a problem hiding this comment.
The Hybrid docs reference use_mlp (and behavior around MLP-based escalation checks), but HybridSelectionConfig doesn’t currently include a use_mlp field. Please remove/clarify this parameter (or add it to the config schema + implementation) so the documented config matches what the router accepts.
Comment on lines
+34
to
+37
| A[Request arrives] --> B[Decision matched] | ||
| B --> C[algorithm.type = static] | ||
| C --> D[Return modelRefs, 0,] | ||
| D --> E[SelectionResult with score=1.0] |
There was a problem hiding this comment.
Mermaid node label has a typo: Return modelRefs, 0, (extra comma) reads like an indexing bug. Suggest updating it to something like “Return modelRefs[0]” to match the description that the first candidate wins.
Comment on lines
+281
to
+293
| <div className={styles.fullSpan}> | ||
| <label className={styles.label} htmlFor="builder-nl-key"> | ||
| Access key | ||
| </label> | ||
| <input | ||
| id="builder-nl-key" | ||
| className={styles.input} | ||
| type="password" | ||
| value={accessKey} | ||
| onChange={(event) => setAccessKey(event.target.value)} | ||
| placeholder="Optional unless the endpoint requires authentication" | ||
| /> | ||
| </div> |
There was a problem hiding this comment.
UI copy says the access key is “Optional unless the endpoint requires authentication”, but the backend requires an access key for the Anthropic provider (anthropic custom connection requires an accessKey). Consider making this field conditionally required/validated when providerKind === "anthropic" (and updating the placeholder/help text) to avoid a confusing round-trip error.
Comment on lines
+175
to
+183
| func readBuilderNLBaseConfig(configPath string) (*routerconfig.CanonicalConfig, error) { | ||
| cfg, err := readCanonicalConfigFile(configPath) | ||
| if err == nil { | ||
| return cfg, nil | ||
| } | ||
| if strings.Contains(strings.ToLower(err.Error()), "no such file") { | ||
| return &routerconfig.CanonicalConfig{}, nil | ||
| } | ||
| return nil, fmt.Errorf("failed to read builder deploy base: %w", err) |
There was a problem hiding this comment.
readBuilderNLBaseConfig detects a missing file by substring-matching the error message ("no such file"). This is brittle across OSes and wrapped errors. Prefer errors.Is(err, os.ErrNotExist) (or os.IsNotExist) based on what readCanonicalConfigFile returns.
Xunzhuo
force-pushed
the
cb/dsl-nl-mode
branch
2 times, most recently
from
4ae0ba4 to
0a03bd0
Compare
Xunzhuo
changed the title
Xunzhuo
force-pushed
the
cb/dsl-nl-mode
branch
4 times, most recently
from
4cbe763 to
a97e9c9
Compare
Contributor
🔍 Tessl Skill Review
|
| Dimension | Score | Detail |
|---|---|---|
| conciseness | ██░ 2/3 | The skill is reasonably efficient for its complexity but includes some redundancy—the structured reflection in step 7 partially repeats the classification from step 3, and some gotchas restate workflow guidance. However, it largely avoids explaining concepts Claude already knows and stays domain-specific. |
| actionability | ███ 3/3 | The skill provides concrete, executable commands in the Standard Commands section, specific API endpoints (PATCH vs PUT /config/router, GET /ready, /api/v1/eval), specific file paths, and clear classification buckets. The guidance is precise enough to be directly followed. |
| workflow clarity | ███ 3/3 | The 8-step workflow is clearly sequenced with explicit validation checkpoints (step 5: local validation before deploy, step 6: wait for ready before trusting eval, re-run probes after deploy). It includes feedback loops (classify failures before changing anything, compare before/after), error recovery guidance, and stop conditions that prevent unsafe operations. |
| progressive disclosure | ███ 3/3 | The skill is well-structured with clear sections (Trigger, Workflow, Gotchas, Standard Commands, Acceptance) and references external files via one-level-deep links (Must Read section, inline file references). Required and conditional surfaces are cleanly separated, and the content appropriately stays at overview level while pointing to detailed resources. |
Overall: This is a strong, well-structured skill that provides clear, actionable guidance for a complex multi-step calibration workflow. Its greatest strengths are the explicit validation checkpoints, failure classification framework, and concrete executable commands. Minor verbosity in the reflection step and some redundancy between gotchas and workflow steps prevent a perfect conciseness score, but overall the content is high quality.
To improve your score, point your agent at the Tessl optimization guide. Need help? Jump on our Discord.
Feedback
Report issues with this review at tesslio/skill-review, or send private feedback from your terminal with tessl feedback.
Xunzhuo
force-pushed
the
cb/dsl-nl-mode
branch
3 times, most recently
from
a3b03d3 to
a8ba93a
Compare
Signed-off-by: xunzhuo <xunzhuo@vllm-semantic-router.ai>
Signed-off-by: xunzhuo <xunzhuo@vllm-semantic-router.ai>
Signed-off-by: xunzhuo <xunzhuo@vllm-semantic-router.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MoMrouting, custom model connections, AI review feedback, and existing compile/deploy integrationdashboard-platform-changedashboard/frontendBuilder page/store/types,dashboard/backendNL generation route and handlers, and minimalsrc/semantic-router/pkg/extproccleanup required to satisfy repository pre-commit hooksyesnoneValidation
cpu-localmake dashboard-checknot runnot runcd dashboard/backend && go test ./...is currently blocked by the existingTestMergeDeployPayload_RoundTripsMaintainedAMDConfigfixture path issue (../../../deploy/amd/config.yamlnot found), which is unrelated to this dashboard changeChecklist
[Bugfix],[CI/Build],[CLI],[Dashboard],[Doc],[Feat],[Router], or[Misc]git commit -s