Skip to content

AI analytics#1339

Open
aadesh18 wants to merge 94 commits intodevfrom
ai-analytics
Open

AI analytics#1339
aadesh18 wants to merge 94 commits intodevfrom
ai-analytics

Conversation

@aadesh18
Copy link
Copy Markdown
Collaborator

@aadesh18 aadesh18 commented Apr 15, 2026

This PR adds an AI analytics dashboard to the internal tool and centralises AI query observability. Every call through the /ai/query and /integrations/ai-proxy endpoints is now logged to SpacetimeDB with token counts, cost, latency, and full conversation replay. The internal tool gains a new "Unified AI Endpoint Analytics" tab backed by a new useAiQueryLogs SpacetimeDB subscription, plus reviewer enrollment and unmark-reviewed flows.

How to review this PR:

  1. Go through the changes in apps/backend/src/app/api/latest/ai/query/[mode]/route.ts and apps/backend/src/app/api/latest/integrations/ai-proxy/[[...path]]/route.ts. These are the entry points to the logging
  2. Go through the apps/backend/src/app/api/latest/internal folder. It contains the routes that interact spacetime db
  3. Other changes in backend:
    1. Under lib/ai - most of these files are helper files that abstract a lot of logic from the ai-proxy and ai-query endpoints, especially the logging logic
    2. .env - added STACK_SPACETIMEDB_SERVICE_TOKEN
  4. Look through the internal tool

Summary by CodeRabbit

  • New Features

    • AI Query Usage dashboard: filtering, sorting, charts, metrics, details view
    • Reviewer enrollment for SpacetimeDB and “Unmark reviewed” action
  • Improvements

    • Separate Usage and Calls tabs with session-persistent selection
    • Call log: pagination, sortable “Human Reviewed” column, optimistic mark/unmark UX
    • Public published Q&A projection for safe read-only consumption
  • Infrastructure

    • Centralized AI proxy, logging, and query telemetry; HTTP-based reducer calls and safer DB access controls

mantrakp04 and others added 30 commits March 23, 2026 10:48
- Added new internal API endpoint for documentation tools, allowing actions such as listing available docs, searching, and fetching specific documentation by ID.
- Updated environment configuration to support optional internal secret for enhanced security.
- Refactored existing search functionality to utilize the new docs tools API instead of the previous MCP server.
- Improved error handling and response parsing for documentation-related requests.
- Expanded documentation to clarify the relationship between the new tools and existing API functionalities.

This update streamlines the documentation access process and enhances the overall developer experience.
- Introduced error capturing for failed HTTP requests in the docs tools API, improving debugging capabilities.
- Updated the API response for unsupported methods to include an 'Allow' header, clarifying the expected request type.

These changes enhance the robustness of the documentation tools integration and improve developer experience.
- Updated the key name in the capabilities section of the API documentation to follow a consistent naming convention, improving clarity and maintainability.
The .gitmodules was updated in d22593d to point at
apps/backend/src/private/implementation, but the gitlink entry (mode
160000) was never added to the tree. This caused
`git clone --recurse-submodules` to silently skip the private submodule.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…on for docs tools

- Added `STACK_DOCS_INTERNAL_BASE_URL` to backend `.env` and `.env.development` files for AI tool bundle configuration.
- Removed references to `STACK_INTERNAL_DOCS_TOOLS_SECRET` from backend and docs environment files and validation logic from the docs tools API route.
- Introduced a new `.env` file for the docs app with essential configuration variables.
@mantrakp04
Copy link
Copy Markdown
Collaborator

@greptile-ai review

Comment thread apps/backend/src/app/api/latest/internal/mcp-review/update-qa-entry/route.ts Outdated
Comment thread apps/backend/src/app/api/latest/ai/query/[mode]/route.ts
Comment thread apps/backend/src/lib/ai/loggers/ai-proxy-logger.ts
const rawCost = extractCostFromUsage(usage);
runAsynchronouslyAndWaitUntil(logAiQuery({
...common,
stepsJson: serializeSteps(steps),
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUG: logging-only serialization can break a successful AI response.

Walkthrough: the AI request can finish successfully, then logAiQuerySuccess tries to prepare the log row. Because serializeSteps(steps) is evaluated before the fire-and-forget logging promise is handed to runAsynchronouslyAndWaitUntil, any JSON.stringify failure here throws synchronously. I reproduced this with a tool result containing a circular object: the answer path is otherwise successful, but logging throws circular before the async boundary.

Expected behavior: logging should be isolated from the user-facing success path. Build/serialize the logging payload inside the async logging task, or catch serialization failures and log a safe placeholder.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread apps/backend/src/app/api/latest/internal/mcp-review/delete/route.ts
const editor = user.display_name ?? user.primary_email ?? user.id;
const qaId = BigInt(body.qaId);
await callReducerStrict("update_qa_entry", [token, qaId, body.question, body.answer, editor]);
await callReducerStrict("set_qa_published", [token, qaId, body.publish]);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUG: this update is not atomic.

Walkthrough: editing text and changing publish state are two separate reducer calls. I reproduced the failure by mocking update_qa_entry to succeed and set_qa_published to fail. The route returns an error, but the question/answer mutation has already been committed.

Expected behavior: use a single reducer for “update text plus publish state” so the whole operation either commits or fails together. Partial review edits are hard for reviewers to reason about because the UI reports failure while the data has changed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread apps/backend/src/lib/ai/prompts.ts
Comment thread apps/dashboard/src/components/vibe-coding/chat-adapters.ts
onSave(row.correlationId, question, answer, publish);
Promise.resolve(onSave(row.id, question, answer, publish))
.catch(err => captureError("knowledge-base-save", err));
setEditingId(null);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUG: failed saves look successful to the reviewer.

Walkthrough: the save starts Promise.resolve(onSave(...)).catch(...), then immediately closes the editor with setEditingId(null). I reproduced this from the component source: a rejected save is only sent to captureError, while the reviewer sees the editor disappear as if the save worked.

Expected behavior: await the save/delete action, keep the editor/dialog open on failure, and show an inline error. Capturing the error is useful for us, but it is not enough feedback for the reviewer performing the action.

{ name: "mark_human_reviewed", args: [wrong, "corr", "reviewer"] },
{ name: "unmark_human_reviewed", args: [wrong, "corr"] },
{
name: "update_human_correction",
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUG: this auth test still references a removed reducer.

Walkthrough: the current correction path uses the newer QA reducers, but this test still calls update_human_correction. I reproduced this statically by checking the test source still contains that reducer name. Against the real module this either fails for the wrong reason or no longer tests the current write surface.

Expected behavior: replace this with the current mutating reducer(s), especially the correction/upsert path that actually ships now.

args: [wrong, "corr", "q", "a", false, "reviewer"],
},
{ name: "add_manual_qa", args: [wrong, "q", "a", false, "reviewer"] },
{ name: "delete_qa_entry", args: [wrong, "corr"] },
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUG: this does not prove delete_qa_entry enforces auth.

Walkthrough: delete_qa_entry expects a numeric/u64 QA id, but the test passes "corr". That can fail while decoding the reducer arguments before the token check is reached. So a failing reducer call here does not necessarily mean the wrong token was rejected.

Expected behavior: pass a valid numeric QA id shape with the wrong token, then assert the failure message is the invalid-token path.

Comment thread apps/e2e/tests/spacetimedb/private-tables.test.ts
Copy link
Copy Markdown
Collaborator Author

@aadesh18 aadesh18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made changes based on comments

Comment thread apps/backend/src/app/api/latest/ai/query/[mode]/route.ts
Comment thread apps/backend/src/app/api/latest/internal/mcp-review/delete/route.ts
const editor = user.display_name ?? user.primary_email ?? user.id;
const qaId = BigInt(body.qaId);
await callReducerStrict("update_qa_entry", [token, qaId, body.question, body.answer, editor]);
await callReducerStrict("set_qa_published", [token, qaId, body.publish]);
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

const rawCost = extractCostFromUsage(usage);
runAsynchronouslyAndWaitUntil(logAiQuery({
...common,
stepsJson: serializeSteps(steps),
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread apps/backend/src/lib/ai/prompts.ts

let chosenIndex: number;
if (edit.occurrenceIndex != null) {
if (edit.occurrenceIndex < 0 || edit.occurrenceIndex >= matches.length) {
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread apps/dashboard/src/components/vibe-coding/chat-adapters.ts
Comment thread apps/e2e/tests/spacetimedb/private-tables.test.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants