Optimize data export and dashboard metrics

Author: JUk1-GH

.gitignore

@@ -9,6 +9,7 @@ __pycache__/
 # Local Codex usage export. This can contain private project names,
 # session ids, timestamps, quota status, and usage patterns.
 data.js
+data.raw.js
 !data.sample.js
 .codexscope-cache.json
 .codexscope-cache.json.stamp

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## v0.1.9 - 2026-05-09
+
+- Switched generated exports to compact `recordsV2` rows and delta timestamps, with catalogs kept in the raw sidecar to reduce first-load `data.js` size.
+- Added precomputed dashboard views for common ranges so preset filters avoid rescanning raw records in the browser.
+- Exported model pricing rules from the generator so precomputed views and custom date ranges share one cost-estimation source.
+- Compact precomputed chart buckets to shrink first-load dashboard views without changing the UI.
+- Split raw event rows into `data.raw.js`, loaded only when custom date ranges need raw records.
+- Added a raw sidecar schema marker so hot-start freshness checks do not scan the full raw export.
+- Fixed custom `--raw-out` paths by writing a browser-relative sidecar path into `data.js`.
+- Refactored preset view generation to aggregate summaries, charts, peak rate, sessions, and model stats in a single token-event pass.
+- Date preset views now slice sorted runtime events by time range before aggregating, avoiding full-history scans for short ranges.
+- Reused the already-loaded parsed cache during regeneration so runs with changed logs do not parse the large cache JSON twice.
+- Bumped cache output stamping to regenerate once after the data-shape upgrade while still reusing readable parsed log cache.
+- Slimmed release packages by removing bundled README docs from platform zips and stripping Go build ids.
+
 ## v0.1.8 - 2026-05-09
 
 - Switched trend chart anchors to market-style adaptive time grains.

README.md

@@ -8,7 +8,7 @@ CodexScope is a local-first dashboard for inspecting Codex usage from local sess
 
 ![CodexScope dashboard](assets/codexscope-dashboard-24h.png)
 
-The dashboard is a static HTML app: no backend, no account connection, and no hosted telemetry. Your real usage export stays local in `data.js`, which is intentionally ignored by git.
+The dashboard is a static HTML app: no backend, no account connection, and no hosted telemetry. Your real usage export stays local in `data.js` and `data.raw.js`, which are intentionally ignored by git. Recent exports keep precomputed dashboard views in `data.js`; compact catalogs and raw rows are loaded from `data.raw.js` only for custom date ranges.
 
 ## Why
 
@@ -34,8 +34,8 @@ To view your real local Codex usage, normal users should download the platform p
 - **macOS**: download `CodexScope-mac.zip`, unzip it, then double-click `Open CodexScope.command` in the extracted folder
 - **Windows**: download `CodexScope-windows.zip`, unzip it, then double-click `Open CodexScope.cmd` in the extracted folder
 
-Release zips include a prebuilt generator, so normal users do not need to install Go. The launcher generates `data.js` from your local Codex logs and then opens `index.html`. Source checkouts can still fall back to `go build` when the prebuilt generator is absent.
-Subsequent runs reuse a local `.codexscope-cache.json` file and only rescan changed session logs, so repeated launches should be much faster.
+Release zips include a prebuilt generator, so normal users do not need to install Go. The launcher generates `data.js` and `data.raw.js` from your local Codex logs and then opens `index.html`. Source checkouts can still fall back to `go build` when the prebuilt generator is absent.
+Subsequent runs reuse a local `.codexscope-cache.json` file and only rescan changed session logs. If nothing changed, the generator can skip rewriting the export files; if a log grew, it appends from the previous cached offset.
 
 Note: GitHub's automatic **Source code (zip)** asset is for developers, not the recommended user download. It may require Go or local compilation. Prefer `CodexScope-mac.zip` / `CodexScope-windows.zip`.
 
@@ -73,15 +73,15 @@ If your Codex sessions are stored elsewhere, pass the path explicitly:
 go run .\generate_codex_data.go --root "$env:USERPROFILE\.codex\sessions"
 ```
 
-The generator writes `data.js` next to `index.html`. Once that file exists, the dashboard automatically uses your real local data instead of the bundled demo. `data.js` and `.codexscope-cache.json` may contain private project names, session ids, timestamps, usage patterns, and quota status, so both are excluded by `.gitignore`.
+The generator writes `data.js` and `data.raw.js` next to `index.html`. Once those files exist, the dashboard automatically uses your real local data instead of the bundled demo. `data.js`, `data.raw.js`, and `.codexscope-cache.json` may contain private project names, session ids, timestamps, usage patterns, and quota status, so they are excluded by `.gitignore`.
 
 ## Project Structure
 
 - `index.html`: the static dashboard shell.
 - `styles.css`: dashboard layout and visual styling.
 - `app.ts`: TypeScript source for charts, filters, rankings, quota display, and cost estimation.
 - `app.js`: compiled browser script loaded by `index.html`.
-- `generate_codex_data.go`: the local data generator. It scans Codex JSONL session logs, extracts usage metadata, and writes `data.js`.
+- `generate_codex_data.go`: the local data generator. It scans Codex JSONL session logs, extracts usage metadata, and writes `data.js` plus `data.raw.js`.
 - `data.sample.js`: bundled demo data used when no local `data.js` exists.
 - `CHANGELOG.md`: release notes for each published version.
 - `macos/open-dashboard.command`: macOS launcher that runs the generator and opens the dashboard.
@@ -94,8 +94,8 @@ The generator writes `data.js` next to `index.html`. Once that file exists, the
 
 1. Codex writes local JSONL session logs under `~/.codex/sessions`.
 2. `generate_codex_data.go` scans local `.jsonl` files and extracts only usage metadata: token counts, model names, session ids, timing, failures, and rate-limit metadata.
-3. The generator writes those records to `data.js` as `window.CODEXSCOPE_DATA`.
-4. `index.html` loads `data.sample.js` first and then `data.js`. If real local data exists, it overrides the sample data.
+3. The generator writes precomputed range views to `data.js` as `window.CODEXSCOPE_DATA`, and compact catalogs plus raw event rows to `data.raw.js` as `window.CODEXSCOPE_RAW_DATA`.
+4. `index.html` loads `data.sample.js` first and then `data.js`. If real local data exists, it overrides the sample data; `data.raw.js` is loaded only when custom date ranges need raw rows.
 5. Date filters, charts, rankings, quota status, and cost estimates are computed in the browser from that local record set.
 
 ## What Gets Displayed
@@ -104,11 +104,11 @@ The generator writes `data.js` next to `index.html`. Once that file exists, the
 - **Quota and risk**: remaining short-window and weekly quota when Codex local logs include rate-limit metadata.
 - **Distribution**: request count or token volume grouped by time bucket.
 - **Rankings**: busiest sessions and models for the selected period.
-- **Cost estimate**: a local estimate using token counts and the built-in model price table.
+- **Cost estimate**: a local estimate using token counts and model pricing rules exported by the generator.
 
 ## Cost Estimates
 
-The cost card is an estimate, not an official bill. It uses local token counts and a built-in table based on OpenAI's published USD model prices. Actual ChatGPT/Codex billing, credits, and subscription quota status should always be checked with the official account or billing page.
+The cost card is an estimate, not an official bill. It uses local token counts and generator-exported rules based on OpenAI's published USD model prices. Actual ChatGPT/Codex billing, credits, and subscription quota status should always be checked with the official account or billing page.
 
 USD is the source currency. The CNY view is only a display conversion. When available, CodexScope fetches the USD/CNY rate from the Frankfurter API with the ECB provider selected. If that request fails, it falls back to the last bundled reference rate and marks the conversion as offline fallback in the UI.
 
@@ -149,7 +149,7 @@ CodexScope does not send data to a server. `generate_codex_data.go` reads local
 
 It does not export prompt text, assistant messages, tool output, or file contents.
 
-Review `data.js` before sharing screenshots or artifacts generated from your own usage.
+Review `data.js` and `data.raw.js` before sharing screenshots or artifacts generated from your own usage.
 
 ## License
 

README.zh-CN.md

@@ -8,7 +8,7 @@ CodexScope 是一个本地优先的 Codex 用量面板，用来查看你本机 C
 
 ![CodexScope dashboard](assets/codexscope-dashboard-24h.png)
 
-它是一个静态 HTML 页面：没有后端、没有账号接入、没有托管遥测。真实用量数据只会导出到本地的 `data.js`，并且这个文件默认不会提交到 git。
+它是一个静态 HTML 页面：没有后端、没有账号接入、没有托管遥测。真实用量数据只会导出到本地的 `data.js` 和 `data.raw.js`，并且这两个文件默认不会提交到 git。新版导出会把预计算视图放在 `data.js`；压缩字典和原始事件只会在自定义日期筛选时从 `data.raw.js` 按需加载。
 
 ## 为什么做
 
@@ -34,8 +34,8 @@ Codex 的用量如果只看零散日志，很难判断 Token 花到哪里、哪
 - **macOS**：下载 `CodexScope-mac.zip`，解压后双击根目录的 `Open CodexScope.command`
 - **Windows**：下载 `CodexScope-windows.zip`，解压后双击根目录的 `Open CodexScope.cmd`
 
-Release zip 会内置预编译生成器，普通用户不需要安装 Go。启动脚本会先在本地生成 `data.js`，然后打开 `index.html` 显示真实数据；源码 checkout 在没有预编译生成器时仍可回退到 `go build`。
-后续运行会复用本地 `.codexscope-cache.json` 缓存，只重扫发生变化的会话日志，所以重复启动会快很多。
+Release zip 会内置预编译生成器，普通用户不需要安装 Go。启动脚本会先在本地生成 `data.js` 和 `data.raw.js`，然后打开 `index.html` 显示真实数据；源码 checkout 在没有预编译生成器时仍可回退到 `go build`。
+后续运行会复用本地 `.codexscope-cache.json` 缓存，只重扫发生变化的会话日志。日志没变时会直接跳过重写导出文件；日志追加时会从上次缓存位置继续解析。
 
 注意：GitHub 自动生成的 **Source code (zip)** 是源码包，不是普通用户推荐入口；它可能需要本机安装 Go 或自己编译。请优先下载上面的 `CodexScope-mac.zip` / `CodexScope-windows.zip`。
 
@@ -75,15 +75,15 @@ start .\index.html
 go run .\generate_codex_data.go --root "$env:USERPROFILE\.codex\sessions"
 ```
 
-生成器会把真实数据写到 `index.html` 同目录下的 `data.js`。`data.js` 和 `.codexscope-cache.json` 都可能包含项目名、会话 id、时间戳、用量模式和额度状态，所以已经被 `.gitignore` 排除。
+生成器会把真实数据写到 `index.html` 同目录下的 `data.js` 和 `data.raw.js`。`data.js`、`data.raw.js` 和 `.codexscope-cache.json` 都可能包含项目名、会话 id、时间戳、用量模式和额度状态，所以已经被 `.gitignore` 排除。
 
 ## 项目结构
 
 - `index.html`：静态面板外壳。
 - `styles.css`：面板布局和视觉样式。
 - `app.ts`：图表、筛选、排行、额度状态和费用估算的 TypeScript 源码。
 - `app.js`：由 TypeScript 编译出的浏览器脚本，`index.html` 会直接加载它。
-- `generate_codex_data.go`：本地数据生成器，扫描 Codex JSONL 会话日志，提取用量元数据并写入 `data.js`。
+- `generate_codex_data.go`：本地数据生成器，扫描 Codex JSONL 会话日志，提取用量元数据并写入 `data.js` 和 `data.raw.js`。
 - `data.sample.js`：内置示例数据。没有本地 `data.js` 时，页面会先显示这份数据。
 - `CHANGELOG.md`：每个公开版本的更新记录。
 - `macos/open-dashboard.command`：macOS 启动脚本，负责运行生成器并打开面板。
@@ -96,8 +96,8 @@ go run .\generate_codex_data.go --root "$env:USERPROFILE\.codex\sessions"
 
 1. Codex 把本机会话日志写到 `~/.codex/sessions`。
 2. `generate_codex_data.go` 扫描本地 `.jsonl` 文件，只提取用量元数据：Token 数、模型名、会话 id、耗时、失败状态和 rate-limit 元数据。
-3. 生成器把这些记录写入 `data.js`，暴露为 `window.CODEXSCOPE_DATA`。
-4. `index.html` 会先加载 `data.sample.js`，再加载 `data.js`。如果真实本地数据存在，它会覆盖示例数据。
+3. 生成器把预计算日期视图写入 `data.js`，暴露为 `window.CODEXSCOPE_DATA`；压缩字典和原始事件行写入 `data.raw.js`，暴露为 `window.CODEXSCOPE_RAW_DATA`。
+4. `index.html` 会先加载 `data.sample.js`，再加载 `data.js`。如果真实本地数据存在，它会覆盖示例数据；只有自定义日期筛选需要时才加载 `data.raw.js`。
 5. 日期筛选、趋势图、排行、额度状态和费用估算都在浏览器里基于这份本地记录计算。
 
 ## 页面里显示什么
@@ -106,11 +106,11 @@ go run .\generate_codex_data.go --root "$env:USERPROFILE\.codex\sessions"
 - **额度与风险**：如果本地日志里有 rate-limit 元数据，会显示短窗口额度、周额度、缓存命中和失败风险。
 - **时间分布**：按时间桶统计调用次数或 Token 消耗。
 - **排行**：当前区间内最活跃的会话和模型。
-- **费用统计**：基于本地 Token 数量和内置模型价格表做估算。
+- **费用统计**：基于本地 Token 数量和生成器导出的模型价格规则做估算。
 
 ## 费用估算说明
 
-费用统计只是估算，不是官方账单。它使用本地 Token 数量和内置的 OpenAI 公开美元价格表计算。实际 ChatGPT/Codex 账单、余额、订阅额度和限制状态，请以官方账号或账单页面为准。
+费用统计只是估算，不是官方账单。它使用本地 Token 数量和生成器导出的 OpenAI 公开美元价格规则计算。实际 ChatGPT/Codex 账单、余额、订阅额度和限制状态，请以官方账号或账单页面为准。
 
 USD 是原始计算币种。CNY 只是展示换算。能联网时，CodexScope 会通过 Frankfurter API 获取 USD/CNY 汇率，并使用 ECB 数据源；如果请求失败，会回退到内置参考汇率，并在页面标记为离线回退。
 
@@ -151,7 +151,7 @@ CodexScope 不会把数据发送到服务器。`generate_codex_data.go` 只读
 
 它不会导出提示词、助手回复、工具输出或文件内容。
 
-分享截图或导出的产物前，建议先检查自己的 `data.js`。
+分享截图或导出的产物前，建议先检查自己的 `data.js` 和 `data.raw.js`。
 
 ## License