docs: refresh gh-pages and document technical constants (#107)#108
Conversation
…engLabResearch#107) Adds a Reference section to the mkdocs documentation with a new technical-constants.md page that documents the hardcoded file explorer limits introduced by the ChengLabResearch#51 out-of-memory mitigation stack (ChengLabResearch#104, ChengLabResearch#105, ChengLabResearch#106). For each constant, records symbol name, current value, source file line, rationale, user-visible behavior when the limit is reached, and guidance on when maintainers should reconsider the value. Values are extracted directly from src/main/event-handlers/filesystem.ts: - FILE_EXPLORER_WATCH_DEPTH = 6 - FILE_EXPLORER_WATCH_LIMIT = 100_000 - FILE_EXPLORER_UPDATE_BATCH_LIMIT = 500 - FILE_EXPLORER_UPDATE_INTERVAL_MS = 100 - IGNORED_PATH_SEGMENTS = {node_modules, __pycache__, venv} plus any segment beginning with "." Also documents the plugin directory-broadcast change from ChengLabResearch#106 (empty nodes payload) and points plugin authors at the reference page. Adds a short user-visible "Large Folder Behavior" section to the docs index so users encountering the watcher limit warning have a place to land, and wires the new page into mkdocs.yml. Setup, development, and packaging docs were audited against main and have no stale claims tied to the ChengLabResearch#104/ChengLabResearch#105/ChengLabResearch#106 changes, so no rewrites in this pass; broader audit work is left as follow-up.
…ackaging (ChengLabResearch#107) Extends documentation/docs/reference/technical-constants.md with the remaining documentation-worthy constants outside the file-explorer block: local server/port defaults (plugin file server 3000, volume server 3001, main server 8000), the docker volume name, main-server compose paths, the renderer's server URL and connection retry, the iframe plugin allowed origins, the slice visualization sampling proportion, the slice/backproject SSE stream endpoints, the Python server host/port pair, the volume-server URL used from inside the container, the bounding-box BSP defaults, the backprojection chunk and process defaults, the shared-memory monitor interval, the cloud-volume flush default, the compose shm_size, the python-poetry base image tag, and the packaging-time server-image and preinstalled-plugin pins.
|
Second sub-agent pass on the technical constants sweep. The extended commit (5078960) documents constants where the intent could be inferred directly from the code and existing PRs. The items below are ones where a short answer from den-sq (or Weaver) would let a follow-up PR either document them alongside the others or turn them into named constants first.
|
…es, invariants, rationales) (ChengLabResearch#107) Landed alongside PR ChengLabResearch#109 (constants/env-config-and-tightening) on the code side. Doc updates in `documentation/docs/reference/technical-constants.md`: - Q1: rename port entries to `PLUGIN_FILE_SERVER_PORT` / `VOLUME_SERVER_PORT`, document env overrides `OUROBOROS_PLUGIN_FILE_SERVER_PORT` / `OUROBOROS_VOLUME_SERVER_PORT`, cross-link PR ChengLabResearch#109. - Q2: add "Invariant" call-out to the `VOLUME='ouroboros-volume'` section: literal is duplicated across TS, Python, and both compose files; compose YAML must be static, so the invariant lives in docs + code review discipline rather than in a code-gen source of truth. - Q4: retitle `allowedOrigins` -> `allowedHostnames`, add a security note describing the prefix -> exact hostname tightening from PR ChengLabResearch#109. - Q6: extend the "Stream endpoint paths" section with the canonical `StepName` enum + `GET /step-names` endpoint, plus the deferred TS runtime-drift check (TODO ChengLabResearch#107). - Q7: document env overrides `OUROBOROS_SERVER_HOST` / `_PORT` and `OUROBOROS_DOCKER_SERVER_HOST` / `_PORT` for the Python server. - Q8: add "Why 'main'" note on `PLUGIN_NAME`: ouroboros is the main app of its own plugin ecosystem; no per-plugin path scheme planned on the Python side. - Q9: add a "User surfacing" note on `DEFAULT_SPLIT_THRESHOLD` - intentionally not exposed through `BoundingBoxParams`; a follow-up would need a matching field on the params schema. - Q10: chunk_size = 160 rationale in Sorotassu's words: enables clean backprojection of the flattened xyz stack; judgment-call sweet spot on memory vs speed for standard data; users tuning this should measure peak memory and iteration time on their own volumes. - Q11: remove the `MEM_INTERVAL_TIMER` entry entirely; the constant is now split into `MEM_LOG_INTERVAL_S` / `SHUTDOWN_GRACE_S` (in PR ChengLabResearch#109) and no longer needs documentation. - Q12: rewrite `shm_size` section with Sorotassu's rationale (Docker's aggressive 64 MB default makes the override necessary; setting it too large shifts OOM from artificial limit to host OOM) and the new packaging-time env override `OUROBOROS_SERVER_SHM_SIZE`. - Q13: add a "Policy" line to the Python base image pin: intentional, driven by transitive-dependency stability, do not bump reactively. - Q14: add a "Bump workflow" note to `productionPluginPins`: they and the fallback `neuroglancerArtifactForTag` / `autosegArtifactForTag` builders in `scripts/prepare-package-flavor.mjs` move together on every plugin release; single-file edit.
|
Thanks for the answers. Landed as two changes:
Q3, Q5, Q15 were "fine" / "no" and needed no action. Verification on the code side: |
… mem interval, env shm_size (refs #107) Follow-up to PR #108 (docs sweep). Addresses the constants-side questions raised in the sub-agent review pass: - Q1: promote plugin-file-server and volume-server port locals to named PLUGIN_FILE_SERVER_PORT / VOLUME_SERVER_PORT constants with env overrides OUROBOROS_PLUGIN_FILE_SERVER_PORT and OUROBOROS_VOLUME_SERVER_PORT (fallback preserved via parseInt || default, which is NaN-safe). - Q4: tighten IFrameContext allowedOrigins from prefix-match to exact hostname equality via new URL(origin).hostname === hostname. Closes the http://localhost.evil.example hole; loopback ports remain unrestricted. - Q6: add StepName enum + step_names_payload in python/ouroboros/common/ step_names.py and expose it via GET /step-names on the FastAPI server. Runtime drift check on the TS side is scoped down to a TODO(#107) because ServerContext does not currently expose baseURL to non-hook callers; wiring that is architecturally invasive. - Q7: make HOST / PORT / DOCKER_HOST / DOCKER_PORT env-configurable via OUROBOROS_SERVER_HOST / _PORT and OUROBOROS_DOCKER_SERVER_HOST / _PORT. - Q11: split MEM_INTERVAL_TIMER into MEM_LOG_INTERVAL_S (polling) and SHUTDOWN_GRACE_S (exit_cleanly sleep). Both stay at 1.0 by design; the split lets one be tuned without silently moving the other. - Q12: env-configurable shm_size at packaging time (OUROBOROS_SERVER_SHM_SIZE, default 64gb) via scripts/prepare-production-server.mjs, plus \${OUROBOROS_SERVER_SHM_SIZE:-64gb} in python/compose.yml and python/compose.dev.yml. Verified with docker-compose config on both files (default and override). Docs updates land in PR #108 (docs/107-gh-pages-refresh). Verification: - npm run typecheck passes. - docker-compose -f python/compose.yml config and docker-compose -f python/compose.dev.yml config both succeed with default substitution and with OUROBOROS_SERVER_SHM_SIZE=128gb override. - node scripts/prepare-production-server.mjs generates compose.yml with default 64gb; OUROBOROS_SERVER_SHM_SIZE=16gb override honored. - Python step_names module imports and step_names_payload() returns the expected mapping. npm run lint is not runnable in this branch: ESLint 9.39 rejects the repo's legacy .eslintrc setup, unrelated to this change.
- Not everything needed an explanation or inclusion here; removed too obvious or not necessary. - Rewrote the chunk size explanation to better reflect specifics.
Closes #107.
References #51, #104, #105, #106.
Summary
Adds a Reference section to the mkdocs documentation with a new
documentation/docs/reference/technical-constants.mdpage thatdocuments the hardcoded file explorer limits introduced by the #51
out-of-memory mitigation stack. Also adds a short user-visible
"Large Folder Behavior" section to the docs index and a plugin-author
note about the changed directory-broadcast payload.
The
gh-pagesbranch is auto-generated by.github/workflows/build-docs.ymlfrom
documentation/onmain, so this PR targetsmainrather thanediting
gh-pagesdirectly.Constants documented
All values extracted from
src/main/event-handlers/filesystem.ts(lines 8-12):
FILE_EXPLORER_WATCH_DEPTH6FILE_EXPLORER_WATCH_LIMIT100_000IGNORED_PATH_SEGMENTS{node_modules, __pycache__, venv}plus any.-prefixed segmentFILE_EXPLORER_UPDATE_BATCH_LIMIT500FILE_EXPLORER_UPDATE_INTERVAL_MS100The plugin directory-broadcast change from #106 (empty
nodespayload)is also documented in the same reference page and cross-linked from
guide/plugins.md.Each entry records: symbol name, current value, source file line,
rationale, user-visible behavior when the limit is reached, and
guidance on when to reconsider the value.
Files changed
documentation/docs/reference/technical-constants.md(new)documentation/docs/index.md(new Reference link + Large FolderBehavior section)
documentation/docs/guide/plugins.md(plugin directory broadcastnote + reference link)
documentation/mkdocs.yml(new Reference nav section)Audit findings and out-of-scope
Setup, development, packaging, and runtime docs on
mainwereaudited against the merged code. No existing pages carried claims
that are made wrong by #104/#105/#106, so no rewrites were needed
in this pass. The three existing mentions of "File Explorer panel"
in
guide/plugins.md,guide/slicing.md, andguide/backproject.mdremain accurate for their drag-and-drop context.
Left out of scope for a follow-up:
docs beyond the file-explorer and plugin-broadcast changes tied
directly to Refresh gh-pages documentation and document technical constants #107's acceptance criteria.
the current empty-
nodesbroadcast.Verification
grep -n 'FILE_EXPLORER_\|IGNORED_PATH_SEGMENTS' src/main/event-handlers/filesystem.ts.branch.
mkdocs buildwill run in CI via.github/workflows/build-docs.ymlon push and redeploy the
gh-pagesbranch automatically.