Skip to content

docs: refresh gh-pages and document technical constants (#107)#108

Merged
den-sq merged 4 commits into
ChengLabResearch:mainfrom
tavateva:docs/107-gh-pages-refresh
Jul 7, 2026
Merged

docs: refresh gh-pages and document technical constants (#107)#108
den-sq merged 4 commits into
ChengLabResearch:mainfrom
tavateva:docs/107-gh-pages-refresh

Conversation

@tavateva

@tavateva tavateva commented Jul 6, 2026

Copy link
Copy Markdown

Closes #107.
References #51, #104, #105, #106.

Summary

Adds a Reference section to the mkdocs documentation with a new
documentation/docs/reference/technical-constants.md page that
documents the hardcoded file explorer limits introduced by the #51
out-of-memory mitigation stack. Also adds a short user-visible
"Large Folder Behavior" section to the docs index and a plugin-author
note about the changed directory-broadcast payload.

The gh-pages branch is auto-generated by .github/workflows/build-docs.yml
from documentation/ on main, so this PR targets main rather than
editing gh-pages directly.

Constants documented

All values extracted from src/main/event-handlers/filesystem.ts
(lines 8-12):

Constant Value Introduced
FILE_EXPLORER_WATCH_DEPTH 6 #104
FILE_EXPLORER_WATCH_LIMIT 100_000 #104
IGNORED_PATH_SEGMENTS {node_modules, __pycache__, venv} plus any .-prefixed segment #104
FILE_EXPLORER_UPDATE_BATCH_LIMIT 500 #105
FILE_EXPLORER_UPDATE_INTERVAL_MS 100 #105

The plugin directory-broadcast change from #106 (empty nodes payload)
is also documented in the same reference page and cross-linked from
guide/plugins.md.

Each entry records: symbol name, current value, source file line,
rationale, user-visible behavior when the limit is reached, and
guidance on when to reconsider the value.

Files changed

  • documentation/docs/reference/technical-constants.md (new)
  • documentation/docs/index.md (new Reference link + Large Folder
    Behavior section)
  • documentation/docs/guide/plugins.md (plugin directory broadcast
    note + reference link)
  • documentation/mkdocs.yml (new Reference nav section)

Audit findings and out-of-scope

Setup, development, packaging, and runtime docs on main were
audited against the merged code. No existing pages carried claims
that are made wrong by #104/#105/#106, so no rewrites were needed
in this pass. The three existing mentions of "File Explorer panel"
in guide/plugins.md, guide/slicing.md, and guide/backproject.md
remain accurate for their drag-and-drop context.

Left out of scope for a follow-up:

Verification

  • Grep confirms every documented value matches the current tree:
    grep -n 'FILE_EXPLORER_\|IGNORED_PATH_SEGMENTS' src/main/event-handlers/filesystem.ts.
  • All internal markdown links resolve to files that exist in the
    branch.
  • mkdocs build will run in CI via .github/workflows/build-docs.yml
    on push and redeploy the gh-pages branch automatically.

Tava added 2 commits July 6, 2026 18:41
…engLabResearch#107)

Adds a Reference section to the mkdocs documentation with a new
technical-constants.md page that documents the hardcoded file explorer
limits introduced by the ChengLabResearch#51 out-of-memory mitigation stack (ChengLabResearch#104, ChengLabResearch#105,
ChengLabResearch#106).

For each constant, records symbol name, current value, source file line,
rationale, user-visible behavior when the limit is reached, and guidance
on when maintainers should reconsider the value. Values are extracted
directly from src/main/event-handlers/filesystem.ts:

- FILE_EXPLORER_WATCH_DEPTH = 6
- FILE_EXPLORER_WATCH_LIMIT = 100_000
- FILE_EXPLORER_UPDATE_BATCH_LIMIT = 500
- FILE_EXPLORER_UPDATE_INTERVAL_MS = 100
- IGNORED_PATH_SEGMENTS = {node_modules, __pycache__, venv} plus any
  segment beginning with "."

Also documents the plugin directory-broadcast change from ChengLabResearch#106 (empty
nodes payload) and points plugin authors at the reference page.

Adds a short user-visible "Large Folder Behavior" section to the docs
index so users encountering the watcher limit warning have a place to
land, and wires the new page into mkdocs.yml.

Setup, development, and packaging docs were audited against main and
have no stale claims tied to the ChengLabResearch#104/ChengLabResearch#105/ChengLabResearch#106 changes, so no rewrites
in this pass; broader audit work is left as follow-up.
…ackaging (ChengLabResearch#107)

Extends documentation/docs/reference/technical-constants.md with the
remaining documentation-worthy constants outside the file-explorer
block: local server/port defaults (plugin file server 3000, volume
server 3001, main server 8000), the docker volume name, main-server
compose paths, the renderer's server URL and connection retry, the
iframe plugin allowed origins, the slice visualization sampling
proportion, the slice/backproject SSE stream endpoints, the Python
server host/port pair, the volume-server URL used from inside the
container, the bounding-box BSP defaults, the backprojection chunk and
process defaults, the shared-memory monitor interval, the cloud-volume
flush default, the compose shm_size, the python-poetry base image tag,
and the packaging-time server-image and preinstalled-plugin pins.
@tavateva

tavateva commented Jul 7, 2026

Copy link
Copy Markdown
Author

Second sub-agent pass on the technical constants sweep. The extended commit (5078960) documents constants where the intent could be inferred directly from the code and existing PRs. The items below are ones where a short answer from den-sq (or Weaver) would let a follow-up PR either document them alongside the others or turn them into named constants first.

  1. src/main/servers/file-server.ts:6 and src/main/servers/volume-server.ts:6port = 3000 (plugin file server) and port = 3001 (volume server) are lowercase locals with no env override. Are these worth promoting to named PLUGIN_FILE_SERVER_PORT / VOLUME_SERVER_PORT constants and/or making them env-configurable for users who already have services on 3000/3001?
  2. src/main/servers/volume-server.ts:9VOLUME = 'ouroboros-volume' is duplicated as string literals in python/compose.yml, python/compose.dev.yml, python/ouroboros/common/volume_server_interface.py, and the compose file written by scripts/prepare-production-server.mjs. Any appetite for a single source of truth (env var or a generated compose fragment)?
  3. src/renderer/src/contexts/ServerContext.tsx:41retryDelay = 5000 (5 s server health poll). Is 5 s the intended UX cadence, or should it be shorter given the connection banner drives visible UI state?
  4. src/renderer/src/contexts/IFrameContext.tsx:23allowedOrigins is a prefix match (text.startsWith(item)), so http://localhost.evil.example would match http://localhost. Is the prefix match deliberate for port-suffix matching, and if so is it worth tightening to origin equality plus a separate port set?
  5. src/renderer/src/routes/SlicesPage/SlicesPage.tsx:26SLICE_RENDER_PROPORTION = 0.008. Is 0.008 an empirically chosen ratio for the SlicesPage, and should it scale with rects.length rather than being flat (very short paths currently render every rect via the Math.floor(...) = 0 fallback)?
  6. src/renderer/src/routes/SlicesPage/SlicesPage.tsx:30SLICE_STEP_NAME = 'SliceParallelPipelineStep' is coupled to a Python class name. Should the Python side expose a canonical step-name enum through the API so a rename doesn't silently break progress rows?
  7. python/ouroboros/common/server.py:10-14HOST/PORT (127.0.0.1:8000) and DOCKER_HOST/DOCKER_PORT (0.0.0.0:8000). Should these be env-configurable (OUROBOROS_SERVER_HOST/OUROBOROS_SERVER_PORT) for users who need to run the desktop server on a non-default port?
  8. python/ouroboros/common/volume_server_interface.py:4PLUGIN_NAME = 'main' is used as a subfolder on the shared volume and hard-coded into every copy_to_volume/copy_to_host/clear_plugin_folder call. Is this intentionally always 'main' from the Python side, or is there a planned per-plugin path scheme?
  9. python/ouroboros/helpers/bounding_boxes.py:6DEFAULT_SPLIT_THRESHOLD = 0.9 is not user-configurable through BoundingBoxParams (only max_depth and target_slices_per_box are). Is that deliberate, or should it also be surfaced through bounding_box_params?
  10. python/ouroboros/helpers/options.py:78chunk_size: int = 160 on BackprojectOptions. What's the rationale for 160 specifically? Would help users tune it if we can quote the reasoning in the docs (peak-memory budget, cache alignment, etc.).
  11. python/ouroboros/helpers/mem.py:14MEM_INTERVAL_TIMER = 1.0. It's both the memory-log poll interval and the pre-exit sleep in exit_cleanly. Is coupling these deliberate, or would separating them (a log interval and a shutdown grace period) be clearer?
  12. python/compose.yml:15, python/compose.dev.yml:28, and scripts/prepare-production-server.mjs:52shm_size: 64gb is hard-coded in all three compose sources. Is 64 GB a measured worst-case for the shipping pipelines, and should it be env-overridable at packaging time (OUROBOROS_SERVER_SHM_SIZE) for constrained hosts?
  13. python/Dockerfile:1 and python/Dockerfile-prod:5thehale/python-poetry:2.1.3-py3.11-slim. Is Poetry 2.1.3 / Python 3.11 an intentional long-term pin, or a "bump when we next touch this" pin? Documenting the policy would help downstream forks.
  14. scripts/prepare-package-flavor.mjs:7-16productionPluginPins for neuroglancer and autoseg are duplicated in the fallback filename builders neuroglancerArtifactForTag / autosegArtifactForTag. Are these pins meant to move together on every release, and is there a preferred single-file-edit workflow when bumping them?
  15. python/ouroboros/helpers/slice.py:12INDEXING = "xy". Assumed to be a NumPy meshgrid indexing convention. Is this ever expected to become "ij" for any code path, and is it worth pulling out as a documented invariant instead?

…es, invariants, rationales) (ChengLabResearch#107)

Landed alongside PR ChengLabResearch#109 (constants/env-config-and-tightening) on the
code side. Doc updates in `documentation/docs/reference/technical-constants.md`:

- Q1: rename port entries to `PLUGIN_FILE_SERVER_PORT` /
  `VOLUME_SERVER_PORT`, document env overrides
  `OUROBOROS_PLUGIN_FILE_SERVER_PORT` /
  `OUROBOROS_VOLUME_SERVER_PORT`, cross-link PR ChengLabResearch#109.
- Q2: add "Invariant" call-out to the `VOLUME='ouroboros-volume'`
  section: literal is duplicated across TS, Python, and both compose
  files; compose YAML must be static, so the invariant lives in docs +
  code review discipline rather than in a code-gen source of truth.
- Q4: retitle `allowedOrigins` -> `allowedHostnames`, add a security
  note describing the prefix -> exact hostname tightening from PR ChengLabResearch#109.
- Q6: extend the "Stream endpoint paths" section with the canonical
  `StepName` enum + `GET /step-names` endpoint, plus the deferred TS
  runtime-drift check (TODO ChengLabResearch#107).
- Q7: document env overrides `OUROBOROS_SERVER_HOST` / `_PORT` and
  `OUROBOROS_DOCKER_SERVER_HOST` / `_PORT` for the Python server.
- Q8: add "Why 'main'" note on `PLUGIN_NAME`: ouroboros is the main
  app of its own plugin ecosystem; no per-plugin path scheme planned
  on the Python side.
- Q9: add a "User surfacing" note on `DEFAULT_SPLIT_THRESHOLD` -
  intentionally not exposed through `BoundingBoxParams`; a follow-up
  would need a matching field on the params schema.
- Q10: chunk_size = 160 rationale in Sorotassu's words: enables clean
  backprojection of the flattened xyz stack; judgment-call sweet spot
  on memory vs speed for standard data; users tuning this should
  measure peak memory and iteration time on their own volumes.
- Q11: remove the `MEM_INTERVAL_TIMER` entry entirely; the constant is
  now split into `MEM_LOG_INTERVAL_S` / `SHUTDOWN_GRACE_S` (in PR ChengLabResearch#109)
  and no longer needs documentation.
- Q12: rewrite `shm_size` section with Sorotassu's rationale (Docker's
  aggressive 64 MB default makes the override necessary; setting it
  too large shifts OOM from artificial limit to host OOM) and the new
  packaging-time env override `OUROBOROS_SERVER_SHM_SIZE`.
- Q13: add a "Policy" line to the Python base image pin: intentional,
  driven by transitive-dependency stability, do not bump reactively.
- Q14: add a "Bump workflow" note to `productionPluginPins`: they and
  the fallback `neuroglancerArtifactForTag` / `autosegArtifactForTag`
  builders in `scripts/prepare-package-flavor.mjs` move together on
  every plugin release; single-file edit.
@tavateva

tavateva commented Jul 7, 2026

Copy link
Copy Markdown
Author

Thanks for the answers. Landed as two changes:

  • Code side: PR #109 - chore(constants): named ports w/ env override, tighten iframe allowlist, split memory interval, env-configurable shm_size. Covers Q1, Q4, Q6, Q7, Q11, and Q12. Q6 runtime drift check was scoped down to a TODO(#107) on the TS side because ServerContext does not currently expose baseURL to non-fetch-hook callers; the Python enum + GET /step-names endpoint is landed.
  • Docs side: one additional commit on this branch (docs/107-gh-pages-refresh), commit 9a58cb9 - docs: address #108 follow-up answers (constants renames, invariants, rationales) (#107). Covers Q1 renames, the Q2 VOLUME invariant call-out, Q7 env vars, Q8 "Why 'main'", Q9 non-user-surfacing note on DEFAULT_SPLIT_THRESHOLD, Q10 chunk_size=160 rationale in your words, Q11 removal of the MEM_INTERVAL_TIMER entry, Q12 shm_size rationale + OUROBOROS_SERVER_SHM_SIZE, Q13 pin policy, and Q14 bump-workflow note.

Q3, Q5, Q15 were "fine" / "no" and needed no action.

Verification on the code side: npm run typecheck passes; docker-compose config succeeds on both python/compose.yml and python/compose.dev.yml with the default substitution and with OUROBOROS_SERVER_SHM_SIZE=128gb; node scripts/prepare-production-server.mjs generates the expected shm_size (64gb default, 16gb under override). npm run lint was not runnable - the repo's .eslintrc predates ESLint 9 and fails config discovery; unrelated to this change. Happy to file a separate lint-config-migration issue if useful.

den-sq pushed a commit that referenced this pull request Jul 7, 2026
… mem interval, env shm_size (refs #107)

Follow-up to PR #108 (docs sweep). Addresses the constants-side questions
raised in the sub-agent review pass:

- Q1: promote plugin-file-server and volume-server port locals to named
  PLUGIN_FILE_SERVER_PORT / VOLUME_SERVER_PORT constants with env
  overrides OUROBOROS_PLUGIN_FILE_SERVER_PORT and
  OUROBOROS_VOLUME_SERVER_PORT (fallback preserved via parseInt || default,
  which is NaN-safe).
- Q4: tighten IFrameContext allowedOrigins from prefix-match to exact
  hostname equality via new URL(origin).hostname === hostname. Closes the
  http://localhost.evil.example hole; loopback ports remain unrestricted.
- Q6: add StepName enum + step_names_payload in python/ouroboros/common/
  step_names.py and expose it via GET /step-names on the FastAPI server.
  Runtime drift check on the TS side is scoped down to a TODO(#107)
  because ServerContext does not currently expose baseURL to non-hook
  callers; wiring that is architecturally invasive.
- Q7: make HOST / PORT / DOCKER_HOST / DOCKER_PORT env-configurable via
  OUROBOROS_SERVER_HOST / _PORT and OUROBOROS_DOCKER_SERVER_HOST / _PORT.
- Q11: split MEM_INTERVAL_TIMER into MEM_LOG_INTERVAL_S (polling) and
  SHUTDOWN_GRACE_S (exit_cleanly sleep). Both stay at 1.0 by design; the
  split lets one be tuned without silently moving the other.
- Q12: env-configurable shm_size at packaging time
  (OUROBOROS_SERVER_SHM_SIZE, default 64gb) via
  scripts/prepare-production-server.mjs, plus \${OUROBOROS_SERVER_SHM_SIZE:-64gb}
  in python/compose.yml and python/compose.dev.yml. Verified with
  docker-compose config on both files (default and override).

Docs updates land in PR #108 (docs/107-gh-pages-refresh).

Verification:
- npm run typecheck passes.
- docker-compose -f python/compose.yml config and
  docker-compose -f python/compose.dev.yml config both succeed with
  default substitution and with OUROBOROS_SERVER_SHM_SIZE=128gb override.
- node scripts/prepare-production-server.mjs generates compose.yml with
  default 64gb; OUROBOROS_SERVER_SHM_SIZE=16gb override honored.
- Python step_names module imports and step_names_payload() returns the
  expected mapping.

npm run lint is not runnable in this branch: ESLint 9.39 rejects the
repo's legacy .eslintrc setup, unrelated to this change.
- Not everything needed an explanation or inclusion here;  removed too obvious or not necessary.
- Rewrote the chunk size explanation to better reflect specifics.
@den-sq
den-sq merged commit 299bb25 into ChengLabResearch:main Jul 7, 2026
1 check passed
@tavateva
tavateva deleted the docs/107-gh-pages-refresh branch July 7, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refresh gh-pages documentation and document technical constants

2 participants