Status: Active — Authoritative implementation plan for all phases.
This project is a POC and prioritizes clarity/simplicity over backward compatibility.
- Cross-epoch backward compatibility is not required by default (wire formats, projected schemas, DB layout, and migrations may change).
- If an old DB/protocol epoch is encountered, fail fast with a clear recreate/reset path instead of adding compatibility shims.
- Add backward compatibility only when explicitly requested for a specific test objective or transition window.
This document preserves historical phase labels, but the build order from a fresh base is authority-first. Build the canonical event graph, invite/accept flow, and projected trust graph before hardening transit security.
Phase 1: CLI + daemon around the current simple prototype.Phase 4: Event schema, recording semantics, and multitenancy foundation.Phase 5: Projector core and dependency blocking (without full queue complexity).Phase 6: Shared signer substrate (signed_bydependency blocking + signature verification ordering).- Pull forward the minimal identity slice of
Phase 12: accepted-workspace binding, invite cascade,peer_shared, and projected trust rows. Phase 7: Multitenancy scoped-projection/query gate (with signer substrate active).Phase 13: Functional multitenancy — one node hosting N tenant identities in a shared DB with one shared QUIC endpoint and per-tenant routing/discovery.Phase 2: authenticated QUIC transport hardening on top of the real identity/trust graph.Phase 3: Multi-workspace/tenant routing smoke on top of real tenant identities, not CLI-supplied placeholder trust.Phase 8: Encrypted events using the same dependency/projector model, tested first with per-instance PSK.Phase 9: Durable queue architecture (ingress,project,egress) and workers.Phase 10: Non-identity special-case projector logic (deletion/emitted-events).Phase 11: Performance hardening, observability, scaling, and low-memory iOS mode.
Scheduling note:
- transport work is downstream of identity work:
- do not start Phase 2 or Phase 3 with manual pinning, static allowlists, or test-only trust seeding,
- start them only after invite/bootstrap and steady-state trust are projected from real events.
- two-tier multitenancy plan:
- Phase 7 proves scoped projection/query separation once projector + signer substrate exist.
- Phase 3 is a later transport/routing smoke over those real tenant identities.
signed_bydependency blocking + signature verification ordering is tackled in Phase 6.- Phase 6 and Phase 7 must be complete before the full identity projector surface in Phase 12, but the minimal invite/accepted-workspace/PeerShared trust slice must be pulled forward before Phase 2.
- Phase 13 depends on Phase 12 identity flows (
create_workspace,join_workspace_as_new_user,add_device_to_workspace) being stable.
The earlier codex-simplified gap audit served bootstrapping and is now historical.
For active work:
- use this PLAN for build order, scope cuts, and phase exit criteria,
- use DESIGN.md for normative behavior and invariants,
- use DESIGN_DIAGRAMS.md for code-accurate runtime control/data-flow topology,
- use git history/PR notes for resolved-gap chronology.
- Terminology:
workspaceis the event and domain term for the peer set and shared context; "network" refers only to transport/networking. - Connection/sync state is protocol/runtime state, not canonical events.
- Canonical events are durable, replayable, and mostly projector-autowritable.
- Local-only events remain canonical events, selected by
event_typepolicy. - Event format stays flat. No universal
depsfield and no universalpayloadobject. - Any schema field that references
event_idis a dependency source. - Use one blocker mechanism for everything, including missing keys for encrypted events.
- Blocked-event normalcy rule:
- blocked events are still canonical/shareable facts and can be expected end-states for some tenants.
- after a sync connection run, appropriately blocked events do not indicate protocol failure.
- expected examples include encrypted content for non-recipients and key-share events for non-recipients.
- No per-event transit wrapper. Authenticated QUIC secures the channel.
- Use separate tables for permanent canonical data vs operational queues.
- Use separate invite event types (
user_invite_shared,peer_invite_shared), not one multimodal invite withmode=*. - Canonical naming for local/shared pairs:
*_secret/*_shared(peer_secret/peer_shared, daemonendpoint_secret/endpoint_shared,invite_secret+ invite shared events,key_secret/key_shared). - Terminology migration: the architecture now uses
accountfor the workspace-scoped app principal andendpointfor the daemon-scoped transport principal. The current wire/schema event name remainspeer_sharedfor compatibility, but it should be read as today'saccount_shared. - Event projector rule: each event writes to its own table; projectors may read dependency-event tables as context, but cross-event effects should flow through emitted events/commands, not direct ad-hoc writes into other event tables. (Operational non-event tables, e.g. bootstrap trust bridges, are separate.)
invite_acceptedis local accepted-workspace binding:- it writes
invites_acceptedrows from carriedworkspace_idin peer scope, - it is not gated by the root-workspace accepted-workspace guard itself,
- authority reads that require one workspace binding normalize by distinct
workspace_id: same-workspace duplicate acceptances collapse to one binding, while distinct-workspace ambiguity rejects rather than selecting a canonical invite event.
- it writes
- Accepted-workspace guards apply to root workspace events (foreign root ids must not become valid).
- Deterministic emitted event types stay inside the emitted-event rule flow but are unsigned for determinism (
no signed_by/signer_type/signature).
These are required, not optional:
- One projector entrypoint for all ingest paths.
local_create,wire_receive,replay, and unblock requeue all converge on the sameproject_one(recorded_by, event_id)path.
- One dependency engine for all event types.
- dependency refs come only from schema metadata on flat fields.
- no per-type ad-hoc dependency checker.
- Split invite types with shared implementation core.
- keep separate event types (
user_invite_shared,peer_invite_shared) for model clarity. - implement with one shared invite projector helper/spec mapping.
- keep separate event types (
- One key-wrap event model across phases.
- PSK bootstrap phase and identity sender-keys phase use the same key-wrap event/projector path.
- only key source and wrapping algorithm differ.
- DRY queue mechanics.
- DB-backed queues such as
project_queueshare generic claim/lease/retry/backoff helper code.
- DB-backed queues such as
- Isomorphism requirement.
- once canonical event bytes are persisted, source differences disappear (
local_create == wire_receive == replay) for projection logic.
- once canonical event bytes are persisted, source differences disappear (
- Shared projection tables with tenant-scoped autowrite.
- all peers write into the same per-event projection tables (no per-peer table fanout).
- subjective/projected rows carry tenant scope and use composite identity (
peer_id,event_id) semantics.
- Emitted-event self-projection rule.
- projector side effects should emit canonical events; each emitted event projects to its own event table via its own projector/autowrite path.
- dependency-event tables may be read for context; they are not write targets.
- direct cross-event table writes are rare operational exceptions only.
- outstanding exception debt to retire:
message_deletionstill issues directDeletewrites intomessages/reactions(target state should instead derive from deletion events + query joins).
- Blocked-is-not-failure requirement.
- blocked rows that are policy-appropriate for a tenant (for example non-recipient encrypted/key-share events) are expected and must not be treated as sync failure.
- Guard-placement requirement.
invite_acceptedis a local anchor-binding event, not a global invite-presence gate.- accepted-workspace gating belongs on root workspace event validity.
- do not use pre-projection raw-blob capture tables as authority for accepted-workspace binding.
- Identity pre-derive.
create_workspace,accept_invite, andaccept_device_linkpre-derive the PeerShared key and write events under the finalrecorded_byfrom first write — nofinalize_identity.- invite acceptance / device link may still install invite-derived bootstrap tenant session-auth creds first (in legacy
local_transport_*replay state), but the live QUIC handshake uses the daemon transport identity; tenant scope key remains final and projection later installs PeerShared-derived tenant session-auth identity. - raw QUIC connections are daemon-scoped, while steady-state peer/workspace routing resolves per logical session via
OpenSessionRoute; bootstrap or new-workspace admission usesOpenSessionAuthInvite. One live daemon connection may host many tenant/workspace sessions over time. create_workspaceis strictly tenant-scoped once local tenant session-auth rows exist:recorded_bymust match a known local tenant peer ID inlocal_transport_creds; unscoped aliases are rejected. Fresh DB bootstrap (no local tenant rows) remains allowed.
- Transport fingerprint bridge.
peer_sharedprojection materializes deterministicpeers_shared.transport_fingerprintand indexes(recorded_by, transport_fingerprint).- trust lookup paths use projected
transport_fingerprintrows and do not fallback to runtime scan+derive overpeers_shared.public_key.
- Context-query planner seam for proof-bearing runtime policy.
- security-sensitive runtime behavior should be structured as
context query -> normalizer -> pure planner -> executor. - the context query is responsible for loading typed raw rows from the SQL state that determines behavior.
- a pure normalizer collapses those rows into the normalized decision boundary.
- here
DecisionContextmeans the normalized post-query decision boundary, not a full database snapshot: the smallest typed value that still determines one runtime decision. - raw typed rows belong to the query boundary; the
DecisionContextis the post-normalization ADT consumed by the planner. - the planner must be side-effect free and emit an explicit plan enum/flags.
- the executor must perform only the side effects authorized by that plan.
- when we want Verus/TLA coverage, prove the planner over the query
DecisionContextand model temporal/runtime composition around it instead of proving directly over interleaved SQL and network code. - do not collapse SQL/query correctness, normalizer correctness, and planner correctness into one obligation:
- SQLite/query execution may be trusted to run as written,
- the SQL text still needs runtime tests proving it returns the intended raw rows,
- the normalizer and planner are the preferred Verus proof targets.
Goal: cover security-sensitive behavior as a composition of proof-bearing seams instead of one monolithic proof over live SQL, network I/O, and concurrency.
End-to-end chain to preserve:
- command parameters or outside wire input become canonical events where possible,
- events project through the normal projector pipeline into SQLite,
- SQL queries return typed
RawRowsfor one decision, - normalizers build constrained
DecisionContextvalues from those rows, - pure planners emit explicit
Planvalues, - executors perform only the runtime actions named by the plan.
Required shape per seam:
RawRows: typed query result(s) representing the actual SQL boundary.DecisionContext: normalized decision context with impossible combinations removed.Plan: explicit bounded authority/effect result.normalize_*: pure collapse fromRawRows -> DecisionContext.decide_*: pure planner fromDecisionContext -> Plan.- executor: runtime code that performs only the effects named by
Plan.
Proof organization (see docs/DESIGN.md §"Verus Proof Model" for the canonical spec):
topo-verus-proofsis a path-dep oftopo, not a parallel mirror. Normalizers and planners are written as executablepub fninsideverus!blocks withrequires/ensuresclauses. The runtime imports them viapub use topo_verus_proofs::...and calls them directly.cargo-verus verifySMT-checks eachensuresagainst the actual function body. A body change that violates the postcondition fails the merge gate.- Where runtime types carry
String/Option<RuntimeEnum>/Vec<RuntimeStruct>payloads that cannot cross the crate boundary, use the verified core + runtime adapter pattern: Verus fn takes/returns a primitives-only*Coreshape; the runtime has a thin unverified adapter that projects → calls verified fn → rehydrates payload. - The Verus tree under
verus-proofs/src/mirrors production ownership (runtime/...,state/...,pipeline/...) so proofs can be reviewed with the code they verify. - Abstract
spec fnmirrors of runtime functions are deprecated — they pass SMT but say nothing about the runtime body. Any remainingspec fns must either (a) be intermediate predicates cited by anensureson a grounded exec fn, or (b) carry a_specsuffix + comment explaining why they have no single-function runtime counterpart (typically protocol frame shapes or emergent cross-function invariants). - The tamper test (
scripts/verus_tamper_test.sh) runs in the primary merge gate and asserts that flipping an ensures-protected function body causescargo-verus verifyto fail with "postcondition not satisfied". This guards against Verus upgrades silently breaking the SMT-on-real-bodies guarantee. - The working seam inventory is
docs/planning/FORMAL_SEAM_COVERAGE.md; update it when a proof-bearing seam is added, moved, or renamed.
Required proof/test split:
- runtime tests prove query correctness:
- the real SQL returns the intended raw rows for representative DB states,
- ambiguity, missing rows, expiry, and removal appear when they should.
- Verus proves normalizer and planner correctness:
- malformed or contradictory raw states fail closed,
- the planner is deterministic,
- irrelevant fields do not change the plan,
- each seam-specific safety property holds over the
DecisionContext.
- runtime tests prove executor conformance:
- each plan variant produces exactly the named effects and no broader authority.
- TLA covers temporal/concurrent properties:
- retries,
- refresh loops,
- cancellation,
- fairness/liveness,
- cross-component temporal composition.
Repo-wide composition invariants:
- Unique current authority or reject.
- No security-relevant runtime action may start unless it is justified by a unique, current, non-revoked binding or authority chain.
- Missing, ambiguous, expired, or removed authority must resolve to rejection.
- Already-local workspaces cannot gain bootstrap power.
- If a workspace is already local, bootstrap-derived link data must not create dial targets, bootstrap auth fallback, bootstrap trust, or bootstrap-started sync for that workspace.
- Workspace confinement.
- Workspace-scoped replay, fanout, and sync may only affect tenants that are bound to that same workspace by valid local state.
- Ambiguity and malformation fail closed.
- Contradictory, conflicting duplicate, malformed, or out-of-range context state must normalize to rejection, ambiguity, or absence, never silent authorization.
- Executors cannot exceed their plan.
- Runtime code may only perform side effects explicitly authorized by the computed plan.
- Executors must not widen scope by re-querying broader authority after planning.
Each proof-bearing seam should declare:
Requires: which upstream invariant(s) it assumes.Provides: which composition invariant(s) it helps uphold.RawRows,DecisionContext, andPlantypes.- Verus lemmas covering normalizer and planner properties.
- runtime tests covering query correctness and executor conformance.
- TLA model coverage when the property is temporal rather than single-step.
Rollout order:
- Auth and admission.
- outbound session auth,
- inbound route/bootstrap admission,
- sync admission.
- Dial and dispatch.
- bootstrap dial target selection,
- bootstrap fallback selection,
- target dispatch reconciliation,
- connection lifecycle retry/eviction.
- Sync scheduling.
- range/window scheduler,
- hot/cold window restrictions,
- dependency-expansion policy.
- State fanout and cleanup.
- shared workspace fanout,
- projection unblock/requeue disposition,
- deletion/purge planning.
- Event-pipeline boundaries.
- command/wire validation inputs,
- projector context-load/query normalization seams.
Hot-path performance gate:
- planner seams on sync/auth hot paths must not move runtime decisions into per-message loops,
- each hot-path refactor must name the benchmark it is guarding and the metric that must not regress,
- any planner change touching warm sync/catchup behavior must run the relevant release benchmark before merge.
Definition of done for a covered seam:
- the runtime decision uses
query/context -> DecisionContext -> planner -> executor, - the planner consumes only the normalized
DecisionContext, - Verus proves the seam-local safety properties,
- runtime tests prove query correctness and executor conformance,
- any touched temporal property is covered in TLA rather than hand-waved,
- the seam is mapped to one or more repo-wide composition invariants above.
The execution plan for finishing the remaining single-node coverage work lives
in
docs/planning/SINGLE_NODE_FORMAL_COVERAGE_EXECUTION_PLAN.md. That document
adds per-area success criteria, required checks, and end-to-end validation so
the remaining work can be tracked as completion criteria instead of open-ended
proof intent.
Primary formal gate rules:
scripts/run_merge_readiness_checks.sh targetedis the primary agent/merge-readiness gate and must run the formal checks before the serial regression cases.- That primary gate must run
cargo test --lib boundary_tests::,cargo test --lib registry_formal_projector_coverage, and strict Verus verification (TOPO_REQUIRE_VERUS=1 scripts/run_verus_proofs.sh). - Repo command entry files (
src/**/commands.rs,src/**/commands_api.rs) must be indexed indocs/planning/COMMAND_FORMAL_COVERAGE.mdwith a Verus mirror and concrete targeted checks. - Registered event projectors must remain assigned to a Verus-covered projector family, and projector TLA/runtime mapping checks must stay in the primary gate path.
- New runtime/state decision seams, repo command entry files, and new Verus source files must fail the primary gate until their coverage inventory and proof mirrors are updated.
- If a covered runtime command/seam/projector path changes, the same diff must also touch one of its mapped Verus mirrors.
Before more transport/auth work lands, the intended authority model is:
endpoint_secretis a local root and may exist before any workspace, invite, or account exists.invite_acceptedis the local accepted-workspace binding that breaks the bootstrap cycle; it is not a normal workspace-presence dependency gate.- Bootstrap connection admission is exact, not node-global:
- the remote side must prove one invite,
- for one local target account/workspace,
- while bound to the actual authenticated remote
irohendpoint id.
endpoint_sharedexists now as a self-signed shared transport identity in endpoint scope; the next step is makingaccount_shared(currently implemented aspeer_shared) depend on it as the steady-state auth-graph binding from account to endpoint.- After
endpoint_sharedis projected, steady-state logical sessions should useOpenSessionRouteonly. - Eventual consistency comes from reject/disconnect/retry after missing projection state arrives; it must not come from permissive bootstrap windows or "some unbound trusted peer exists" heuristics.
The focused safety model for this target is EndpointBootstrapRoute.tla, checked with endpoint_bootstrap_route_fast.cfg. It proves the key safety invariants we want before rewriting the runtime:
- bootstrap proof is exact to the authenticated remote endpoint,
- bootstrap proof is exact to the invite target and workspace,
- routed admission requires exact shared-workspace and endpoint binding,
- route authority comes from projected graph state, not bootstrap side tables.
Every CLI instance is a real peer-to-peer device. All user-facing commands go through daemon RPC to the service layer:
- One service layer: business logic lives in event modules (
src/event_modules/) for event-specific concerns, in state-owned local infra modules for non-event facilities (for examplesrc/state/subscriptions/*), and insrc/runtime/control/service.rsfor orchestration. CLI subcommands are thin UI adapters that call daemon RPC, which dispatches to those owner modules. - Real networking: invite acceptance uses real QUIC bootstrap sync, not in-process event copying. The daemon manages ongoing sync with discovered peers.
- Testing equivalence: CLI integration tests exercise the full path (CLI binary → RPC → service → DB/sync). No separate interactive surface exists; the daemon-backed CLI is the single command interface.
- No synthetic shortcuts: no
copy_event_chain, no direct DB-to-DB event transfers, no bypass of the sync/projection pipeline. Every event flows through the same ingest path it would in production. - New test preference: new user-visible and runtime behavior should prefer CLI end-to-end coverage. If a test needs metadata not already exposed by CLI/RPC, add first-class daemon/CLI observability instead of directly querying internal tables from the test.
- One CLI instance = one device: each running
topoprocess is a device with its own transport identity and persistent state. - Multiple tenants per device: a single device can host many tenants, each participating in arbitrary (potentially overlapping) workspaces.
- Zeroconf discovery: daemon-scoped
irohmDNS discovers peer daemons on the same local machine or LAN. - Single-port QUIC endpoint: one shared
irohendpoint serves all tenants with one daemon-scoped identity replayed fromendpoint_secret. The daemon creates that local endpoint root before any workspace exists so the transport principal is stable across replay and restart. Inbound daemon connections are quarantined until the first routed/bootstrap control frame proves one exact tenant/workspace scope; if that proof does not arrive quickly, the connection is dropped.- the raw daemon connection is shared across tenants; routed range sessions carry current sync work on that one connection, and dependency/auth/key catchup is prioritized by range selection rather than a separate dependency session.
- same-workspace sibling tenants should reuse the same daemon connection, not compete with extra QUIC congestion controllers to the same remote daemon.
- the direct-connect timing and hole-punch job belongs to
iroh. The current branch accepts n0 relays for rendezvous instead of keeping a bespoketopo intro/IntroOffersubsystem. - known drawback: anyone who knows the daemon's
irohaddress or relay-reachable endpoint can hit the pre-proofiroh/toposurface and exercise timeout/parser/admission paths. Membership proof is still required before any workspace-scoped sync is allowed.
- Shared batch writer: all tenants on a device share one batch writer for projection, grouped by
recorded_by. - Same-workspace shared-DB convergence: if multiple local tenants share one DB and one workspace, shared events must fan out locally across sibling tenant scopes after local create or wire ingest, and newly accepted tenants/devices must replay already-present shared workspace history into their own scope.
- Already-local workspace bootstrap isolation: if invite acceptance targets a workspace that is already present locally anywhere in the DB, the accept path must treat that join as local-only. Link-supplied bootstrap endpoint/address/relay data must not create bootstrap trust rows, bootstrap dial targets, or outbound sync authority for that workspace. Regression coverage for this rule must prove that local fanout/replay stays intra-DB and that forged or stale bootstrap link data does not cause raw shared events from the already-local workspace to be sent outward.
We also considered removing daemon-to-daemon logical sessions entirely and instead signing every control message individually, routing by signer/workspace mapping at ingest time.
Pros:
- simpler connection sharing model,
- no explicit session-open state,
- easier to reason about stateless control handlers in isolation.
Cons:
- negentropy still needs round correlation, replay windows, and response matching, so the design is not truly stateless in practice,
- asymmetric verification would move onto the hot path for every control message instead of once per logical session,
- larger signed envelopes would be paid repeatedly for the same peer/workspace conversation,
- prioritized lanes become harder because admission and replay protection have to be re-solved per message rather than once per session.
Current plan: keep daemon-scoped QUIC connections plus routed logical sessions inside them. That pays the signature cost only when bootstrap or new-workspace admission is needed, preserves explicit priority lanes, and still allows many tenants to share one raw connection.
For direct peer-to-peer optimization, the plan is also daemon-scoped: iroh relays provide rendezvous and NAT traversal, and any later direct-path upgrade happens inside the same daemon connection surface. We do not maintain a custom introduction or punched-session protocol anymore.
Build this before queue complexity.
- One daemon per profile/peer with local RPC control socket.
- Thin CLI (
topo) for non-interactive control. - Stable JSON responses and exit codes.
- Assertion-first testing commands:
assert-now <predicate>assert-eventually <predicate> --timeout-ms <n> --interval-ms <n>- optional
assert-stable <predicate> --for-ms <n>
- Locks in operational UX for human + agent workflows.
- Lets LLM self-play loops run through real commands instead of ad-hoc waits.
- Provides a stable harness before deeper internal refactors.
- Two daemons exchange chat events over QUIC.
- CLI can create/query/assert deterministically in scripts.
- Real-time feel is acceptable with current simple path.
Phase 1 is functionally complete. All deliverables are met:
start,send,messages,status,generateCLI commands work.generatesupportshistory_span; generated messages are spread across that interval and default to a 3-year span when unspecified.assert-nowandassert-eventuallycommands enable deterministic scripting.- CLI integration tests use assert commands (no ad-hoc wait helpers).
- JSON output is not required; human-readable output is sufficient.
Phase 1 CLI/daemon shape must preserve:
- versioned request/response envelopes and bounded frame size (
<= 16 MiB) over length-prefixed JSON RPC framing, - bounded concurrent RPC connection handling in daemon,
--dbtakes a literal file path (default:topo.dbin CWD). No registry or alias resolution.
- Peer-authenticated QUIC sessions with exact transport-fingerprint validation against projected trust rows.
- Runtime protocol messages (sync and routed/bootstrap session admission) handled outside canonical events.
- Accepted-workspace binding, invite acceptance, and
peer_sharedprojection already exist. - Bootstrap and steady-state trust rows are projection-owned SQL state.
- At least one realistic flow proves
create_workspace -> invite/link -> accept -> bootstrap sync -> steady-state direct syncwithout manual trust seeding.
- No transit event wrapping layer in this model.
- Phase 2 is not a substitute for identity bootstrap; it consumes the projected trust graph built earlier.
- Event signature/dependency enforcement is delivered in Phase 6 (
signed_byblocking + signature verification ordering). - Transport authentication must remain separate from event authorization semantics.
- No placeholder pinning or ad-hoc transport authority is permitted in this phase.
- Reconnect/retry behavior is stable across daemon restarts.
- Daemon transport identity is plumbed into peer/session context.
Phase 2 transport hardening is implemented and should be maintained as strict authenticated QUIC with exact transport-fingerprint checks backed by projected trust rows.
Normative transport requirements now live in DESIGN.md:
- transport/auth model: §2.1-§2.5,
- trust-source lifecycle: §9.5,
- multitenant runtime integration: §3.2.1-§3.2.3.
When touching transport in future phases:
- preserve strict exact-fingerprint checks and reject-any-untrusted behavior on both client and server,
- keep transport trust SQL-backed and projection-owned (no ad-hoc authority path),
- keep session identity mapping explicit (
recorded_by,via_peer_id), - keep tenant session-auth cert/key materialization behind the established transport-identity contract boundary, while the live daemon endpoint root continues to replay from
endpoint_secret.
Policy for future transport work:
- do not add manual peer pinning or test-only trust seeding paths,
- build invite/trust-graph realism before adding transport security layers so transport starts from the same authority model used in production,
- prefer projected trust-row queries and exact transport fingerprints over ad-hoc in-memory trust state.
Sync keeps one authenticated daemon connection per remote daemon and opens routed range sessions inside it.
- discovery remains round-scoped (
NegOpen/NegMsg), - outbound rounds cycle
last day -> last week -> last twelve weeks -> old, then repeat per peer, - the data stream carries
Eventblobs for the active range, - live suppression mode also carries batched
SuppressIdsandRangeDataDoneframes on the data stream, - auth/removal-frontier and key ranges are scheduled ahead of hot message ranges,
- duplicate raw daemon connections are resolved by a deterministic tie-breaker only after both sides have authenticated.
The active sync path still uses stock Negentropy-style range fingerprints, which is the simpler production choice today, but it has a known security caveat.
- The original range-based set reconciliation paper and current stock Negentropy implementations use an algebraically composable fingerprint for each range.
- That hashing scheme admits attacker-chosen collision attacks: a malicious peer can interfere with syncing of other peers' events by causing unequal ranges to look equal and therefore preventing deeper reconciliation in those ranges.
- The known stronger alternative is Range-Based Set Reconciliation without Homomorphic Fingerprints, which computes range fingerprints from a clamped Merkle tree instead of the classic additive / homomorphic-style fingerprint.
- The tradeoff is a somewhat slower snapshot rebuild when a durable shared-index epoch changes.
- Repo guidance for now: keep one immutable in-memory dep-aware snapshot per standard window and reuse it across repeated sync checks until the durable shared-index epoch for that window changes. The protocol still leaves room for a stronger Merkle-style auxiliary structure later, but the current baseline is cached snapshot reuse, not rebuild-per-session.
For any transport-affecting PR, require:
- positive/negative handshake tests,
- reconnect/retry tests across daemon restarts,
- transport trust policy tests (new handshakes follow current SQL trust state; no user-removal teardown behavior is in scope).
- No permissive verifier in production paths.
- No socket-address-based trust decisions.
- No separate transport identity authority outside projected trust state.
Goal: validate basic workspace/tenant separation on top of real tenant identity and trust state.
- Start two provisional workspace/tenant contexts from CLI/profile supplied key material.
- Route transport ingress into the correct tenant scope (
recorded_by) for each context. - Expose CLI workspace selector/scope so reads show each workspace independently.
- Demonstrate separate
recorded_eventshistory per workspace with no cross-display.
- This is a routing/scope smoke phase, not a placeholder-trust phase.
- It uses the active transport trust policy from DESIGN.md §2.1-§2.2 and does not introduce a parallel trust path.
- Signature/dependency enforcement is still deferred to Phase 6.
- Two workspace contexts can run concurrently and exchange events in isolation.
- Event created/received in workspace A does not appear in workspace B scoped CLI queries.
- Basic scoped DB checks pass for
recorded_eventsand at least one projected table.
Define event shape once and drive these from it:
- wire encode/decode
- canonical signing bytes metadata (consumed by signer substrate in Phase 6)
- signer metadata fields (
signed_by,signer_type,signature) - validation scaffolding
- projector auto-row mapping metadata
- dependency extraction metadata (
is_event_ref,required)
Field encoding kinds:
fixed_bytes(N)u8/u16/u32/u64fixed_text(N, utf8=true, zero_pad=true)— fixed-size UTF-8 text slot, zero-padded after contentfixed_ciphertext(N)— fixed-size opaque byte slot
Removed (no longer canonical):
var_bytes(len_prefix=u16|max_len=...)var_string(len_prefix=u16|max_len=..., utf8=true)
No canonical event parser uses a length or count field to determine body boundaries.
- Flat fields per type.
- Deterministic field order from schema.
- Every canonical event type has a fixed total wire size (deterministic by type code).
- Encrypted events have wire size deterministic by
inner_type_code. - Length-prefixed framing for sync transport (frame-level
payload_lenonly; no in-event length fields).
This supports deterministic parsing by type dispatch and fixed offsets (langsec-first).
payload_len is a framing delimiter, not semantic authority:
- it must exactly match the schema-defined fixed size for the event type,
- for encrypted events it must match the size determined by
inner_type_code, - any mismatch rejects the frame.
Frame safety bounds (required):
- enforce a global max frame payload length before allocation/decode,
- enforce per-frame-type max lengths,
- reject any frame where declared
payload_lenexceeds either bound.
signed_by: canonical signer reference (event id).signer_type: signer keyspace discriminator (peer | user | workspace | invite).recorded_by: local tenant peer identity that recorded/projected the event.signed_by/signer_typeandrecorded_byare intentionally separate concerns.- No
recorded_viafield. recorded_byis derived from authenticated local connection/profile identity, not from event payload claims.- Remote transport identity for metadata is
via_peer_id, resolved from the authenticated daemon endpoint today and eventually from projected endpoint identity in the auth graph.
- Canonical shared events: durable + shareable.
- Canonical local-only events: durable + replayable, not shared.
- Protocol/runtime messages: non-canonical, not in event DAG.
Shareability is by event_type policy, not by separate event tables.
CREATE TABLE events (
event_id TEXT PRIMARY KEY,
event_type TEXT NOT NULL,
blob BLOB NOT NULL,
share_scope TEXT NOT NULL, -- 'shared' | 'local' from event_type policy
created_at INTEGER NOT NULL,
inserted_at INTEGER NOT NULL
);
CREATE TABLE recorded_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
peer_id TEXT NOT NULL, -- semantic: recorded_by
event_id TEXT NOT NULL,
recorded_at INTEGER NOT NULL, -- local receive/create time for this tenant
source TEXT NOT NULL, -- local_create | emitted | import | quic_recv:<peer_id>@<ip:port>
UNIQUE(peer_id, event_id)
);
CREATE INDEX idx_recorded_peer_order ON recorded_events(peer_id, id);
CREATE INDEX idx_recorded_peer_event ON recorded_events(peer_id, event_id);
CREATE TABLE valid_events (
peer_id TEXT NOT NULL,
event_id TEXT NOT NULL,
PRIMARY KEY (peer_id, event_id)
);
CREATE TABLE rejected_events (
peer_id TEXT NOT NULL,
event_id TEXT NOT NULL,
reason TEXT NOT NULL,
rejected_at INTEGER NOT NULL,
PRIMARY KEY (peer_id, event_id)
);
CREATE TABLE peer_endpoint_observations (
recorded_by TEXT NOT NULL,
via_peer_id TEXT NOT NULL,
origin_ip TEXT NOT NULL,
origin_port INTEGER NOT NULL,
observed_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL,
PRIMARY KEY (recorded_by, via_peer_id, origin_ip, origin_port, observed_at)
);
CREATE INDEX idx_peer_endpoint_expires
ON peer_endpoint_observations(recorded_by, via_peer_id, expires_at);
CREATE INDEX idx_peer_endpoint_lookup
ON peer_endpoint_observations(recorded_by, via_peer_id, origin_ip, origin_port);- Subjective tables are keyed by tenant identity first (
peer_id/recorded_by). - Query/projection APIs should use a tenant-bound wrapper (
TenantDb { peer_id, tx }) rather than raw DB handles. recorded_eventsis the per-event receive journal (recorded_at~= localreceived_at).- Endpoint/IP metadata is intentionally separate and append-only in
peer_endpoint_observationsfor frequent TTL purge and reconnect hinting. - If needed,
first_seen/last_seenare derived byMIN(observed_at)/MAX(observed_at)queries, not stored via in-place updates.
Table lifecycle:
- Schema creation uses deterministic owner
ensure_schema(conn)calls (no migration history playback in this POC). - Core tables are created by core owner modules (
state/db/*); event projection tables are created by event-module ownerensure_schemafunctions. - Startup runs deterministic schema bootstrap plus registry/schema consistency checks and fails fast on mismatch.
- Prototype schema epoch is explicit (
schema_epoch) and checked at startup. - No backward compatibility is provided across prototype epochs: if an old DB is detected (legacy
schema_migrationswithout current epoch marker), startup fails with a clear "recreate DB" error.
Naming and ownership:
- Do not infer table names by pluralization heuristics.
- Each event module declares explicit constants/metadata:
event_type(for examplemessage)projection_table(for examplemessages)
- Default shape is one event module -> one projection table -> one autowrite mapping.
- Exceptions are explicit and documented (deletion/tombstones, join tables, operational queue tables).
Multitenant scoping in shared tables:
- Keep one shared physical table per event type (no per-peer table fanout).
- Subjective/projected rows must include tenant scope key (
peer_id/recorded_by). - Use composite keys/indexes with tenant first (for example
(peer_id, event_id)). - For subjective autowrite tables, default uniqueness is exactly one row per
(peer_id, event_id)unless the event spec explicitly defines a different shape. - Shared canonical blobs in one DB may legitimately project into multiple tenant scopes only when those tenants share the same
workspace_id; cross-workspace fanout is always a bug.
Why this rule exists:
- keeps schema ergonomic and queryable,
- preserves
poc-6-style tenant scoping guarantees, - prevents assistants from inventing per-tenant table proliferation.
Phase 4 must introduce a standard event-store replay harness and make it mandatory for scenario coverage.
Required checks per tenant scope (recorded_by):
- replay-once: rebuild projection state from canonical event store order and compare with baseline.
- replay-twice idempotency: run replay again on already replayed state; no additional changes.
- reverse-order replay: replay canonical events in reverse order; final projected state matches baseline.
Harness policy:
- these checks run automatically after every scenario test that writes canonical events.
- source of truth is canonical event store rows (
events+ scoped subjective state), not transient in-memory state. - comparisons use deterministic table-state fingerprints (same mechanism as replay/reproject/reorder invariants).
Implement projection semantics before adding heavy queue machinery.
Projectors are pure functions over (ParsedEvent, ProjectorDecisionContext). They return
a deterministic ProjectorResult containing write operations and emitted commands.
They do not execute SQL or any other side effects directly.
All code paths call the same projector entrypoint:
project_one(recorded_by, event_id) -> ProjectionDecision
struct ProjectorResult {
decision: ProjectionDecision, // Valid | Block | Reject | AlreadyProcessed
write_ops: Vec<WriteOp>, // deterministic state mutations
emit_commands: Vec<EmitCommand>, // follow-on actions for command executor
}
enum WriteOp {
InsertOrIgnore { table, columns, values },
Delete { table, where_clause },
}
enum EmitCommand {
HardPurgeMessageGraph { message_event_id },
RetryWorkspaceEvent { workspace_id },
MaterializeTransportIdentity { spec },
EmitDeterministicBlob { blob },
}Entry-point requirement:
local_create,wire_receive,replay, and unblock retries must all invokeproject_one.- no alternate projection code paths for specific ingestion sources.
- Internal cascade optimization:
project_one_step(the 7-step algorithm without cascade) is used by the Kahn cascade worklist to avoid redundant recursive cascade.apply_projectionexecutesemit_commands(which handles guard retries), so the cascade only manages Kahn dependency unblocking. All projection stages are shared; the split is a performance optimization, not an alternate path.
Apply engine execution stages:
- Pipeline resolves
EventTypeMetaand calls the event-module-ownedcontext_loader(conn, recorded_by, event_id_b64, parsed)to buildProjectorDecisionContext. - Pipeline calls pure projector → receives
ProjectorResult. - Pipeline executes
write_opstransactionally only onValid. - Pipeline executes
emit_commandsvia explicit handlers onValidandBlock. RejectandAlreadyProcessedexecute no write ops and no emitted commands.
DRY split (required):
- Shared projection pipeline code owns:
- canonical event load/decode dispatch,
- dependency extraction + missing-dependency block writes,
- signer resolution + signature verification ordering (Phase 6),
- orchestrating context loading via
EventTypeMeta.context_loader(no projector-specific SQL in pipeline files), - executing
write_opsandemit_commands, - terminal state writes (
valid/block/reject) + queue transitions.
- Per-event projector code owns only:
- event-specific predicate/policy checks,
- returning
ProjectorResultwith deterministicwrite_opsandemit_commands.
- Event modules own projector-specific SQL decision-context queries through module-local context loaders (
queries.rsor projector-local helpers). - Per-event projector functions (
project_pure) must not access the database, implement their own dependency walker, signer verifier, queue handling, or terminal-state writer.
- Most event types use predicate + auto-write.
- Auto-write is typically
InsertOrIgnoreWriteOps of flat event fields + metadata. - Auto-write is tenant-scoped in shared tables (
peer_id/recorded_byincluded in subjective rows and keys). - Validation order for signed events is fixed:
- dependency extraction/check (including signer dependency),
- signature verification using resolved signer key,
- authorization/policy predicate checks,
- pure projector dispatch → WriteOps + EmitCommands.
When a projector emits event X:
- emit canonical event
Xonly (toevents+ normal queue flow), - let
Xproject throughX's own projector/autowrite intoX's table.
Do not directly write into another event type's projection table as a side effect, except for rare non-event operational writes explicitly documented in that projector.
Deterministic emitted-event exception (still under this rule):
- deterministic emitted event types remain canonical events and still follow
emit -> persist -> self-project. - for deterministic event ids/bytes, these types are unsigned:
- schema omits
signed_by,signer_type, andsignature, - signer dependency/signature stages are skipped for those types only.
- schema omits
- validation for these types is deterministic-derivation checks from dependencies/context, not signature checks.
message_deletionuses the two-stage deletion intent + tombstone model (see Phase 10).- deterministic emitted-event patterns (for example key material derivations) using the unsigned deterministic exception above.
- identity-specific exceptions (
invite_acceptedaccepted-workspace binding viaRetryWorkspaceEvent { workspace_id }, transport identity intent application) implemented via explicit projector decisions +WriteOp/EmitCommandhandling.
Two-stage model so deletes stay deterministic when events arrive out of order:
MessageDeletionprojector emits an idempotentdeletion_intentwrite keyed by(recorded_by, target_id).- If target exists in projected state, projector also emits the tombstone WriteOps plus a
HardPurgeMessageGraphfollow-up in the same projection transaction. - If target does not exist yet, projector only records intent; no imperative retries.
- Target-creation projectors check for matching
deletion_intentrows in theirProjectorDecisionContextand immediately tombstone on first materialization, using the original deletion event's identity for replay invariance. - Hard purge owns all live-state cleanup for the deleted message graph.
- Deletion state is monotonic:
active → tombstonedallowed,tombstoned → activeforbidden. - Hard purge removes deleted event blobs and dependent rows in the same projection transaction; only minimal tombstones remain (
deletion_intents,deleted_messages). - Late
reaction/filearrivals on tombstoned messages return a terminal prereq/context purge outcome instead of entering their projectors. - Late encrypted
file_slicearrivals purge by the same owner-message rule: the wrapper carriesowner_event_id = root_message_id, and purge finds those recorded blobs throughrecorded_event_ownersinstead of scanning or decrypting tenant event storage. - If hard purge or verification fails, the whole projection transaction rolls back and the normal queue retry path handles it later.
Deletion invariants validated by tests:
- Duplicate replay leaves state unchanged after first application.
- Delete-before-create converges to identical final state as create-before-delete.
- Full replay reproduces identical tombstone state.
- Authorization failure paths are deterministic from projected context.
- No live reactions, file descriptors, or file slices remain for a tombstoned message.
- Late dependents (
reaction,file,file_slice) on an already tombstoned message purge themselves instead of surviving in recorded/live state. - Command execution idempotence: intent identities are stable, re-running does not mutate state.
Start with blocker-edge persistence plus a small blocked-header table.
CREATE TABLE blocked_event_deps (
peer_id TEXT NOT NULL,
event_id TEXT NOT NULL,
blocker_event_id TEXT NOT NULL,
PRIMARY KEY (peer_id, event_id, blocker_event_id)
);
CREATE INDEX idx_blocked_by_dep_covering
ON blocked_event_deps(peer_id, blocker_event_id, event_id);
CREATE TABLE blocked_events (
peer_id TEXT NOT NULL,
event_id TEXT NOT NULL,
deps_remaining INTEGER NOT NULL,
PRIMARY KEY (peer_id, event_id)
);Rules:
- Extract refs from schema-marked fields on each projection attempt.
- If required refs are present: continue projection.
- If any required refs are missing: dedupe blockers, write rows in
blocked_event_deps, writeblocked_events.deps_remainingfrom unique blocker count, and returnBlock. - Signer refs (
signed_by+signer_type) are dependency metadata and use the same blocking/unblocking path. - Signature verification is attempted only after signer deps and other required deps are available (signed event types only).
blocked_event_depsremains the canonical blockedness check for queue admission guards.blocked_events.deps_remainingis the performance counter used by the cascade scheduler.- Do not persist full
event_dependenciesyet. - Use one dependency resolver for all event families (content, identity, encrypted wrappers, invites).
- Dependency extraction is driven by event schema metadata only (
is_event_ref,required, conditional requirement flags).
When full dependency table is justified later:
- reverse-edge analytics/debugging,
- heavy dependency introspection,
- or proven perf bottleneck from repeated lookups.
- An event can have N blocker rows.
- It is runnable when
blocked_events.deps_remainingreaches zero.
Use counter-based unblock when blocker X becomes valid:
Implementation shape:
- Read candidates from
blocked_event_depsby (peer_id,blocker_event_id) using the covering index. - For each candidate, decrement
blocked_events.deps_remaining. - If remaining deps are > 0, keep it blocked.
- If remaining deps reach 0, delete the
blocked_eventsrow and project the event through the canonical entrypoint. - When a projected event becomes
Valid, use it as the next blocker in the same cascade worklist. - If projection returns
Block { missing }(for example encrypted inner deps), blocker rows are written by the shared dependency stage (check_deps_and_block) before returningBlock. - Keep
blocked_event_depsread-only inside the per-step cascade loop. - After cascade transitions occur, bulk-clean
blocked_event_depsrows for events now terminal (validorrejected). - Run guard retries after dep cleanup so guard queries see current dep state.
Design tradeoff:
- SQL-only unblock (
DELETE ... RETURNING+ blocker-row existence checks) is simpler. - Current code keeps the counter path because branch-local topo-cascade measurements showed roughly 2x higher throughput.
Do not route in-call cascade fanout through
project_queue.
Use one primitive with two entry points:
persist_and_enqueue(event_blob, peer_id) -> event_idcreate_event_synchronous(...) -> event_idcreate_events_batch(events, project_now: bool) -> Vec<EventId>
create_event_synchronous should inline projection until terminal (valid, blocked, or rejected) so command chains can immediately use prior event ids.
Imperative orchestration contract (poc-6 style ergonomics):
- default
create_event_synchronousmust return success only when the created event isvalidforrecorded_by. - if terminal state is
blockedorrejected, return an error containingevent_id+ terminal reason. - this guarantees call-site sequencing:
a = create_event_synchronous(...)b = create_event_synchronous(depends_on=a, ...)
Implementation note:
- even synchronous create uses the same internal pipeline:
persist_and_enqueue->project_oneloop (same code as worker mode)
- do not add a separate "local fast-path" projector.
Product code should prefer two real create modes:
-
create_event_synchronous(strict, PLAN-normative): returnsOk(event_id)only when projection reachesValidorAlreadyProcessed. ReturnsErr(Blocked { event_id, missing })orErr(Rejected { event_id, reason })otherwise. All user-facing service commands (svc_send,svc_react,svc_delete_message,svc_generate) use this API. -
store_*_then_project(...)(rare, explicit): used only when projection needs event-id-dependent context to be written after canonical store but before the first projection attempt. Current example: bootstrap-context invite creation writes transport bootstrap context keyed by the stored invite event id, then projects in the same transaction.
event_id_or_blocked(...) and the create_*_staged(...) helpers are bootstrap/test conveniences layered on top of those two modes. They do not represent a separate storage model.
Local synchronous create no longer depends on create-side origin project_queue rows or create-side pending fanout rows for crash safety. Atomicity comes from committing canonical store plus the first projection attempt together; blocked rows are the durable recovery path for blocked outcomes.
Test index for this contract:
test_create_signed_event_synchronous_returns_blocked_error— strict API blocked→Errtest_create_signed_event_staged_returns_ok_on_blocked— staged API blocked→Oktest_create_reaction_before_target— blocked event DB statetest_react_errors_on_blocked— service layer propagationtest_delete_of_missing_target_writes_intent— service-layer delete behaviortest_send_succeeds_on_valid— valid→Oktest_ensure_identity_chain_tolerates_workspace_blocked— staged bootstraptest_create_event_sync_contract_valid_only— explicit §6.4 contracttest_create_event_sync_contract_blocked_returns_err_with_event_id— error shape contract
Usually not required at this stage, but useful if blocker behavior gets ambiguous:
- model
valid,blocked_event_deps,blocked_events.deps_remaining, and unblock transitions, - verify multi-blocker convergence and no-lost-unblock behavior,
- then map those guards directly into projector dependency checks.
TLA sync status (done):
EventGraphSchema.tlamodelstenantas root (no deps); local peer identity material now flows throughpeer_secret(the old standalonepeerlocal event is retired).key_sharedmodels direct dependency on both recipient invite event andinvite_secretunwrap event.- Re-run TLC configs and refresh
docs/tla/projector_spec.mdmapping notes when dependency semantics change next.
Implement one signer pipeline for all signed event types:
- signer metadata is schema-declared (
signed_by,signer_type,signature). - missing signer dependency uses normal blocking/unblocking (
blocked_event_deps+blocked_events). - resolve signer key by (
signer_type,signed_by) only after dependency resolution. - invalid signature is
Reject, neverBlock. - signer verification helper path is shared across signed event families (no identity-specific signer path later).
- deterministic emitted event types are explicitly schema-marked unsigned (
signer_required=false) and excluded from signer-stage enforcement.
This phase should be completed immediately after Phase 5 and before Phase 8/Phase 12 work.
Goal: validate full tenant/workspace scoping after projector + signer substrate are active.
Required checks:
- Run two workspace/tenant contexts in one DB with shared physical projection tables.
- Project overlapping event shapes in both tenants and verify subjective rows remain isolated by tenant key (
peer_id/recorded_by). - Confirm signer pipeline behavior remains tenant-scoped (missing/invalid signer in tenant A does not leak effects into tenant B).
- Verify CLI workspace selection only reads tenant-scoped data.
- Add/keep a DB scoping checker that fails on unscoped reads/writes.
Exit criteria:
- Cross-tenant leak tests fail correctly when scope guards are removed.
- Re-enable scope guards and pass full scoped projection/query suite.
- This gate passes before identity projector implementation (Phase 12).
Goal: encrypted events behave like normal events for dependencies and projection. Precondition: Phase 6 signer substrate and Phase 7 multitenancy gate are already active.
- Encrypted wrapper is a normal event type in the same event registry.
- It uses flat fields, for example:
key_event_id,inner_type_code,ciphertext,nonce,auth_tag. inner_type_codeis mandatory (fixed-width) for this phase.- Do not make
inner_type_codeoptional whileciphertextremains variable-length. - No separate encryption registry.
"Materialization" means converting an encrypted wrapper into a transient normal event view:
- parse and verify the encrypted wrapper envelope,
- resolve/decrypt using dependency-provided key material,
- decode plaintext into a typed flat
EventViewfrom the same registry, - pass that
EventViewinto the normal projector path.
Important:
- materialization is an adapter step, not a second projection system.
- no persisted plaintext queue is required for baseline.
- after materialization, projection is isomorphic to cleartext event projection.
Rule:
- If decrypted inner type resolves to encrypted wrapper type,
Reject(nested_encrypted_not_allowed).
- Parse outer encrypted event from canonical
events.blob. - Extract outer deps from its flat event-ref fields (
key_event_id, plus any other refs). - If outer deps missing: write
blocked_event_deps+blocked_eventsand returnBlock. - Verify signature/auth over canonical encrypted bytes.
- Decrypt ciphertext using key from
key_event_id. - Decode inner event with normal registry.
- Verify decoded inner type matches outer
inner_type_code; mismatch ->Reject(inner_type_mismatch). - If inner type is encrypted wrapper: reject.
- Extract inner deps from inner schema metadata.
- If inner deps missing: write
blocked_event_deps+blocked_eventsusing outerevent_idand returnBlock. - For decrypted inner events, enforce dep type checks against tenant-scoped
valid_events.semantic_type_code, so inner deps may target encrypted wrappers while still validating by semantic inner type. - Run the normal shared signer+projector stage for the inner type.
- Mark outer event
validonly after inner projection succeeds.
- Default: no persisted plaintext queue.
- Decrypted plaintext exists in memory only for projection.
- Optional later optimization: short-lived decrypted cache with TTL.
This preserves one blocker model and one projector model.
Encryption tests use deterministic local key materialization (shared key bytes + deterministic created_at_ms from BLAKE2b hash) to set up key state on both sender and recipient sides. This matches the production invite-key wrap/unwrap flow where both parties derive identical key_secret event IDs.
Test harness contract:
- Materialize key as a local
key_secretevent in the correct tenant scope (recorded_by). - Run encrypted projection through the standard block/unblock flow:
- missing key event →
Block - key present + decrypt/auth failure →
Reject - key present + decrypt/auth success → inner dep/signer/dispatch stages
- missing key event →
- Keep all replay/reorder invariants enabled.
The same key_shared event type and wrap/unwrap projector logic is used for both bootstrap (wrap to invite key) and runtime (wrap to peer key) key distribution. Only the recipient key source differs.
Add full queue machinery after projection + signer + encryption semantics are stable.
CREATE TABLE project_queue (
peer_id TEXT NOT NULL,
event_id TEXT NOT NULL,
available_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
lease_until INTEGER,
priority_lane INTEGER NOT NULL DEFAULT 1,
priority_ts INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (peer_id, event_id)
);
CREATE INDEX idx_project_queue_claim
ON project_queue(peer_id, priority_lane, priority_ts DESC, available_at, event_id);Keep canonical and queue data separate:
- permanent:
events,recorded_events, projection outputs - operational/transient:
project_queue,blocked_event_deps,blocked_events - removed:
ingress_queuewas dropped in migration 28 because no runtime writer or reader used it
A single table sounds simple but mixes incompatible concerns:
- very different retention and purge policies,
- different dedupe keys and lease semantics,
- harder indexing and worse observability.
Separate queue tables stay simpler operationally.
Runtime flow reference: DESIGN_DIAGRAMS.md sections 1, 2, and 4.
range receive path(current runtime): QUIC frame -> hash event id -> publish live suppression when enabled -> enqueue bounded in-memory direct-ingest batch -> canonical insert -> record by tenant -> enqueue project.project worker: claim row -> project path (valid/block/reject) -> dequeue.range session worker: dep-aware Phase 1 over one selected root window discovers missing roots and dependency-probe roots, an explicit candidate-dep exchange names the dep universe, stock Negentropy Phase 2 confirms missing deps, then the session streams event blobs and live-suppression frames when enabled.cleanup worker: reclaim expired project leases and TTL-purge old endpoint observations.
Queue DRY requirement:
- implement generic queue helper traits/functions once (
claim_batch,renew_lease,mark_done,mark_retry/backoff). - keep them shared for DB-backed queues such as
project_queue, rather than duplicating retry/lease semantics.
Sync event transfer is range-owned:
- each session selects one explicit range from the tier ladder,
- both sides reuse the cached dep-aware snapshot for that durable root range,
- dep-aware Phase 1 compares combined fingerprints for root slices and exacts only root ids for small mismatched slices,
- both sides expand dep candidates from their Phase 1 local-only and dep-probe roots, exchange those candidate ids, and run stock Negentropy Phase 2 over that dep universe,
- each side streams the resulting missing roots plus confirmed-missing deps directly on the session data stream,
- live suppression mode lets receivers quickly advertise event ids they have already received or are receiving from sibling sessions,
- a bounded direct-ingest queue applies backpressure when SQLite falls behind instead of spilling to a file-backed receive log,
- the next sync session with a given peer waits until the prior session's direct-ingest batches have drained for that same peer,
- there are no durable
wanted/wanted_sourcestables, noResponseCredit, no pending-receive overlay, and no per-event pull coordinator in the current plan.
project_queueis transient, not history.- Do not enqueue if already terminal or currently blocked.
- If duplicate enqueue races happen, worker should fast-drop after terminal check.
- On terminal decision (
valid,reject,block), remove row fromproject_queuein the same transaction as state write. - Queue classification uses outer semantic type. For encrypted wrappers the runtime reads the wrapper-exposed
inner_type_code, so encryptedfile_slicerows still enter the bulk lane without waiting for decryption. project_queuestorespriority_tsfor policy metadata, but current claim order is lane-first and FIFO within the lane so largefile_slicebursts cannot monopolize projection while dependency-heavy foreground chains retain arrival order.
Outstanding follow-up debt for CLI/integration tests:
- use one explicit product-visible readiness predicate for local authoring:
ready. - define tenant visibility as the acceptance signal; do not add a second public
acceptedpredicate when the tenant already appears inview/tenant lists. - make
readymean only "writable now on this device", derived from the same authoring-context helper used by write commands. - continue replacing bespoke polling/retry helpers with shared
assert_eventually/value-eventually helpers over explicit invariants:- tenant visible, when the test cares about acceptance,
ready == true, when the test cares about sending/chaining,- specific event/message presence, when the test cares about sync.
- split deterministic bootstrap coverage from discovery-enabled coverage:
- deterministic tests use fixed bind + discovery disabled + explicit public address,
- discovery-enabled tests exist specifically to exercise duplicate-connection/idempotency pressure.
- keep sleeps only as poll intervals inside eventual assertions or when intentionally stretching a race to reproduce a bug.
- do not use fixed waits as protection against correctness failures.
Recommended enqueue guard:
INSERT OR IGNORE INTO project_queue (peer_id, event_id, available_at)
SELECT ?, ?, ?
WHERE NOT EXISTS (
SELECT 1 FROM valid_events v
WHERE v.peer_id = ? AND v.event_id = ?
)
AND NOT EXISTS (
SELECT 1 FROM rejected_events r
WHERE r.peer_id = ? AND r.event_id = ?
)
AND NOT EXISTS (
SELECT 1 FROM blocked_event_deps b
WHERE b.peer_id = ? AND b.event_id = ?
);attempts is retry bookkeeping, not business state.
Use it for:
- backoff scheduling (
available_at), - lease recovery after crashes/timeouts,
- dead-letter/alert threshold if needed.
If a queue is always synchronous at first, this can remain mostly zero until retry logic is enabled.
Must be atomic:
- canonical event insert + recorded insert + project enqueue.
- projection state transition (
valid/block/reject) + project dequeue. - unblock updates + requeue.
Need not be atomic with projection:
- transport send,
- cleanup/purge,
- metrics/logging.
Use INSERT OR IGNORE for:
- immutable canonical events,
- idempotent projection materialization,
- queue dedupe insertions,
- append-only endpoint observation rows.
Use ON CONFLICT DO UPDATE for:
- mutable cursor/checkpoint state,
- lease/heartbeat/retry metadata,
- sync state snapshots.
Avoid broad INSERT OR REPLACE.
Do not add in-memory dedup sets in front of INSERT OR IGNORE writers:
- pre-writer dedup causes data loss if the writer transaction rolls back (event marked "seen" but never persisted; peer retransmissions silently dropped),
- the set grows without bound in long-running daemons,
- redundant
INSERT OR IGNOREfrom concurrent sessions is cheap and self-correcting.
These should not be forced into generic auto-write behavior.
message_deletionand cascade/tombstone semantics.- deterministic emitted-event flows where projection emits another event.
- these flows still obey emitted-event rule and are unsigned by schema policy for determinism.
Deletion is special and should remain explicit.
Start simple, then tune.
- Prefer SQLite durable queues over a pure in-memory cascade pipeline.
- Tune batch sizes and indexes from measurements.
- Enable WAL and prepared statements.
- Keep queue purges explicit and predictable.
- Add invariants/metrics for blocked counts, retry growth, and queue age.
- Add a dedicated
low_mem_iosruntime mode for iOS NSE constraints.
When investigating sync performance degradation at high cardinality (e.g. 500k+ events):
- Baseline capture: run serial perf suite (
scripts/run_perf_serial.sh core) and the target tail benchmark in isolation with--test-threads=1. Record wall time, msgs/s, peak RSS, and environment details (filesystem type, hardware). - Tail profiling: add per-batch timing instrumentation to the direct-ingest queue and canonical ingest path (persist_ms, commit+effects_ms, epoch_10k_ms via profiling log lines). Identify whether tail slowdown is enqueue/backpressure-side or protocol-side (negentropy reconciliation stalls causing data starvation).
- Root-cause ranking: rank bottlenecks by measured contribution. Common patterns:
- Negentropy round latency scaling super-linearly with item count (protocol-level, deep fix).
- Autocommit overhead in projection drain (transaction batching fix).
- SQLite page cache pressure when dataset exceeds
cache_size(tuning knob).
- Fix one bottleneck: apply only the top-ranked implementable fix. Verify no regression on core suite before measuring tail improvement.
- Evidence artifact: document baseline, profiling data, root-cause ranking, fix rationale, and post-fix comparison in
docs/planning/for auditability.
Operational constraint: serial perf measurements (--test-threads=1) must be used for tail profiling to avoid cross-test interference. Stale test processes (e.g. from timed-out sync_graph_test) must be killed before re-running, as they hold tmpfs resources and cause spurious failures.
Performance tuning should assume the current preferred maintenance model:
- Keep one immutable in-memory dep-aware snapshot per
(db_path, workspace_id, window_kind, bounds, epoch)and reuse it across repeated sync checks. - Rebuild that snapshot only after the durable shared-index epoch for the window changes or the window bounds roll over.
LastDay,LastWeek, andLastTwelveWeekssnapshots include recursive dependency contribution in Phase 1;Olddoes not expand deps.- Stock Negentropy still remains the exact possession engine for Phase 2 candidate-dependency confirmation.
- If future work revisits a persisted auxiliary structure, it must beat cached-snapshot reuse on measured end-to-end stability and throughput, not just on isolated index-build cost.
- If future work adopts the non-homomorphic Merkle approach from rbsr_nonhomomorphic.pdf, treat the extra initial build or maintenance cost as an explicit security/performance tradeoff rather than a transparent drop-in.
The projection drain path (drain_project_queue_on_connection + drain_with_limit) applies two optimizations to reduce per-batch overhead:
-
Batch dequeue:
drain_with_limitcollects successfully-projected event IDs and dequeues them viamark_done_batch(oneBEGIN/COMMITper claim cycle) instead of individualmark_doneDELETEs per event. Reduces ~1000 autocommit DELETEs per writer batch to ~10 batch transactions. -
Deferred WAL checkpoint:
drain_project_queue_on_connectionsetsPRAGMA wal_autocheckpoint = 0during the drain, restoring to 1000 after. This prevents WAL checkpoint stalls between autocommit projection writes. Skipped inlow_memmode where WAL growth must remain bounded.
- Projection writes remain autocommit (each
project_onecall commits independently), preserving cascade-unblock correctness. - On projection failure, events are retried with exponential backoff via
mark_retry. - Crash safety:
project_queueprovides recovery; interrupted drains leave events in the queue for re-projection. - Operational note: WAL grows during drain proportional to pending event count. Disk space must accommodate this. The next persist-phase COMMIT triggers a checkpoint that processes accumulated WAL pages.
Profiling evidence: 500k one-way sync improved from 170.93s (2,925 msgs/s) to 106.75s (4,684 msgs/s) — 37.5% wall time reduction, 60% throughput improvement. Core suite (10k, 50k) showed no regressions.
Note: wrapping all projection writes in a single transaction was attempted first but abandoned — it caused ~0.06% of events to be left unprojected at 500k scale due to cascade_unblocked bulk cleanup interacting with the transaction scope.
low_mem_ios requirements:
- target steady-state RSS at or below
24 MiBduring sustained sync/projection. - keep memory bounded with strict in-flight caps (queue claims, decode buffers, batch sizes).
- prefer one writer connection + minimal readers; avoid large in-memory caches/prefetch.
- do not keep full trust/key sets in memory in low-memory mode.
- use SQL canonical trust/key tables with indexed point lookups and a bounded hot cache only.
- degrade throughput before violating memory ceiling (memory safety over speed).
- validate at scale (
>= 1_000_000canonical events on disk) with stable memory. - include large-identity-set validation (for example
>= 100_000peer trust keys) while preserving memory ceiling. - Linux realism gate for this phase:
- lowmem harness perf scripts are Linux-only (
/proc+ cgroup v2 dependencies), - hard-cap validation should run receiver (
bob) with cgroup v2memory.max=22528 KiBandmemory.swap.max=0, - passing runs must show both
PASS_UNDER_24MB=1andCGROUP_OOM_KILL=0, - RSS-sampling budget tests in
low_mem_test.rsare sanity-only and ignored by default.
- lowmem harness perf scripts are Linux-only (
Recommended initial size policy:
EVENT_MAX_BLOB_BYTES = 1_048_576(1 MiB soft cap)FILE_SLICE_TARGET_BYTES = 262_144(256 KiB)FILE_SLICE_CIPHERTEXT_BYTES = 279_552(fixed payload: 256 KiB logical slice data + bao proof budget sized for files up to 10 GiB)FILE_SLICE_MAX_BYTES = 279_552
file_slice events can be much larger than legacy simulator limits and are signed/verified like other events.
The FILE_SLICE_CIPHERTEXT_BYTES constant is derived so a full 256 KiB logical slice plus the
worst-case bao proof for a 10 GiB file fits in a fixed slot, while the maximum encoded
file_slice event still fits comfortably within
EVENT_MAX_BLOB_BYTES.
Two new event types (migration 13):
file(type 24, signed): Descriptor linking a file to a message. Deps:message_id,key_event_id,signed_by. Fields includeblob_bytes,total_slices,slice_bytes,root_hash,filename,mime_type.file_slice(type 25, signed): Individual encrypted chunk of a file. Dep:signed_by. Fields includefile_id,slice_number,ciphertext.
Both event families are normal content and therefore require outer encrypted
wrappers in steady state. valid_events.semantic_type_code records their inner
semantic type for dep checks, and file-slice authorization additionally requires
the wrapper key_event_id to match the parent file descriptor's
key_event_id.
Primary key for file_slices: (recorded_by, file_id, slice_number) — optimized for
sequential IO locality when reassembling files.
file events are validated at parse time:
blob_bytes > 0requirestotal_slices > 0total_slices > 0requiresslice_bytes > 0total_slicesmust equalceil(blob_bytes / slice_bytes)- Zero-byte files (
blob_bytes == 0, total_slices == 0, slice_bytes == 0) are valid. - Signer metadata (
signed_by,signer_type,signature) is required and then validated by the shared signer pipeline after dependency resolution.
file_slice projection uses INSERT OR IGNORE with post-insert conflict detection:
- If insert succeeds (rows > 0):
Valid - If insert is ignored: check existing row's
event_id- Same
event_id: idempotent replay →Valid - Different
event_id: conflict →Rejectwith durable rejection record
- Same
Projection authorization also rejects when:
- the file descriptor signer does not match the file-slice signer,
- the file-slice outer wrapper
key_event_iddoes not match the file descriptorkey_event_id.
send-fileand synthetic file generation stream slices one at a time instead of buffering the entire file in memory.save-filestreams decrypt/write into a temp file in the destination directory, truncates to the descriptor byte length, and atomically renames on success.- Effective download rate is derived from synced
file_sliceevents only. It is available in attachment/file query output, butLOW_MEM_IOSdisables receive-side capture so completed files may legitimately have no MiB/s there.
- Merkle-proof extension: Attachment carries
merkle_root, eachfile_slicecarries proof path, projector verifies proof against descriptor root. Overhead ~log2(N) * 32bytes per slice proof. - Full DAG encoding: Deferred unless needed. Too heavy for current phase.
Multi-source download allows a sink to reconcile overlapping event ranges with multiple sources concurrently. The current plan is suppression-based rather than planner-based: every peer runs the same per-peer range cadence, and receivers advertise event ids they have already received or are receiving so other sources can stop sending duplicates.
Current runtime shape:
- Per-peer range cadence. Each active peer independently cycles through
last day -> last week -> last twelve weeks -> old. Peer count, priority-owner flags, sorted peer rank, and hash partitions do not decide who owns a range. - Dep-aware negentropy per selected range. Each range session reuses a cached dep-aware snapshot for one durable root window, runs dep-aware Phase 1 over combined root-plus-dependency fingerprints, exchanges candidate dep ids, runs stock Negentropy Phase 2 over that candidate-dep universe, and then streams the resulting missing event blobs for that explicit range.
- Receive-side event ids. The receiver hashes each incoming blob once before enqueue, publishes suppression immediately when enabled, and hands bounded batches straight to canonical ingest. Backpressure now comes from the direct-ingest queue instead of a file-backed receive log.
- Live suppression cohort. When
TOPO_ENABLE_LIVE_SUPPRESSION=1, range sessions in the same(db_path, recorded_by, workspace_id)cohort share a bounded recent event-id list. New sessions replay the recent list, and active sessions publish new event ids immediately. - Broad suppression broadcast. Suppression is sent to all connected same-workspace sessions in the cohort, not filtered by range overlap. This is intentionally broad because false suppression is cheaper than missing a suppression opportunity; later range rounds recover holes.
- Per-peer ingest barrier. A new range session with peer
Pdoes not start until the previous range session with peerPhas finished both network transfer and direct ingest. This preserves the invariant that one peer-specific negentropy session is fully ingested before the next starts. - Large bounded lists. The default live suppression cap is large
(
TOPO_LIVE_SUPPRESSION_EVENT_ID_CAP, default 512k ids; lower in low-memory mode). Batch size is controlled byTOPO_LIVE_SUPPRESSION_SEND_BATCH_SIZE. - Deterministic scatter after an ordered head. Send order keeps the first
sender-prefetch head in strict timestamp order, then deterministically
scatters the remaining tail using
blake3(seed || event_id), seeded by the local peer/recorder id. This preserves early descriptor/message visibility while still preventing identical sources from sending the same bulk tail first. - Settle only when useful.
TOPO_LIVE_SUPPRESSION_BATCH_SETTLE_MSdefaults to a small delay between batches, but the runtime skips it for a 1:1 cohort. It is only useful when another distinct source peer can act on suppression. - Connection idempotency after auth. Either side may dial. After daemon identity extraction, at most one live daemon connection slot survives per remote daemon; the deterministic preferred direction is only a simultaneous duplicate tie-breaker.
Deliberately removed approaches:
- durable
wanted/wanted_sourcestables, ResponseCreditor a per-event pull coordinator,- peer-rank, priority-owner, or hash-partition ownership of historical ranges,
- peer-local SQL event ownership or sticky assignment state,
- a separate
PeerReplicatorobserver/request split for this phase.
Test families:
tests/daemon_multi_source_tiered_window_perf_test.rsis the daemon-facing benchmark/proof harness. It records projected catchup, active source count, downloader event frames, delivery efficiency, and per-source sync-log receive deltas.- The 2-source 1k replicated rejoin test is the stable connectivity smoke: it proves sources are known, connections are live, useful sync happens, and the basic live-suppression path is enabled.
- Ignored larger daemon tests scale that same harness to more sources and larger replicated histories for manual throughput/efficiency comparisons.
tests/sync_graph_test.rsremains useful for in-process graph probes, but daemon behavior should be proved through the daemon/RPC/CLI path when the assertion is about real connections or runtime sync.
Invariants this model is meant to preserve:
- at most one steady-state live daemon connection slot per remote daemon,
- any number of authenticated logical sessions may share that daemon connection over time,
- no balancing logic is embedded in Negentropy,
- false suppression is an efficiency risk, not a correctness authority,
- multi-source fairness is across distinct peers, not multiplied by duplicate connections,
- low-memory mode bounds blob and id-list residency,
- tiered cadence does not split the event universe or create a separate completion protocol.
This is a final functional phase after the core projection/queue path is stable. Prerequisite: Phase 6 signer substrate is complete before identity projector implementation begins.
Before writing identity/encryption projectors in Rust:
- Confirm signer pipeline from Phase 6 is active:
- missing
signed_bydependency blocks, - unblocked signer enables signature verification,
- invalid signature rejects (not block).
- missing
- Build/update a TLA+ model of causal relationships and guards for this phase.
- Model split invite types (
user_invite_shared,peer_invite_shared) and accepted-workspace semantics. - Model workspace binding: workspace events must be parameterized by workspace id, and the accepted binding must resolve to one distinct accepted workspace id or reject ambiguity. The model must prove that foreign workspace events (for workspaces the peer did not accept an invite for) can never become valid. Without this, the model cannot distinguish between valid and invalid workspace events, making it insufficiently expressive for multi-workspace scenarios. See
InvWorkspaceAnchor,InvSingleWorkspace,InvForeignWorkspaceExcludedinvariants. - Model invite-derived accepted-workspace binding: the accepted workspace binding must derive from event-carried
workspace_idvalues, collapse same-workspace duplicate acceptances, and reject multiple distinct workspace ids rather than choose a canonical invite event. This ensures the binding mechanism is faithful to the real protocol where accepted rows carryworkspace_id. SeeInvTrustAnchorMatchesCarriedinvariant. - Model guard placement explicitly (poc-6 parity):
- accepted-workspace guard applies to root workspace events,
invite_acceptedis local binding from carriedworkspace_id(no invite-presence dep gate),- downstream identity admission (
user/device/peer) still requires signer/dependency chain validity in the same peer scope.
- Verify bootstrap/self-invite, join, and device-link safety invariants.
- Freeze a projector-spec mapping table: each projector predicate/check maps to a named TLA guard.
- Record TLA scope boundary for this phase:
- current identity-causality models may abstract away transport TLS credential/session-key lifecycle.
- this abstraction must be tracked explicitly and followed by a transport-key model extension milestone before final identity/transport convergence sign-off.
Projector implementations should mirror TLA conditions as directly as possible.
Only include identity and policy needed for:
- accepted-workspace bootstrap/join cascade
- self-invite bootstrap flow
- device linking
- device-link authorization scoped to the signer's own user identity; adding new users stays admin-only
- recipient selection for encrypted message key wraps
- transport trust policy derived from identity-backed SQL trust state (PeerShared-derived SPKIs + accepted-invite bootstrap trust), not static CLI/file pin sources
User removal is out of scope in this phase because safe removal requires key rotation and group key agreement beyond this PoC.
Use separate types:
user_invite_shared(invites a user identity)peer_invite_shared(invites/links a peer device to a user)invite_accepted(records accepted-workspace binding; local SQL also stores accepted invite-link bootstrap metadata)
Do not use one invite type with mode=user|peer.
Implementation requirement:
- keep one shared invite projector helper with per-type policy tables (signer/dependency checks).
- this keeps logical separation for TLA/model checking while avoiding duplicated Rust control flow.
Required behavior:
invite_acceptedrecords accepted-workspace binding rows forworkspace_id(perrecorded_bypeer scope).invite_acceptedis a local binding step from event-carriedworkspace_id(no invite-presence dep gate).- downstream identity events (
user,peer_invite_shared,peer_shared) remain dependency/signer-gated in peer scope. - root
workspaceevents are not valid until a corresponding accepted-workspace binding exists and matches the root id. - accepted-workspace binding must come from validated projector input fields, not pre-projection capture tables.
- invites are never force-valid; they validate only through signer/dependency chain.
- accepted invite links produce bootstrap transport trust tuples in SQL via projection:
- service layer writes local
bootstrap_contextrows (inviter address + SPKI fingerprint from invite link), - invite projectors read
bootstrap_contextand write bootstrap trust rows via deterministicWriteOp::InsertOrIgnore, - peer_shared projector consumes matching bootstrap trust rows via deterministic
WriteOp::Delete, - trust rows are looked up by sync on each connection/handshake (no in-memory-only trust authority).
- this follows the same poc-6 cascade pattern where
invite_acceptedprojection drives accepted-workspace establishment and workspace event unblocking.
- service layer writes local
Current state (implemented):
invite_acceptedownsinvites_acceptedprojection rows directly (event-to-own-table clarity).- Workspace authority reads normalize
invites_acceptedby distinctworkspace_id; same-workspace duplicate acceptances collapse to one binding, while distinct-workspace ambiguity rejects. workspacecontext loading validates that a matching accepted-workspace binding exists before projection.- End state authority is
invite_acceptedprojection rows only; there is no separatetrust_anchorstable.
Self-invite bootstrap sequence must stay explicit:
- create
workspaceevent (integrity self-sign only). - accept workspace locally ->
invite_accepted(workspace_id=workspace_event_id). - create bootstrap
user_invite_sharedsigned by workspace authority. - normal cascade unblocks:
workspace -> user_invite_shared -> user -> peer_invite_shared -> peer_shared.
Use poc-6 as reference behavior for end-to-end test setup:
- Alice creates workspace + identity chain via high-level bootstrap API.
- Alice creates invite link in plaintext format (all fields hex-encoded, slash-delimited, labeled — see DESIGN.md §9.2 "Invite link wire format"). Contains bootstrap address, inviter SPKI fingerprint, invite event ID, invite private key, and workspace ID. Wrapped content-key material is delivered via
key_sharedevents during bootstrap sync, not embedded in the invite link payload. - Bob accepts invite link via high-level accept API:
- records local
invite_accepted, - writes accepted-workspace binding (
workspace_id), - stores accepted-invite bootstrap transport trust tuple in SQL,
- unwraps bootstrap content-key material using invite private key and inviter public key,
- materializes local
key_secretevents with deterministic event IDs (matching inviter's key IDs).
- records local
- Sync bootstrap trust is read from SQL at connection creation (no in-memory-only trust authority).
- Encrypted events received during bootstrap sync block until local key materialization (step 3) completes, then unblock via normal cascade.
- Connection state follows
poc-6ordering:- invite-auth request (
connReq) -> ack (connAck) -> - invite-labeled connection (
connInvite) -> - peer-labeled upgraded connection (
connPeer).
- invite-auth request (
- Tests migrate to this flow:
- remove direct SPKI pin setup from test harnesses,
- remove synthetic low-level bootstrap stubs where high-level APIs exist,
- require invite-link acceptance path for multi-peer bootstrap.
- Keep low-level tests only for explicit adversarial/property coverage, not baseline bootstrap.
Use the sender-subjective O(n) baseline ("maximally simple phase-1/phase-4 style key broadcast"):
- sender creates a fresh local-only
secretkey event per message, - sender emits one
key_sharedkey-wrap event per perceived eligible recipient peer pubkey, - encrypted content event references the key event id through normal dependency fields,
- each sender wraps to all perceived eligible members for each message (intentionally crude).
Bootstrap and runtime wrapping share the same key_shared event type and wrap/unwrap projector path. Bootstrap wraps target the invite public key (X25519-derived from Ed25519 invite signing key); runtime wraps target PeerShared public keys. Recipients materialize local key_secret events with deterministic event IDs derived from key bytes (BLAKE2b hash → created_at_ms), ensuring inviter and joiner agree on key_event_id values.
Key modeling requirements for this phase:
- All protocol-level key material that projectors depend on must be represented as events and resolved by event-id dependencies (for example sender
secretkeys and recipient key-wrap events). Do not introduce out-of-band key stores for event-graph key dependencies. - This requirement also applies to transport TLS keying in end-state: cert/public-key trust mapping and local TLS private key material are event-backed state with dependency ordering.
- Runtime file artifacts for TLS keys/certs are optional cache/materialization outputs only; they are not authoritative policy state.
- For fanout to many recipients, produce one canonical encrypted content event and many recipient-specific key-wrap events for the same key/decrypt target; recipients that are eligible and have key material must decrypt to the same plaintext event bytes (deterministic materialization target across recipients).
Not in scope yet:
- key history availability/backfill guarantees,
- optimized tree cover (TreeKEM update paths),
- advanced healing policies.
- The PoC does not implement
user_removed,peer_removed, orban. - Safe removal requires key rotation and group key agreement, which are deferred beyond this phase.
- Sender-subjective encryption in this phase wraps to the currently known peer set only.
- When removal returns, live daemon connections must revoke any admitted tenant/peer routes that match the removed subject and close the daemon connection if no tenant scopes remain.
- Removal must be enforced before sync/session work starts; ingest rejection alone is not sufficient because it still leaks metadata and wastes transport work.
- Keep model alignment with
docs/tla/EventGraphSchema.tla(includingevent_graph_schema_bootstrap.cfg),docs/tla/TransportCredentialLifecycle.tla, anddocs/tla/UnifiedBridge.tla. - Treat
BootstrapGraph.tlaas retired/deleted in this epoch; do not add new checks there. - Extend/adjust model events for split invites (
user_invite_shared,peer_invite_shared). - For each identity-phase projector, include a referenced guard list in comments/docs.
- Treat divergence between projector logic and TLA guards as a spec bug that must be resolved before adding behavior.
Four CI gate scripts enforce bidirectional coverage between TLA+ guards and runtime checks:
scripts/check_projector_tla_conformance.py— everyspec_idin the conformance matrix has at least one linked test; guard-levelspec_ids have both pass and break polarity; everycheck_idexists in the catalog; everyspec_idmaps to at least onecheck_id.scripts/check_projector_tla_bijection.py— every TLA guard inprojector_spec.mdmaps to at least onecheck_idin the runtime check catalog; everycheck_idmaps to atla_guard_idor explicitNON_MODELED::waiver; everycheck_idhas at least one linked test row.scripts/check_bridge_conformance.py— everyCHK_BRIDGE_*check in the runtime catalog has non-waiver pass coverage and (except liveness checks) non-waiver break coverage; disallows bridge regressions to TLC-only waivers.scripts/check_tcl_conformance.py— everyCHK_TCL_*check in the runtime catalog has non-waiver pass+break coverage and disallows TCL regressions to integration-effect-only waivers.
Artifacts: docs/tla/runtime_check_catalog.md (check_id → guard mapping), docs/tla/projector_conformance_matrix.md (spec_id → check_id → test_id with polarity). Waivers (NON_MODELED::) require inline rationale and are reviewed as spec debt.
Status: Closed. docs/tla/TransportCredentialLifecycle.tla models the runtime transport credential/trust-store layer, and docs/tla/UnifiedBridge.tla models the cross-layer trust-to-connection invariants and progress properties.
Previous gap: TLA models were identity/event-causality models that did not encode mTLS credential lifecycle (SPKI generation/rotation/revocation, projected trust-set state transitions).
What is now modeled (TransportCredentialLifecycle.tla):
- Local credential lifecycle: single credential per peer (no rotation/revocation in POC).
- Three-source trust store: PeerShared-derived SPKIs, invite_bootstrap_trust, pending_invite_bootstrap_trust.
- PeerShared trust source is represented as projected
peers_shared.transport_fingerprintvalues (indexed by(recorded_by, transport_fingerprint)), matching runtime lookup shape. - Consumption: PeerShared projector emits deterministic
WriteOp::Deleterows at projection time, removing matching bootstrap/pending entries. Trust check reads are pure (no write side-effects). - TTL expiry of bootstrap trust sources.
- Trust state is the union of PeerShared-derived SPKIs plus bootstrap trust rows; user-removal deny state is intentionally absent in this PoC.
- Invite ownership:
inviteCreatorvariable tracks which peer created each invite SPKI. TheCreateInviteaction establishes ownership;AddPendingBootstrapTrustis guarded byinviteCreator[s] = pso only the invite creator can materialize pending trust. - 7 invariants verified by TLC (11.5M states, 771K distinct at 2-peer/3-SPKI), mapped to Rust checks in
docs/tla/projector_spec.md. TheInvPendingTrustOnlyOnInviterinvariant catches the joiner-side pending trust emission bug (verified: buggy model violates in 5 steps).
What remains abstract (by design):
- TLS handshake and session-key derivation.
- No manual trust source; authorization is modeled as invite/bootstrap or steady-state projected trust only.
- Event-graph causality for trust-source inputs (nondeterministic in this module; covered by EventGraphSchema.tla).
Config files: transport_credential_lifecycle_fast.cfg (2 peers, 3 SPKIs), transport_credential_lifecycle.cfg (2 peers, 4 SPKIs).
- CLI commands are deterministic and script-friendly.
- Prefer
assert-eventuallyover ad-hoc sleeps/waits. create_event_synchronouschaining works for imperative orchestration:- event A created+projected synchronously,
- event B created next line with dependency on A,
- B succeeds without manual waits.
- Happy path: all peers share PSK, encrypted events project end-to-end.
- Missing key dep: encrypted event blocks until key event appears.
- Wrong PSK/decrypt failure: event is rejected, not blocked.
- Reorder: ciphertext before key events converges after unblock.
- Replay/reproject: same final state from canonical events.
- Tenant-scoped PSK materialization: each local PSK key event is recorded by the intended
recorded_bytenant only. - Two-set isolation harness:
- set A: peer+peer+PSK_A,
- set B: peer+peer+PSK_B,
- verify no decrypt/projection crossover and no cross-tenant read leakage.
- Valid/block/reject decisions per event type.
- Blocked-only dependency behavior with multiple blockers.
- Set-based unblock correctness.
- Blocked-normalcy invariants:
- policy-appropriate blocked rows may remain after sync and are not test failures by themselves.
- non-recipient encrypted messages/key-shares are expected blocked cases.
- Signer substrate invariants:
- missing signer dependency blocks,
- invalid signature rejects,
- valid signature passes and continues policy checks.
- Deterministic emitted unsigned invariants:
- emitted deterministic types omit signer fields by schema,
- deterministic derivation checks gate validity instead of signature checks.
- Encrypted wrapper flow, including nested-encryption rejection.
- Encrypted wrapper
inner_type_codeinvariants:- mandatory field present for every encrypted wrapper event,
- mismatch between outer
inner_type_codeand decrypted inner type rejects.
- Source-isomorphism checks:
local_create,wire_receive, andreplayconverge through the sameproject_onesemantics and yield identical projected state.
- Replay invariance: replay from canonical events yields same projected end state.
- Replay idempotency: replaying again on already replayed state (2x replay) yields no state change.
- Reverse-order replay invariance: replaying canonical events in reverse order yields same end state.
- Reproject invariance: wipe projections and reproject yields same state.
- Reorder invariance: out-of-order ingest converges to same state.
- Operational queues are excluded from end-state equality checks.
Harness rule:
- run these replay/reproject/reorder checks automatically after every scenario test that mutates the event store.
Use deterministic table-state fingerprints for comparisons.
- Single-DB scoping checker test (like
poc-6): ensure subjective queries are tenant-scoped. - Two-daemon integration test: separate peer identities record/project without overlap.
- Cross-tenant leak tests: fail if rows from peer A appear in peer B scoped reads.
TLA-led acceptance:
- Model invariants pass for split-invite causal graph before projector implementation is finalized.
- Rust projector predicates map 1:1 to TLA guard conditions for identity/encryption gate checks.
Behavior tests:
- Self-invite bootstrap: accepted-workspace binding recorded, then normal cascade to first
peer_shared. - User-join flow via
user_invite_sharedkeeps signer/dependency blocking semantics. - Device-link flow via
peer_invite_sharedkeeps signer/dependency blocking semantics. - No force-valid invites: invite remains blocked until signer path is valid.
- Removal enforcement: removed peers stop receiving new key wraps.
- Sender-subjective baseline: each sent encrypted message yields wraps for all currently eligible recipients.
Topology reference: DESIGN_DIAGRAMS.md sections 2 (one sync connection) and 4 (runtime topology).
- 2-node bootstrap and sync.
- 3-node out-of-order convergence.
- reconnect/retry/backoff behavior.
Topology reference: DESIGN_DIAGRAMS.md section 4 (threads/queues/DB topology) for interpreting runtime memory shape during these tests.
- Fast default lowmem lane:
cargo test --release --test low_mem_test -- --nocapture --test-threads=1(functional lowmem smoke, no RSS gate),scripts/run_perf_serial.sh lowmem(fast cgroup-enforced matrix).
- RSS-sampling sanity lane (ignored by default):
cargo test --release --test low_mem_test -- --ignored --nocapture --test-threads=1
- Run lowmem realism through daemon-based harness, not in-process unit tests:
scripts/run_lowmem.sh delta10kscripts/run_lowmem.sh deltafiles
- For release/perf evidence at larger scales, enable optional serial lowmem POC matrix:
PERF_LOWMEM_POC_ENABLE=1 PERF_LOWMEM_RUN_LARGE_TARGET=1 PERF_LOWMEM_RUN_SMALL_BRACKET=1 scripts/run_perf_serial.sh lowmem
- Linux-only contract for these runs:
- requires
/proc/<pid>/statusand/proc/<pid>/smaps, - cgroup hard-cap mode requires cgroup v2 (
memory.max,memory.events), - when enforced cap is enabled, treat any
CGROUP_OOM_KILL > 0as failure.
- requires
- Loopback/simulator paths in production runtime.
- Connection/sync canonical event types.
- Ad-hoc bootstrap reprojection paths that bypass blocker logic.
Keep:
- local-only canonical events where replay matters,
- recorded-event model for replayability and tenant-scoped history.
Fastest coherent milestone:
- Finish Phase
1,2,4, and5with a small event set. - Complete Phase 6 signer substrate.
- Add Phase 8 encrypted wrapper with PSK test harness for one core content path.
- Add minimal Phase 9 queues.
- Add deletion special-case behavior after baseline sync is stable.
- Add final Phase 12 identity + invite cascade + sender-subjective key wraps.
Use this section as an execution-process checklist only. Normative protocol/runtime behavior is defined in DESIGN.md.
- Build in the phase order from §1 unless explicitly overridden.
- Treat phase exit criteria in this PLAN as release gates.
- Use DESIGN as the normative source for protocol semantics and invariants.
- Keep one projection path, one dependency engine, one signer pipeline, and tenant-scoped shared tables (see DESIGN §4-§5, §14).
- For transport work, preserve strict authenticated transport and SQL-backed trust evaluation (see DESIGN §2, §9.5).
Recommended sequence:
- transport-only changes,
- schema/wire changes,
- projection and dependency changes,
- signer substrate,
- encrypted adapter,
- queue/worker changes,
- identity/TLA changes.
Rule:
- each PR must include failing tests made to pass by that PR.
- tests cover the new behavior and key regressions,
- replay/reorder/reproject invariants still pass,
- tenant-isolation checks still pass,
- DESIGN references are updated if semantics change.
Normative relay/NAT behavior is defined in DESIGN.md §2.4. This PLAN keeps only execution-level expectations:
- keep relay-backed
irohrendezvous and NAT traversal runtime-only (non-canonical), - do not reintroduce an explicit
topo introAPI or a parallel punch scheduler, - keep endpoint observation TTL + cleanup behavior where still useful for reconnect hints,
- preserve coverage with integration + netns NAT tests.
One node can host N tenant identities in one DB with one shared QUIC endpoint and tenant-scoped routing/trust. Detailed architecture and invariants are documented in DESIGN.md §3.2.1-§3.2.3, §2.5, §8, and §14.
- tenant session-auth credentials/state are DB-resident and tenant-discoverable,
- runtime supports shared endpoint + tenant routing + dynamic trust checks,
- shared ingest/batch writer supports tenant-tagged ingest items,
irohmDNS supports daemon-scoped discovery with self-filtering.
- no filesystem cert authority regression,
- no cross-tenant projection/query leakage regression,
- no reintroduction of event-count-based convergence assertions,
- no bypass of workspace command ownership boundaries,
- no reintroduction of duplicate live daemon connections/workers for the same remote daemon.
- multi-tenant shared-DB scenario coverage,
- mDNS/discovery smoke coverage,
- cross-tenant leakage checks in scenario tests,
- application-level convergence/assertion style in tests,
- same-workspace sibling fanout coverage for both local create and wire ingest,
- join/link replay coverage for historical shared events already present in one DB,
- real CLI coverage for reused invites, fresh invite creation, and mixed user/device-link growth patterns.
- loopback-bound daemons may advertise non-loopback IPs for mDNS reachability,
- browse-side must normalize discovered non-loopback addresses back to loopback when local daemon is loopback-bound,
- advertised IP must be explicit runtime input, not inferred implicitly by discovery internals.
Goal: establish a test suite where successful P2P bootstrap and sync cannot be faked by local process shortcuts, shared filesystem state, or manual operator dials.
- Out-of-band input is limited to invite links (
topo://...) with standard bootstrap data. - Nodes run in non-interactive daemon mode (
topo start) and are asserted through CLI command results (topo assert-*). - Desired steady-state connectivity is invite/discovery-driven autodial, not manual
--connect. - Multi-network topologies are required (local discovery + internet bootstrap-address mode).
- Add executable realism contract tests under
tests/cheat_proof_realism_test.rs. - Include a passing baseline that proves invite bootstrap + daemon autodial transport path works, so failures are scoped to realism gaps, not broken transport.
- Include strict contract tests for desired behavior:
- invite-only daemon autodial after invite acceptance,
- daemon CLI invite lifecycle surface (
topo invite/topo accept).
- Keep these tests as non-negotiable regression guards for future refactors.
- Placeholder startup autodial manager in
node.rsthat consumes persisted invite/bootstrap address rows first.- Must be explicitly labeled in code/logs as placeholder (for example
PLACEHOLDER AUTODIAL) so it is not mistaken for end-state design.
- Must be explicitly labeled in code/logs as placeholder (for example
- Keep daemon startup invite/discovery-driven only (manual
--connectremoved). - Daemon CLI parity for invite lifecycle and tenant-targeted operations (so realism tests can stay daemon-first).
- Scope note: in this branch, daemon-first operation requires at least one local tenant. Fresh DB first-invite acceptance is still a pre-daemon bootstrap step.
- For realism in this POC, a naive autodialer is the critical requirement.
- Full peer connection management (address scoring/prioritization, lifecycle policy, and advanced dial orchestration) is explicitly out of scope for this POC.
- Add segmented multi-network harness (container or netns) with at least 3 networks and overlapping peers.
- Exercise all three behaviors in one scenario:
- multitenancy overlap,
- local mDNS discovery on shared LAN segments,
- invite-link bootstrap address dialing across non-LAN segments.
- Start with netns harness for fast iteration on network realism:
- three LAN segments,
- routed bootstrap reachability,
- daemon-only assertions.
- Treat netns harness as insufficient for strict anti-cheat filesystem isolation:
- netns isolates network stack, not host filesystem visibility.
- Container phase is required for strict "no filesystem/shared-memory cheat path" guarantees:
- one container per peer,
- no shared writable volumes,
- isolated IPC/PID namespaces,
- capability drop + no-new-privileges.
- Current branch uses netns as a stepping stone and keeps this limitation explicit.
- Netns realism caveat observed in practice: multiple peers can share the same OS hostname, which can confuse host-oriented discovery stacks.
- Mitigation in this branch: rely on daemon-scoped
irohmDNS endpoint ids, not hostnames or per-tenant host labels.
- Mitigation in this branch: rely on daemon-scoped
- Automatic port mapping is now transport-owned (
irohportmapper), not a bespoketopo upnpsurface. - This remains outside strict protocol scope: protocol correctness does not depend on port mapping, and explicit bootstrap addresses remain supported.
- Expected limitation: some networks (for example CGNAT or non-UPnP routers) still require manual bootstrap endpoints even with best-effort transport-owned port mapping.
Tracked in docs/bugs/qa-findings.md. Key items:
Replace the single-identity-only transport-identity command with transport-keys that lists all rows from local_transport_creds with peer_id and source. Works for both single and multi-tenant DBs.
Duplicate reactions (same user, same message, same emoji) are accepted as separate events. Display queries should deduplicate by picking the winning reaction per (user, message, emoji) tuple — winner by earliest timestamp. The event layer continues to accept duplicates; deduplication is query-side only.
messages currently renders up to 50 items with no truncation indicator. Change to show the 50 newest messages, and when more exist, display a note like "(25 older messages not shown)". Pagination is out of scope for now.
See docs/bugs/qa-findings.md for the full list: identity command wrong IDs, workspace creation atomicity, input validation (empty username/message/reaction), error message improvements.
Normative locality/layering rules are defined in DESIGN.md §14. Plan-level enforcement remains:
- event-type-specific commands/queries/projectors stay in event modules,
src/runtime/control/service.rsremains orchestration-only,- projection pipeline remains orchestration-only,
- local-only subscription lifecycle/feed storage stays in
src/state/subscriptions/*(notevent_modules/*), - event-specific subscription matching/payload shaping stays in
event_modules/<type>/subscription_filter.rs(orsubscription_filters.rs), - module boundary checks stay automated in CI where available.
Verus formal verification (verus-proofs/src/bug_hunt.rs) identified that workspace members with tampered clients can perform detectable misbehavior. The system should automatically detect and respond to such behavior.
- GC
deletion_intentrows where the referenced deletion event was rejected - Prevents unbounded storage growth from wrong-author peer deletion spam
- When
MessageDeletionprojector detects author mismatch, record thesigned_bypeer as misbehaving in a newmisbehavior_logtable - When signature verification fails for a signed event, log the source peer
- When a peer accumulates N misbehavior incidents, emit a
Removalevent targeting that peer (requires admin authority on the removing device) - The removal propagates via normal event sync, revoking transport trust across all peers in the workspace
- Periodic re-authorization checks during long-running sync sessions to close the TOCTOU window between auth check and data exchange
- Bootstrap auth cache entries should carry TTL matching the DB trust TTL
- Add
created_attimestamp toAcceptedBootstrapAuthCacheentries inDaemonConnection(peering_boundary.rs) - Check TTL on cache lookup; expire entries older than bootstrap trust TTL
- Prevents long-lived QUIC connections from bypassing expired DB trust