feat(compose): worker-compose daemon — declare a project, start it as a graph, supervise it - #2034
Open
guibeira wants to merge 62 commits into
Open
feat(compose): worker-compose daemon — declare a project, start it as a graph, supervise it#2034guibeira wants to merge 62 commits into
guibeira wants to merge 62 commits into
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — docs5 verified, 776 skipped.
Four for four. Nicely done. |
anthonyiscoding
requested changes
Aug 4, 2026
anthonyiscoding
left a comment
anthonyiscoding
left a comment
Contributor
There was a problem hiding this comment.
Looking good, definitely have some runtime/behavior and configuration specifics to figure out first though.
Also I added some additional docs to this branch that I felt were needed for using-iii/compose and understanding-iii/compose. They'll need further updating if behavior or other specifics change as I wrote them to currently reflect how the branch is at this moment.
guibeira
force-pushed
the
feat/compose-daemon
branch
from
August 4, 2026 23:36
e37dbef to
b45721d
Compare
guibeira
force-pushed
the
feat/compose-daemon
branch
from
August 5, 2026 12:18
458b790 to
fd7b4a6
Compare
…awn contract Greenfield crate behind `iii compose`, sharing no runtime code with crates/iii-worker. Covers plan_d tasks 1-3 and 5: - strict worker-compose.yaml v1 parsing: unknown fields are hard errors at every level, duplicate container keys are rejected (serde_yaml would otherwise keep the last one and silently drop a worker); - dependency graph: unknown/self dependencies and cycles rejected, cycle path reported in declaration order, start order dependencies-first; - optional iii.worker.yaml subset parser with compose `run` winning over `scripts.start`, and a manifest that may not rename its container; - deterministic project namespace: --namespace, else <project>-<sha256(compose path)[..8]>; - child spawn contract: III_URL, III_NAMESPACE, III_CONFIG and III_WORKER_NAME are injected and stripped from the inherited environment, so a stale value in the operator's shell can never reach a child. Every rejection carries a stable machine code; the offline path is complete and daemon supervision reports DAEMON_NOT_IMPLEMENTED.
`iii compose` delegates straight to iii_compose::run: the arm never touches EngineBuilder, so the subcommand cannot boot an engine by accident. Parse tests cover both modes plus a guard that the new subcommand does not shadow the `iii trigger` alias. CLI reference regenerated so the cli-docs-built job stays green.
Covers plan_d tasks 6 and 7 plus the offline half of task 4: - every child is spawned as its own process group leader, so teardown (SIGTERM -> 10s grace -> SIGKILL) reaches grandchildren too; verified by a test whose worker spawns its own child, and by one that ignores SIGTERM; - a child is fingerprinted at birth (linux /proc start time) and a PID whose fingerprint cannot be verified is never signalled, so a restarting daemon can never kill a stranger holding a recycled PID. Non-linux unix reports the identity as unverifiable until the libproc path is written and tested; - pre_start blocks with a budget and kills its own group on timeout; its pipes are drained while it runs, so a chatty hook cannot deadlock into a false timeout. post_run fires after exit and is never awaited; - config_override merge (maps recurse, arrays/scalars/null replace, key positions preserved) and delivery as an owner-only 0600 file. Fetching the base entry from the configuration worker still needs an engine connection. `iii compose validate` now also reports the namespace the project would register under and, per container, its working directory and config entry, so the namespace derivation and spawn contract are reviewable without a daemon.
…s supervision plan_d tasks 10 and 8. State (`~/.iii/compose/<id>/state.json`, 0600 in a 0700 dir, written through a temp file and renamed): - reconciliation only ever *reads*. A recorded child resolves to Adopt (same process, re-adopt without restarting), Gone (process exited while the daemon was away) or Unverifiable — a live PID whose birth fingerprint does not match, i.e. a recycled PID. Unverifiable is never signalled, only reported; - a platform that cannot fingerprint (macOS today) therefore never adopts, which is the safe direction; - a corrupt state file is an error, never a silent reset: that is exactly when children may be running unaccounted for; - state recorded for one compose file cannot be rebound to another under the same --id, so a daemon can never adopt another project's children. Windows supervision mirrors the unix module: a Job Object per child replaces the process group, CTRL_BREAK replaces SIGTERM, and TerminateJobObject replaces SIGKILL. Hooks get a job too, so a hung pre_start is killed instead of waited on forever. JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE is deliberately not set: it would kill every child when the daemon exits, and re-adoption after a daemon crash is a requirement. The windows path is compile-verified only (cargo check against x86_64-pc-windows-msvc); it has never been executed. The new compose-test-matrix CI job runs the suite on ubuntu, macos and windows so that gap closes on the next run.
Closes the v1 schema decision: plan_d's field set plus the four fields the other draft carried. `config_name`/`config_override` keep their names; `schema_version`, inline `config` and `image://` stay out. New fields: - `environment` and `env_file` per container. Env files apply in declared order (a later file wins), then literal `environment` on top; - `startup_timeout` (file-level default plus a per-container override) and `stop_timeout`. Both were hardcoded at 60s/10s and are now the operator's. The environment is now **built rather than inherited**, per IMPL_PLAN §2.4 and §6.5: a child gets a host baseline (PATH/HOME/... on unix, the longer set windows genuinely needs), then its declared env, then the reserved contract — and nothing else. A compose project therefore starts the same way whatever shell launched the daemon. This is a behaviour change: a worker that relied on an inherited variable must now declare it. A user value can never shadow a reserved key. It is refused at parse time (`RESERVED_ENV_OVERRIDE`) rather than dropped, in the compose file and in env files alike, because a silently ignored `III_URL` looks like it took effect. Env files are read at spawn time, never at parse time — they hold secrets — but a missing one fails validation, so `up` cannot die halfway through a graph. `validate` reports env variable *names* and env_file paths, never values.
Routing has been strict since the namespace work landed: an invoke without a namespace resolves only in `default`. That left every function registered outside `default` unreachable from the CLI — a hole in the namespace feature itself, not just a compose problem. `iii trigger <fn> --namespace <ns>` now resolves in that namespace, and `--help` passes it to `engine::functions::info` so the schema shown is the one that would actually be invoked. Verified against a live engine: `--namespace default` resolves, `--namespace nao-existe` returns the engine's structured miss, which already names where the id does exist: Function engine::functions::list not found in namespace nao-existe. It is registered in namespace(s): default. This is what decision #2 needs: each compose daemon owns `compose::*` in its own namespace, so two daemons never collide, and the operator reaches one with `iii trigger compose::up --namespace host-a`.
Phases 4 and 5: the crate stops being a validator and becomes a daemon. **Connection.** The daemon registers as a worker named `--id`, in the namespace of the same name, and exposes `compose::*` there. Children go to the *project* namespace, which is derived separately — the two are independent, which is what lets several daemons share an engine without competing for function ids. A second daemon claiming the same `--id` loses the `(namespace, worker_name)` lease and is told so; it exits instead of serving half a project. **Readiness is registration.** A container is up when the engine reports it under `(project_namespace, container)` — not when its process spawned. The two differ exactly when it matters: a worker that starts and dies, or starts and never connects, must not count as started for whatever depends on it. A child that exits while we wait short-circuits the wait, so the operator is told the child died rather than that it was slow. **Failure undoes this operation, not the world.** A failed `up` stops what it started, in reverse order, and leaves already-running containers alone; a repeated `up` reports `changed: false`. `down` walks the graph backwards, so dependents stop before what they depend on, and a targeted `down` takes its local dependents with it. Config is now fetched before the spawn (fetch-or-fail: a container that declares `config_name` does not start on defaults), merged with `config_override`, and handed over as the `0600` file the spawn contract already carried. Verified against a real engine in `engine/tests/compose_daemon_e2e.rs`: `compose::*` answers only in the daemon's namespace, a mismatched `id=` is refused with WRONG_DAEMON naming the right invocation, a child that never registers times out and rolls back leaving nothing ready, and a duplicate `--id` is rejected rather than silently shadowed. Two gaps, both declared rather than papered over: `compose::logs` returns LOGS_NOT_IMPLEMENTED because children still inherit the daemon's stdio, and no e2e covers a container that actually reaches ready — that needs a fixture binary which speaks the SDK.
`down` reversed the plan unconditionally. For the whole project that is right — the start order is dependencies-first, so its mirror is dependents-first. For a single target it was wrong: `transitive_dependents` already returns them nearest-first, so reversing put the target ahead of the things that depend on it. `down container=api` stopped `api` and only then `web`, which is the state `down` exists to prevent. The reverse now happens only on the whole-project branch. Found by running the smoke-test project in smoke_compose, not by a unit test — the regression test added here is the one that would have caught it.
The daemon awaited `ctrl_c()` alone, so it handled SIGINT and nothing else. Every other way a daemon is stopped — systemd, docker, a `kill` or `pkill` in a script — sends SIGTERM, and on that path the process died without running its shutdown: the children were orphaned with ppid 1, still connected to the engine, with no daemon left that could stop them. Found while cleaning up after a smoke run: `pkill -f 'compose --id smoke-a'` left four demo-workers behind. The daemon now selects on both signals and runs the same local `down` either way, which is what IMPL_PLAN §2.5 means by "an intentional daemon shutdown performs a local down". Verified by hand: SIGTERM to a running daemon now prints "stopping 3 container(s)..." and leaves no process behind.
guibeira
added a commit
that referenced
this pull request
Aug 10, 2026
The review pointed out that one container failing answers differently depending on when it fails: during up the first failure ends the operation and rolls back everything it started, while after readiness the supervisor takes only that container's dependents. Same declaration, two blast radii. The diagnosis is right and the docs did not say it. The explanation page now does, along with the fact that there is no restart policy at all, so a crash is permanent until the next up. Declining to add the policy in this PR. is small but changes what means, and a restart policy needs backoff, a cap, and a state compose::status does not have. Both are filed: #2051 and #2052. Reported by anthonyiscoding in review of #2034.
The review pointed out that one container failing answers differently depending on when it fails: during up the first failure ends the operation and rolls back everything it started, while after readiness the supervisor takes only that container's dependents. Same declaration, two blast radii. The diagnosis is right and the docs did not say it. The explanation page now does, along with the fact that there is no restart policy at all, so a crash is permanent until the next up. Declining to add the policy in this PR. `required: false` is small but changes what `status: ok` means, and a restart policy needs backoff, a cap, and a state compose::status does not have. Both are filed: #2051 and #2052. Reported by anthonyiscoding in review of #2034.
guibeira
force-pushed
the
feat/compose-daemon
branch
from
August 10, 2026 21:02
de45c92 to
b09338e
Compare
Terminology.SlopMarketing rejects it. Two uses in the state section, both replaced with plain verbs.
The artifact carried an injected '# Compose' heading that iii-skill-render no longer emits for this file, so it was 11 bytes off and the docs check reported drift. Hand-syncing the .skill.md alongside the .mdx keeps the body right and cannot get the wrapper right; rendered with iii-skill-render 0.4.32 instead.
Left over from the sanitize removal: the schema section still said the name was lowercased and reduced to [a-z0-9_-], which is the behaviour 849669d deleted.
The capture exists for one window: a worker that dies before it connects has no way to tell the engine anything, so its own stderr is the only account of itself. That is why log_tail reads the file, and it is the only thing that reads it. The window closes at readiness, and the capture did not. It ran for the life of the container, appending, with no rotation, for a purpose that ended at boot. A single trigger against a chatty worker grew the file from 496 bytes to 149 KB in the smoke suite. Registered, a worker has the engine to log through, and that is where a worker's log belongs. So capture stops there. What stays on disk is the boot, which is the part the engine never saw. Draining does not stop, only writing. A pipe nobody reads fills at 64 KiB and the child blocks forever on its next println, so ending the capture by dropping the reader would hang exactly the workers that log the most. 13-logs now prints 5000 lines after readiness and asserts both halves: nothing reaches the file, and the worker still answers.
The SDK now sends a worker's calls into its own namespace, and the daemon has one: whatever `--ns` says. So its five calls to configuration::get, configuration::register, configuration::schema, engine::functions::list and engine::workers::list would have resolved in the daemon's namespace, where the engine does not serve them. That is not a small break. Those five are how the daemon reads a container's configuration and how it observes readiness, so a daemon on the new SDK would have failed every `up` at the first fetch and never seen a container register. The constant was already there for the readiness checks; its doc now says why outbound calls use it too. The smoke fixture already named `default` on its own configuration::register, so it needed nothing. 19/19 scenarios, 371 checks, against a local build of this branch merged with its base.
Left out of 745da04: that commit staged `crates/` only, and `cargo fmt --all` formats the workspace, so this file stayed unformatted in the repo and would have failed Engine Formatting.
…a blank namespace is refused (#2057)
…I uses The daemon took `--ns` while `iii trigger` took `--namespace`, so the two commands an operator uses together spelled the same concept two ways and neither accepted the other's. Starting a daemon with `--ns loja` and then reaching for `iii trigger compose::up --ns loja` is an unknown-argument error; the line the daemon prints already corrects it, but only after the trip. `--ns` stays as an unadvertised alias so a script that already spells it that way keeps running. It can go once nothing does.
Resolves the two conflicts in `engine-protocol.mdx` and its skill artifact by taking the base branch's side, which is the corrected one: - the `RegisterTrigger` example carried `"namespace": null` here, which the paragraph two lines below rejects as invalid; - the paragraph on an omitted `namespace` said the SDKs simply stopped omitting it, without the exception for a worker that declared no namespace of its own.
Compose accepted `binary` and refused everything else, with a comment that explained only the `engine` case. The check was really "compose runs native executables", which left `bundle` out — a kind `iii add` installs and the registry publishes today. A bundle is publisher-controlled shell: its `iii.worker.yaml` names the start command, and nothing here reads or runs it. It boots in a libkrun VM on iii-worker's own rails, so the operator kill switch, the manifest re-validation and the resource clamp are the same ones `iii add` applies rather than a second copy that drifts. Compose supervises its own children, so it needs the command rather than a detached VM tracked by a pid file. `build_vm_command` splits the boot arguments out of `run_dev`, which keeps its behaviour by calling it; both go through one place, so the isolation the guest gets cannot differ between them. `setsid` moves to the caller: `run_dev` detaches because its VMs outlive it, compose puts its children in a group it can signal, and applying both makes the spawn fail with EINVAL. Identity is the caller's too. Keying VM state by worker name would have two projects running the same bundle under one rootfs, and a start would kill the other project's VM by name — compose keys it by project instead. Configuration reaches the guest. `--config` was ignored for bundles for want of a mount; the container's config directory is published at /run/iii/config and `III_CONFIG` names the file inside it, so a worker reads its configuration the same way whichever side of the boundary it runs on. The directory is per-container because virtiofs shares a whole tree, and a flat one would put every sibling's resolved secrets inside every guest.
Two changes to one surface, both about telling an operator the truth. `--ns` is gone rather than kept as a hidden alias. The daemon said `--ns` while `iii trigger` said `--namespace` for the same idea, and nothing has shipped that calls it the old way. The sweep is the interesting part: four unit tests, a systemd `ExecStart` in the docs, and 23 invocations in compose-smoke-tests all spelled it the old way and would have broken at the moment the alias came out. And bundles now refuse windows with a reason. That was a build break, not a missing message: libkrun's VMM does not compile there, so depending on iii-worker unconditionally stopped `iii` itself from building for x86_64-pc-windows-msvc. The dependency moves under `cfg(unix)` and the refusal takes its place, raised when the registry names the kind and before the archive is fetched — megabytes an operator learns nothing from. It is its own error rather than UNSUPPORTED_PACKAGE_KIND, whose message claims compose installs bundle workers. True on linux, false here, and it would send someone looking for a fault in their compose file. It points at WSL, which is where that machine has a KVM to run a VM on; if WSL has none, the KVM check downstream already says so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets
feat/namespare(#1984): compose is the first real consumer of namespace routing, and does not build without it.What this is
iii composeruns a declared set of workers as one project. Aworker-compose.yamlnames the containers, their dependencies and how each one starts. The daemon validates the declaration, resolves configuration, starts the graph in order, and supervises what it created.There is one command, and it does not background itself:
That connects to an engine and serves the
compose::*functions in the foreground. Every operation on a project is a normal trigger, addressed to one daemon:The foreground is the shape a process supervisor already wants. systemd, launchd or a shell redirect gives restart-on-failure and log rotation, so compose does not have to grow either.
A greenfield crate (
crates/iii-compose). It shares no runtime code withcrates/iii-worker, whose lifecycle system it is meant to replace rather than extend.Why it needs namespaces
Two namespaces are in play, and they answer different questions.
The daemon's namespace is its address. The worker name is always
compose, so the engine leases(namespace, compose)to one connection.--nsis what tells two daemons on one engine apart, and it is what an operator types into--namespaceto reach exactly one of them. A second daemon claiming a namespace that is taken is refused at registration withDAEMON_ALREADY_SERVING. Without--nsthe daemon generates one and prints it, because a shared well-known default is the collision the namespace exists to prevent.The project's namespace is where its containers register. It comes from the
namespace:field in the compose file, and a project that declares none lands indefault. Nothing about the file's path enters it. A value outside[a-z0-9_-]is refused withINVALID_NAMESPACEinstead of being rewritten to fit, so what the file declares is what the operator types.Readiness is
(namespace, container)as seen inengine::workers::list. It is the pair, never the name alone, or another project's worker would satisfy it.The parts
Declaration. Strict schema (unknown fields are errors, duplicate container keys rejected), dependency graph with cycle paths in declaration order, Kahn ordering.
compose::validateneeds nothing but the file: it does not take the project on, andpackage://containers are reported underdeferred_packagesrather than resolved. So a CI job that only validates leaves the daemon owning nothing.Identity. A project is its compose file and nothing else names one. The same file reached twice is the same project however the path was spelled, so there is no second identity to keep in sync. The container key is the worker name, and it wins over the manifest's
name: the key is what reaches the child asIII_WORKER_NAME.Spawn contract. The child's environment is built, not inherited: a host baseline, then
env_filein declared order, thenenvironment, then five reserved variables the daemon owns (III_URL,III_NAMESPACE,III_WORKER_NAME,III_CONFIG,III_CONFIG_NAME). A staleIII_URLin the operator's shell can never point a child at the wrong engine, and a project starts the same way whoever launched it. Declaring a reserved variable is caught at validation, in anenv_fileas well as inenvironment.Configuration. A container names the entry it owns with
config_name, andconfig_overrideis merged on top. Precedence is what the package ships, then what the configuration worker holds, then the override. The merged result is written to an owner-only file and passed asIII_CONFIG, so the worker reads configuration the way it always does. A container that declaresconfig_namedoes not start when the fetch fails.Supervision. Each child leads its own process group, so teardown reaches grandchildren. A pid is never signalled without a birth-identity match, because pids are recycled. After readiness a poll loop notices an exit nobody asked for, fires
post_run, and cascades to the transitive dependents in reverse-topological order. Nothing is restarted: v1 has no restart policy. On restart, survivors are adopted back, because a process left out of the children map is onedownwalks past while reporting success.Registry.
package://resolves against the registry, verifies the sha256 before anything reaches the final directory, and caches by(name, version, target).State. One place per machine,
~/.iii/composeor$III_COMPOSE_STATE_DIR, keyed by daemon namespace and a derived project directory. It holds the child records, the resolved configuration files, and what each container printed while starting. Capture stops at readiness: before it, the container's own output is the only account of itself; after it, the worker's logging is the record and compose keeps no second copy.Diagnosis over silence. A worker that ignores
III_NAMESPACE, one that registers under its own name, and one whose functions were filed in another namespace are all healthy processes that readiness can never satisfy. Each is named the moment it can be known, instead of twenty seconds of nothing followed bySTARTUP_TIMEOUT. Codes cross the wire, stable and documented.Verification
engine/tests/compose_daemon_e2e.rs) against a real in-process engine:compose::*answers only in the daemon's namespace, a request aimed at another daemon is refused, one daemon holds several projects, one file is one project however it is spelled, a child that never registers times out and rolls back, a duplicate namespace is refused while two distinct ones both serve, and a configuration that cannot be read stops the container.compose-test-matrixCI job runs the crate suite on Linux, macOS and Windows, because supervision is platform code.downin the wrong order, orphaned grandchildren, a stale worker satisfying readiness, hooks shredding the spinner, and teardown returning on a dead leader while its worker kept serving.Known gaps
CTRL_BREAKreplacesSIGTERM, but the unix suites are cfg-gated, so the Windows job proves compilation of the job-object path and little of its behaviour.libprocthere is no birth fingerprint, so every live pid is unverifiable. That is safe, but a restart there reports children for manual cleanup.upends the operation; a failure after readiness takes the dependents. Neither is configurable in v1.path://andpackage://resolve. There is no container runtime behindimage://, which is out of scope for v1.