docs(multi-agent): how notification delivery actually behaves, with n… #1158

Workflow file for this run

	name: M3 Memory CI

	on:
	push:
	branches: [ "main" ]
	pull_request:
	branches: [ "main" ]

	permissions:
	contents: read

	jobs:
	lint:
	name: Lint (Ruff)
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
	with:
	persist-credentials: false
	- name: Set up Python
	uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
	with:
	python-version: "3.12"
	cache: 'pip'
	- name: Install Ruff
	run: pip install ruff
	- name: Run Ruff
	# bench_.py / benchmark_.py are excluded: they carry the same mechanical
	# lint debt, but editing them trips the pre-push bench-data guard (their
	# source references private benchmark dataset paths). Lint them via a
	# separate bench-aware change, not the standard CI gate.
	run: ruff check bin/ memory/ --exclude 'bench_.py' --exclude 'benchmark_.py'

	typecheck:
	name: Type Check (Mypy)
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
	with:
	persist-credentials: false
	- name: Set up Python
	uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
	with:
	python-version: "3.12"
	cache: 'pip'
	- name: Install dependencies
	run: \|
	pip install -r requirements.txt
	pip install types-PyYAML
	- name: Run Mypy
	run: mypy bin/ --ignore-missing-imports

	security:
	name: Security scan (Bandit + pip-audit on core deps)
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
	with:
	persist-credentials: false
	- name: Set up Python
	uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
	with:
	python-version: "3.12"
	cache: 'pip'

	# Bandit: static analysis on all code paths. Config (skips, exclusions)
	# lives in pyproject.toml under [tool.bandit].
	- name: Install Bandit
	run: pip install "bandit[toml]"
	- name: Run Bandit
	run: bandit -c pyproject.toml -r bin/ m3_memory/

	# pip-audit: scoped to CORE deps only (no [dev] / no opt-in rerank
	# path). Bench/dev transitive CVEs (e.g. transformers, lxml) shouldn't
	# gate merges since they don't reach end users; but a CVE in a real
	# shipped dep should fail the build immediately. We install the
	# package itself (core deps only — no extras) into a clean venv, then
	# audit only the locally-installed transitive tree (-l). The local
	# flag also skips m3-memory itself, which would otherwise fail audit
	# before its release lands on PyPI (chicken-and-egg in pre-release CI).
	- name: Install M3 (core deps only) into a clean venv
	run: \|
	python -m venv /tmp/m3-core
	/tmp/m3-core/bin/pip install --upgrade pip
	# Install M3 to pull its core dependencies, then uninstall the
	# package itself. pip-audit -l would otherwise try to look up
	# m3-memory on PyPI and fail with "not found" before each release
	# tag is published (chicken-and-egg in pre-release CI).
	/tmp/m3-core/bin/pip install .
	/tmp/m3-core/bin/pip uninstall -y m3-memory
	/tmp/m3-core/bin/pip install pip-audit
	- name: pip-audit (core deps, locally installed)
	# CVE-2026-3219: pip itself — present because pip is in every venv,
	# but pip is not a shipped runtime dependency of m3-memory. Ignored
	# so the audit signal stays focused on actual shipped-library CVEs.
	#
	# PYSEC-2025-183: pyjwt "weak encryption" — affects ALL pyjwt versions
	# (0.1.1–2.12.1) with no fixed version published, so it cannot be
	# resolved by upgrading. The advisory is disputed by the pyjwt
	# maintainers: the concern is the key length chosen by the calling
	# application, not a flaw in the library. m3-memory's own code never
	# imports `jwt`; pyjwt is present only transitively via the `mcp`
	# core dependency, so the weak-key-length path is not reachable
	# through m3-memory. Ignored; revisit if pyjwt ships a real fix.
	run: /tmp/m3-core/bin/pip-audit --strict -l --ignore-vuln CVE-2026-3219 --ignore-vuln PYSEC-2025-183

	# Emit the test matrix based on event type. Pull requests get a single fast
	# cell (ubuntu + 3.12) for quick feedback; pushes to main/release run the
	# full OS × Python grid so cross-platform coverage still gates the branch
	# everything merges into. A bug that only shows on macOS/Windows or 3.11 is
	# caught at merge time, not on every PR iteration.
	matrix-setup:
	name: Resolve test matrix
	runs-on: ubuntu-latest
	outputs:
	matrix: ${{ steps.set.outputs.matrix }}
	steps:
	- id: set
	run: \|
	if [ "${{ github.event_name }}" = "pull_request" ]; then
	echo 'matrix={"os":["ubuntu-latest"],"python-version":["3.12"]}' >> "$GITHUB_OUTPUT"
	else
	echo 'matrix={"os":["ubuntu-latest","macos-latest","windows-latest"],"python-version":["3.11","3.12"]}' >> "$GITHUB_OUTPUT"
	fi

	test:
	name: Test on ${{ matrix.os }} (py${{ matrix.python-version }})
	needs: [lint, typecheck, matrix-setup]
	runs-on: ${{ matrix.os }}
	strategy:
	fail-fast: false
	matrix: ${{ fromJSON(needs.matrix-setup.outputs.matrix) }}

	steps:
	- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
	with:
	persist-credentials: false

	- name: Set up Python ${{ matrix.python-version }}
	uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
	with:
	python-version: ${{ matrix.python-version }}
	cache: 'pip'

	- name: Install Dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -r requirements.txt

	- name: Run Pytest (hermetic lane)
	# The hermetic unit lane: everything that needs no external service and is
	# not timing-sensitive.
	# - `not integration` excludes tests needing a live service (requires_pg /
	# requires_embedder / requires_gguf / requires_files_db) — GH runners
	# provision none, so they'd only self-skip. Run them on a dev box (or a
	# future services job) with `pytest -m integration`.
	# - `not slow` excludes the perf tests (test_chatlog_perf). Their wall-clock
	# budgets (e.g. enqueue < 200ms) assume a quiet, dedicated host; a shared
	# ubuntu-latest runner misses them intermittently (measured 213ms vs 200ms
	# budget — a scheduling flake, not a regression). Perf belongs on a
	# baseline-stable host, not the PR gate. Run locally with `pytest -m slow`.
	# --strict-markers makes a typo'd marker a hard error.
	# See docs/design/TEST_SUITE_DESIGN.md.
	run: \|
	pytest tests/ -m "not integration and not slow"

	# ── End-to-end install ──────────────────────────────────────────────────────
	# The unit lane covers the install PIECES; nothing exercised the whole path.
	# test_install_no_clobber deliberately swallows wiring failures
	# ("post-fetch wiring may raise in the isolated env; data safety is what we
	# assert"), so an install that completes and leaves a NON-FUNCTIONAL system
	# passes every existing test. That is exactly how a broken install reaches a
	# user: `m3 setup` used to discard the doctor's verdict and exit 0 regardless.
	#
	# This job installs the package the way a user does, runs the real wizard
	# unattended against THROWAWAY roots, and requires it to exit 0 — which now
	# means "installed AND `m3 doctor` passed". Exit 3 (installed but unverified)
	# fails the build.
	#
	# Runs on all three OSes on push: the failures that motivated it were
	# platform-specific (Windows file locks, venv launcher stubs, POSIX-only
	# asyncio signal handlers), so a single-OS lane would have missed them.
	e2e-install:
	# A HANG must not burn a 6-hour runner. The Windows lane sat in_progress for
	# 25+ minutes on a step whose ubuntu/macOS twins finish in well under 10 —
	# the failure mode is a BLOCK, not a crash, and without a cap every failing
	# run costs six hours and delays the post-mortem steps that explain it.
	timeout-minutes: 25
	name: E2E install + upgrade on ${{ matrix.os }}
	needs: [lint, typecheck, matrix-setup]
	runs-on: ${{ matrix.os }}
	strategy:
	fail-fast: false
	matrix: ${{ fromJSON(needs.matrix-setup.outputs.matrix) }}

	steps:
	- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
	with:
	persist-credentials: false

	- name: Set up Python ${{ matrix.python-version }}
	uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
	with:
	python-version: ${{ matrix.python-version }}
	cache: 'pip'

	- name: Install m3-memory from source (as a user would)
	run: \|
	python -m pip install --upgrade pip
	pip install .

	# Roots live under the runner temp dir, never the default ~/.m3, so the
	# wizard cannot touch anything outside this job and each run starts clean.
	- name: Run the setup wizard unattended (throwaway roots)
	shell: bash
	env:
	# No agent is installed on a runner, so wire none and disable capture:
	# the point is the install path itself, not hook wiring into a client
	# that is not there.
	M3_E2E_ROOT: ${{ runner.temp }}/m3-e2e
	# Breadcrumbs to a FILE: the Windows failure kills setup with no output
	# on either stream, so anything relying on the dying process flushing
	# its own stdio is precisely what cannot be trusted here.
	M3_TRACE_FILE: ${{ runner.temp }}/m3-setup-trace.log
	run: \|
	mkdir -p "$M3_E2E_ROOT/config" "$M3_E2E_ROOT/engine"
	# --no-native-wheel: runners have no GPU and a source build is slow;
	# the pure-Python path is what we are gating here.
	# --no-shared-embedder / --no-dashboard / --no-governor-migration:
	# long-lived services and OS scheduler entries are not the subject
	# of this test and would leak state on a shared runner.
	# Invoke the MODULE, not the `m3` console script. On Windows the script's
	# UTF-8 re-exec SPAWNS a child rather than replacing the image, so the
	# launcher generation stays alive as a registered m3 process for the whole
	# run. Setup's preflight then finds its own PARENT holding the DB and
	# correctly refuses to kill it — exit 2, "cannot quiesce mcp(pid N): it is
	# this install's own parent process". `python -m m3_memory.cli` has no
	# launcher generation. This is exactly the advice setup itself prints for
	# this case, so CI should follow it.
	python -m m3_memory.cli setup \
	--non-interactive --terminal \
	--agents "" --capture-mode none \
	--no-native-wheel --no-shared-embedder \
	--no-dashboard --no-governor-migration \
	--force-quiesce \
	--decouple-roots \
	--config-root "$M3_E2E_ROOT/config" \
	--engine-root "$M3_E2E_ROOT/engine"

	# The Windows lane has never been green and its failure mode is a SILENT
	# one: the log ends at "Step 5/5" with exit 1 and no verdict. Print the
	# doctor's own exit code and output on that runner so the next failure
	# names itself instead of needing another CI round-trip to diagnose.
	- name: Dump the setup trace (always — this is the post-mortem)
	if: always()
	shell: bash
	run: \|
	f="${{ runner.temp }}/m3-setup-trace.log"
	echo "--- setup trace ($f) ---"
	if [ -f "$f" ]; then cat "$f"; else echo "(no trace file written)"; fi

	- name: Diagnose verification (Windows only, never fails the job)
	# `always()` is REQUIRED: without it a step is skipped once an earlier one
	# fails, and the whole point of this step is to explain the setup failure
	# that just happened. Placed after setup, it never ran on the very runs it
	# exists to diagnose.
	if: always() && runner.os == 'Windows'
	shell: bash
	env:
	M3_E2E_ROOT: ${{ runner.temp }}/m3-e2e
	run: \|
	export M3_CONFIG_ROOT="$M3_E2E_ROOT/config"
	export M3_ENGINE_ROOT="$M3_E2E_ROOT/engine"
	echo "--- m3 doctor (scoped, exactly as setup runs it) ---"
	m3 doctor --skip-shared-embedder --skip-cascade; echo "doctor exit=$?"
	echo "--- the same invocation setup uses, via -m ---"
	python -m m3_memory.cli doctor --skip-shared-embedder --skip-cascade
	echo "python -m doctor exit=$?"
	# The setup step's child dies writing NOTHING — not one byte reaches the
	# log, on a step whose ubuntu twin streams the full doctor report. The
	# steps above run that same command standalone and both exit 0, so the
	# difference is the CHILD-OF-SETUP context, not the command. Reproduce
	# it: spawn exactly as _step_doctor does, with the parent's re-exec
	# sentinel present, and CAPTURE both streams instead of streaming them.
	echo "--- as a CHILD, with the parent's UTF-8 re-exec sentinel set ---"
	cat > "$RUNNER_TEMP/probe_child.py" <<'PYEOF'
	import os, subprocess, sys
	argv = [sys.executable, "-m", "m3_memory.cli", "doctor",
	"--skip-shared-embedder", "--skip-cascade"]
	print("sentinel:", os.environ.get("_M3_UTF8_REEXEC"))
	r = subprocess.run(argv, capture_output=True, text=True,
	errors="backslashreplace")
	print("rc:", r.returncode)
	print("stdout[:400]:", repr(r.stdout[:400]))
	print("stderr[:800]:", repr(r.stderr[:800]))
	PYEOF
	sed -i 's/^ //' "$RUNNER_TEMP/probe_child.py"
	_M3_UTF8_REEXEC=1 python "$RUNNER_TEMP/probe_child.py" \|\| true
	echo "--- engine root ---"
	ls -la "$M3_E2E_ROOT/engine" \|\| true
	echo "--- config root ---"
	ls -la "$M3_E2E_ROOT/config" \|\| true

	# Belt-and-braces: setup's own exit code already encodes this, but assert
	# the resulting install is independently usable rather than trusting the
	# installer to grade its own work.
	- name: Verify the install is actually functional
	shell: bash
	env:
	M3_E2E_ROOT: ${{ runner.temp }}/m3-e2e
	run: \|
	export M3_CONFIG_ROOT="$M3_E2E_ROOT/config"
	export M3_ENGINE_ROOT="$M3_E2E_ROOT/engine"
	set -x
	m3 --version
	# `m3 status` returns 1 on "degraded" BY DESIGN (installer.status:
	# "Returns 0 healthy, 1 degraded/broken"). This job installs with
	# --no-native-wheel --no-shared-embedder, so the result is degraded on
	# purpose: pure-Python embeds, no :8082. Asserting exit 0 here demands
	# the opposite of what the flags above requested — and under `set -e`
	# it killed the step before `m3 doctor` even ran. Keep the output for
	# the log, but do not gate on a verdict this job engineered.
	m3 status \|\| true
	# Grade only what this job installed. The setup above passes
	# --no-shared-embedder, so the shared-embedder probe and the
	# embedder-dependent probes (embedding-cascade, and the file-extraction
	# LLM probe nested inside it) all report the ABSENCE we asked for and
	# return 1. Demanding them here would fail the job for honouring its own
	# flags. Every other probe still runs and still gates this step.
	m3 doctor --skip-shared-embedder --skip-cascade
	# The store must be writable and searchable end to end, not merely present.
	m3 memory memory_write --content "e2e canary" --type note
	m3 memory memory_search --query "e2e canary" --k 1 \| grep -q "e2e canary"

	# ── Upgrade over the install we just made ────────────────────────────────
	# This is the path that actually breaks in the field: a reinstall on top of
	# a POPULATED install, with the previous version's artifacts still on disk.
	# A fresh install exercises none of it — no existing DBs to migrate or
	# clobber, no already-registered schedules, no stale bytecode, no running
	# writers. Every incident that motivated this job happened on an upgrade.
	#
	# Two things must hold, and they are different claims:
	# 1. the upgrade completes AND verifies (exit 0, not 3)
	# 2. the data written before it is still there afterwards
	# (2) is the one that matters most and the one a "did it exit 0" check
	# cannot see: a wipe-and-recreate would pass every other assertion here.
	- name: Re-run setup over the existing install (upgrade path)
	shell: bash
	env:
	M3_E2E_ROOT: ${{ runner.temp }}/m3-e2e
	run: \|
	# Record the pre-upgrade state so a silent data loss is visible even if
	# the canary search were to pass for the wrong reason.
	export M3_CONFIG_ROOT="$M3_E2E_ROOT/config"
	export M3_ENGINE_ROOT="$M3_E2E_ROOT/engine"
	echo "engine root before upgrade:"
	ls -la "$M3_E2E_ROOT/engine"
	# Invoke the MODULE, not the `m3` console script. On Windows the script's
	# UTF-8 re-exec SPAWNS a child rather than replacing the image, so the
	# launcher generation stays alive as a registered m3 process for the whole
	# run. Setup's preflight then finds its own PARENT holding the DB and
	# correctly refuses to kill it — exit 2, "cannot quiesce mcp(pid N): it is
	# this install's own parent process". `python -m m3_memory.cli` has no
	# launcher generation. This is exactly the advice setup itself prints for
	# this case, so CI should follow it.
	python -m m3_memory.cli setup \
	--non-interactive --terminal \
	--agents "" --capture-mode none \
	--no-native-wheel --no-shared-embedder \
	--no-dashboard --no-governor-migration \
	--force-quiesce \
	--decouple-roots \
	--config-root "$M3_E2E_ROOT/config" \
	--engine-root "$M3_E2E_ROOT/engine"

	- name: Verify the upgrade preserved data and left a working install
	shell: bash
	env:
	M3_E2E_ROOT: ${{ runner.temp }}/m3-e2e
	run: \|
	export M3_CONFIG_ROOT="$M3_E2E_ROOT/config"
	export M3_ENGINE_ROOT="$M3_E2E_ROOT/engine"
	set -x
	# The memory written BEFORE the upgrade must still be searchable. This
	# is the load-bearing assertion: it fails on a wipe-and-recreate, on a
	# migration that drops rows, and on an engine root silently relocated
	# out from under the data.
	m3 memory memory_search --query "e2e canary" --k 1 \| grep -q "e2e canary"
	# And the install must still be healthy, not merely intact.
	# Same scoping as the first verify step — see the note there.
	m3 doctor --skip-shared-embedder --skip-cascade
	# A second write must work too — a store that survived but is now
	# read-only or schema-broken is still a broken upgrade.
	m3 memory memory_write --content "post-upgrade canary" --type note
	m3 memory memory_search --query "post-upgrade canary" --k 1 \| grep -q "post-upgrade canary"

	- name: Show the engine root on failure
	if: failure()
	shell: bash
	env:
	M3_E2E_ROOT: ${{ runner.temp }}/m3-e2e
	run: \|
	ls -la "$M3_E2E_ROOT/engine" \|\| true
	ls -la "$M3_E2E_ROOT/config" \|\| true

Navigation Menu

docs(multi-agent): how notification delivery actually behaves, with n… #1158

Workflow file

docs(multi-agent): how notification delivery actually behaves, with n… #1158

Uh oh!

Workflow file for this run