R1.4 Client callbacks for delivery and engagement events

[alexeygrigorev]

Plan issue: R1.4 (DataTalksClub/community-base docs/plan/phase-1.md). Mirrored as #17. Depends on R1.2 (merged in #9).

What this does

• Per-client callback endpoint (CallbackEndpoint: URL, dedicated HMAC signing secret with rotation overlap, contract version, enable/disable) and a durable ClientCallback outbox deduplicated on (client, event_id), with stable uuid5 event ids per transition.
• Events: delivery.accepted, delivery.delivered, delivery.bounced (with bounce_type hard/soft), delivery.complained, delivery.suppressed, engagement.opened, engagement.clicked, subscription.changed — matching the contract pinned by the community-base package client (callback_ingress.py: timestamp field, bounce_type selector, event_id/message_id/client_reference/reason_code shapes).
• Signature scheme identical to webhook tasks: sha256= HMAC-SHA256 over <timestamp>.<raw body>, headers X-Relay-Timestamp / X-Relay-Signature (+ event id, type, contract version, attempt). No Bearer credential.
• Retry with bounded exponential backoff (60 s base, 6 h cap, deterministic jitter); other-4xx, redirects, endpoint disablement and retry exhaustion are terminal. Per-message sequence counters for best-effort ordering; duplicate event_id emission is deduplicated (unique constraint + test).
• GET /api/transactional/messages?since= reconciliation: Bearer-authenticated, scoped to the authenticated client, updated_at >= since, oldest first, capped at 1000 rows, package reconciliation vocabulary (queued, retrying, sent, delivered, suppressed, failed, bounced, complained) with pattern-safe reason codes. template_version reports 1 until R1.3 introduces versions.
• docs/api.md documents callback payloads, signature headers, reference receiver, retry semantics, and the reconciliation endpoint. tests/fixtures/client_callback_contract_v1.json gives consumers a deterministic verification vector.

Scope decision: CMP callbacks stay

The first WIP commit on this branch deleted the CMP callback system. It is restored here unchanged: CMP (DataTalksClub/course-management-platform) is a production consumer of those callbacks (docs/context.md: "CMP production depends on it. Breaking it stops course email"), and R1.4 does not scope its removal. The two channels now run side by side:

• Client callbacks carry transactional-message transitions and client-level subscription changes (the transitions whose receivers own a client_reference).
• Campaign-recipient transitions and terminal send failures stay on the CMP channel; failed sends reach plan clients through reconciliation instead.

Ambiguity reading (per the plan): emission scope follows the receiver contract in community-base, which has no handling for campaign-scoped or unknown (delivery.failed) event types; emitting them would poison the outbox with terminal 4xx failures.

Verification

• uv run --frozen pytest -> 608 passed (full suite; includes duplicate event_id dedup, signature/replay-window receiver tests, retry classification, rotation overlap, reconciliation status mapping, auth and validation errors).
• uv run --frozen ruff check . -> all checks passed.
• uv run --frozen python scripts/validate_infra.py -> validated.
• uv run --frozen python manage.py makemigrations --check --dry-run -> no changes detected.

Not run here, needs: the sandbox verification (send to the SES simulator bounce address and watch a reference receiver log delivery.bounced with the sent client_reference) after this is deployed to the sandbox environment.

[alexeygrigorev]

Coordination note: this branch now conflicts with main (R1.1 #15, R1.3 #16 and R1.5 #19 merged since it diverged), so it shows CONFLICTING and no CI has run. A merge of origin/main into r1.4-client-callbacks is needed before it can land; CI will then run on the updated head.

[alexeygrigorev]

Coordination note: this branch now conflicts with main (R1.1 #15, R1.3 #16 and R1.5 #19 merged since it diverged), so it shows CONFLICTING and no CI has run. A merge of origin/main into r1.4-client-callbacks is needed before it can land; CI will then run on the updated head.

[alexeygrigorev]

Pre-merge verification from a read-only pass at head 5d7bfbb:

Issue coverage (R1.4 card) is complete at this head:

• All nine event types map in CALLBACK_EVENT_TYPES (mailing/services/client_callbacks.py), including UNSUBSCRIBE -> subscription.changed.
• Reconciliation endpoint exists: GET /api/transactional/messages with required ISO since (validate_reconciliation_since, get_transactional_messages_since_for_client, route in mailing/urls.py), documented in docs/api.md.
• Duplicate event_id dedupe test, X-Relay-Signature HMAC, and bounded retry with backoff (max_attempts, next_attempt_at) are present and tested.
• uv run --frozen ruff check . passes at this head.

Before merging: GitHub reports this branch CONFLICTING with current main after the R1.5 merge. Overlapping files: docs/api.md, mailing/models.py, mailing/services/api.py, mailing/services/transactional.py, mailing/tests/test_client_api.py, mailing/tests/test_transactional_api.py, mailing/urls.py, mailing/views.py, relay/settings.py.

One trap while resolving: both sides add a migration numbered 0026 (this branch: 0026_callbackendpoint_clientcallback_and_more; main: 0026_emailtemplate_category_...). Renumber this branch's to 0027 when merging main in, otherwise Django ends up with two same-numbered leaves on 0025.

The local full pytest battery was interrupted mid-run (verification worktree torn down before completion), so CI plus the targeted checks above are the evidence recorded here.

[alexeygrigorev]

Merged origin/main (R1.1, R1.3, R1.5) into the branch and re-verified. Post-merge adaptations:

1. The branch migration was renumbered 0026_callbackendpoint... -> 0027_callbackendpoint_clientcallback_and_more and chained on R1.3's 0026_emailtemplate_category... (both branches had created a 0026; the collision broke the migration graph).
2. The reconciliation payload now reports the real template_version recorded by R1.3's send pipeline, falling back to 1 for messages sent before versions were recorded; docs/api.md updated to match.
3. Conflict resolutions were import blocks in mailing/services/api.py, mailing/services/transactional.py, and mailing/tests/test_transactional_api.py (union of the client-callback and category/versions imports).

Re-verification on the merged tree:

• uv run --frozen pytest -q -> 645 passed in 408.77s
• uv run --frozen ruff check . -> all checks passed
• uv run --frozen python manage.py makemigrations --check --dry-run -> no changes detected
• migrate from empty on a fresh sqlite database -> applies clean through 0027

[alexeygrigorev]

Merged origin/main (R1.1, R1.3, R1.5) into the branch and re-verified. Post-merge adaptations:

1. The branch migration was renumbered 0026_callbackendpoint... -> 0027_callbackendpoint_clientcallback_and_more and chained on R1.3's 0026_emailtemplate_category... (both branches had created a 0026; the collision broke the migration graph).
2. The reconciliation payload now reports the real template_version recorded by R1.3's send pipeline, falling back to 1 for messages sent before versions were recorded; docs/api.md updated to match.
3. Conflict resolutions were import blocks in mailing/services/api.py, mailing/services/transactional.py, and mailing/tests/test_transactional_api.py (union of the client-callback and category/versions imports).

Re-verification on the merged tree:

• uv run --frozen pytest -q -> 645 passed in 408.77s
• uv run --frozen ruff check . -> all checks passed
• uv run --frozen python manage.py makemigrations --check --dry-run -> no changes detected
• migrate from empty on a fresh sqlite database -> applies clean through 0027