Skip to content

api.md

Latest commit

 

History

History
1203 lines (957 loc) · 43.3 KB

api.md

File metadata and controls

1203 lines (957 loc) · 43.3 KB
Raw

API Design

Datamailer client apps use the native API to sync contacts, check subscription and verification state, import/export contacts, send transactional emails, and retrieve scoped contact history.

The in-app staff API docs at /api-docs/ are the primary runnable reference. They include copy-pasteable local examples, request/response bodies, common errors, and the endpoint reference generated alongside OpenAPI JSON.

Authentication

All client API endpoints under /api/... require Bearer authentication:

Authorization: Bearer <client-api-key>

Clients can have multiple named API keys for separate integrations. Datamailer stores only key hashes and displays each key's safe dm_<prefix> identifier for support and audit trails. Staff users create and revoke keys from the client detail page in the operator UI.

Local demo data creates stable named keys for examples:

  • dtc-courses / Course platform transactional: dm_dtccourses_demo_transactional_email_key
  • dtc-newsletter / Newsletter import/export: dm_dtcnews_demo_newsletter_import_export_key
  • asl-platform / ASL platform transactional: dm_aslplatform_demo_transactional_email_key

Sandbox deployment provisions the CMP scope explicitly:

python manage.py provision_client_scope \
  --organization datatalksclub \
  --organization-name DataTalksClub \
  --audience dtc-courses \
  --audience-name "DataTalksClub Courses" \
  --client dtc-courses \
  --client-name "DTC Courses"

python manage.py set_client_senders dtc-courses \
  --organization datatalksclub \
  --default-sender courses \
  --sender 'courses=DataTalks.Club Courses <courses@dtcdev.click>'

That keeps DATAMAILER_AUDIENCE=dtc-courses and DATAMAILER_CLIENT=dtc-courses valid for CMP contact sync, status lookups, history lookups, recipient lists, and transactional sends. The sender command keeps CMP payloads using from_email=courses while making delivered mail show DataTalks.Club Courses.

The same sender mapping is available through the client-scoped API:

curl -sS -X PUT "$DATAMAILER_URL/api/client/senders" \
  -H "Authorization: Bearer $DATAMAILER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "default_sender_id": "courses",
    "senders": [
      {
        "id": "courses",
        "email": "DataTalks.Club Courses <courses@dtcdev.click>"
      }
    ]
  }'

The in-app examples default to DATAMAILER_API_DOCS_BASE_URL, falling back to PUBLIC_BASE_URL. Override DATAMAILER_URL when running an example against a different environment:

export DATAMAILER_URL="${DATAMAILER_URL:-https://datamailer.example.com}"
export DATAMAILER_API_KEY="dm_dtccourses_demo_transactional_email_key"

Contact APIs

Upsert Contact

POST /api/contacts

{
  "email": "learner@example.com",
  "audience": "dtc-courses",
  "client": "dtc-courses",
  "status": "subscribed",
  "tags": ["course-ml-zoomcamp"],
  "verified": true,
  "email_validation": {
    "status": "externally_validated",
    "reason": "client signup validation"
  }
}

Creates or updates the global contact, creates or updates the audience/client subscription, and adds audience-scoped tags.

verified=true marks the audience/client subscription as verified for the authenticated client scope. Marketing, campaign, and recipient-list eligibility treat a contact as verified when the global contact, audience subscription, or client subscription has a verification timestamp.

Contact Status

GET /api/contacts/status?email=learner@example.com&audience=dtc-courses&client=dtc-courses

Returns contact existence, subscription, verification, validation, suppression, and sendability state for the authenticated client scope.

Verification, Validation, and Suppression

PATCH /api/contacts/{contact_id}/verification
PATCH /api/contacts/{contact_id}/validation
PATCH /api/contacts/{contact_id}/suppression

These endpoints update state for an existing contact visible to the authenticated client scope. Verification is typically called after the user verifies in the client app. Validation stores external hygiene decisions. Suppression records global unsubscribe, hard bounce, or complaint state.

Tags

PUT /api/contacts/{contact_id}/tags
POST /api/contacts/{contact_id}/tags/{tag_slug}
DELETE /api/contacts/{contact_id}/tags/{tag_slug}

Use PUT to replace the audience tag set. Use single-tag POST and DELETE for toggle-style client workflows.

Subscription APIs

POST /api/subscriptions/subscribe
POST /api/subscriptions/unsubscribe
POST /api/subscriptions/request-verification
POST /api/subscriptions/confirm

Canonical preference categories

Subscription preferences use one canonical category vocabulary. A canonical category name is stored as the CategoryPreference tag, so send-time suppression works through the same tag path as free-form category_tag values.

Category Opt-out Meaning
newsletter yes Newsletter and marketing digest sends.
events yes Event announcements, reminders, and recaps.
courses yes Course announcements and cohort communication.
product yes Product updates and release notes.
transactional never Account, security, and delivery-critical messages. Always on.

The transactional category is always on: a contact cannot opt out of it, and a transactional send tagged with category_tag: "transactional" is never suppressed by the category check.

Subscribe and unsubscribe with a category

Subscribe creates the contact if needed and marks the client-scoped subscription subscribed. The optional category field also enables the canonical category preference for the scoped contact:

{
  "email": "subscriber@example.com",
  "audience": "datatalks-club",
  "client": "dtc-newsletter",
  "category": "events"
}

Unsubscribe accepts scope values:

  • client: unsubscribe from one client.
  • audience: unsubscribe from the whole audience.
  • global: unsubscribe from all marketing email managed by Datamailer.

The optional category field also disables the canonical category preference for the scoped contact:

{
  "email": "subscriber@example.com",
  "audience": "datatalks-club",
  "client": "dtc-newsletter",
  "scope": "client",
  "category": "events",
  "reason": "user_requested"
}

Validation rules for category on both endpoints:

  • Unknown values are rejected with 400 and {"category": "unknown"}.
  • unsubscribe with category: "transactional" is rejected with 400 and {"category": "transactional_cannot_be_disabled"}.

Requests without category behave exactly as before: no preference row is created or changed.

Double opt-in

Double opt-in proves a recipient wants a category before it is enabled. The flow is stateless on the Datamailer side: the confirmation token is signed and expiring, carries only ids and the category name (never a raw email), and requires no database row.

  1. The client calls POST /api/subscriptions/request-verification:
{
  "email": "subscriber@example.com",
  "audience": "datatalks-club",
  "client": "dtc-newsletter",
  "category": "newsletter",
  "template_key": "newsletter-double-opt-in"
}

Datamailer validates the scope, rejects category: "transactional" with 400 and {"category": "transactional_not_allowed"}, and fails closed with 404 and {"template_key": "not_found"} when the template is missing, inactive, or owned by another client. It then enqueues one transactional message through that template. The message context carries confirm_url, verification_token, and category; confirm_url is built from the SUBSCRIPTION_CONFIRM_BASE_URL setting plus the token, so the link lands on the client site's public confirm page:

{SUBSCRIPTION_CONFIRM_BASE_URL}?token=<opaque signed token>

The response repeats the request scope without the token:

{
  "status": "verification_requested",
  "email": "subscriber@example.com",
  "audience": "datatalks-club",
  "client": "dtc-newsletter",
  "category": "newsletter",
  "template_key": "newsletter-double-opt-in"
}
  1. The recipient opens confirm_url on the client site. The site extracts the token query parameter and calls POST /api/subscriptions/confirm with its own API key:
{
  "token": "<opaque signed token from confirm_url>"
}

Datamailer validates the signature, expiry (48 hours), and that the token was issued for the authenticated client, then enables the category preference (creating it when needed) and returns the resulting preference state:

{
  "email": "subscriber@example.com",
  "audience": "datatalks-club",
  "client": "dtc-newsletter",
  "category": {
    "tag": "newsletter",
    "label": "Newsletter",
    "enabled": true
  }
}

Confirmation is idempotent: repeating it returns the same state. A malformed, tampered, expired, or foreign-client token is rejected with 400 and {"token": "invalid"}.

Import and Export APIs

POST /api/contacts/imports
POST /api/contacts/imports/csv
GET /api/contacts
GET /api/contacts.csv

JSON and CSV imports are idempotent by normalized email plus audience/client scope. Invalid items are returned in partial errors while valid items continue. CSV export returns safe recreatable contact, subscription, tag, verification, validation, suppression, unsubscribe, and update timestamp columns.

Recipient List APIs

Recipient lists are client-scoped audience nodes for later list sends. CMP writes path keys such as ml-zoomcamp-2026, ml-zoomcamp-2026:@e, and ml-zoomcamp-2026:@e:@homework:homework-1. Adding a member to a child path also adds cascade memberships to each parent path, including <all>. CMP writes the most specific node, and Datamailer keeps the ancestors current.

PUT /api/recipient-lists/{list_key}
GET /api/recipient-lists/{list_key}
PUT /api/recipient-lists/{list_key}/members/{source_object_key}
POST /api/recipient-lists/{list_key}/members/bulk-upsert
POST /api/recipient-lists/{list_key}/members/reconcile
POST /api/recipient-lists/{list_key}/transactional-send

Single member upsert creates the parent list when it does not exist, so CMP does not need a separate "does this batch exist?" call when the first learner submits:

{
  "audience": "dtc-courses",
  "client": "dtc-courses",
  "list": {
    "type": "homework_submitters",
    "name": "ML Zoomcamp 2026 Homework 1 submitters",
    "metadata": {
      "course": "ml-zoomcamp-2026",
      "homework": "homework-1"
    }
  },
  "member": {
    "email": "learner@example.com",
    "status": "active",
    "metadata": {
      "submission_id": 42
    }
  }
}

Bulk upsert accepts the same scope and list metadata plus a members array. It is intended for retroactive creation from CMP:

{
  "audience": "dtc-courses",
  "client": "dtc-courses",
  "list": {
    "type": "registrants",
    "name": "ML Zoomcamp 2026 registrants"
  },
  "members": [
    {
      "source_object_key": "registration:1",
      "email": "one@example.com",
      "metadata": {"user_id": 1}
    },
    {
      "source_object_key": "registration:2",
      "email": "two@example.com",
      "metadata": {"user_id": 2}
    }
  ]
}

Reconcile uses the provided member array as the current desired list. With remove_absent=true, existing members missing from the payload are marked removed instead of deleted. This gives CMP an idempotent backfill path for "everyone registered" and "everyone who submitted this homework/project" lists.

List transactional send creates one transactional message per active list member using a shared template and context. The caller must provide a base idempotency_key; Datamailer appends each member source_object_key so retrying the same list send does not duplicate per-member email.

The request can include list and members. When members is present, Datamailer syncs the recipient list before sending. The default member_sync mode is reconcile, which marks existing active members absent from the request as removed. Use member_sync=upsert to only add or update the provided members. Each member's metadata is merged into that recipient's template context and is also available under member.

{
  "audience": "dtc-courses",
  "client": "dtc-courses",
  "template_key": "homework-score-notification",
  "idempotency_key": "homework-score:ml-zoomcamp-2026:homework-1",
  "context": {
    "course_title": "ML Zoomcamp 2026",
    "homework_title": "Homework 1",
    "scores_url": "https://courses.example.com/courses/ml-zoomcamp-2026/"
  },
  "list": {
    "type": "homework_submitters",
    "name": "ML Zoomcamp 2026 Homework 1 submitters"
  },
  "members": [
    {
      "source_object_key": "homework-submission:123",
      "email": "learner@example.com",
      "status": "active",
      "metadata": {
        "submission_id": 123,
        "questions_score": 6,
        "learning_in_public_score": 2,
        "faq_score": 1,
        "total_score": 9
      }
    }
  ],
  "metadata": {
    "source": "course-management-platform",
    "event": "homework_score_publication"
  }
}

Transactional Email API

Transactional Template Upsert

PUT /api/transactional/templates/{template_key}
GET /api/transactional/templates/{template_key}

Templates are scoped to the authenticated client. Use PUT to create or update a transactional template, and GET to verify the current configuration. PUT replaces the whole editable draft: a field omitted from the payload is cleared.

{
  "name": "Homework Submission Confirmation",
  "description": "Confirm that the course platform saved a homework submission.",
  "subject": "Homework submission received: {{ homework_title }}",
  "html_body": "<p>Your homework submission for <strong>{{ homework_title }}</strong> in {{ course_title }} was saved.</p>",
  "text_body": "Your homework submission for {{ homework_title }} in {{ course_title }} was saved.",
  "markdown_body": "",
  "category": "homework",
  "required_context": [
    {"name": "course_title", "description": "Course title."},
    {"name": "homework_title", "description": "Homework title."}
  ],
  "example_context": {
    "course_title": "ML Zoomcamp",
    "homework_title": "Homework 1"
  },
  "is_active": true
}

Set markdown_body to store a markdown draft instead of raw html_body/text_body. Markdown drafts are rendered at send and preview time with the shared email shell (header, footer, and external links opened in a new tab); the text part carries the substituted markdown source. category is a free-form label (for example events or onboarding). The GET response also reports latest_version, the newest published version number, or null when the template was never published.

CMP transactional templates can be provisioned through the API with:

DATAMAILER_URL="https://datamailer.example.com" \
DATAMAILER_API_KEY="<client-api-key>" \
python scripts/upsert_cmp_templates.py

The script provisions these template keys:

homework-submission-confirmation
project-submission-confirmation
homework-score-notification
project-score-notification
certificate-availability-notification
deadline-reminder

Markdown Template Format

Markdown templates are markdown bodies with a YAML frontmatter block. The frontmatter keys map to template fields:

Frontmatter key Template field Required
subject subject yes
required_context required_context no
category category no
name name no, defaults to the file stem
example_context example_context no
body markdown_body yes 
---
subject: "You're registered: {{ event_title }}"
category: events
required_context:
  - name: user_name
    description: "Recipient name."
  - join_url
---

Hi {{ user_name }},

You're registered for **{{ event_title }}**.

Join link: {{ join_url }}

required_context entries are either names or name/description mappings. Sends validate that every required key is present and non-empty in the context; missing keys fail the send with one context.<name>: required error per key before any contact, message, or queue work happens.

Import a directory of these files as drafts with:

uv run python manage.py import_templates --dir <templates-directory> --client <client-slug>

The command creates one draft per *.md file, keyed by file stem. Re-running it updates the existing drafts. Files without a frontmatter subject are reported as errors and the command exits non-zero.

Publish a Template Version

POST /api/transactional/templates/{template_key}/publish

Publishing snapshots the current draft into a new immutable version numbered 1, 2, ... Response 201:

{
  "template_key": "event-registration",
  "latest_version": 2,
  "version": {
    "version": 2,
    "subject": "You're registered: {{ event_title }}",
    "html_body": "",
    "text_body": "",
    "markdown_body": "Hi {{ user_name }} ...",
    "required_context": [
      {"name": "user_name", "description": "Recipient name."},
      {"name": "join_url", "description": "Join link."}
    ],
    "category": "events",
    "created_at": "2026-09-08T12:00:00Z"
  }
}

Published versions never change: editing the draft does not touch existing versions, and version rows cannot be updated or deleted.

List Template Versions

GET /api/transactional/templates/{template_key}/versions

{
  "template_key": "event-registration",
  "latest_version": 2,
  "versions": [
    {"version": 2, "subject": "...", "...": "..."},
    {"version": 1, "subject": "...", "...": "..."}
  ]
}

Versions are listed newest first. A template that was never published returns "latest_version": null and an empty list.

Preview a Template

POST /api/transactional/templates/{template_key}/preview

{
  "context": {"user_name": "Ada", "event_title": "Community Lunch", "join_url": "https://example.com/join"},
  "template_version": 1
}

template_version is optional; omit it to preview the current draft, or name an explicit version. The preview writes nothing and returns the rendered message plus which required context keys would have produced values:

{
  "template_key": "event-registration",
  "template_version": 1,
  "published": true,
  "subject": "You're registered: Community Lunch",
  "html_body": "<!DOCTYPE html> ... </html>",
  "text_body": "Hi Ada, ...",
  "missing_context": []
}

Preview rendering matches sends exactly: for the same version and context, subject, html_body, and text_body equal what a send stores. published is false when the preview rendered the draft of a never-published template. Missing required context keys do not fail a preview; they are listed in missing_context (the affected placeholders render empty).

Test Send a Template

POST /api/transactional/templates/{template_key}/test-send

{
  "email": "staff@example.com",
  "template_version": 1,
  "context": {"user_name": "Ada", "event_title": "Community Lunch", "join_url": "https://example.com/join"}
}

Sends one message through the normal transactional pipeline to the given address, with "test_send": true in the message metadata. The recipient must belong to the TRANSACTIONAL_TEST_SEND_ALLOWLIST deployment setting; an empty allowlist disables the endpoint entirely (403 test_send: allowlist_not_configured), and addresses outside the allowlist are rejected with 403 email: not_in_test_send_allowlist. Required context is validated like a normal send.

Send Transactional Email

POST /api/transactional/send

{
  "email": "learner@example.com",
  "template_key": "registration-welcome",
  "template_version": 2,
  "idempotency_key": "registration-user-123",
  "context": {
    "name": "Learner",
    "course_name": "ML Zoomcamp"
  },
  "metadata": {
    "source": "registration"
  }
}

template_version is optional. With it, the send renders exactly that published version; without it, the send renders the latest published version, or the draft for templates that were never published. An unknown version fails with 404 template_version: not_found. The created message records the version it rendered, and the queue payload carries the same template_version.

Transactional sends validate required template context before Datamailer creates a contact, message, event, or queue payload. Reusing an idempotency key returns the existing message.

Local transactional send examples require the Datamailer server to be started with SQS_TRANSACTIONAL_EMAIL_QUEUE_URL configured, for example through LocalStack. With the default empty queue URL, the endpoint is documented but is not runnable because queueing provider work will fail.

Transactional sends do not require marketing subscription, but they are blocked for hard bounces and complaints.

A send with a category_tag is also suppressed when the contact opted out of that category (the canonical categories from the Subscription APIs section use the same tag path). The send is rejected with 409, the message is stored as skipped, and the error payload names the reason and confirms the suppression:

{
  "error": {
    "code": "transactional_suppressed",
    "message": "Contact is hard-suppressed for transactional email.",
    "reason": "category_unsubscribe",
    "suppressed": true
  }
}

A send with category_tag: "transactional" is never suppressed by the category check.

Transactional Message Status

GET /api/transactional/messages/{message_id}

Returns the current status for a transactional message created by the authenticated client, plus its event timeline. Use the message.id returned by POST /api/transactional/send.

Task API

Relay runs background work on behalf of clients. A client submits a task, gets a task id, and reads status from the same service. See the README for a copy-pasteable system.echo example.

POST /api/tasks
GET  /api/tasks
GET  /api/tasks/{task_id}
POST /api/tasks/{task_id}/complete
POST /api/tasks/{task_id}/fail

Every submission needs an idempotency_key. Repeating the same request returns the original task; reusing the key for different work returns 409 Conflict.

Webhook tasks

type: "webhook" sends a signed HTTPS request to a registered client origin. The origin must be registered on the client (with the webhook signing secret) before submission; every other origin is rejected.

Relay adds these headers to each request:

Header Meaning
X-Relay-Task-Id task id, so the receiver can deduplicate retries
X-Relay-Correlation-Id stable id shared by all attempts of one logical task
X-Relay-Timestamp unix timestamp of the attempt, used in the signature
X-Relay-Attempt 1-based attempt counter (1 on the first delivery)
X-Relay-Signature sha256= + HMAC-SHA256 over <timestamp>.<raw-json-body> with the client's webhook secret

The signature scheme and body layout are identical on every attempt; only the timestamp, signature, and attempt header change.

Timeout ceiling and the 202 lease protocol

A synchronous webhook may run for at most 60 seconds (timeout_seconds, default 30). This ceiling is decided: work that cannot finish in 60 seconds does not belong in the synchronous path.

For longer work the receiver can acknowledge instead of finishing. When it responds 202 with {"lease_seconds": N}, Relay marks the task running under a lease that expires after N seconds (cap: 3600). The receiver then resolves the task with a client-authenticated callback:

POST /api/tasks/{task_id}/complete    {"result": {...}}   # optional body
POST /api/tasks/{task_id}/fail       {"error": "...", "retryable": false}
  • complete marks the task succeeded and stores the optional result.
  • fail marks the task failed with the optional error. By default it does not retry: the receiver reported on work it already owns. Passing "retryable": true asks Relay to redeliver the task: while attempts remain it becomes retrying with the usual backoff, and once attempts are exhausted it fails.
  • Repeating the same callback is a harmless idempotent 200; the opposite resolution on a finished task is 409 task_already_finished; a callback for a task with no lease in progress is 409 no_lease_in_progress.
  • If the lease expires without a callback, Relay fails the task with lease expired without a completion callback. An expired lease never re-executes the request: the receiver already took the work, and re-running it could repeat a side effect. Callbacks that arrive after expiry are rejected with 409 lease_expired.

Retry table

Retries use exponential backoff (base 5 seconds: 5s, 10s, 20s, ...) up to the per-task max_attempts (default 5, range 1-10).

Outcome Class
Connection error or response timeout retry with backoff
HTTP 429 retry with backoff
HTTP 5xx retry with backoff
fail callback with "retryable": true, attempts left retry with backoff
Any other 4xx (including 408, 425) fail immediately
202 without a valid lease_seconds fail immediately
Lease expiry without a callback fail, no retry

On every failure Relay stores the HTTP response status and the response body, truncated to 2 KB, on the task. A webhook task that fails with no detail is undebuggable.

Per-client limits on webhook tasks

Limit Default Effect
Concurrent in-flight tasks 4 a submission that would exceed it is rejected with 429 concurrency_limit_exceeded
Task creation rate 60 per rolling minute a submission beyond it is rejected with 429 rate_limit_exceeded

Tasks created by Relay schedules do not consume these limits; schedules are operator-configured and bounded by their cron expression. Replays of an existing idempotency key do not consume them either — they return the original task instead of creating a new one.

Tasks that end in the failed state after their attempts (or a client fail callback, or lease expiry) appear in the staff dead-letter list at /jobs/dead-letters/ with a retry button.

Reference receiver

A receiver verifies origin and freshness with the shared webhook secret. The window is part of the receiver's contract, not Relay's; 5 minutes is a workable default. A captured request replayed later fails the timestamp check.

import hashlib
import hmac
import time

REPLAY_WINDOW_SECONDS = 300


class Reject(Exception):
    pass


def verify_relay_webhook(body: bytes, headers: dict, secret: str, *, now=None) -> None:
    """Verify X-Relay-* headers on a webhook task delivery. Raises Reject."""
    now = now or time.time()
    timestamp = headers.get("X-Relay-Timestamp", "")
    signature = headers.get("X-Relay-Signature", "")
    try:
        stamp = int(timestamp)
    except ValueError:
        raise Reject("missing or malformed timestamp") from None
    if abs(now - stamp) > REPLAY_WINDOW_SECONDS:
        raise Reject("timestamp outside replay window")
    expected = hmac.new(
        secret.encode(), timestamp.encode() + b"." + body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest("sha256=" + expected, signature):
        raise Reject("bad signature")

Relay's own test suite replays a correctly signed request with an old timestamp against this exact function and asserts the rejection.

Contact History

GET /api/contacts/{contact_id}/history?audience=dtc-courses&client=dtc-courses&limit=25

Returns safe scoped campaign recipient, transactional message, and event history. Secret hashes and delivery link tokens are never returned.

Transactional Dry-Run (test-only)

POST /api/transactional/send      { ..., "dry_run": true }

For end-to-end tests, the real send endpoint accepts a "dry_run": true flag. It is the same endpoint, path, auth, request body, and validation as a normal send -- a test hits /api/transactional/send exactly as production does, adding only this one flag -- so the whole production send path is exercised. With dry_run set, Datamailer runs the identical validate/render pipeline but stops before delivery: it sends nothing, queues nothing, and persists nothing, so there are no message rows to poll and no teardown to run.

The response is a superset of the real send response. Alongside the usual message, idempotent_replay, and enqueued fields it adds the rendered subject/html_body/text_body plus a would_deliver flag and a delivery_decision explaining whether the real send would have been allowed. The returned message.id and message.created_at are always null because nothing is stored, and enqueued is always false.

{
  "message": {
    "id": null,
    "email": "learner@example.com",
    "from_email": "courses",
    "from_email_address": "DataTalks.Club Courses <courses@dtcdev.click>",
    "reply_to": "",
    "cc": [],
    "bcc": [],
    "status": "queued",
    "template_key": "registration-welcome",
    "idempotency_key": "registration-user-123",
    "created_at": null
  },
  "idempotent_replay": false,
  "enqueued": false,
  "rendered": {
    "subject": "Welcome to ML Zoomcamp",
    "html_body": "<p>Thanks ...</p>",
    "text_body": "Thanks ..."
  },
  "would_deliver": true,
  "delivery_decision": {"allowed": true, "reason": ""}
}

status is queued when the real send would deliver and skipped when it would be suppressed; in the suppressed case would_deliver is false and delivery_decision.reason names the reason.

Public and Provider Routes

Public tracking, unsubscribe, and provider webhook routes are documented in OpenAPI and the in-app endpoint reference for integration context:

GET /t/o/{tracking_token}.gif
GET /t/c/{tracking_token}
GET /unsubscribe/{unsubscribe_token}
POST /unsubscribe/{unsubscribe_token}
POST /webhooks/ses

These are not client Bearer API routes.

CMP Contact Event Callbacks

When configured, Datamailer queues hard-bounce, complaint, public unsubscribe, resubscribe, and transactional skipped/failed events for CMP after the local Datamailer transaction commits. A background dispatcher posts queued callbacks and retries failures with backoff.

CMP_WEBHOOK_URL=https://courses.example.com/api/datamailer/events
CMP_WEBHOOK_TOKEN=shared-secret

These global settings are the fallback. A Datamailer client can also configure its own CMP webhook URL and token from the operator client form.

Datamailer sends:

Authorization: Bearer <CMP_WEBHOOK_TOKEN>

Implemented callback event types:

contact.hard_bounced
contact.complained
subscription.unsubscribed
subscription.resubscribed
transactional.skipped
transactional.failed

Callbacks are stored in the cmp_callbacks outbox. Run the dispatcher with:

python manage.py process_cmp_callbacks --batch-size 25

Sandbox deploys install this dispatcher as datamailer-cmp-callbacks-worker. Operators can inspect recent callback status, attempt counts, next retry time, delivery time, and the last error from the client detail page and Django admin.

Client Callbacks

When a callback endpoint is configured for a client, Relay announces delivery and engagement transitions for that client with signed, versioned, redacted JSON events. This channel is additive to the CMP callback contract above. It carries transactional deliveries and client-level subscription changes for unification-plan clients; campaign-recipient transitions stay on the CMP channel. Payloads carry identifiers and safe reason codes only, so a consumer can log and store them as-is.

Configuration

One endpoint per client, managed by operators on the client record:

Field Meaning
URL The one HTTPS endpoint Relay posts to. Per-message URLs and redirects to other origins are not permitted.
Signing secret Dedicated HMAC signing secret, separate from every client API key. Write-only: it is shown once at provisioning and never returned afterwards.
Previous signing secret Retained during a rotation for a bounded verification overlap; the receiver accepts either secret until the overlap ends.
Contract version Integer version of the event payload, currently 1.
Enabled Disablement stops all deliveries immediately; in-flight pending callbacks fail terminally with endpoint_disabled.

Real secret provisioning is infrastructure work and uses approved server-side secret handling; secrets never appear in URLs, logs, errors, or docs examples.

Event types and reason codes

Relay posts these event types after the corresponding transport state commits:

Event type Fired on Reason codes
delivery.accepted message queued, then handed to the provider none
delivery.delivered provider confirmed delivery none
delivery.bounced provider reported a bounce hard_bounce, soft_bounce
delivery.complained recipient filed a complaint complaint
delivery.suppressed send was suppressed before delivery unverified, invalid_email, global_unsubscribe, client_unsubscribe, audience_unsubscribe, hard_bounce, complaint, duplicate, category_unsubscribe, missing_category_scope, suppressed
engagement.opened tracked open none
engagement.clicked tracked click none
subscription.changed subscribe or unsubscribe state changed subscribed, unsubscribed

reason_code is omitted when the table shows "none". A permanently failed send is not announced on this channel; the synchronous send response and the reconciliation endpoint below carry that outcome.

Payload

The body is the canonical JSON serialization of the event (sorted keys, no whitespace), signed exactly as sent on every attempt:

{
  "bounce_type": "hard",
  "client_reference": "registration-user-123",
  "contract_version": 1,
  "event_id": "dd8b8f17-6d55-5094-ae45-83cd8ac37b96",
  "event_type": "delivery.bounced",
  "message_id": "1042",
  "reason_code": "hard_bounce",
  "sequence": 3,
  "template_key": "registration-welcome",
  "timestamp": "2026-09-08T12:00:00+00:00"
}
Field Meaning
contract_version Payload contract version. Additive changes do not bump it; removing or repurposing a field does.
event_id Stable UUID for one transition. Duplicate deliveries of the same event reuse the id, so receivers can deduplicate on it.
event_type One of the types above.
timestamp When the transition committed (ISO 8601).
sequence 1-based counter per message. Ordering of deliveries is best-effort (retries reorder); use sequence to reorder if you need strict ordering.
message_id Relay transactional message id as a string; null for contact-level transitions such as subscription changes.
client_reference Your idempotency key from the original send; null when the transition has no transactional message.
template_key Template key for transactional messages, empty otherwise.
bounce_type hard or soft; present only on delivery.bounced.
reason_code Safe reason code from the table above, when applicable.

The payload contains nothing else. No recipient address, subject, body, template context, custom headers, credentials, raw provider payload, or arbitrary metadata is ever included.

Signature headers

Every request carries:

Header Meaning
X-Relay-Timestamp Unix timestamp of the attempt
X-Relay-Signature sha256= + HMAC-SHA256 over <timestamp>.<raw-json-body> with the callback signing secret
X-Relay-Event-Id the event_id, for quick routing and dedup
X-Relay-Event-Type the event_type
X-Relay-Contract-Version the contract_version
X-Relay-Attempt 1-based attempt counter

There is no Authorization header and no Bearer credential. The signature scheme is identical to webhook task deliveries.

A deterministic verification fixture lives at tests/fixtures/client_callback_contract_v1.json: it contains a canonical body, a signing secret, a timestamp, and the expected signature, plus a rotated-secret signature. Consumer contract tests should verify the fixture instead of inventing their own vectors.

Reference receiver

The receiver verifies origin and freshness with the callback signing secret. The replay window is part of the receiver's contract; 5 minutes is a workable default. During a rotation, verify against the current secret and then the previous one.

import hashlib
import hmac
import time

REPLAY_WINDOW_SECONDS = 300


class Reject(Exception):
    pass


def verify_relay_callback(body: bytes, headers: dict, secrets: list[str], *, now=None) -> None:
    """Verify X-Relay-* headers on a client callback delivery. Raises Reject."""
    now = now or time.time()
    timestamp = headers.get("X-Relay-Timestamp", "")
    signature = headers.get("X-Relay-Signature", "")
    try:
        stamp = int(timestamp)
    except ValueError:
        raise Reject("missing or malformed timestamp") from None
    if abs(now - stamp) > REPLAY_WINDOW_SECONDS:
        raise Reject("timestamp outside replay window")
    expected = [
        "sha256="
        + hmac.new(secret.encode(), timestamp.encode() + b"." + body, hashlib.sha256).hexdigest()
        for secret in secrets
    ]
    if not any(hmac.compare_digest(candidate, signature) for candidate in expected):
        raise Reject("bad signature")

Delivery, retries, and deduplication

Callback rows are created in the same transaction as the transport state they announce and dispatched only after that transaction commits, so a committed transition is never silently missing its callback. A duplicate transition never creates a second row: work is deduplicated on client plus event_id.

Delivery responds to a receiver that answers 2xx. Duplicate successful acknowledgements are treated as success. Relay retries:

Outcome Class
Connection error or timeout retry with bounded exponential backoff
HTTP 429 retry with bounded exponential backoff
HTTP 5xx retry with bounded exponential backoff
Any other 4xx fail terminally
Redirect (3xx) fail terminally; redirects are never followed
Endpoint disabled fail terminally as endpoint_disabled
Retry exhaustion fail terminally

Backoff starts at 60 seconds and doubles per attempt, capped at 6 hours, with a small deterministic jitter. Relay records the attempt count, the response status class (2xx/3xx/4xx/5xx), a safe error code, and the next attempt time. Response bodies and headers are never stored.

Callback failure never regresses Relay transport state: a delivered message stays delivered whether or not the callback succeeded. If callbacks are delayed or lost, reconcile against GET /api/transactional/messages?since= (described next).

Run the dispatcher with:

python manage.py process_client_callbacks --batch-size 25

Sandbox deploys install this dispatcher as datamailer-client-callbacks-worker. Operators can inspect recent callback status, attempt counts, next retry time, delivery time, and the last safe error from the client detail page and Django admin.

Transactional Message Reconciliation

GET /api/transactional/messages?since=<iso8601> returns every transactional message of the authenticated client whose updated_at is at or after since, oldest first, capped at 1000 rows. It is the pull-based complement to client callbacks and the authoritative view for recovery after missed or failed callback deliveries.

Authentication and errors are the same as the rest of the client API: Bearer client API key; a missing or unparseable since returns validation_error.

GET /api/transactional/messages?since=2026-09-08T00:00:00%2B00:00

Response:

{
  "messages": [
    {
      "id": "1042",
      "client_reference": "registration-user-123",
      "status": "bounced",
      "template_key": "registration-welcome",
      "template_version": 1,
      "reason_code": "hard_bounce",
      "updated_at": "2026-09-08T12:00:00+00:00"
    }
  ]
}
Field Meaning
id Transactional message id as a string.
client_reference The idempotency key from the original send.
status queued, retrying (in flight to the provider), sent (accepted by the provider), delivered, suppressed, failed, bounced (hard bounce), or complained.
template_key Template key the message was sent with.
template_version Published template version the message was sent with; 1 for messages sent before versions were recorded.
reason_code Safe reason code, empty when there is nothing to report; provider diagnostics are never included.
updated_at ISO 8601; use it as the next poll's since.

Messages without an idempotency key are never reported.

Mailchimp Sync

Datamailer can push contacts into a client's Mailchimp audience with a tag when they join a recipient-list tree node. This is a one-way sync (Datamailer → Mailchimp); Datamailer never reads Mailchimp state back.

Each client configures its own Mailchimp credentials, so different clients sync into different Mailchimp accounts. Configuration is write-only: the key can be set but is never returned.

Configure credentials (set-only)

PUT /api/client/mailchimp

{
  "api_key": "abc123def456xxxxxxxxxxxxxxxxxxxx-us21",
  "list_id": "1a2b3c4d5e",
  "enabled": true
}
  • api_key — Mailchimp API key including the -<datacenter> suffix (e.g. -us21). The datacenter is derived from the key; there is no separate field.
  • list_id — the Mailchimp audience (list) ID to sync tagged contacts into.
  • enabled — turn sync on/off without resending the key.

Only the fields present in the body are updated. The response is a non-secret status; the stored key is never echoed back:

{
  "client": "dtc-courses",
  "enabled": true,
  "configured": true,
  "list_id": "1a2b3c4d5e",
  "datacenter": "us21",
  "api_key_set": true
}

There is no GET for this endpoint. Operators can also set the key, audience ID, and enabled flag on the client edit form in the product UI.

Map audience subtrees to tags (set-only)

Each recipient-list tree node can carry a Mailchimp tag. When a contact becomes an active member of that node — including every ancestor node the cascade creates, up to the audience root <all> — the mapped tag is applied in Mailchimp. So registering for ai-dev-tools-zoomcamp-2026:@registered applies that node's tag, and a mapping on <all> applies an audience-wide tag to anyone added to any list.

PUT /api/client/mailchimp/tag-mappings

{
  "audience": "dtc-courses",
  "mappings": [
    {"list_key": "ai-dev-tools-zoomcamp-2026:@registered", "tag": "ai-dev-tools-zoomcamp-2026"},
    {"list_key": "<all>", "tag": "dtc-courses-audience", "enabled": true}
  ]
}

The request is a full reconcile for that audience: mappings not present in the payload are removed. The response returns the resulting mapping set. The same mappings can be edited per audience from the client detail page.

Delivery

Syncs are queued in the mailchimp_syncs outbox after the triggering transaction commits, then dispatched with exponential backoff (Mailchimp PUT /lists/{id}/members/{hash} upsert followed by POST .../members/{hash}/tags). 4xx responses other than 429 are treated as permanent failures; 429, 5xx, and transport errors retry.

python manage.py process_mailchimp_syncs --batch-size 25

Operators can inspect recent sync status, attempt counts, next retry time, and the last error from the client detail page and Django admin.