D1.2c Retire the datamailer client, the data app and the studio datamailer views

[alexeygrigorev]

Split out of the combined D1.2 mail-adoption issue (community-base plan phase 1). Depends on #370 (D1.2b, landed).

Scope:

• Delete course_management/datamailer* (the client package, its payloads, sync modules, templates registry), email_app/ (donor PendingUnsubscribe, replay jobs, relay links), the data app (DatamailerOutboxEvent, DatamailerSendAudit, dispatch runs), and studio_courses/views/datamailer*.py with their templates. Studio email pages come from the package.
• P6 second PR: delete the donor models kept read-only since D1.2a/D1.2b; migrations drop their tables. Migration rehearsal with counts (P14) before landing; development-copy counts recorded in the PR.
• Link bridge URLs keep their exact paths; RELAY_LINK_BRIDGE_* settings become COMMUNITY_BASE keys declared by the mail app.
• Retire the retained datamailer outbox storage from D1.2b (P6 step 4): storage goes read-only-to-dropped with the rollback window closed by the D1.2a/D1.2b dev deploys.

Non-goals:

• No Relay-side changes; the R1.x catalog/callback contracts stay as pinned.
• No new mail purposes; the five D1.2b purposes are unchanged.

Verification:

• grep -rln datamailer --include=*.py . | grep -v migrations -> nothing.
• Open pixel and click bridge respond as before (existing contract tests).
• Fresh-database and development-copy migrations; table-drop rehearsal with counts.
• Full gate set: ruff check/format, makemigrations --check, mypy, Django suites per the change-selection plan.

Not run here: production freeze activities (D1.3, separate issue).

[alexeygrigorev]

Status after the Sep 9 evening / Sep 10 early-AM repair session (independent session, dtc lane).

Restored:

• CI is GREEN on main for the first time since Sep 9 ~18:00 (run 34424685883, commit b5b8ca9). All jobs: django, quality, container, screenshots, playwright, ci-gate.
• Two fix waves landed on main to get there (178265a, 7ccee14 via merges f2293ff/10bc38ab): canonical catalogue route expectations, tour-stripe band in the entrance contract, donor generation column on unsubscribe-copy rollback, entry-point/exclusion registry reconciliation, adoption inventory pin (learner_duplicate_preflight), workflow-isolation seal bump for 7a1b3f0's typecheck opt-ins, login-as policy test modernization, tour-card interactive primitive, and the playwright bound 60->90 minutes for CI and the scheduled regression (both Sep 9 full-profile runs died at the 60-minute cap with every completed test still passing).

Current blocker - dev deploy, migration task:

• Deploy Dev for b5b8ca9 (run 34423083099) reached the deploy job for the first time in this window (earlier deploys were skipped at verify-ci) and failed at prepare_deployment: "Migration task exited 1; services were not changed". The recovery receipt confirms no service was mutated; dev still serves the prior release.
• The actual traceback is only in CloudWatch (awslogs), stream for the website-dev-migration task of version 20260910-010419-b5b8ca9. Someone with AWS access needs to read it; agents hold no AWS credentials.
• Static audit of the migrations that are new on main since deploys last ran (courses 0006 compound content IDs, accounts 0005 / courses 0007 CMP binding models, email_app 0002 generation + 0003 merge, D1.2a copy migration) found no obvious data hazard: the compound-ID validator accepts the UUID strings the copy step writes, the copy migration guards on both tables existing, and the AddField migrations are data-independent. The failure therefore needs the real traceback to classify.
• Note: this failure predates tonight's fixes (the earlier deploys never got past verify-ci, so the pending migration chain has been accumulating on dev since the last green deploy).

[alexeygrigorev]

Status after the Sep 9 evening / Sep 10 early-AM repair session (independent session, dtc lane).

Restored:

• CI is GREEN on main for the first time since Sep 9 ~18:00 (run 34424685883, commit b5b8ca9). All jobs: django, quality, container, screenshots, playwright, ci-gate.
• Two fix waves landed on main to get there (178265a, 7ccee14 via merges f2293ff/10bc38ab): canonical catalogue route expectations, tour-stripe band in the entrance contract, donor generation column on unsubscribe-copy rollback, entry-point/exclusion registry reconciliation, adoption inventory pin (learner_duplicate_preflight), workflow-isolation seal bump for 7a1b3f0's typecheck opt-ins, login-as policy test modernization, tour-card interactive primitive, and the playwright bound 60->90 minutes for CI and the scheduled regression (both Sep 9 full-profile runs died at the 60-minute cap with every completed test still passing).

Current blocker - dev deploy, migration task:

• Deploy Dev for b5b8ca9 (run 34423083099) reached the deploy job for the first time in this window (earlier deploys were skipped at verify-ci) and failed at prepare_deployment: "Migration task exited 1; services were not changed". The recovery receipt confirms no service was mutated; dev still serves the prior release.
• The actual traceback is only in CloudWatch (awslogs), stream for the website-dev-migration task of version 20260910-010419-b5b8ca9. Someone with AWS access needs to read it; agents hold no AWS credentials.
• Static audit of the migrations that are new on main since deploys last ran (courses 0006 compound content IDs, accounts 0005 / courses 0007 CMP binding models, email_app 0002 generation + 0003 merge, D1.2a copy migration) found no obvious data hazard: the compound-ID validator accepts the UUID strings the copy step writes, the copy migration guards on both tables existing, and the AddField migrations are data-independent. The failure therefore needs the real traceback to classify.
• Note: this failure predates tonight's fixes (the earlier deploys never got past verify-ci, so the pending migration chain has been accumulating on dev since the last green deploy).

[alexeygrigorev]

Migration blocker resolved — deploy now reaches ECS and fails only on an IAM grant someone with infra access must apply (DataTalksClub/aws-infra#49).

What changed since the last update:

• The migration-graph fix landed on main (b6f39e3 + d16f7de): events.0001 no longer depends on cb_jobs, and events.0006 converges provisioning_job to the package FK keyed off database state (fresh databases rebuild the column via portable schema-editor ops; deployed databases swap the legacy jobs_durablejob key for the package key). CI is green (run 34430085112) including the database-portability gate, which rejected the first cut's connection.vendor branch.
• Both shapes were proven against PostgreSQL replicas built locally: a database migrated at the last green dev deploy (4e7d218) with seeded donor rows passes the entire pending chain, and a fresh database migrates from zero; both end with the package FK plus the OneToOne unique index.
• The d16f7de deploy got further than any run in this window: verify-ci and publish green, the migration task ran to completion — the dev database now carries the full pending chain (compound content IDs, CMP bindings, email_app generation + copy, D1.2a copy) — and both services were promoted. The new failure is the worker self-check one-off task: the deployer role's ecs:RunTask grant covers the migration task definition only (main/dev/deployment_iam.tf), so the self-check on website-dev-worker:35 is denied; bounded recovery restored both services and dev is serving the prior release again.
• Residual note for the window until the next green deploy: dev runs the old image against the migrated schema. Additive migrations are backward-compatible by design, and durable-job ids were preserved by the D1.1 copy, but the old image's new durable jobs land only in jobs_durablejob — a QnA provisioning write referencing such a job would hit the new package FK. Dev-only, bounded by the next deploy.

The infra ask is DataTalksClub/aws-infra#49 (widen the RunTask resource list to the three task-definition families). Once applied, re-running the Deploy Dev workflow should go green end to end; D1.2ca (branch issue-372) is merged up to d16f7de with all gates green and lands immediately after.

[alexeygrigorev]

Migration blocker resolved — deploy now reaches ECS and fails only on an IAM grant someone with infra access must apply (DataTalksClub/aws-infra#49).

What changed since the last update:

• The migration-graph fix landed on main (b6f39e3 + d16f7de): events.0001 no longer depends on cb_jobs, and events.0006 converges provisioning_job to the package FK keyed off database state (fresh databases rebuild the column via portable schema-editor ops; deployed databases swap the legacy jobs_durablejob key for the package key). CI is green (run 34430085112) including the database-portability gate, which rejected the first cut's connection.vendor branch.
• Both shapes were proven against PostgreSQL replicas built locally: a database migrated at the last green dev deploy (4e7d218) with seeded donor rows passes the entire pending chain, and a fresh database migrates from zero; both end with the package FK plus the OneToOne unique index.
• The d16f7de deploy got further than any run in this window: verify-ci and publish green, the migration task ran to completion — the dev database now carries the full pending chain (compound content IDs, CMP bindings, email_app generation + copy, D1.2a copy) — and both services were promoted. The new failure is the worker self-check one-off task: the deployer role's ecs:RunTask grant covers the migration task definition only (main/dev/deployment_iam.tf), so the self-check on website-dev-worker:35 is denied; bounded recovery restored both services and dev is serving the prior release again.
• Residual note for the window until the next green deploy: dev runs the old image against the migrated schema. Additive migrations are backward-compatible by design, and durable-job ids were preserved by the D1.1 copy, but the old image's new durable jobs land only in jobs_durablejob — a QnA provisioning write referencing such a job would hit the new package FK. Dev-only, bounded by the next deploy.

The infra ask is DataTalksClub/aws-infra#49 (widen the RunTask resource list to the three task-definition families). Once applied, re-running the Deploy Dev workflow should go green end to end; D1.2ca (branch issue-372) is merged up to d16f7de with all gates green and lands immediately after.