fix(infra): stop Terraform clobbering app runtime secret on every apply

[arielr-lt]

Problem

The app runtime secret (ctdl-xtra/<env>/app) is managed out-of-band in AWS Secrets Manager — by the Update app env vars workflow and by manual ops — but aws_secretsmanager_secret_version.app rewrote the entire JSON blob from local.app_secret_values on every terraform apply.

Because ENCRYPTION_KEY is sourced from random_password.encryption_key, any apply that recreated/refreshed that resource silently rotated the key. The app uses it for AES-256-CBC, so all DB rows encrypted under the previous key became undecryptable.

This is exactly what bit production: the live key had drifted to a random value and encrypted DB secrets could no longer be decrypted (restored manually by setting the correct ENCRYPTION_KEY back into Secrets Manager + rolling api/worker).

Fix

Add lifecycle { ignore_changes = [secret_string] } to aws_secretsmanager_secret_version.app in test, sandbox, and production. Terraform now seeds the secret on first creation only and never overwrites runtime values afterward — Secrets Manager + the Update app env vars workflow are the single source of truth.

Trade-off / follow-up

secret_string is now fully ignored, so infra-derived values (DATABASE_URL, REDIS_URL) are no longer auto-propagated by Terraform. If the RDS endpoint or db_password/redis_password change, the secret must be updated out-of-band (via the workflow) to match. This matches the existing operational model where that workflow already owns runtime env. Worth a short runbook note.

Verification

• terraform fmt clean on changed files.
• Reviewers should run terraform plan per env and confirm the only secret-related change is the in-place lifecycle update (no secret_string replacement).

🤖 Generated with Claude Code