feat(ci): update-app-env workflow + Secrets Manager IAM#268
Open
arielr-lt wants to merge 1 commit into
Open
Conversation
Adds a workflow_dispatch job that mutates the AWS Secrets Manager
secret feeding the ctdl-xtra-app-env k8s Secret (via External Secrets
Operator), then rolls the api and worker Deployments so they pick up
the new values. Sandbox/prod are gated through GitHub Environments,
matching promote-* workflows.
Why
- ESO already syncs ctdl-xtra/{env}/app from Secrets Manager into the
ctdl-xtra-app-env k8s Secret (5m refresh). The api/worker Deployments
consume it via envFrom. Today there is no audited path to add/change/
remove keys other than the AWS console; result is no run log, no
Slack notice, and no automatic pod refresh.
- The workflow closes that gap with a controlled, scriptable surface
that never echoes values to logs and produces a key-level diff in
the run summary and Slack.
How
- Inputs: environment, changes (JSON {"KEY":"value" | null}), restart,
dry_run. null deletes the key; non-string values are rejected; keys
must match [A-Za-z_][A-Za-z0-9_]*.
- All values (both incoming and existing) are passed through
::add-mask:: before any further step runs.
- jq merge yields the next secret JSON, written to a tmp file; diff is
emitted as added/changed/removed key lists only (never values).
- aws secretsmanager put-secret-value creates a new version; previous
versions remain available for one-step revert via the AWS console.
- ESO is force-refreshed with a force-sync annotation on the
ExternalSecret; we poll the target Secret's resourceVersion (up to
60s) to confirm it actually picked up the new version before any
rollout. Avoids the silent "values still old, pods bounced for
nothing" failure mode.
- Rollout: kubectl rollout restart deploy/ctdl-xtra-api and
deploy/ctdl-xtra-worker, then rollout status with a 5m timeout each.
- Slack always notifies (success or failure) with the key-level diff,
new VersionId, and outcome (applied / applied+restarted / dry-run /
no-op).
IAM
- New aws_iam_policy.github_actions_secretsmanager grants
Describe/Get/Put/ListSecretVersionIds scoped to
ctdl-xtra/{test,sandbox,prod}/app-* only. Attached to the existing
ctdl-xtra-github-actions-ci role used by every other workflow.
- Must be applied with `terraform apply` in
infra/terraform/github-ci-oidc/ before the workflow can succeed.
Test plan
- Dispatch on test with dry_run=true and changes={"LOG_LEVEL":"debug"}
-> step summary shows changed: ["LOG_LEVEL"], no put-secret-value.
- Dispatch on test for real with the same input -> new VersionId,
ctdl-xtra-app-env resourceVersion bumps, api/worker rollout restart
+ rollout status OK, pods come up with new LOG_LEVEL.
- Dispatch on test with changes={"LOG_LEVEL":null} -> key removed.
- Dispatch on test with changes={"LOG_LEVEL":"info"} and restart=false
-> secret updated, pods not bounced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
workflow_dispatchworkflow that updates the AWS Secrets Manager secret feeding thectdl-xtra-app-envk8s Secret (via External Secrets Operator), then rollsctdl-xtra-apiandctdl-xtra-workerso they pick up the new values. Sandbox and prod runs are gated through GitHub Environments (same pattern aspromote-*).Why
Today
ctdl-xtra/{env}/appin Secrets Manager can only be changed from the AWS console. No run log, no Slack notice, no automatic pod refresh — and the 5-minute ESO refresh window means people sometimes "redeploy" just to pick up env changes. This closes that gap with an audited, scriptable path.What the workflow does
Inputs:
environment— test / sandbox / prod.changes— JSON map, e.g.{"LOG_LEVEL":"debug","OLD_FLAG":null}.nullremoves the key.restart— bool, default true.dry_run— bool, default false.Pipeline:
changesis a{string: string|null}object and keys match[A-Za-z_][A-Za-z0-9_]*.::add-mask::every value (incoming and existing) before anything else runs.jq(nulldeletes), write the next JSON to a tmp file.dry_runor no-op → exit cleanly.aws secretsmanager put-secret-value→ new VersionId (previous versions remain available for one-step revert).externalsecret/ctdl-xtra-appwithforce-sync=<ts>and poll the targetctdl-xtra-app-envSecret'sresourceVersion(up to 60s) to confirm ESO actually picked up the new version — avoids the "pods bounced but values still old" failure mode.restart=true:kubectl rollout restart deploy/ctdl-xtra-api deploy/ctdl-xtra-workerthenrollout status(5m timeout each).applied/applied + restarted/dry-run/no-op).Only
ctdl-xtra-apiandctdl-xtra-workerare restarted;db-migrate-joband other consumers are out of scope per the design ask.IAM (prerequisite — must
terraform applybefore the workflow can succeed)infra/terraform/github-ci-oidc/main.tfaddsaws_iam_policy.github_actions_secretsmanagerwithDescribe/Get/Put/ListSecretVersionIds, scoped strictly toctdl-xtra/{test,sandbox,prod}/app-*. Attached to the existingctdl-xtra-github-actions-cirole. No other resources touched.