Skip to content

Build a Redis cache replica set to migrate to#1631

Merged
ryanjjung merged 6 commits intomainfrom
redis-replicaset
Apr 17, 2026
Merged

Build a Redis cache replica set to migrate to#1631
ryanjjung merged 6 commits intomainfrom
redis-replicaset

Conversation

@ryanjjung
Copy link
Copy Markdown
Contributor

@ryanjjung ryanjjung commented Apr 16, 2026

This PR builds new Redis replica sets which we will soon migrate the Appointment backend containers to. It also outlines through commentary what the next steps are here.

Just to create some parity, I fleshed out the upper variable section of the dev config as well based on the recent stage config changes, but I haven't gone out and set all the secrets and updated the task definitions yet. That's some tedious work that's irrelevant to this issue.

I put a condition in around DNS in prod so that rolling these changes out will not result in DNS getting swapped out before I'm ready to make that change. However, with this PR, we will get these changes in prod (summary of preview):

  • New log destination
  • Appointment app logs go to that log destination
  • New Redis cache gets built (in a way that I've verified in stage works with both the backend API and Celery/Flower)

After approval of this PR, I'll merge and apply these changes to prod. Then I will prep the next (much smaller) PR to swap DNS over. I will force a rollout of the services without changing the image and will verify both the continued functionality of the app and that we see a corresponding shift in load through the CloudWatch metrics for the caches.

Ref: #1611

@ryanjjung ryanjjung self-assigned this Apr 16, 2026
@ryanjjung ryanjjung added backend This affects the backend application infra labels Apr 16, 2026
@ryanjjung ryanjjung marked this pull request as draft April 17, 2026 15:37
@ryanjjung
Copy link
Copy Markdown
Contributor Author

Holding off. Need to fix a few features on Redis.

@ryanjjung ryanjjung marked this pull request as ready for review April 17, 2026 17:57
Copy link
Copy Markdown

@mzeier mzeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Limited context aside,

  1. Dev cuts over immediately along with stage. Uses if project.stack == 'prod' else redis_replica_group_primary_endpoint → maybe intentional.

  2. Empty valueFrom: "" secret placeholders (SECRET_APP_ADMIN_ALLOW_LIST, SECRET_DB_SECRET, SECRET_FXA_*, etc). Maybe harmless.

Copy link
Copy Markdown
Contributor

@davinotdavid davinotdavid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after comment!

Comment thread pulumi/config.dev.yaml
.log_level: &VAR_LOG_LEVEL {name: "LOG_LEVEL", value: "ERROR"}
.log_use_stream: &VAR_LOG_USE_STREAM {name: "LOG_USE_STREAM", value: "True"}
.oids_exp_grace_period: &VAR_OIDC_EXP_GRACE_PERIOD {name: "OIDC_EXP_GRACE_PERIOD", value: "60"}
.oidc_exp_grace_period: &VAR_OIDC_EXP_GRACE_PERIOD {name: "OIDC_EXP_GRACE_PERIOD", value: "60"}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should also change this to oidc_exp_grace_period for stage and prod for consistency

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did I miss it? I'll double-check.

@ryanjjung
Copy link
Copy Markdown
Contributor Author

ryanjjung commented Apr 17, 2026

@mzeier dev isn't an environment that normally exists, and I tested the cutover procedure there yesterday. The reason the variables are not used is because this env is not part of our normal deployment flows and we don't run fully working services here; normally just testing infra changes. The values are empty because I would otherwise have to go and create all of those secrets, which is tedious work that didn't progress this issue at all, so I opted not to go that far with the work. And harmless because those variables are not referenced in the container definitions.

@ryanjjung ryanjjung merged commit 4716c63 into main Apr 17, 2026
7 checks passed
@ryanjjung ryanjjung deleted the redis-replicaset branch April 17, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend This affects the backend application infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants