ci: cut per-PR cost from deploy, bundle analysis, and unit tests - #8464
ci: cut per-PR cost from deploy, bundle analysis, and unit tests#8464Fbartoli (Fbartoli) wants to merge 4 commits into
Conversation
The deploy-dev job built storybook on every PR. A scope step now diffs against the base branch for story, storybook, or dependency changes and skips the build otherwise. The S3 sync excludes storybook/* in that case so --delete keeps the previously deployed storybook. Push deploys are unchanged.
Every apps/web PR paid an install plus a full build for a comment few PRs read. The job now runs on PRs only when the bundle-analysis label is set. The push run on dev stays because it produces the base artifact PRs compare against. Adds workflow_dispatch and a 20 minute timeout.
Jest applies --shard after --changedSince selection, so each shard runs a quarter of the selected set. Push runs stay single on shard 1 with coverage, Datadog upload, and Slack alert unchanged; shards 2-4 skip install on push.
|
CI results for the costdown changes on this PR. Unit tests ran as 4 shards and all passed. Durations: shard 1 took 6m24s, shard 2 took 6m14s, shard 3 took 6m13s, shard 4 took 6m08s. The scope guard output full=false because this diff only touches .github/** and apps/web/scripts/**, so every shard ran the affected-tests step with --changedSince and Jest matched zero tests. Shard time here is almost entirely checkout, yarn install, and typings generation. The single-job baseline from #8451 was 6 to 10 minutes, so this PR measures the fixed per-shard setup cost of about 6 minutes. It does not exercise the full-suite split, so a PR that trips the full-suite guard is still needed to measure the real sharding gain. Bundle analysis did not execute. Its run sits at action_required with zero jobs because this PR edits the workflow file and needs manual approve-and-run. The label gate was not exercised, and no analyze job consumed minutes. Deploy to dev/staging is in the same state: action_required, zero jobs, awaiting manual approval. The storybook-scope output could not be observed. The expected value for this diff is build=false, since it touches apps/web/scripts/github/s3_upload.sh and no storybook-relevant paths. Total wall time for the runs that executed was 13m24s (07:18:55 to 07:32:19 UTC), with Web Smoke tests finishing last. Everything except smoke completed by 07:25:30, about 6m35s after trigger. |
Branch preview✅ Deploy successful! Storybook: |
|
Results from the two approval-gated runs. Deploy (run 31157175021): the scope step classified this diff as no storybook work and the build-storybook step was skipped. The S3 sync took the exclude branch; the log contains zero storybook/* uploads and zero storybook/* deletes. Job time 7m09s against the ~11 min baseline, so the skip saves about 4 minutes per PR deploy that touches no stories. One known effect: this branch never built a storybook, so the preview comment's storybook link 404s until a story-touching commit produces one. Bundle analysis (run 31157173389): the analyze job skipped with zero steps because the PR has no bundle-analysis label. Cost on unlabeled PRs is now zero against the ~8 min baseline. The labeled path and the dev push path are still unexercised on this PR. Sharding remains unmeasured here; this diff selects zero tests, so the 4 shards only show fixed setup cost (~6m10s each). A src-touching PR in the stack will provide the real number. |
Security review of the stack found the scope-guard steps splicing
${{ github.base_ref }} into run scripts. base_ref is limited to
existing branch names, so fork PRs cannot exploit it, but a refname
may legally contain ;, $(), and |. The runner env var GITHUB_BASE_REF
carries the same value as data. All four sites switched.
1 participant
Part of the test suite experiment stack. Base is test-msw-handlers (#8463).
Three changes, one commit each.
1. Conditional storybook build on PR deploys. deploy-dev built storybook on every PR. A scope step diffs the PR against its base for
.stories., storybook config,src/stories/,package.json, oryarn.lock. No match skips the build. The S3 sync then excludesstorybook/*so--deletekeeps the previously deployed storybook. Push deploys to dev and main are unchanged. Known tradeoff: the PR comment's storybook link shows the last deployed storybook when the build is skipped.2. Label gate on bundle analysis. Every apps/web PR paid install plus full build for the bundle comment. The job now runs on PRs only with the
bundle-analysislabel (trigger types includelabeled, so adding the label starts a run). The push run on dev stays because it produces the base artifact PRs compare against. Added workflow_dispatch and a 20 minute timeout.3. Shard unit PR runs 4 ways. Jest applies
--shardafter--changedSinceselection, so each shard runs a quarter of the selected set. The scope guard from #8451 is unchanged and runs per shard. Push runs stay single on shard 1 with coverage, Datadog, and Slack untouched; shards 2-4 skip install on push.A fourth item from the review (dropping the apps/web trigger from web-tanstack checks) was withdrawn: web-tanstack declares
"@safe-global/web": "workspace:*"in its package.json, so the trigger is correct.Expected effect: PR deploy drops ~4-5 min when storybook is skipped, bundle analysis (~8 min) disappears from unlabeled PRs, and the unit gate wall time drops roughly in proportion to shard count on full runs. Numbers will be posted from this PR's own runs.
Draft, do not merge. Measurement comment follows.