test(cli): snapshot the cdk synth output contract - #1865
Open
iankhou wants to merge 3 commits into
Open
Conversation
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
iankhou
added a commit
that referenced
this pull request
Aug 20, 2026
The only differences against the #1865 baseline are message codes: the template and success messages now carry their toolkit-lib codes (CDK_TOOLKIT_I1901/I1902) instead of no code. Message text, levels, streams, and ordering are unchanged across all 16 scenarios.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds baseline IO snapshots to protect the user-visible cdk synth output contract ahead of the toolkit-lib migration.
Changes:
- Adds 16 synth scenarios covering output, selection, validation, and CLI flags.
- Adds NDJSON snapshots for each scenario.
- Scrubs randomized assembly output-directory suffixes.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
packages/aws-cdk/test/commands/synth.test.ts |
Adds synth contract tests. |
packages/aws-cdk/test/_helpers/io-recorder.ts |
Adds output-directory suffix scrubbing. |
.../synth/cdk_synth_stack_ids_containing_glob_metacharacters_are_handed_to_toolkit-lib_literally.ndjson |
Snapshots literal stack-ID handling. |
.../synth/cdk_synth_single_stack_with_--quiet_prints_no_template_but_keeps_the_flags_warning.ndjson |
Snapshots quiet output. |
.../synth/cdk_synth_single_stack_with_--json_prints_the_JSON_template.ndjson |
Snapshots JSON output. |
.../synth/cdk_synth_single_stack_prints_the_obscured_YAML_template_and_the_flags_warning.ndjson |
Snapshots YAML output. |
.../synth/cdk_synth_non-exclusive_selection_expands_to_upstream_dependencies.ndjson |
Snapshots dependency expansion. |
.../synth/cdk_synth_nested-assembly_stacks_are_addressed_by_hierarchical_id_in_the_supply_line.ndjson |
Snapshots nested stack IDs. |
.../synth/cdk_synth_multiple_stacks_print_the_success_and_supply-a-stack-id_lines.ndjson |
Snapshots multi-stack output. |
.../synth/cdk_synth_explicitly_selected_stack_with_error_annotations_fails_synthesis.ndjson |
Snapshots annotation errors. |
.../synth/cdk_synth_CI_mode_single_stack_with_--quiet_allows_the_flags_warning.ndjson |
Snapshots quiet CI behavior. |
.../synth/cdk_synth_CI_mode_single_stack_skips_the_flags_warning_to_keep_stdout_valid_YAML.ndjson |
Snapshots CI YAML behavior. |
.../synth/cdk_synth_an_unmatched_pattern_fails_with_the_historical_error_message.ndjson |
Snapshots unmatched selection. |
.../synth/cdk_synth_a_validateOnSynth_stack_with_errors_is_tolerated_with_--no-validation.ndjson |
Snapshots disabled validation. |
.../synth/cdk_synth_a_validateOnSynth_stack_with_errors_fails_synth_when_validation_is_on.ndjson |
Snapshots enabled validation. |
.../synth/cdk_synth_a_stage-only_app_selects_no_stacks_and_still_succeeds.ndjson |
Snapshots stage-only apps. |
.../synth/cdk_synth_--strict_fails_synthesis_on_warning_annotations.ndjson |
Snapshots strict-mode warnings. |
.../synth/cdk_synth_--ignore-errors_tolerates_error_annotations_on_a_selected_stack.ndjson |
Snapshots ignored errors. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.
Suppressed comments (1)
packages/aws-cdk/test/_helpers/io-recorder.ts:47
- This global scrubber also rewrites any stable path segment whose name merely starts with
cdk.outplus six alphanumerics (for example,/project/cdk.outABC123archive). That can hide real output-contract changes in unrelated snapshots, despite the comment saying the match is specifically for anmkdtempdirectory directly under the temp directory. Anchor the match to the already-scrubbed<TMP>prefix and require the generated directory name to end (or be followed by a path separator).
{ pattern: /[\\/]cdk\.out[a-zA-Z0-9]{6}/g, replacement: '/cdk.out<RANDOM>' },
Adds IoHostRecorder NDJSON snapshots exercising every `cdk synth` CLI path, generated against the current implementation, so any change to synth's user-visible output shows up as a committed snapshot diff (the protection list/validate/destroy/deploy/metadata/refactor already have). 16 scenarios: single-stack YAML/JSON template printing with Rules.CheckBootstrapVersion obscuring, --quiet, multi-stack and nested-assembly output, stack ids containing glob metacharacters, the CI-mode flags-message gating that keeps stdout valid YAML, stage-only (pipeline) apps with an empty selection, non-exclusive upstream dependency expansion, validation failures (explicit selection, validateOnSynth, --no-validation), --strict on warning annotations, --ignore-errors, and the unmatched-pattern error. Also teaches the recorder's default scrubbers about random mkdtemp assembly dirs (cdk.outXXXXXX), which the multi-stack success line embeds.
- Anchor the cdk.out scrubber on the path separator so it can only match
mkdtemp assembly dirs, never prose that happens to start with "cdk.out"
(it lives in the shared defaults, so over-matching would affect every
suite).
- Spread the error-stack fixture before the validateOnSynth override so a
future `properties` key on the fixture cannot silently discard the
load-bearing override; drop the override entirely where it was dead
configuration (tests that never pass autoValidate).
- Give the nested-assembly test its own small fixture instead of reusing
the error fixture with its errors blanked out.
- Correct the header comment: the suite enters at CdkToolkit.synth, so the
cli.ts argument mapping (settings-file quiet, --exclusively branch) is
not pinned here; cross-reference the spy-style describe('synth') block.
- Note that the stage-only snapshot deliberately pins the empty-parens
"Supply a stack id ()" output as a baseline, not an endorsement.
- Drop the redundant afterEach isCI reset.
No snapshot changes: all 16 scenarios still match the committed baselines
byte-for-byte.
The anchored scrubber only matched POSIX separators; on Windows the temp-dir scrubber leaves '<TMP>\cdk.outAb12Cd', so the random suffix survived and the snapshots would be nondeterministic there (and differ from the committed '/' form). Match either separator and normalize to '/' in the replacement so the committed snapshots hold on both platforms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for this change
Baseline IO snapshots for every
cdk synthCLI path, generated against the current implementation, to validate changes in #1852.This is the same protection
list/validate/destroy/deploy/metadata/refactoralready have viaIoHostRecorderNDJSON snapshots.No production code changes.
Describe any new or updated permissions being added
None.
Description of how you validated changes
Snapshots generated against the current (
main) implementation and verified stable across repeated runs; suite and lint pass.Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license