Reduce maintenance cost of Source-Build SDK Diff Tests baselines and exclusions

[MichaelSimons]

Summary

The Source-Build SDK Diff Tests (comparing binaries/files/APIs between source-built and Microsoft-built SDKs) generate frequent baseline update PRs that require human review and merge. The maintenance cost of this process is disproportionate to the regressions it catches.

The Problem

The CreateBaselineUpdatePR tooling generates PRs titled "Update Source-Build SDK Diff Tests Baselines and Exclusions" against dotnet/dotnet on a regular basis. These PRs update files in test/Microsoft.DotNet.SourceBuild.Tests/assets/SdkContentTests/.

Data (Jan 1 - Jun 15, 2026)

Metric	Value
Total baseline update PRs created	49
Merged	37
Closed without merge	9
Currently open	3
Average frequency	~6.7 PRs/month (~1.6/week)

These PRs are authored by dotnet-sb-bot and primarily reviewed/merged by a single maintainer. Most are merged same-day, indicating routine approval rather than deep review catching issues.

Sample PR

dotnet/dotnet#7162

Concern

1. High frequency, low signal: ~37 merged baseline updates in ~5.5 months means the team is spending review/merge cycles nearly twice a week on what are largely expected differences.
2. Low regression catch rate: The vast majority of these PRs are routine acknowledgements of known/expected differences - not regressions being caught and fixed.
3. Evolved landscape: Source build has matured significantly since these tests were originally implemented. The diff surface has changed, but the testing approach hasn't adapted.

Desired Outcome

We need to find a way to lower the maintenance cost while preserving the ability to catch meaningful regressions.

[github-actions]

🏷️ Source-build triage pass — 2026-06-15 20:12 UTC

Area: area-testing
Additional area(s): area-dev-ux (impacts developer workflow and maintenance burden)
Kind: process
Repro: n/a — process improvement proposal with supporting data
Affected version(s): current
Severity: S3 — causes developer friction and maintenance burden; no broken scenario but significant time investment

Blocking: not blocking

Urgency: 🟡 next release — backlog
Reason: Process improvement that would reduce ongoing maintenance cost; should be addressed but not urgent for current release.

Suggested routing: @dotnet/source-build
Possible SME(s): mthalman, MichaelSimons — based on testing area activity
Evidence:

• mthalman: Active on 15+ area-testing issues (Capture scenario test nuget.configs in the test project binlogs #5350, Generated baseline update PRs for internal branches #5352, Define a reproducibility validation test for the SDK tarball #5486, SdkContentTests.CompareMsftToSbTemplatePackageVersions fails: 11.0 MSTest templates aren't valid standalone XML #5565, Re-enable WebScenarioTests #5422)
• MichaelSimons: Issue author, extensive experience with SDK diff tests and automation

Recommended labels: add area-testing, add area-dev-ux

Related issues: none found
Possible duplicate of: none

Confidence: high
Needs human?: no

Restricted mode — no labels or milestone applied. A maintainer should manually apply any accepted labels and milestone.

Note

🔒 Integrity filter blocked 22 items

The following items were blocked because they don't meet the GitHub integrity level.

• Add testing to source-build-reference-packages #1609 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• 1ES PT extraction fails: expected copilot-instructions.md missing from dotnet source tarball #5573 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Generate SBoM Manifest fails in source-only validation due to missing BuildDropPath #5515 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• 1ES PT extraction fails: runtime pack binaries missing from SDK/source-build tarballs #5574 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• WPF - sRGB Color Space Profile.icm is non-free #4590 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Release missing for 11.0.0-preview.4 #5562 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• SignToolTask failed in Source-Built Artifacts (Round0-Sign.proj exit code 1) #5555 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NU1301 (401 Unauthorized) when restoring from dnceng internal feeds inside container (ubuntuArmContainer) #5556 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Guardian TSAUpload failed: TSA Error 999 – insufficient disk space on build agent #5547 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NU1903: System.Security.Cryptography.Xml 8.0.0 vulnerability fails Sign Source-Built Artifacts #5544 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NU1603 failure due to missing Microsoft.DotNet.SourceBuild.Tasks 10.0.0-beta.26219.105 #5552 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• 1ES PT Error: Using an image without 1es-pt-prerequisites artifact is not allowed #5543 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• NuGet restore is returning 401 Unauthorized from internal Azure DevOps feeds #5532 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Run Tests times out even though all source-build tests pass #5514 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• VMR Source-Build on CentOS Stream 9 / Fedora 43 fails with agent timeout and exit code 134 #5525 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• runtime build fails with CS9368 (RequiresUnsafeAttribute is only valid under the updated memory safety rules) #5527 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 6 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Triage · ● 7M · ◷

[mthalman]

[Triage]: Making use of AI workflows is something to consider as well.

Another consideration: make use of manual invocation instead of automated pipeline.