Add StorageVersionMigrator controller to clean up storedVersions

[ChrisJBurns]

Summary

Adds a new StorageVersionMigrator controller to the ToolHive operator that watches opted-in toolhive.stacklok.dev CRDs and reconciles each one's status.storedVersions down to the current storage version, so a future release can drop deprecated API versions (e.g. v1alpha1) from spec.versions without orphaning etcd objects.

Closes #4969

Medium level

• Controller — cmd/thv-operator/controllers/storageversionmigrator_controller.go. Watches apiextensions.k8s.io/v1/CustomResourceDefinition (metadata-only), filtered by opt-in label + group. Per-CRD: reads spec.versions to find the storage version, lists every CR via paginated unstructured list, then for each CR does a plain Get + Update of the unmodified live object. The apiserver re-encodes the request body at the current storage version and byte-compares against etcd; when the CR was stored at a different apiVersion (the migration scenario) the comparison fails and the write proceeds, re-encoding the object. When the CR is already at the current storage version, the comparison matches and the apiserver harmlessly elides the write — there was nothing to migrate. This matches upstream kube-storage-version-migrator's mechanism. Once every CR has been processed, patches status.storedVersions via optimistic-lock merge. 1h migration cache (with periodic background GC) dedupes Get+Update RPCs across reconciles.
• Opt-in label — every v1beta1 root type carries a +kubebuilder:metadata:labels=toolhive.stacklok.dev/auto-migrate-storage-version=true marker, baked into the generated CRD YAML by task operator-manifests. New CRDs opt in by adding the same marker; CRDs that shouldn't be migrated omit it. A CI test (TestV1beta1TypesMarkerCoverage) fails the build if a root type is added without either this marker or an explicit +thv:storage-version-migrator:exclude sibling.
• Feature flag — operator.features.storageVersionMigrator in the helm chart (default true) wires to ENABLE_STORAGE_VERSION_MIGRATOR following the existing ENABLE_SERVER / ENABLE_REGISTRY / ENABLE_VMCP pattern. Admins who prefer to run kube-storage-version-migrator externally can disable it.
• Conflict-safe trim — if any per-CR re-store hits an IsConflict (concurrent writer), the controller refuses to trim storedVersions on that pass and returns a sentinel error so the next reconcile retries cleanly. The post-conflict state of the affected object is unverified, so trimming would be reasoning from an unproven premise.
• Tests — envtest suite at cmd/thv-operator/test-integration/storageversionmigrator/ with 8 scenarios: noop-when-clean, happy-path-trims-storedVersions, re-encodes CRs that are stored at a prior storage version (the load-bearing test that proves the migration mechanism actually rewrites etcd — installs a CRD with v1alpha1 storage, creates a CR, flips storage to v1beta1, asserts the CR's resourceVersion bumped after reconcile), partial-failure leaves storedVersions untouched, conflict-during-restore leaves storedVersions untouched, skips-foreign-CRDs, skips-unlabelled-CRDs, pagination (continue-token loop verified via a counting APIReader wrapper). Each test installs its own CRD fixture and calls Reconcile directly — no background manager, no timing races.
• Docs — docs/operator/storage-version-migration.md covers the actual write mechanism, webhook interaction, the opt-in label contract, feature-flag opt-out, the admin per-CRD escape hatch, and coordination with the future v1alpha1-removal release.

Low level

File	Change
cmd/thv-operator/controllers/storageversionmigrator_controller.go	Reconciler, SetupWithManager, migration cache + periodic GC, conflict-safe trim, RBAC markers
cmd/thv-operator/controllers/marker_coverage_test.go	Scans api/v1beta1/*_types.go and fails if a root type lacks either the migrate marker or an explicit exclude marker
cmd/thv-operator/test-integration/storageversionmigrator/suite_test.go	envtest bootstrap (no manager; direct Reconcile calls)
cmd/thv-operator/test-integration/storageversionmigrator/controller_test.go	8 ginkgo scenarios incl. cross-version re-encode empirical proof
cmd/thv-operator/main.go	Register apiextensions/v1 scheme, add featureStorageVersionMigrator const, setupStorageVersionMigrator() wiring
cmd/thv-operator/api/v1beta1/*_types.go (12 files)	Add +kubebuilder:metadata:labels=... marker to each root type
deploy/charts/operator-crds/files/crds/*.yaml + templates/*.yaml (12 × 2)	Regenerated CRDs — label now present in each CRD's metadata.labels
deploy/charts/operator/templates/clusterrole/role.yaml	Regenerated RBAC — adds customresourcedefinitions get/list/watch + /status update;patch + toolhive.stacklok.dev/* get/list/update + */status update
deploy/charts/operator/values.yaml	New operator.features.storageVersionMigrator: true
deploy/charts/operator/templates/deployment.yaml	New ENABLE_STORAGE_VERSION_MIGRATOR env var
deploy/charts/operator/README.md	Regenerated via task helm-docs
docs/operator/storage-version-migration.md	New user-facing docs

Type of change

• Bug fix
• New feature
• Breaking change
• Refactoring
• Documentation only

Test plan

• task lint-fix — 0 issues
• task build — clean
• Operator unit tests (go test $(go list ./cmd/thv-operator/... | grep -v /test-integration)) — all green
• Envtest suite — 8/8 pass (task operator-test-integration path, or directly with KUBEBUILDER_ASSETS=$(setup-envtest use 1.31.0 -p path) go test ./cmd/thv-operator/test-integration/storageversionmigrator/)
• Marker coverage test — passes against all 12 current v1beta1 root types
• task operator-manifests + task helm-docs produce no drift against committed state
• helm template deploy/charts/operator renders the new env var correctly

Special notes for reviewers

The write mechanism in this PR has been through three iterations during review — final state described below; the journey is captured in the inline comment threads.

The current implementation is plain Get + Update of the unmodified CR — exactly what upstream kube-storage-version-migrator does. The Kubernetes apiserver's storage layer re-encodes the request body at the current storage version, then byte-compares against etcd. When the CR was originally stored at an older apiVersion (the actual migration scenario), the encoded apiVersion stamp differs from etcd's, the comparison fails, and the write proceeds — re-encoding the object. When the CR is already at the current storage version, the bytes match and the apiserver elides the write harmlessly. This is the upstream maintainers' explicit reasoning, captured at kube-storage-version-migrator#65.

A new envtest case (re-encodes CRs that are stored at a prior storage version) proves the migration mechanism empirically: install a CRD with v1alpha1 as storage, create a CR (etcd writes apiVersion: v1alpha1 bytes), flip storage to v1beta1, run the reconciler, and assert the CR's resourceVersion bumped. Observed bump 202→204; the elision sanity check on a same-storage-version CR shows RV stays at 204 as expected.

PR size: ~450 production lines across 15 files, slightly over the 400-line / 10-file convention. 12 of those files are 1-line kubebuilder marker additions in api/v1beta1/*_types.go; the substantive Go work is 3 files (controller, main.go wiring, helm values/deployment). I did not split because the controller + opt-in labels + marker-coverage CI test are one cohesive feature — splitting would mean landing unused labels or an unused controller mid-stack.

Why this PR must ship at least one release before v1alpha1 is removed: the migrator needs time to walk every cluster's storedVersions during the deprecation window. Shipping it in the same release as the version removal provides zero benefit because clusters jumping straight to that release won't have run the migrator yet. Documented in docs/operator/storage-version-migration.md.

Follow-ups not in scope for this PR:

• Optional chainsaw smoke test against a real Kind cluster (envtest covers the same behaviour at the apiserver level; chainsaw would add coverage for the helm chart wiring and admission-webhook compatibility specifically).
• Audit existing operator webhooks (MCPServer, MCPGroup, VirtualMCPServer) for compatibility with migration-induced Updates. Migration writes go through normal admission flow on objects that need re-encoding; webhooks should already tolerate metadata-stable Updates but worth empirical validation in a kind cluster.

🤖 Generated with Claude Code

[jhrozek]

A few things I'd like to flag. The two bigger ones are about the SSA body being empty and about swallowed conflicts before we trim storedVersions, curious whether those are intentional. The rest are smaller.

[jhrozek]

IsNotFound is fine to swallow, but I'm not sure IsConflict is. The comment says "storage is already fresh", but a concurrent writer using the deprecated API version (still served during the deprecation window) produces a write encoded at the old version. If our SSA races with that writer and comes back Conflict, the CR is still stored at the old version, we silently skip it, and then patchStoredVersions trims storedVersions to [storageVersion]. That leaves storedVersions claiming we're clean while there's still an old-version-encoded object in etcd.

Options: retry the apply on Conflict (bounded), or track a "swallowed-conflict" flag and refuse to trim storedVersions on the pass where any occurred. Either way I don't think we should treat Conflict the same as NotFound here.

[ChrisJBurns]

Agreed — fixed in dad30c9. restoreCRs now tracks per-pass conflict count in a function-local counter. IsConflict is logged at V(1) and skipped per-CR (still treated as transient), but if any conflicts occurred and no other errors did, the function returns a sentinel errMigrationRetriedDueToConflicts. Reconcile propagates it without calling patchStoredVersions, so storedVersions stays at [v1alpha1, v1beta1] until a conflict-free pass succeeds.

Added a new test ConflictDuringRestoreLeavesStoredVersionsUntouched that injects an apierrors.NewConflict(...) for one CR via a wrapping client, asserts the sentinel error is returned, asserts storedVersions is untouched, then drops the injection and verifies the next reconcile completes cleanly.

IsNotFound is still a clean swallow with no error — the object is gone, it can't be at the old version anymore.

[jhrozek]

Minor, just wanted to surface this for sign-off. The wildcard grants get/list/patch across every current and future kind in the toolhive group, not just the opted-in ones. I get that the opted-in set isn't known at codegen time, so the alternative (explicit per-kind rules) is awkward. Probably fine, but worth a brief comment on the marker noting the intentional wildcard plus the isManagedCRD gate as defense-in-depth.

[ChrisJBurns]

Added the comment in dad30c9 — explicit note above the kubebuilder marker block explaining the wildcard is intentional (opt-in set isn't known at codegen time), with isManagedCRD documented as the runtime gate forming defence in depth.

While I was there: the verbs themselves changed too — patch → update for both the main resource and /status to match the new write path (Update + annotation toggle, see your other comment). RBAC regenerated via task operator-manifests.

[ChrisJBurns]

Design correction — withdrawing the "upstream is broken" claim

I owe a correction on the design pivot from dad30c90. That commit message and the inline replies on this PR both claimed kube-storage-version-migrator upstream and Cluster-API's crdmigrator were silently broken, based on probes that showed plain Update of an unmodified CR was elided by the apiserver. That conclusion was wrong — I was testing the wrong scenario.

The corrected understanding (verified against ksvm#65 and confirmed empirically against our envtest in ab03d5e1):

• The apiserver's storage layer elision happens after the storage encoder runs, comparing freshly-encoded bytes against etcd bytes.
• The encoded form includes the storage version's apiVersion stamp.
• When etcd holds bytes stamped at an older version (the actual migration scenario) and the controller writes via the current storage version, the encoded bytes differ → write proceeds → RV bumps → etcd re-encoded.
• When etcd is already at the current storage version (no migration needed), the bytes match → write elides — and that's correct, not a bug.

My earlier probes only tested the second case (CR already at storage version), saw the elision, and overgeneralised. The probe in ab03d5e1's new envtest case exercises the actual cross-version migration scenario and confirms plain Update bumps RV correctly.

Practical consequence for this PR: the annotation toggle (toolhive.stacklok.dev/storage-version-migrated-at) added in dad30c90 was unnecessary. ab03d5e1 removes it — the controller now does plain Get + Update, which matches upstream kube-storage-version-migrator exactly. CRs no longer carry the persistent migration annotation that earlier review surfaced as a concern.

Apologies to the upstream maintainers of kube-storage-version-migrator and cluster-api/crdmigrator for the mischaracterisation. Both projects' mechanisms are correct for the migration scenario they target.

[github-actions]

Large PR Detected

This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.

How to unblock this PR:

Add a section to your PR description with the following format:

## Large PR Justification

[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformation

Alternative:

Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.

See our Contributing Guidelines for more details.

---

This review will be automatically dismissed once you add the justification section.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 201 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.18%. Comparing base (bd73817) to head (9b80858).

Files with missing lines	Patch %	Lines
...r/controllers/storageversionmigrator_controller.go	0.00%	174 Missing ⚠️
cmd/thv-operator/app/app.go	0.00%	27 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5011      +/-   ##
==========================================
- Coverage   68.37%   68.18%   -0.20%     
==========================================
  Files         624      625       +1     
  Lines       63442    63643     +201     
==========================================
+ Hits        43381    43393      +12     
- Misses      16820    17015     +195     
+ Partials     3241     3235       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.