CLOUDP-384711: Add kubectl import-tool plugin for migration epic#872
Draft
filipcirtog wants to merge 23 commits intovm-migration-feature-branchfrom
Draft
CLOUDP-384711: Add kubectl import-tool plugin for migration epic#872filipcirtog wants to merge 23 commits intovm-migration-feature-branchfrom
filipcirtog wants to merge 23 commits intovm-migration-feature-branchfrom
Conversation
Implement the kubectl-mongodb migrate generate command that reads an Ops Manager automation config and produces a MongoDB CR for the Kubernetes operator. Includes validation, security/TLS mapping, connection string handling, and unit tests. Made-with: Cursor
Document every automation config field, its migration category (mapped, default, ignored, blocker), and how the tool handles it when generating the MongoDB CR. Made-with: Cursor
Add vm_migration_kubectl_plugin.py e2e test that exercises the migrate generate command end-to-end. Wire up e2e_vm_migration and e2e_vm_migration_kubectl_plugin tasks in a new Evergreen variant. Include kubectl-mongodb binary in the test Docker image. Made-with: Cursor
- Remove pruneEmptyValues (was incorrectly pruning votes: 0 and other intentional zero values from the generated CR) - Surface buildProcessMap errors in validation instead of silently discarding them - Fail early when user CRs need password secrets but no Kubernetes client is available - Validate PEMKeyFile independently of certificateKeyFile so legacy configs using only PEMKeyFile are caught - Return error from normalizeK8sName when input yields an empty string - Fix README markdown typos and blocker summary constant paths - Update test fixture PEMKeyFile to use operator default path Made-with: Cursor
Made-with: Cursor
filipcirtog
changed the title
There was a problem hiding this comment.
Pull request overview
Adds a new kubectl-mongodb migrate generate command to support VM-to-Kubernetes migration by generating MongoDB Operator CRs from Ops Manager automation configs, and wires new CI/e2e coverage around migration scenarios (auth/TLS/heterogeneous configs). It also standardizes local/CI tooling to use the kubectl-mongodb binary name/path (replacing the old multi-cluster kubeconfig creator naming).
Changes:
- Introduce
kubectl-mongodb migratecommand, including OM project resolution, Kubernetes client creation (with in-cluster fallback), CR generation, validation, and supporting testdata. - Add multiple VM migration e2e test variants (no-auth, TLS, mongod TLS, heterogeneous config, SCRAM+X509), plus Evergreen task wiring.
- Update scripts and test harnesses to build/use
kubectl-mongodbinstead ofmulti-cluster-kube-config-creator.
Reviewed changes
Copilot reviewed 71 out of 72 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/release/kubectl_mongodb/download_kubectl_plugin.py | Adjust local test plugin artifact path to kubectl-mongodb |
| scripts/funcs/multicluster | Build/invoke kubectl-mongodb multicluster via KUBECTL_MONGODB_PATH |
| scripts/dev/prepare-multi-cluster/main.go | Switch env var to KUBECTL_MONGODB_PATH for tool path |
| scripts/dev/contexts/root-context | Export KUBECTL_MONGODB_PATH for local dev contexts |
| scripts/dev/contexts/e2e_vm_migration_kind_ubi_cloudqa | Add a new VM migration test context for cloud_qa |
| pkg/util/maputil/maputil.go | Add flat-map helpers used for config comparisons/intersections |
| mongodb-community-operator/pkg/automationconfig/automation_config.go | Add helpers for log-rotate conversion and systemLog map parsing |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_helpers.py | Shared helpers for generate-based VM migration e2e tests |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_generate.py | New generate-based VM migration e2e (SCRAM/users/roles/etc.) |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_generate_tls.py | New generate-based e2e validating OM TLS via CA bundle |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_generate_mongod_tls.py | New generate-based e2e covering mongod requireSSL + certs |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_generate_scram_x509.py | New generate-based e2e covering SCRAM + X509 dual-mode auth |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_generate_no_auth.py | New generate-based e2e covering auth-disabled path |
| docker/mongodb-kubernetes-tests/tests/tls/vm_migration_generate_heterogeneous.py | New generate-based e2e covering heterogeneous member config intersection |
| docker/mongodb-kubernetes-tests/tests/conftest.py | Update multicluster tool invocation to kubectl-mongodb |
| docker/mongodb-kubernetes-tests/Dockerfile | Install kubectl-mongodb into test image |
| controllers/operator/authentication/scramsha_credentials.go | Refine SCRAM credential reuse/regeneration behavior |
| controllers/om/shardedcluster.go | Export shard/shardedcluster accessors (Shards, Id) |
| controllers/om/replicaset.go | Add helper to extract member/version/FCV and improve member field access |
| controllers/om/process.go | Add safer accessors (disabled/auth schema/systemLog/tls sections/etc.) |
| controllers/om/omclient.go | Add OM API reads for process/audit log rotation configs |
| controllers/om/monitoring_agent_config.go | Add helpers to read logPath/logRotate from monitoring agent config |
| controllers/om/mockedomclient.go | Extend mock OM connection to support new log-rotation reads |
| controllers/om/deployment.go | Add helpers for downloadBase/prometheus/process/sharding access |
| controllers/om/backup_agent_config.go | Add helper to read backup agent logPath |
| cmd/kubectl-mongodb/root/root.go | Register new migrate command on the plugin root |
| cmd/kubectl-mongodb/migrate/util.go | Add version helper + Kubernetes name normalization |
| cmd/kubectl-mongodb/migrate/connection.go | Implement OM+Kubernetes connection setup (incl. in-cluster kubeconfig fallback) |
| cmd/kubectl-mongodb/migrate/connection_test.go | Unit tests for OM org/project resolution |
| cmd/kubectl-mongodb/migrate/command.go | Implement migrate generate command flow (validate, generate, create secrets, emit YAML) |
| cmd/kubectl-mongodb/migrate/README.md | Document mapping/validation model and migration blockers |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/tls/require.json | Input fixture for TLS require mode |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/tls/disabled.json | Input fixture for TLS disabled mode |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/tls/allow.json | Input fixture for TLS allowTLS mode |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/member_options.json | Input fixture for member tags/hidden/slaveDelay |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/heterogeneous.json | Input fixture for heterogeneous process configs |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/full.json | Full input fixture covering many AC sections |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/authentication/x509_only.json | Input fixture for X509-only auth mode |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/authentication/x509.json | Input fixture for SCRAM+X509 auth |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/authentication/scram_only.json | Input fixture for SCRAM-only auth |
| cmd/kubectl-mongodb/migrate/testdata/input/singlecluster/replicaset/authentication/disabled.json | Input fixture for auth disabled |
| cmd/kubectl-mongodb/migrate/testdata/input/sharded_cluster.json | Input fixture for sharded cluster detection/blocking |
| cmd/kubectl-mongodb/migrate/testdata/input/multicluster/replicaset/simple.json | Input fixture for multicluster RS generation |
| cmd/kubectl-mongodb/migrate/testdata/input/multi_replicaset.json | Input fixture for multi-RS detection/handling |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/user_crs.yaml | Golden output for generated user CRs |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/tls/require_cr.yaml | Golden output for TLS require CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/tls/disabled_cr.yaml | Golden output for TLS disabled CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/tls/allow_cr.yaml | Golden output for TLS allowTLS CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/prometheus_password.yaml | Golden output for Prometheus password secret |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/password_secrets.yaml | Golden output for SCRAM password secret |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/mongodb_cr.yaml | Golden output for main MongoDB CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/member_options_cr.yaml | Golden output for member options CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/ldap_resources.yaml | Golden output for LDAP secret/configmap |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/heterogeneous_cr.yaml | Golden output for heterogeneous intersection behavior |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/authentication/x509_user_crs.yaml | Golden output for X509 user CRs |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/authentication/x509_only_cr.yaml | Golden output for X509-only CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/authentication/x509_cr.yaml | Golden output for SCRAM+X509 CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/authentication/scram_only_user_crs.yaml | Golden output for SCRAM-only user CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/authentication/scram_only_cr.yaml | Golden output for SCRAM-only CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/singlecluster/replicaset/authentication/disabled_cr.yaml | Golden output for auth-disabled CR |
| cmd/kubectl-mongodb/migrate/testdata/golden/multicluster/replicaset/ldap_resources.yaml | Golden output for multicluster LDAP secret |
| cmd/kubectl-mongodb/migrate/testdata/golden/multicluster/replicaset/3_clusters.yaml | Golden output for 3-cluster MongoDBMultiCluster |
| cmd/kubectl-mongodb/migrate/testdata/golden/multicluster/replicaset/2_clusters.yaml | Golden output for 2-cluster MongoDBMultiCluster |
| .gitignore | Ignore new kubectl-mongodb artifacts |
| .evergreen.yml | Add VM migration generate tasks into task groups |
| .evergreen-tasks.yml | Define new e2e VM migration generate tasks |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+158
to
+162
| } | ||
| } else { | ||
| b, _ := json.Marshal(v) | ||
| result[key] = string(b) | ||
| } |
There was a problem hiding this comment.
flattenToMap ignores json.Marshal errors (b, _ := json.Marshal(v)). If marshaling ever fails, the value will silently become an empty string, which can make FlatMapsEqual return incorrect results. Please handle the error (e.g., propagate it or record a sentinel/error) instead of discarding it.
Comment on lines
+352
to
+361
| func ExtractMemberInfo(members []ReplicaSetMember, processMap map[string]Process) ([]mdbv1.ExternalMember, string, string) { | ||
| firstProc := processMap[members[0].Name()] | ||
| version := firstProc.Version() | ||
| fcv := firstProc.FeatureCompatibilityVersion() | ||
|
|
||
| var externalMembers []mdbv1.ExternalMember | ||
| for _, m := range members { | ||
| host := m.Name() | ||
| proc := processMap[host] | ||
| port := maputil.ReadMapValueAsInt(proc.Args(), "net", "port") |
There was a problem hiding this comment.
ExtractMemberInfo assumes members is non-empty (members[0]) and that every member name exists in processMap (direct index without ok). With a malformed/partial automation config this will panic or produce zero-value Process data. Please add explicit checks (len(members)>0, process existence) and return an error up the call chain instead of panicking.
Collaborator
Author
There was a problem hiding this comment.
should we even account for malformed input?
Comment on lines
+90
to
+103
| yamlOut, resourceName, err := GenerateMongoDBCR(ac, opts) | ||
| if err != nil { | ||
| return fmt.Errorf("error generating CR: %w", err) | ||
| } | ||
| fmt.Print(yamlOut) | ||
|
|
||
| if err := ensurePrometheus(ctx, ac, kubeClient); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if err := ensureUsers(ctx, ac, resourceName, kubeClient); err != nil { | ||
| return err | ||
| } | ||
|
|
There was a problem hiding this comment.
runGenerate prints the generated CR YAML to stdout before prompting for/creating Prometheus and user password secrets. If secret creation later fails, callers can end up with partially-successful output (CR emitted, but required secrets not created), which is hard to use in automation. Consider performing validation + all required secret creation steps first, then emit YAML only after all side effects succeed (or provide a --no-apply/--print-only mode).
Comment on lines
+560
to
+564
| // marshalCRToYAML marshals a Kubernetes resource to YAML, stripping the | ||
| // "status" block, "creationTimestamp: null", and all zero-value fields that | ||
| // are artifacts of Go struct serialization (empty strings, 0, false, nil, | ||
| // empty maps/slices). This produces clean YAML matching hand-written CRs. | ||
| func marshalCRToYAML(obj interface{}) (string, error) { |
There was a problem hiding this comment.
The marshalCRToYAML comment says zero-value stripping removes 0 and false, but isZeroValue does not treat numbers/booleans as zero values. Please update the comment (or change the implementation) so expectations match behavior, since fields like votes: 0 and logAppend: false depend on this.
filipcirtog
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a kubectl-mongodb migrate generate subcommand that reads an Ops Manager automation config and produces Kubernetes CRs (MongoDB/MongoDBMultiCluster + MongoDBUser)
skip-changeloglabel if not needed