roachtest: schemachange/mixed-versions failed [NOT NULL constraint validation failure]

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on release-25.3 @ d5712a7cbbfa674980ae0501f402173219bd3456:

(test_runner.go:1420).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• mvtDeploymentMode=separate-process
• mvtVersions=v25.2.10 → release-25.3
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed #160368 roachtest: schemachange/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-sql-foundations branch-release-25.4 release-blocker]
• roachtest: schemachange/mixed-versions failed #160295 roachtest: schemachange/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.3.25-rc]
• roachtest: schemachange/mixed-versions failed [index_key_decoding_error: single entry value with no default column id] #160000 roachtest: schemachange/mixed-versions failed [index_key_decoding_error: single entry value with no default column id] [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.3]
• roachtest: schemachange/mixed-versions failed [failed generating operation: dropPolicy: descriptor is being dropped] #159509 roachtest: schemachange/mixed-versions failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-26.1]
• roachtest: schemachange/mixed-versions failed [failed generating operation: createSchema: descriptor is being dropped] #159458 roachtest: schemachange/mixed-versions failed [relation: depends-on relation is dropped during DROP SCHEMA CASCADE] [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]
• roachtest: schemachange/mixed-versions failed [index_key_decoding_error: single entry value with no default column id] #158838 roachtest: schemachange/mixed-versions failed [index_key_decoding_error: single entry value with no default column id] [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.3.24-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-58358

[spilchen]

The test times out because a job (1138228159327174660) keeps trying. That job is doing an ALTER TABLE of some kind and it keeps failing during the validation step:

cockroach.teamcity-20923833-1767422299-15-n4cpu4-0001.ubuntu.2026-01-03T08_29_09Z.046338.log:36525:W260103 11:28:15.435473 1234063 sql/schemachanger/scrun/scrun.go:140 ⋮ [T3,Vmixed-version-tenant-lwmj6,nsql1,job=NEW SCHEMA CHANGE id=1138228159327174660]
 9321  failed executing declarative schema change PostCommitPhase stage 1 of 2 with 1 ValidationType op (rollback=false) for ALTER TABLE with error:
 validation of column ‹"col976_w2_988"› NOT NULL failed on row:
 ‹col976_w2_982='227.69.47.76/7', col976_w2_983=NULL, col976_w2_984=-0.2389256223828774, col976_w2_985='18:55:57.155647', col976_w2_986='{"IYUH31EOF": 0.4182471470099219, "baz": [false, 0.03420639074251733], "qL8k": [{"b": false, "c": null, "foo": {"baz": true}}, null, false, null]}', col976_w2_987='2020-05-08 08:27:20.000344', col976_w2_988=NULL, col976_w2_989='12:55:53.923915', col976_w2_990='1971-04-17', col976_w2_991=-66, col976_w2_992=e'\x1a', col976_w2_993='\x94b2826c', col976_w2_994='1984-04-28 20:02:27.000172', col976_w2_995=B'11101010100001011000100100010111110000100111', col976_w2_996='"', col976_w2_997='227.69.47.76/7', col976_w2_998='12:55:53.923915'›

It resumed over 2500 times before the test timed out.

[spilchen]

What error is being returned? The thinking is that it shouldn't be retried. Need to check the error with the whitelist of errors for the validation phase.

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.