stress test failure with trie index

[xingbowang]

BlockBasedTableIterator::SeekImpl Assertion Failure with Trie UDI

Failure Summary

Assertion: block_based_table_iterator.cc:187

assert(!Valid() || icomp_.Compare(*target, key()) <= 0);

After a Seek(target), the iterator is Valid() but pointing to a key less than the target, violating the Seek contract.

Seed: 5436581147554408775
Exit code: -6 (SIGABRT)

Stack Trace (Key Frames)

#11 BlockBasedTableIterator::SeekImpl           (block_based_table_iterator.cc:187)
#12 IteratorWrapperBase::Seek                   (iterator_wrapper.h:137)
#13 LevelIterator::Seek                         (version_set.cc:1433)
#14-15 MergingIterator::SeekImpl                (merging_iterator.cc:835)
#16 MergingIterator::Seek                       (merging_iterator.cc:323)
#17-18 DBIter::FindNextUserEntryInternal        (db_iter.cc:605)
#19 DBIter::FindNextUserEntry                   (db_iter.cc:366)
#20 DBIter::Next                                (db_iter.cc:207)
#22 TestPrefixScan                              (no_batched_ops_stress.cc:1656)

Relevant db_stress Flags

Flag	Value	Significance
--use_trie_index	1	Enables trie-based User Defined Index (UDI) — new feature (#14310)
--nooverwritepercent	1	99% of keys are overwritten → multiple seqno versions per user key
--prefix_size	1	Crash occurs during TestPrefixScan
--max_key	100000	Small key space increases same-key multi-version probability
--max_key_len	3	Short keys
--separate_key_value_in_data_block	1	New feature (#14287), not root cause
--index_block_search_type	1	Interpolation search, not root cause (UDI path bypasses it)

Root Cause: Trie Index (UDI) Separator Key Mismatch

The bug is caused by a fundamental mismatch between how the trie index (UDI) and the internal block-based index compute separator keys between data blocks.

How the Internal Index Works

ShortenedIndexBuilder (in index_builder.cc) computes separators from internal keys (user_key + sequence_number + type). When two adjacent blocks share the same user key but differ by sequence number, the internal index uses the full internal key as the separator, preserving correct ordering by sequence number (higher seqno = smaller in internal key order).

How the Trie Index (UDI) Works

TrieIndexBuilder (in trie_index_factory.cc) receives only user keys (stripped of sequence number and type) from UserDefinedIndexBuilderWrapper::AddIndexEntry():

// user_defined_index_wrapper.h, lines 62-71
user_defined_index_builder_->AddIndexEntry(
    pkey_last.user_key,    // USER KEY ONLY — no seqno!
    first_key_in_next_block ? &pkey_first.user_key : nullptr,
    handle, separator_scratch);

return internal_index_builder_->AddIndexEntry(
    last_key_in_current_block,  // FULL INTERNAL KEY
    first_key_in_next_block, block_handle,
    separator_scratch, skip_delta_encoding);

The trie then calls FindShortestSeparator on user keys only, and stores the result.

The Failure Scenario

When the same user key spans two adjacent data blocks with different sequence numbers:

• Block 1 last key: "foo" | seq=100
• Block 2 first key: "foo" | seq=50

Internal index: Separator is the full internal key "foo" | seq=100. Internal key comparison correctly places "foo" | seq=75 after this separator (since seq=100 > seq=75, and higher seqno = smaller in internal order), directing the seek to Block 2.

Trie index: Separator is user key "foo". FindShortestSeparator("foo", "foo") produces "foo". The trie seek for user key "foo" finds this separator and returns Block 1 (wrong).

The iterator then seeks within Block 1, doesn't find the target, advances via FindKeyForward() to the next block, and ends up positioned on a key that is less than the target in internal key order — triggering the assertion.

Additional Factor: Synthetic Internal Key in UDI Wrapper

UserDefinedIndexIteratorWrapper::Seek creates a synthetic internal key at line 208:

ikey_.Set(result_.key, 0, ValueType::kTypeValue);

This assigns sequence number 0 to the separator key. In internal key ordering, seq=0 is the largest (last) for a given user key, which can further confuse the reseek optimization at lines 114-118 that compares index_iter_->user_key() against the target.

Code Acknowledgement

The code already acknowledges this limitation in user_defined_index_wrapper.h lines 121-124:

// Pass the user key to the UDI. We don't expect multiple entries with
// different sequence numbers for the same key in the file. RocksDB may
// enforce it in the future by allowing UDIs only for read only
// bulkloaded use cases, and only allow ingestion of files with
// sequence number 0.

However, this assumption is not enforced — the stress test freely creates SST files with multiple versions of the same user key.

Suggested Fixes

Option 1: Enforce at Write Time (Recommended)

In UserDefinedIndexBuilderWrapper::OnKeyAdded(), track the previous user key and return an error (or fall back to the internal index) if the same user key appears with a different sequence number. This prevents building a UDI for files that violate the single-version assumption.

Option 2: Enforce in Stress Test

In db_crashtest.py, when use_trie_index=1, set nooverwritepercent=100 to prevent overwrites, ensuring each user key has only one version per SST file.

Option 3: Fix the UDI API

Extend the UserDefinedIndexBuilder API to accept internal keys (or at least sequence numbers) so the trie can produce correct separators that respect internal key ordering. This is a larger change.

Evidence from Crash DB Artifact

Analysis of the SST files from the crashed DB (Sandcastle workflow 1711367858414982505) confirms the root cause.

SST files with duplicate user keys (data blocks)

SST File	Entries	Dup User Keys	Cross Block Boundary
007560.sst	137,178	38,007	260
007559.sst	514	2	2
007564.sst	—	1	—
All other SSTs	—	0	0

Block boundary analysis: same user key in index entries

In 007560.sst, 314 out of 867 block boundaries (36%) have the identical user key on both sides of the boundary, differing only by sequence number. At each such boundary, the trie index produces a duplicate separator that cannot distinguish the two blocks.

Block boundary at entry 162:
  Last in block 0:  user_key=...0093787878, seq=171841
  First in block 1: user_key=...0093787878, seq=139577
  Internal index separator: ...0093787878 | seq=171841  ← correctly ordered
  Trie index separator:     ...0093787878               ← loses seqno

Block boundary at entry 4270:
  Last in block 26:  user_key=...002D7878787878, seq=142116
  First in block 27: user_key=...002D7878787878, seq=37225
  Internal index separator: ...002D7878787878 | seq=142116
  Trie index separator:     ...002D7878787878

Block boundary at entry 8226:
  Last in block 51:  user_key=...012B0000000000000177, seq=171908
  First in block 52: user_key=...012B0000000000000177, seq=171756
  Internal index separator: ...012B0000000000000177 | seq=171908
  Trie index separator:     ...012B0000000000000177

In each case, if a seek targets this user key with a sequence number between the two sides (e.g., seq=100000 when the boundary is seq=142116 / seq=37225), the trie returns the wrong block.

SST file properties

007560.sst properties show:

• 702 data blocks, 137,178 entries
• Sequence range: 101 – 184,797
• Index type: partitioned (index_type=2)
• Comparator: leveldb.BytewiseComparator

Reproduction

./db_stress \
  --use_trie_index=1 \
  --nooverwritepercent=1 \
  --prefix_size=1 \
  --max_key=100000 \
  --max_key_len=3 \
  --prefixpercent=5 \
  --ops_per_thread=100000000 \
  # ... (full flag list from stress test output)

Use random seed for iteration 5436581147554408775 Running db_stress with pid=226512: ./db_stress --WAL_size_limit_MB=1 --WAL_ttl_seconds=0 --abort_and_resume_compactions_one_in=1000000 --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --adm_policy=1 --advise_random_on_open=0 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=0 --allow_resumption_one_in=0 --allow_unprepared_value=1 --async_io=1 --auto_readahead_size=0 --auto_refresh_iterator_with_snapshot=1 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=1 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=8 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=2 --block_size=16384 --bloom_before_level=2147483646 --bloom_bits=127.88634023494654 --bottommost_compression_type=disable --bottommost_file_compaction_delay=600 --bytes_per_sync=262144 --cache_index_and_filter_blocks=0 --cache_index_and_filter_blocks_with_high_priority=0 --cache_size=33554432 --cache_type=auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=1 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=0 --checksum_type=kCRC32c --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_pri=2 --compaction_readahead_size=1048576 --compaction_style=0 --compaction_ttl=2 --compressed_secondary_cache_ratio=0.0 --compressed_secondary_cache_size=0 --compression_checksum=0 --compression_manager=autoskip --compression_max_dict_buffer_bytes=8191 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=zstd --compression_use_zstd_dict_trainer=0 --compression_zstd_max_train_bytes=65536 --continuous_verification_interval=0 --daily_offpeak_time_utc=04:00-08:00 --data_block_index_type=1 --db=/dev/shm/rocksdb_test/rocksdb_crashtest_blackbox --db_write_buffer_size=8388608 --decouple_partitioned_filters=0 --default_temperature=kCold --default_write_temperature=kCool --delete_obsolete_files_period_micros=30000000 --delpercent=0 --delrangepercent=0 --destroy_db_initially=0 --detect_filter_construct_corruption=0 --disable_file_deletions_one_in=1000000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_compaction_on_deletion_trigger=1 --enable_custom_split_merge=1 --enable_do_not_compress_roles=1 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=0 --enable_thread_tracking=0 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=1 --exclude_wal_from_write_fault_injection=0 --expected_values_dir=/dev/shm/rocksdb_test/rocksdb_crashtest_expected --fifo_allow_compaction=1 --fifo_compaction_max_data_files_size_mb=0 --fifo_compaction_use_kv_ratio_compaction=0 --file_checksum_impl=xxh64 --file_temperature_age_thresholds='' --fill_cache=0 --flush_one_in=1000000 --format_version=2 --get_all_column_family_metadata_one_in=10000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=10000 --get_properties_of_all_tables_one_in=100000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0 --index_block_restart_interval=1 --index_block_search_type=1 --index_shortening=1 --index_type=2 --ingest_external_file_one_in=1000 --ingest_wbwi_one_in=0 --initial_auto_readahead_size=16384 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=1 --lock_wal_one_in=1000000 --log_file_time_to_roll=0 --log_readahead_size=0 --long_running_snapshots=1 --low_pri_pool_ratio=0 --lowest_used_cache_tier=1 --manifest_preallocation_size=0 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=16384 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=8192 --max_manifest_space_amp_pct=1000 --max_sequential_skip_in_iterations=2 --max_total_wal_size=0 --max_write_batch_group_size_bytes=16777216 --max_write_buffer_number=3 --max_write_buffer_size_to_maintain=1048576 --memtable_avg_op_scan_flush_trigger=0 --memtable_insert_hint_per_batch=0 --memtable_max_range_deletions=1000 --memtable_op_scan_flush_trigger=100 --memtable_prefix_bloom_size_ratio=0.001 --memtable_protection_bytes_per_key=4 --memtable_veirfy_per_key_checksum_on_seek=0 --memtable_whole_key_filtering=1 --memtablerep=skip_list --metadata_charge_policy=0 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=128 --min_write_buffer_number_to_merge=2 --mmap_read=1 --mock_direct_io=False --multiscan_use_async_io=0 --nooverwritepercent=1 --num_bottom_pri_threads=1 --num_file_reads_for_auto_readahead=1 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=0 --ops_per_thread=100000000 --optimize_filters_for_hits=0 --optimize_filters_for_memory=0 --optimize_multiget_for_io=1 --paranoid_file_checks=1 --paranoid_memory_checks=0 --partition_filters=0 --partition_pinning=0 --pause_background_one_in=1000000 --periodic_compaction_seconds=1 --prefix_size=1 --prefixpercent=5 --prepopulate_block_cache=0 --preserve_internal_time_seconds=36000 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=32 --read_fault_one_in=0 --readahead_size=524288 --readpercent=45 --recycle_log_file_num=0 --remote_compaction_failure_fall_back_to_local=1 --remote_compaction_worker_threads=0 --reopen=0 --report_bg_io_stats=0 --reset_stats_one_in=10000 --sample_for_compression=0 --secondary_cache_fault_one_in=32 --secondary_cache_uri='compressed_secondary_cache://capacity=8388608;enable_custom_split_merge=true' --separate_key_value_in_data_block=1 --set_options_one_in=0 --skip_stats_update_on_db_open=1 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=1048576 --sqfc_name=bar --sqfc_version=1 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --statistics=1 --stats_dump_period_sec=0 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=2 --super_block_alignment_size=131072 --super_block_alignment_space_overhead_ratio=32 --sync=0 --sync_fault_injection=0 --table_cache_numshardbits=-1 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --test_ingest_standalone_range_deletion_one_in=0 --test_secondary=1 --top_level_index_pinning=3 --track_and_verify_wals=0 --uncache_aggressiveness=56 --universal_max_read_amp=-1 --universal_reduce_file_locking=1 --unpartitioned_pinning=0 --use_adaptive_mutex=0 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=1 --use_multi_get_entity=0 --use_multiget=0 --use_multiscan=0 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=0 --use_trie_index=1 --use_write_buffer_manager=1 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=0 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=zstd --write_buffer_size=33554432 --write_dbid_to_manifest=1 --write_fault_one_in=0 --write_identity_file=1 --writepercent=40

WARNING: db_stress ended before kill: exitcode=-6

Exit Before Killing stdout:

stderr: db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice*, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) Invoking LLDB for stack trace... db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) Received signal 6 (Aborted) Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(target, key()) <= 0' failed. Received signal 6 (Aborted) db_stress: table/block_based/block_based_table_iterator.cc:187: void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool): Assertion !Valid() || icomp_.Compare(*target, key()) <= 0' failed. Received signal 6 (Aborted) Process 226512 stopped

thread #1, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfe972c1 libc.so.6__futex_abstimed_wait_common64(private=, cancel=true, abstime=0x0000000000000000, op=393, expected=0, futex_word=0x00006160000003d8) at futex-internal.c:57:12 thread #2, name = 'rocksdb:low', stop reason = signal SIGSTOP frame #0: 0x00007f22dfe972c1 libc.so.6__futex_abstimed_wait_common64(private=, cancel=true, abstime=0x0000000000000000, op=393, expected=0, futex_word=0x0000611000031354) at futex-internal.c:57:12 thread #3, name = 'rocksdb:bottom', stop reason = signal SIGSTOP frame #0: 0x00007f22dfe972c1 libc.so.6__futex_abstimed_wait_common64(private=, cancel=true, abstime=0x0000000000000000, op=393, expected=0, futex_word=0x0000611000031214) at futex-internal.c:57:12 thread #4, name = 'rocksdb:high', stop reason = signal SIGSTOP frame #0: 0x00007f22dfe972c1 libc.so.6__futex_abstimed_wait_common64(private=, cancel=true, abstime=0x0000000000000000, op=393, expected=0, futex_word=0x0000611000031494) at futex-internal.c:57:12 thread #5, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfe972c1 libc.so.6__futex_abstimed_wait_common64(private=, cancel=true, abstime=0x00007f22d44b6020, op=393, expected=0, futex_word=0x00007f22e337dd38) at futex-internal.c:57:12 thread #6, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22cce2a270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #7, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22cd62b270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #8, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22cde2c270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #9, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22ce62d270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #10, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22d051a270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #11, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22cf210270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #12, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22cbb20270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #13, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22ca816270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #14, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c950c270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #15, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c8202270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #16, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c6ef8270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #17, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c5bee270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #18, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c3da5270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #19, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfef0e13 libc.so.6__GI___wait4(pid=226909, stat_loc=0x00007f22c1cf6830, options=0, usage=0x0000000000000000) at wait4.c:30:10 thread #20, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c1791270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #21, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22c0487270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #22, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22bf17d270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #23, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22bde73270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #24, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22bcb69270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #25, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22bb85f270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #26, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22ba555270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #27, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b924b270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #28, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b7f41270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #29, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b6c37270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #30, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b592d270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #31, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b4623270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #32, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b3319270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #33, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b200f270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #34, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22b0d05270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #35, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22af9fb270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #36, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22ae6f1270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 thread #37, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007f22dfeec3a3 libc.so.6__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f22ad3e7270, rem=0x0000000000000000) at clock_nanosleep.c:48:7 Executable binary set to "/data/sandcastle/boxes/trunk-hg-full-fbsource/fbcode/internal_repo_rocksdb/repo/db_stress". Architecture set to: x86_64-unknown-linux-gnu. True frame #4: 0x00007f22dfe44560 libc.so.6__restore_rt at libc_sigaction.c:13 frame #5: 0x00007f22dfe9c993 libc.so.6__pthread_kill_internal(signo=6, threadid=) at pthread_kill.c:46:37 frame #6: 0x00007f22dfe9c968 libc.so.6__GI___pthread_kill(threadid=, signo=6) at pthread_kill.c:62:10 frame #7: 0x00007f22dfe444ad libc.so.6__GI_raise(sig=6) at raise.c:26:13 frame #8: 0x00007f22dfe2c433 libc.so.6_GI_abort at abort.c:79:7 frame #9: 0x00007f22dfe3bc28 libc.so.6__assert_fail_base(fmt="%s%s%s:%u: %s%sAssertion %s' failed.\n%n", assertion="!Valid() || icomp.Compare(*target, key()) <= 0", file="table/block_based/block_based_table_iterator.cc", line=187, function=) at assert.c:92:3 frame #10: 0x00007f22dfe3bc93 libc.so.6__GI___assert_fail(assertion="!Valid() || icomp_.Compare(target, key()) <= 0", file="table/block_based/block_based_table_iterator.cc", line=187, function="void rocksdb::BlockBasedTableIterator::SeekImpl(const rocksdb::Slice, bool)") at assert.c:101:3 frame #11: 0x00007f22e2170618 librocksdb.so.11.1rocksdb::BlockBasedTableIterator::SeekImpl(this=0x00006190000e4c80, target=, async_prefetch=) at block_based_table_iterator.cc:187:5 frame #12: 0x00007f22e1ac1ef8 librocksdb.so.11.1rocksdb::IteratorWrapperBaserocksdb::Slice::Seek(k=, this=0x00006210000c31b0) at iterator_wrapper.h:137:16 frame #13: 0x00007f22e1ac1ef5 librocksdb.so.11.1rocksdb::(anonymous namespace)::LevelIterator::Seek(this=0x00006210000c3100, target=) const at version_set.cc:1433:20 frame #14: 0x00007f22e23ad553 librocksdb.so.11.1rocksdb::IteratorWrapperBaserocksdb::Slice::Seek(k=0x00007f22c1e12670, this=0x000061a000096550) at iterator_wrapper.h:137:16 frame #15: 0x00007f22e23ad500 librocksdb.so.11.1rocksdb::MergingIterator::SeekImpl(this=0x000061e00007fd90, target=, starting_level=, range_tombstone_reseek=false) at merging_iterator.cc:835:33 frame #16: 0x00007f22e23c7d6f librocksdb.so.11.1rocksdb::MergingIterator::Seek(this=0x000061e00007fd90, target=0x00007f22c1ccd240) at merging_iterator.cc:323:13 frame #17: 0x00007f22e15d8b3c librocksdb.so.11.1rocksdb::IteratorWrapperBaserocksdb::Slice::Seek(k=0x00007f22c1ccd240, this=0x000061e00007f918) at iterator_wrapper.h:137:16 frame #18: 0x00007f22e15d8ae2 librocksdb.so.11.1rocksdb::DBIter::FindNextUserEntryInternal(this=0x000061e00007f8c0, skipping_saved_key=true, prefix=) at db_iter.cc:605:17 frame #19: 0x00007f22e15da8ac librocksdb.so.11.1rocksdb::DBIter::FindNextUserEntry(this=, skipping_saved_key=, prefix=) at db_iter.cc:366:35 frame #20: 0x00007f22e15db9d7 librocksdb.so.11.1rocksdb::DBIter::Next(this=0x000061e00007f8c0) at db_iter.cc:207:24 frame #21: 0x00007f22e0e21d7b librocksdb.so.11.1rocksdb::ArenaWrappedDBIter::Next(this=0x000061e00007f880) at arena_wrapped_db_iter.h:78:19 frame #22: 0x00000000004ed4a7 db_stressrocksdb::NonBatchedOpsStressTest::TestPrefixScan(this=, thread=, read_opts=, rand_column_families=, rand_keys=) at no_batched_ops_stress.cc:1656:55 frame #23: 0x00000000006bf0db db_stressrocksdb::StressTest::OperateDb(this=, thread=0x000061a000018080) at db_stress_test_base.cc:1446:23 frame #24: 0x000000000060606f db_stressrocksdb::ThreadBody(v=0x000061a000018080) at db_stress_driver.cc:39:47 frame #25: 0x00007f22e1ccf72b librocksdb.so.11.1rocksdb::(anonymous namespace)::StartThreadWrapper(arg=0x00006020000387f0) at env_posix.cc:471:23 frame #26: 0x00007f22dfe9abc9 libc.so.6start_thread(arg=) at pthread_create.c:434:8 frame #27: 0x00007f22dff2ce4c libc.so.6__clone3 at clone3.S:81

[zaidoon1]

so the bug is: when the same user key exists in two adjacent data blocks with different sequence numbers (e.g., "foo"|seq=100 ends Block N, "foo"|seq=50 starts Block N+1), the UDI's AddIndexEntry receives identical user keys for both sides of the boundary. The UDI produces a separator that equals the key itself (e.g., trie's FindShortestSeparator("foo", "foo") = "foo"), which cannot distinguish the two blocks. A subsequent Seek for a sequence number between the two (e.g., "foo"|seq=75) lands on the wrong block, causing the assertion !Valid() || icomp_.Compare(*target, key()) <= 0 to fire in block_based_table_iterator.cc:187.
This happens because the UserDefinedIndexBuilderWrapper strips sequence numbers before forwarding to the UDI plugin — the plugin only sees user keys. When two consecutive blocks share the same user key (which occurs legitimately when snapshots keep multiple versions), the UDI has no way to produce a correct separator. This is a limitation of the UDI wrapper layer, not any specific plugin.

we have a few options to deal with this:

Option 1 — Flush block policy (prevent at write time): When UDI is enabled, defer flushing a data block if the next key shares the same user key as the block's last key. This ensures every block boundary sits between distinct user keys, so the UDI separator is always unambiguous. Blocks may slightly exceed the block_size target when a key has multiple versions at the boundary. Pros: simple, correct by construction, no UDI interface changes, no read-path changes, works for all UDI implementations. Cons: blocks can exceed block_size (though block_size is already a soft target); a pathological case (thousands of versions of one key) could create a very large block, so a safety cap may be warranted.
Option 2 — Reject at write time: Detect the condition in AddIndexEntry and return Status::NotSupported, failing the SST build. The crashtest config would also need nooverwritepercent=100 and delpercent=0 to prevent the condition in testing. Pros: simple, no UDI or read-path changes. Cons: flush and compaction fail in production whenever snapshots or overwrites cause the same user key to span a block boundary — this trades a crash for an operational failure, which limits UDI to bulkload/ingest-only workloads with unique user keys.
Option 3 — Handle in the wrapper (transparent to plugins): When AddIndexEntry sees the same user key spanning blocks, don't forward that boundary to the UDI plugin. Instead, store a mini side-index of (user_key, seqno_range) → block_handle in a separate SST meta block. At read time, the wrapper queries the UDI for the user key, then consults the side-index to pick the correct block using the target's sequence number. Pros: fully lifts the limitation, UDI plugin interface unchanged, no block size inflation. Cons: significant implementation complexity in the wrapper, extra meta block in SSTs, re-invents a small piece of the internal index for an edge case that's bounded and rare in practice.
Option 4 — Extend the UDI interface: Add sequence number awareness to AddIndexEntry and Seek (e.g., pass seqno alongside user keys for same-key boundaries). Pros: each UDI plugin can handle the case optimally. Cons: leaks RocksDB internals (sequence numbers) into the plugin API, every plugin must implement the logic, increases interface complexity for a case most plugins won't encounter.

Any preference?

[xingbowang]

Discussed offline, we will try option 4.