sra-rust

A pure-Rust, from-scratch reimplementation of NCBI's SRA/VDB file format for read-only parsing of .sra files. No bindings, no linking against ncbi-vdb/sra-tools — just a Rust library that reads the binary format directly.

Writing .sra files and the sra-tools CLI equivalents (fastq-dump, vdb-dump, etc.) are explicitly out of scope.

Status

Layered bottom-up, each layer with its own test suite (synthetic fixtures always; real-fixture tests gated behind a feature flag):

Layer	Status
KAR container (header + TOC)	Implemented
KDB metadata tree, column headers, idx0 blob index	Implemented
.vschema parsing + column encoding resolution	Implemented
VDB cursor / typed-column read API	Implemented
Compression codecs (zip, RLE, bit-pack, bzip2, zstd)	Implemented
idx1/idx2 block-index decoding (Random/Uniform/Magnitude/Predictable)	Implemented — verified against real .sra fixtures
VDB blob framing + multi-frame BlobHeaders transform chain + PageMap	Implemented — verified against real .sra fixtures
izip (v0) and irzip (v2/v3) integer codecs	Implemented (all 8 int widths), gated behind the izip feature
zip_encoding (framed raw-DEFLATE) and delta_average_zip_encoding (QUALITY)	Implemented — verified against real .sra fixtures
Schema production-graph evaluator for computed columns (SPOT_ID, NAME, READ_LEN/START/TYPE, QUALITY)	Implemented — verified against real .sra fixtures
Aligned READ reconstruction (align_restore_read/seq_restore_read CIGAR+mismatch splice)	Implemented — verified byte-for-byte against real .sra fixtures
External reference retrieval (NCBI efetch) for aligned READ	Implemented behind the optional fetch-reference feature; pluggable ReferenceProvider

Reading real column data works end-to-end across integer, text, and quality columns. A .sra column is read by: locating the covering blob via idx0/idx1+idx2; verifying its NCBI CRC-32 (MSB-first, not reflected IEEE); stripping the v2 KColumn framing (byte order + BlobHeaders + PageMap); replaying the transform chain frame-by-frame down to the leaf codec; then expanding the deduplicated, variable-length element array back to per-row values via the blob's PageMap. Codecs wired into the chain:

• izip (self-describing, header version 0) and irzip (byte-plane range-zip, version 2/3: a linear/delta predictor plus per-plane raw-DEFLATE residuals, with min/slope/planes from the header frame),
• zip_encoding (raw-DEFLATE unzip),
• delta_average_zip_encoding (two-frame chain: unzip then the undelta_average per-position average-table inverse) used by QUALITY.

Decoded output is byte-for-byte cross-checked against sra-tools vdb-dump: ERR225922's MAPQ (all 45,521 rows), REF_LEN, GLOBAL_REF_START (spanning multiple idx2 blocks), SEQUENCE.QUALITY (152-base phred strings), and the variable-length MISMATCH column all match exactly.

Computed columns (production-graph evaluator)

Most columns users want — READ, NAME, SPOT_ID, READ_LEN/READ_START/ READ_TYPE — are not physically stored. The schema defines each as a production: a pipeline of transform functions over physical columns, sometimes spanning tables. The parser (src/schema) retains these production expressions as an AST; a row-at-a-time evaluator (src/vdb/{prod,eval}.rs) resolves a readable column into a node graph (through the table's inheritance chain) and walks it per row, dispatching to a library of vxf/axf functions (src/vxf/func.rs: map, bit_or, echo, sum, add_row_id, format_spot_name, align_restore_read, seq_restore_read, local_ref_id, 2na (un)packing, …). Cross-table joins (SEQUENCE → PRIMARY_ALIGNMENT → REFERENCE) recurse through the same evaluator at the target row id.

Cross-checked against vdb-dump on ERR225922: SPOT_ID (= row id), NAME ($R template → row id), READ_LEN, READ_START, READ_TYPE, and QUALITY all match exactly, as does aligned READ.

Aligned READ and the external reference

Both test fixtures are reference-compressed cSRA whose reference genome is stored externally (SEQ_ID = CM000663.1, GRCh38 chr1), not in the .sra. SEQUENCE.READ is reconstructed by seq_restore_read from each read's PRIMARY_ALIGNMENT entry (reverse-complemented per READ_TYPE), whose own READ is align_restore_read(ref_window, HAS_MISMATCH, MISMATCH, …) — a CIGAR/mismatch splice over a window of the reference. That window comes from a pluggable ReferenceProvider:

• the default null provider leaves aligned READ unavailable offline (a clean error, no network I/O);
• the optional fetch-reference feature adds NcbiEfetchProvider, which fetches just the needed base subrange by accession + coordinates from NCBI efetch (db=nuccore, covering both refseq and WGS accessions) over HTTPS — no whole-chromosome download.

Verified byte-for-byte against vdb-dump, fully offline via an in-memory provider seeded with the captured reference windows: PRIMARY_ALIGNMENT.READ (including the mismatch splice) and SEQUENCE.READ (forward + reverse-complement mate). An opt-in #[ignore]d test reconstructs the same through live efetch.

# reconstruct reads, fetching the reference from NCBI
cargo run --example dump_reads --features fetch-reference -- <file.sra>

Build & Test

cargo build
cargo test

Enable optional codecs:

cargo build --features zlib,bzip2,zstd,izip

Real-fixture tests

Most tests run against small hand-crafted synthetic fixtures with no network dependency. A second tier of tests runs against real .sra files:

./scripts/fetch-fixtures.sh fixtures/real
SRA_TEST_FIXTURES_DIR="$(pwd)/fixtures/real" cargo test --features real-fixtures

Fixtures are ERR225922 (~2.4 MB, human) and SRR520124 (~11.9 MB, bacterial) — both first-party test accessions used by sra-tools itself, downloaded from the AWS Open Data Program SRA mirror. Not committed to the repo.

The aligned-READ reconstruction tests run fully offline (captured reference windows). To also run the opt-in test that fetches the reference live from NCBI:

SRA_TEST_FIXTURES_DIR="$(pwd)/fixtures/real" \
  cargo test --features fetch-reference,real-fixtures -- --ignored

Reference

Ground truth for the binary format is NCBI's own C source, since no written specification exists:

• ncbi/ncbi-vdb — the VDB engine and KDB physical storage layer
• ncbi/sra-tools — CLI utilities built on top

See AGENTS.md for more detail on project conventions and contribution workflow.