Harden UMI correctness, SAM/BAM I/O, and performance

[justinblethrow-cloud]

Summary

This PR consolidates the generally useful correctness, I/O-safety, and
performance work developed and validated in the
dUMI maintained fork into one
upstream submission.

It:

• includes the exact commit already proposed in Remove redundant BAM file open in paired mode #32, preserving
  @siddharthab's authorship, so paired mode does not reopen an indexed BAM at
  every reference change;
• adds a guarded, opt-in streaming path for coordinate-sorted, single-end
  SAM/BAM input;
• fixes ambiguous-N, missing/short-UMI, edit-bound, overflow, equality, and
  deterministic-tie behavior;
• makes graph and BK-tree traversal stack-safe and prevents redundant
  scheduling in dense parallel components;
• reduces singleton-group, UMI parsing, alignment-map, and NgramBKTree
  overhead;
• makes FASTQ/SAM/BAM output transactional and hardens reader, iterator,
  writer, and paired-mate cleanup/recovery;
• upgrades HTSJDK and snappy-java beyond reviewed advisory ranges; and
• hardens the checksum-locked build, launcher, artifact verification, and
  Linux/macOS Java 11/21 CI.

Closes #31.

Reviewable commit series

The PR remains one submission, with changes divided by concern:

1. the unchanged Remove redundant BAM file open in paired mode #32 commit;
2. guarded streaming and hot-path optimization;
3. deterministic representative selection;
4. clustering, encoded-UMI, threshold, scheduling, and iterative-traversal
   hardening;
5. transactional I/O, CLI validation, resource cleanup, and paired recovery;
6. dependency, build, launcher, artifact, and CI hardening; and
7. an isolated Bash 3 compatibility correction for the macOS build gate.

Streaming mode

A new option controls the fast path:

--streaming-mode off|auto|on

• off remains the upstream default and preserves the existing path and
  output ordering.
• auto streams compatible input and safely restarts through the existing
  path if coordinate-order metadata proves inaccurate or the clipping window
  is insufficient.
• on requires streaming and fails explicitly when its contract is not met.

Streaming is eligible only for coordinate-declared, actually
coordinate-ordered, single-end SAM/BAM input using sequential algorithms and
data structures with cluster tracking disabled. Paired, parallel, tagging,
two-pass, FASTQ, and declared-unsorted inputs retain the existing paths.

The implementation retains alignment groups only within the active coordinate
window. Positive-strand unclipped starts use a conservative 10,000-base lag
window, configurable with
-Dumicollapse.streaming.positiveLag=...; unsafe inputs fall back in auto
mode or fail in on mode.

Because completed groups can be emitted outside coordinate order, streaming
output is correctly declared SO:unsorted. Record content is checked against
the existing path, but record order is not guaranteed to match.

Correctness and failure safety

• Encoded UMIs now preserve N state through cloning, mutation, equality,
  hashing, ordering, and generated neighbor keys.
• Missing and shorter-than-requested UMIs fail instead of silently aliasing a
  different UMI; custom separators are treated literally.
• Invalid edit bounds, non-finite/negative thresholds, counter overflow, and
  malformed input fail explicitly.
• Built-in algorithms use stable UMI and read-content tie-breaks. This can
  intentionally change which equal-scoring representative is selected;
  any remains arbitrary.
• Component and BK-tree walks are iterative; dense parallel traversals enqueue
  a node once per query.
• Every advertised FASTQ/SAM/BAM route stages output beside its destination
  and promotes it only after successful processing and close. Existing output
  survives a failure, and input/output aliases and hard links are rejected.
• Paired mate recovery uses indexed queries when available and a single
  sequential fallback for SAM, unindexed BAM, and unresolved cross-reference
  mates, with exact read-name/reference/start matching.

Performance work

• Lazy one-entry alignment groups and an exact-class singleton bypass avoid
  constructing a clustering structure for the common one-UMI case while
  preserving custom Java extension semantics.
• Default underscore-delimited UMI extraction avoids regex allocation.
• NgramBKTree uses packed primitive keys where representable, falls back
  safely outside the packing range, and caps initial capacity by the finite
  five-base n-gram universe.
• The original Remove redundant BAM file open in paired mode #32 persistent-reader improvement is retained and extended
  with failure-safe resource ownership and non-indexed recovery.

The maintained fork's
sealed synthetic benchmark
compares canonical upstream efeab35 with the corresponding dUMI production
implementation at 2995329. All compared outputs passed record-multiset and
header-semantic gates. Selected median matched-repetition results:

Workload	Raw speedup	Raw + downstream-ready speedup	Peak RSS change
Sparse, 100,000 reads	2.56x	1.96x	-46%
Sparse, 1,000,000 reads	2.75x	1.89x	-83%
Moderate-density single-end	1.48x	1.40x	-23%
Hotspot single-end	1.30x	1.27x	-16%
Paired, 1,000 reference groups	4.83x	4.77x	-80%

The small paired fixture was 8.9% slower, so this is not presented as a
universal speedup. These are fixed-seed synthetic scaling results on one
non-exclusive host, not production-throughput guarantees. The large paired
gain is principally attributable to #32.

Dependencies and build

• HTSJDK is upgraded from 2.19.0 to 3.0.5.
• snappy-java is upgraded from 1.1.7.3 to 1.1.10.8.
• Dependency filenames, URLs, hashes, compile/test classpaths, and manifest
  classpaths are derived from dependencies.lock; unexpected lock syntax,
  duplicate names, unsafe paths, non-HTTPS URLs, and checksum mismatches fail.
• Production and test classes are compiled separately with
  --release 11 -Xlint:all -Werror.
• Artifact verification checks current production-source identity, dependency
  bytes, manifest identity, Java target, and exclusion of test classes.
• The symlink-safe launcher no longer reserves a fixed 12 GB heap or 20 MB
  thread stack.

The tracked JAR is retained to respect the current upstream distribution
policy. The maintained fork's release packaging, SBOM, provenance,
reproducible-archive framework, benchmarks, profiles, and presentation
materials are deliberately not included here.

Validation

The exact PR head passed:

• complete ./scripts/check.sh runs on Java 11 and Java 21;
• an independent fresh-worktree Java 11 gate;
• strict compilation and Java 11 class-file-major verification;
• artifact/source/dependency verification;
• launcher and symlink-launcher smoke tests;
• git diff --check and identity/private-collateral scans; and
• independent Java and build/test adversarial reviews with no P0/P1/P2
  findings.

The regression suite covers sequential and parallel data structures,
ambiguous/generated UMIs, thresholds and overflow, deterministic
representatives, deep graph/BK-tree traversal, dense scheduling, n-gram
differential parity, singleton/custom-extension behavior, SAM/BAM streaming
compatibility and fallback, FASTQ/CLI transactions, malformed-input cleanup,
same-file/hard-link rejection, and paired recovery from indexed BAM, unindexed
BAM, and SAM.

The public
v2.0.0 validation record
and release
retain the full fork-level evidence.

Compatibility and intentional behavior changes

• Streaming remains disabled by default upstream.
• The tracked JAR and existing upstream distribution policy remain intact.
• Builds require JDK 11 or newer and target Java 11 bytecode.
• Invalid input that previously produced ambiguous or unsafe results now
  fails early.
• UMI separators are literal rather than regular expressions.
• Stable tie rules can change the chosen representative when scores are
  exactly equal, without changing the intended cluster membership.
• The Java API remains CLI-oriented and is not made concurrently reentrant by
  this PR.

Prior work and attribution

• The first commit is @siddharthab's original Remove redundant BAM file open in paired mode #32 commit, unchanged and with
  its original authorship. It addresses the repeated indexed-BAM-open cost
  reported in Very slow paired reads mode for transcriptome  #31.
• optimize objects redundancy #34 by @0jvh398j independently identified Interval allocation as an
  NgramBKTree hotspot. This implementation addresses the same hotspot with
  packed keys that include sequence and interval bounds, a safe fallback
  outside representable limits, no new dependency, and differential testing
  against Naive. Credit to @0jvh398j for identifying and reporting the
  hotspot.