feat(cli,build): convert/downsample/filter/pool commands, Parquet output, uv bootstrap

CLI: add the legacy-style data commands — `convert` (any supported format → canonical
TSV/Parquet), `downsample`, `filter` (coding/frequency/segment), `pool` (flat or
incidence --join). `-o` is now format-aware (.parquet/.pq → Parquet, else TSV).

Packaging: switch the dev bootstrap to uv (repo-local .venv, no conda); rewrite setup.sh
to be uv-first with a `python -m venv` fallback and portable across bash and zsh.
environment.yml demoted to optional (mmseqs2 + slow arda tests only).

Docs/skills: v3.0.0 status line; README + docs/index + usage gain the new CLI commands,
Parquet output, the uv install path, a Longitudinal-dynamics usage section, and the
AnnData/zarr bridge note. CHANGELOG updated; test_cli covers the four new commands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: mikessh

CHANGELOG.md

@@ -35,6 +35,18 @@ Notable changes to vdjtools v2. Releases before 3.0.0 are recorded in the git ta
   samples) and `--cohort DIR` (one streamed pass over a pre-ingested Parquet cohort); the `overlap`
   command now pre-aggregates each sample once.
 
+### Added — CLI & packaging
+
+- New `vdjtools` subcommands: **`convert`** (read any supported format — native / AIRR / Parquet /
+  MiXcr / MiGec / MiTCR / immunoSEQ / IMGT / Vidjil / RTCR / TRUST4 / arda — and write the canonical
+  table), **`downsample`**, **`filter`** (coding / non-coding / frequency / V-J segment), and
+  **`pool`** (flat pool or incidence `--join`).
+- Every command's `-o` is now **format-aware**: a `.parquet` / `.pq` path writes Parquet, anything
+  else (or stdout) writes TSV.
+- Development switched to **uv** — one repo-local `.venv`, no conda. `setup.sh` is rewritten to be
+  uv-first (with a `python -m venv` fallback) and **portable across bash and zsh**. `environment.yml`
+  is now optional, needed only for MMseqs2 (arda's aligner) + the slow arda round-trip tests.
+
 ### Added — notebooks (marimo, `[examples]` extra)
 
 - `notebooks/vaccination_tracking.py` — clonotype tracking + recapture model across YFV / influenza

CLAUDE.md

@@ -32,8 +32,8 @@ and are run ad hoc; they were never importable from here, so nothing in the libr
 
 ## Build / test / run
 ```bash
-conda env create -f environment.yml && conda activate vdjtools   # or reuse .venv
-pip install -e ".[dev,test]"                                      # builds _core
+uv venv && source .venv/bin/activate && uv pip install -e ".[dev,test]"   # builds _core (default)
+# or `bash setup.sh` (uv-first, portable bash/zsh); conda env.yml only for mmseqs2 + slow arda tests
 pytest tests/python -q
 cmake -S . -B build -DVDJTOOLS_TESTS=ON && cmake --build build && ctest --test-dir build
 sphinx-build -W --keep-going -b html docs docs/_build/html        # docs gate (zero warnings)

README.md

@@ -27,12 +27,13 @@ Built on the antigenomics ecosystem:
 [vdjmatch](https://github.com/antigenomics/vdjmatch) (overlap + TCRnet),
 [arda](https://github.com/antigenomics/arda) (AIRR annotation + markup repair).
 
-> **Status: `v2.2.0`** — the native V(D)J model engine plus the full analytics suite (diversity,
-> overlap/TCRnet, preprocessing, biomarkers, single-cell), CDR features, and legacy-format ingestion
-> (MiXcr, MiGec, immunoSEQ, IMGT/HighV-QUEST, Vidjil, RTCR, TRUST4, arda). Clonotype columns follow the AIRR
-> **junction** convention (`junction_nt` / `junction_aa`). The legacy v1.x tool lives on the
-> [`legacy-1.x`](https://github.com/antigenomics/vdjtools/tree/legacy-1.x) branch and its releases
-> remain available under the repository tags (`v0.0.1` … `1.2.1`).
+> **Status: `v3.0.0`** — the native V(D)J model engine plus the full analytics suite (diversity,
+> overlap/TCRnet, preprocessing, biomarkers, single-cell), **longitudinal clonotype dynamics**
+> (paired expansion testing + the VDJtrack recapture model), CDR features, and legacy-format
+> ingestion (MiXcr, MiGec, immunoSEQ, IMGT/HighV-QUEST, Vidjil, RTCR, TRUST4, arda). Clonotype
+> columns follow the AIRR **junction** convention (`junction_nt` / `junction_aa`). The legacy v1.x
+> tool lives on the [`legacy-1.x`](https://github.com/antigenomics/vdjtools/tree/legacy-1.x) branch
+> and its releases remain available under the repository tags (`v0.0.1` … `1.2.1`).
 
 ## Install
 
@@ -71,16 +72,23 @@ for germline lookup, Pgen, generation, or the analytics. Install it via conda/br
 
 ### Development
 
+Uses [uv](https://docs.astral.sh/uv/) — one repo-local `.venv`, no conda:
+
 ```bash
-conda env create -f environment.yml   # python + C++ toolchain + mmseqs2 (arda's aligner)
-conda activate vdjtools
-pip install -e ".[dev,test]"          # builds the _core C++ extension
+uv venv && source .venv/bin/activate
+uv pip install -e ".[dev,test]"       # builds the _core C++ extension (scikit-build-core)
 ```
 
-The conda env is a convenience, not a requirement — `pip install -e ".[dev,test]"` in any venv
-works. It supplies MMseqs2 so the slow arda annotation round-trips in the test suite run too.
+Or run the bootstrap script (portable across bash/zsh, uv-first with a `python -m venv` fallback):
+
+```bash
+bash setup.sh --dev-parents --tests   # or: zsh setup.sh
+```
 
-Or run the bootstrap script: `bash setup.sh --dev-parents --tests`.
+You need a C++ toolchain (Xcode CLT on macOS, build-essential on Linux) for the native `_core`
+extension. MMseqs2 is arda's aligner, needed **only** for the annotation path and the slow arda
+round-trip tests — `brew install mmseqs2`, or use the optional `environment.yml` conda env which
+bundles it.
 
 ## Quickstart — recombination model engine
 
@@ -130,15 +138,28 @@ vdjtools generate -m TRB -n 1000 -o gen.tsv    # sample sequences   (cf. olga-ge
 vdjtools pgen seqs.tsv -m TRB -o pgen.tsv      # Pgen per CDR3       (cf. olga-compute_pgen)
 vdjtools pgen seqs.tsv -m TRB --mismatches 1   # + the Hamming-1 ball; --v-col/--j-col to condition
 
+# data — convert any format to the canonical table (TSV, or Parquet by extension), preprocess
+vdjtools convert mixcr.txt.gz -o clones.parquet   # MiXcr/immunoSEQ/AIRR/… → canonical Parquet
+vdjtools downsample clones.parquet 100000 -o ds.tsv
+vdjtools filter clones.parquet --coding --min-freq 1e-4 -o coding.tsv
+vdjtools pool s1.tsv s2.tsv s3.tsv --join --min-samples 2 -o joint.tsv
+
 # repertoire analytics — sample files, or a cohort via -m/--metadata + --base-dir
 vdjtools diversity      sampleA.tsv sampleB.tsv -o diversity.tsv
 vdjtools overlap        *.tsv -o overlap.tsv
 vdjtools segment-usage  *.tsv --segment v -o usage.tsv
 vdjtools spectratype    *.tsv -o spectra.tsv
+vdjtools diversity      -m metadata.txt --base-dir samples/ --threads 8 -o div.tsv   # parallel cohort
+vdjtools spectratype    --cohort cohort_parquet/ -o spectra.tsv                       # one streamed pass
+
+# longitudinal — paired within-donor expansion test between two timepoints
+vdjtools dynamics day0.tsv day15.tsv -o tracked.tsv
 ```
 
-Native vdjtools and AIRR Rearrangement inputs are auto-detected; every command writes TSV to `-o`
-(or stdout, so it pipes). Run `vdjtools <command> --help` for options.
+Native vdjtools, AIRR Rearrangement, Parquet, and third-party inputs are auto-detected; every
+command writes to `-o` — **TSV, or Parquet when the path ends in `.parquet` / `.pq`** — or to stdout
+(so it pipes). Cohort commands parallelise over samples with `-t/--threads` or stream a pre-ingested
+Parquet cohort with `--cohort`. Run `vdjtools <command> --help` for options.
 
 ## Analytics (Python API)
 
@@ -175,6 +196,22 @@ usage = preprocess.correct_vj_usage(cohort, batch_col="batch", transform="sigmoi
 fixed = preprocess.apply_vj_correction(sampleA, usage, sample_id="A0")
 ```
 
+Longitudinal tracking — which clonotypes changed between two timepoints, and the VDJtrack recapture
+model (Pavlova, Zvyagin & Shugay 2024):
+
+```python
+from vdjtools import dynamics
+
+# paired within-donor test: emergent / expanded / persistent / contracted / vanishing
+tracked = dynamics.test_pair(day0, day15)                  # depth handled per-pair (effective N)
+grouped = dynamics.test_metaclonotypes(day0, day15, scope="1,0,0,1")  # 1-Hamming CDR3 ball first
+called  = dynamics.expansion_test(day0, day15)             # edgeR NB-exact caller (log2FC + p)
+
+# VDJtrack size-bucket recapture model — recapture fraction per clone-size class (Beta bands);
+# split by a group column + capture_test() for the group effect (see notebooks/vaccination_tracking.py)
+rates = dynamics.capture_rates(pre, post)
+```
+
 Incidence-based clonotype association (Emerson 2017 / Howie 2015 / De Witt 2018 / Vlasova 2026)
 — a choice of test, condition, and co-occurrence — and single-cell paired-chain Pgen:
 
@@ -232,7 +269,12 @@ suites (`RUN_BENCHMARK=1`).
 - **Overlap** — sample overlap and TCRnet (via vdjmatch/seqtree), similarity-aware overlap, clustering.
 - **Preprocess** — downsampling, error-correction, VJ-usage batch-effect correction, pooling/joining.
 - **Biomarker** — incidence association (Fisher / χ² / Bayesian / permutation) vs binary / HLA-allele / CMH-stratified conditions; α-β & same-chain co-occurrence pairing; metaclonotypes.
-- **Single-cell** — AIRR Cell / 10x interoperability, chain pairing + QC, and paired α/β Pgen.
+- **Dynamics** — longitudinal clonotype tracking between timepoints: the paired within-donor
+  expansion test (emergent / expanded / persistent / contracted / vanishing), the VDJtrack
+  size-bucket **recapture model**, metaclonotype-grouped testing, and an edgeR NB-exact caller
+  ([`vdjtools.dynamics`](python/vdjtools/dynamics)).
+- **Single-cell** — AIRR Cell / 10x interoperability, chain pairing + QC, paired α/β Pgen, and a
+  `to_anndata` bridge into the scverse ecosystem (writes `.h5ad` / `.zarr` via AnnData).
 
 ## License
 

docs/index.rst

@@ -10,8 +10,9 @@ built on the antigenomics ecosystem (`seqtree <https://github.com/antigenomics/s
 
 .. note::
 
-   **v2.2.0** — the native V(D)J model engine plus the full analytics suite (diversity, overlap/TCRnet,
-   preprocessing, biomarkers, single-cell), CDR features, and legacy-format ingestion (MiXcr, MiGec,
+   **v3.0.0** — the native V(D)J model engine plus the full analytics suite (diversity, overlap/TCRnet,
+   preprocessing, biomarkers, single-cell), longitudinal clonotype **dynamics** (paired expansion
+   testing + the VDJtrack recapture model), CDR features, and legacy-format ingestion (MiXcr, MiGec,
    immunoSEQ, IMGT/HighV-QUEST, Vidjil, RTCR, TRUST4, arda). Clonotype columns follow the AIRR **junction**
    convention (``junction_nt`` / ``junction_aa``). The legacy Groovy/Java vdjtools (v1.x) lives on the
    ``legacy-1.x`` branch and its releases remain available under the repository's tags.
@@ -51,15 +52,21 @@ the repertoire analytics (sample files or a metadata table, like the legacy tool
    vdjtools models                                # list the bundled models
    vdjtools generate -m TRB -n 1000 -o gen.tsv    # sample sequences   (cf. olga-generate_sequences)
    vdjtools pgen seqs.tsv -m TRB -o pgen.tsv      # Pgen per CDR3       (cf. olga-compute_pgen)
-   vdjtools pgen seqs.tsv -m TRB --mismatches 1   # + the Hamming-1 ball; --v-col/--j-col to condition
+
+   vdjtools convert mixcr.txt.gz -o clones.parquet   # any format → canonical TSV / Parquet
+   vdjtools filter clones.parquet --coding --min-freq 1e-4 -o coding.tsv
+   vdjtools downsample clones.parquet 100000 -o ds.tsv
 
    vdjtools diversity     sampleA.tsv sampleB.tsv -o diversity.tsv
    vdjtools overlap       *.tsv -o overlap.tsv
    vdjtools segment-usage *.tsv --segment v -o usage.tsv
-   vdjtools spectratype   *.tsv -o spectra.tsv
+   vdjtools spectratype   --cohort cohort_parquet/ -o spectra.tsv     # one streamed pass
+   vdjtools dynamics      day0.tsv day15.tsv -o tracked.tsv           # paired expansion test
 
-Native vdjtools and AIRR inputs are auto-detected; every command writes TSV to ``-o`` (or stdout).
-Run ``vdjtools <command> --help`` for options.
+Native vdjtools, AIRR, Parquet, and third-party inputs are auto-detected; every command writes to
+``-o`` — TSV, or Parquet when the path ends in ``.parquet`` / ``.pq`` — or to stdout. Cohort commands
+parallelise over samples with ``-t/--threads`` or stream a Parquet cohort with ``--cohort``. Run
+``vdjtools <command> --help`` for options.
 
 Performance
 -----------
@@ -89,7 +96,10 @@ Capabilities (see the :doc:`API reference <api>` and the project ROADMAP):
 - **Overlap** — sample overlap and TCRnet (via vdjmatch/seqtree), similarity-aware overlap, clustering.
 - **Preprocess** — downsampling, error-correction, batch-effect correction, pooling/joining.
 - **Biomarker** — incidence-based association (Fisher) and metaclonotype grouping.
-- **Single-cell** — AIRR Cell / 10x interoperability, chain pairing + QC, and paired α/β Pgen.
+- **Dynamics** — longitudinal clonotype tracking: paired within-donor expansion testing, the VDJtrack
+  size-bucket recapture model, metaclonotype-grouped testing, and an edgeR NB-exact caller.
+- **Single-cell** — AIRR Cell / 10x interoperability, chain pairing + QC, paired α/β Pgen, and an
+  AnnData / scverse bridge (``.h5ad`` / ``.zarr``).
 
 .. toctree::
    :hidden:

docs/usage.rst

@@ -167,6 +167,56 @@ the paper's own FMBA covid TCRβ cohort (deep repertoires, ~3.3M reads/sample):
 explained by batch drops from η²≈0.11 to ≈0.002 while the grand-mean usage and per-sample read
 depth are preserved.
 
+Longitudinal dynamics
+---------------------
+
+:mod:`vdjtools.dynamics` compares two samples of the **same donor** — which clonotypes changed
+between timepoints. Depth is handled *per pair* via a two-step effective sample size (never by
+normalising a whole cohort to a common depth, which is not defined here):
+
+.. code-block:: python
+
+   from vdjtools import dynamics, preprocess
+
+   day0 = demo_sample(seed=1)
+   day15 = preprocess.downsample(day0, day0["duplicate_count"].sum() // 2, seed=7)   # a later draw
+
+   # paired within-donor test: every clonotype -> emergent / expanded / persistent /
+   # contracted / vanishing (or `untested` when the pair is too shallow to call it)
+   tracked = dynamics.test_pair(day0, day15)
+   tracked["dynamics"].value_counts()
+
+   # group near-identical CDR3s (a 1-Hamming ball, V/J-pinned) BEFORE testing, for power on
+   # convergent expansions — scope "1,0,0,1" = 1 substitution, "1,1,1,1" = 1 edit (Levenshtein)
+   grouped = dynamics.test_metaclonotypes(day0, day15, scope="1,0,0,1")
+
+   # complementary edgeR NB-exact caller (TMM + qCML common dispersion) -> log2FC + p
+   called = dynamics.expansion_test(day0, day15)
+
+The **recapture model** (Pavlova, Zvyagin & Shugay, *Front Immunol* 2024) asks a different
+question: do *particular* clonotypes reappear more than the background rate, and does that depend
+on clone size? Clonotypes are binned by size (singleton / doubleton / tripleton / large), each
+bin's recapture fraction gets a ``Beta`` credible interval, and a log-linear model tests the
+**group** effect:
+
+.. code-block:: python
+
+   import polars as pl
+
+   # tag the clonotypes of interest (e.g. antigen-specific) vs the rest, in the PRE sample
+   specific = set(day0["junction_aa"].head(20))
+   pre = day0.with_columns(
+       pl.when(pl.col("junction_aa").is_in(specific)).then(pl.lit("specific"))
+         .otherwise(pl.lit("background")).alias("group"))
+
+   rates = dynamics.capture_rates(pre, day15, group_col="group")   # recapture per (group, size)
+   dynamics.capture_test(rates)                                    # log-linear group effect + p
+
+Across several donors, concatenate the per-donor ``capture_rates(..., donor=<id>)`` frames and use
+:func:`~vdjtools.dynamics.capture_paired_test` for the per-size-class paired t-test. The full
+worked example — sunken/alluvial tracking plots, a VDJdb overlay, and the capture ribbons — is
+``marimo edit notebooks/vaccination_tracking.py``.
+
 Biomarker association
 ---------------------
 
@@ -346,17 +396,30 @@ surface.
 Command line
 ------------
 
-Every workflow above has a CLI counterpart; inputs are auto-detected and results are written
-as TSV to ``-o`` (or stdout):
+Every workflow above has a CLI counterpart; inputs are auto-detected and results are written to
+``-o`` — TSV, or **Parquet when the path ends in** ``.parquet`` / ``.pq`` — or to stdout:
 
 .. code-block:: bash
 
    vdjtools models                                # list the bundled models
    vdjtools generate -m TRB -n 1000 -o gen.tsv    # (cf. olga-generate_sequences)
    vdjtools pgen seqs.tsv -m TRB -o pgen.tsv      # (cf. olga-compute_pgen)
+
+   # data: convert any format to the canonical table, and preprocess
+   vdjtools convert mixcr.txt.gz -o clones.parquet    # → canonical Parquet (or .tsv)
+   vdjtools filter clones.parquet --coding --min-freq 1e-4 -o coding.tsv
+   vdjtools downsample clones.parquet 100000 -o ds.tsv
+   vdjtools pool s1.tsv s2.tsv --join --min-samples 2 -o joint.tsv
+
+   # analytics (sample files, or -m metadata + --base-dir; parallel with -t, or stream --cohort)
    vdjtools diversity     sampleA.tsv sampleB.tsv -o diversity.tsv
    vdjtools overlap       *.tsv -o overlap.tsv
    vdjtools segment-usage *.tsv --segment v -o usage.tsv
-   vdjtools spectratype   *.tsv -o spectra.tsv
+   vdjtools spectratype   -m metadata.txt --base-dir samples/ -t 8 -o spectra.tsv
+   vdjtools diversity     --cohort cohort_parquet/ -o div.tsv
+
+   # longitudinal + enrichment
+   vdjtools dynamics day0.tsv day15.tsv -o tracked.tsv    # paired within-donor expansion test
+   vdjtools tcrnet   sample.tsv -o net.tsv                 # neighbourhood enrichment (control cohort)
 
 Run ``vdjtools <command> --help`` for options.

environment.yml

@@ -1,9 +1,15 @@
-# Dedicated conda environment for vdjtools.
-# Provides Python, a C++ toolchain (to build the _core pybind11 extension), and MMseqs2.
+# OPTIONAL conda environment for vdjtools — only if you want MMseqs2 (arda's aligner) and a
+# bundled C++ toolchain from conda. The default/recommended dev setup is uv (no conda needed):
+#
+#   bash setup.sh            # uv venv + editable install (builds _core); see setup.sh
+#   # or by hand:
+#   uv venv && source .venv/bin/activate && uv pip install -e ".[dev,test]"
+#
+# Use this file only when you need MMseqs2 for the annotation / slow arda round-trip tests:
 #
 #   conda env create -f environment.yml
 #   conda activate vdjtools
-#   pip install -e ".[dev,test]"
+#   uv pip install -e ".[dev,test]"      # (or: pip install -e ".[dev,test]")
 #
 # NB: this conda env is a convenience for DEVELOPMENT, not a requirement. arda (the germline
 # reference + VDJ-model backend) is a plain pip base dependency, so `pip install vdjtools`