Skip to content

Commit 056845a

Browse files
gesttalttclaude
andcommitted
Surface the cytochrome c negative result in CLAUDE.md itself
The internal hierarchy/ARI/Q metrics documented throughout this file (V21-V24) measure loss-function self-consistency, not real-world usefulness -- that distinction was previously only visible in docs/plans/EXTERNAL-VALIDATION-ROADMAP.md, which nothing forces a reader of this file to open. Adding the 2026-07-17 result here directly (none of A/B/C beat a zero-model baseline on real cytochrome c phylogeny; the full p-adic curriculum scored lowest of the three) so anyone reading this file for architecture claims sees the caveat in the same place, not several links away. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent ea04cb6 commit 056845a

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

CLAUDE.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -679,3 +679,24 @@ Changes applied:
679679
Identical to V23.0 in all other respects. The fix isolates the `tangent_scale` change.
680680

681681
Expected improvement: VAE-B develops diverse directions (ARI_B should be > 0), confirming both VAEs contribute genuinely to the hierarchy signal.
682+
683+
---
684+
685+
## External Validation Result — Cytochrome C Phylogeny (2026-07-17)
686+
687+
**Important caveat on every hierarchy/ARI/Q number above**: they measure whether the loss functions do what they were written to do (`v_3(index)` is handed to the loss as a target, not inferred). `docs/plans/EXTERNAL-VALIDATION-ROADMAP.md` asked whether this buys anything on real, non-synthetic data where the hierarchy is *not* injected into the loss. First real test, run 2026-07-17:
688+
689+
**Setup:** 39 real cytochrome c orthologs (bacteria → human, UniProt Pfam PF00034, aligned to human reference), evaluated against real NCBI taxonomic distance via a Mantel permutation test. Three trained conditions (A: flat Euclidean VAE; B: same architecture as C with all p-adic-specific losses/structure off; C: full p-adic/hyperbolic curriculum, config ≈ `v24.0_tangent_fix.yaml`) plus a zero-model control (`raw_encoding_baseline`: raw hydropathy-encoded sequence distance, no VAE).
690+
691+
**Result — none of the three trained conditions beat doing no training at all:**
692+
693+
| Condition | Spearman vs. real taxonomy |
694+
|---|---|
695+
| raw_encoding_baseline (zero model) | **0.7228** |
696+
| B_hyperbolic_generic | 0.6538 |
697+
| A_euclidean | 0.6285 |
698+
| C_padic | 0.4955 |
699+
700+
**C_padic — the full p-adic curriculum — scored lowest**, despite hitting its own best internal objective (Q=1.943, hierarchy Spearman=0.8185, matching this doc's V21+ ceiling). Consistent with the roadmap's core worry: `v_3(index)` for a windowed-amino-acid ternary index has no causal relationship to species identity, and optimizing hard for it appears to actively pull the embedding away from real taxonomic structure that A/B captured comparatively better.
701+
702+
**Do not read this as "the architecture is broken."** It's one dataset, one coarse 3-symbol amino-acid encoding, one snapshot. It does mean: don't cite this project's internal hierarchy/ARI/Q metrics as evidence of real-world usefulness without this result alongside them. Full methodology, caveats (no species-level holdout yet, 73.9% cross-species index collision rate), and numbers: `docs/plans/EXTERNAL-VALIDATION-ROADMAP.md`. Checkpoints + configs + results JSON: https://huggingface.co/geestaltt/3-adic-vae-cytochrome-c.

0 commit comments

Comments
 (0)