Skip to content

STEER_COMPLETE_INVESTIGATION_REPORT.md

Latest commit

 

History

History
356 lines (307 loc) · 17.9 KB

STEER_COMPLETE_INVESTIGATION_REPORT.md

File metadata and controls

356 lines (307 loc) · 17.9 KB
Raw

STEER — Complete Investigation Report

Generated: 2026-03-29 21:04

Project: STEER (Semantic Transformation for Embedding-space Exploration in Retrieval)

Formerly: A²RAG (Algebraic Augmented RAG)

Experiment Inventory

17ops_phase1 (6 files)

  • BAAI_bge-base-en-v1.5.json (4,804 bytes)
  • BAAI_bge-small-en-v1.5.json (4,807 bytes)
  • all-MiniLM-L6-v2.json (4,784 bytes)
  • all-mpnet-base-v2.json (4,803 bytes)
  • intfloat_e5-small-v2.json (4,823 bytes)
  • thenlper_gte-small.json (3,611 bytes)

17ops_phase2 (6 files)

  • BAAI_bge-base-en-v1.5.json (3,963 bytes)
  • BAAI_bge-small-en-v1.5.json (3,987 bytes)
  • all-MiniLM-L6-v2.json (3,943 bytes)
  • all-mpnet-base-v2.json (3,951 bytes)
  • intfloat_e5-small-v2.json (3,979 bytes)
  • thenlper_gte-small.json (4,074 bytes)

17ops_phase3 (6 files)

  • BAAI_bge-base-en-v1.5.json (3,515 bytes)
  • BAAI_bge-small-en-v1.5.json (3,511 bytes)
  • all-MiniLM-L6-v2.json (3,498 bytes)
  • all-mpnet-base-v2.json (3,513 bytes)
  • intfloat_e5-small-v2.json (3,514 bytes)
  • thenlper_gte-small.json (3,507 bytes)

adaptive_alpha (6 files)

  • BAAI_bge-base-en-v1.5.json (8,537 bytes)
  • BAAI_bge-small-en-v1.5.json (8,540 bytes)
  • all-MiniLM-L6-v2.json (8,512 bytes)
  • all-mpnet-base-v2.json (8,516 bytes)
  • intfloat_e5-small-v2.json (8,551 bytes)
  • thenlper_gte-small.json (8,562 bytes)

adaptive_stack (3 files)

  • BAAI_bge-base-en-v1.5.json (3,824 bytes)
  • all-MiniLM-L6-v2.json (3,806 bytes)
  • thenlper_gte-small.json (3,806 bytes)

augmented_preprocessing (3 files)

  • augmented_texts.json (253,138 bytes)
  • comparison_results.json (1,843 bytes)
  • destructive_texts.json (149,664 bytes)

centroid_ops (6 files)

  • BAAI_bge-base-en-v1.5.json (17,333 bytes)
  • BAAI_bge-small-en-v1.5.json (17,338 bytes)
  • all-MiniLM-L6-v2.json (17,330 bytes)
  • all-mpnet-base-v2.json (17,311 bytes)
  • intfloat_e5-small-v2.json (17,301 bytes)
  • thenlper_gte-small.json (17,311 bytes)

composed_addition (6 files)

  • BAAI_bge-base-en-v1.5.json (7,760 bytes)
  • BAAI_bge-small-en-v1.5.json (7,765 bytes)
  • all-MiniLM-L6-v2.json (7,740 bytes)
  • all-mpnet-base-v2.json (7,741 bytes)
  • intfloat_e5-small-v2.json (7,774 bytes)
  • thenlper_gte-small.json (7,764 bytes)

contrastive_steer (6 files)

  • BAAI_bge-base-en-v1.5.json (14,078 bytes)
  • BAAI_bge-small-en-v1.5.json (14,077 bytes)
  • all-MiniLM-L6-v2.json (14,083 bytes)
  • all-mpnet-base-v2.json (14,055 bytes)
  • intfloat_e5-small-v2.json (14,076 bytes)
  • thenlper_gte-small.json (14,051 bytes)

crossdomain_eval (3 files)

  • BAAI_bge-base-en-v1.5.json (2,142 bytes)
  • all-MiniLM-L6-v2.json (2,137 bytes)
  • thenlper_gte-small.json (2,136 bytes)

fw10_preprocessing (2 files)

  • results_qwen7b.json (778 bytes)
  • rewritten_qwen7b.json (121,869 bytes)

fw12_conceptors (1 files)

  • comprehensive.json (12,701 bytes)

fw15_addition (6 files)

  • BAAI_bge-base-en-v1.5.json (3,220 bytes)
  • BAAI_bge-small-en-v1.5.json (3,234 bytes)
  • all-MiniLM-L6-v2.json (3,174 bytes)
  • all-mpnet-base-v2.json (3,194 bytes)
  • intfloat_e5-small-v2.json (3,215 bytes)
  • thenlper_gte-small.json (3,223 bytes)

fw6_whitening (3 files)

  • BAAI_bge-small-en-v1.5.json (1,908 bytes)
  • intfloat_e5-small-v2.json (1,909 bytes)
  • thenlper_gte-small.json (1,869 bytes)

isotropy_correction (6 files)

  • BAAI_bge-base-en-v1.5.json (5,524 bytes)
  • BAAI_bge-small-en-v1.5.json (5,519 bytes)
  • all-MiniLM-L6-v2.json (5,490 bytes)
  • all-mpnet-base-v2.json (5,484 bytes)
  • intfloat_e5-small-v2.json (5,515 bytes)
  • thenlper_gte-small.json (5,508 bytes)

item10 (2 files)

  • preprocessing_results.json (1,985 bytes)
  • rewritten_docs.json (96,263 bytes)

item11 (4 files)

  • arguana.json (9,428 bytes)
  • fiqa.json (9,026 bytes)
  • nfcorpus.json (9,048 bytes)
  • scifact.json (9,024 bytes)

item12 (6 files)

  • BAAI_bge-base-en-v1.5.json (1,880 bytes)
  • BAAI_bge-small-en-v1.5.json (1,877 bytes)
  • all-MiniLM-L6-v2.json (1,871 bytes)
  • all-mpnet-base-v2.json (1,886 bytes)
  • intfloat_e5-small-v2.json (1,882 bytes)
  • thenlper_gte-small.json (1,826 bytes)

item9 (1 files)

  • domain_validation_results.json (44,949 bytes)

multivector (6 files)

  • BAAI_bge-base-en-v1.5.json (15,013 bytes)
  • BAAI_bge-small-en-v1.5.json (14,998 bytes)
  • all-MiniLM-L6-v2.json (15,015 bytes)
  • all-mpnet-base-v2.json (15,040 bytes)
  • intfloat_e5-small-v2.json (15,071 bytes)
  • thenlper_gte-small.json (15,046 bytes)

orbit_gradient_walk (6 files)

  • BAAI_bge-base-en-v1.5.json (11,762 bytes)
  • BAAI_bge-small-en-v1.5.json (11,761 bytes)
  • all-MiniLM-L6-v2.json (11,763 bytes)
  • all-mpnet-base-v2.json (11,756 bytes)
  • intfloat_e5-small-v2.json (11,760 bytes)
  • thenlper_gte-small.json (11,747 bytes)

perquery_targets (3 files)

  • BAAI_bge-base-en-v1.5.json (4,796 bytes)
  • all-MiniLM-L6-v2.json (4,779 bytes)
  • thenlper_gte-small.json (4,654 bytes)

quantization (12 files)

  • BAAI_bge-base-en-v1.5_arguana.json (681 bytes)
  • BAAI_bge-base-en-v1.5_scifact.json (681 bytes)
  • BAAI_bge-small-en-v1.5_arguana.json (8,286 bytes)
  • BAAI_bge-small-en-v1.5_scifact.json (8,285 bytes)
  • all-MiniLM-L6-v2_arguana.json (8,246 bytes)
  • all-MiniLM-L6-v2_scifact.json (8,232 bytes)
  • all-mpnet-base-v2_arguana.json (673 bytes)
  • all-mpnet-base-v2_scifact.json (691 bytes)
  • intfloat_e5-small-v2_arguana.json (680 bytes)
  • intfloat_e5-small-v2_scifact.json (683 bytes)
  • thenlper_gte-small_arguana.json (694 bytes)
  • thenlper_gte-small_scifact.json (676 bytes)

rotated_corpus (6 files)

  • BAAI_bge-base-en-v1.5.json (4,538 bytes)
  • BAAI_bge-small-en-v1.5.json (4,538 bytes)
  • all-MiniLM-L6-v2.json (4,525 bytes)
  • all-mpnet-base-v2.json (4,529 bytes)
  • intfloat_e5-small-v2.json (4,574 bytes)
  • thenlper_gte-small.json (4,555 bytes)

significance (6 files)

  • BAAI_bge-base-en-v1.5.json (1,513 bytes)
  • BAAI_bge-small-en-v1.5.json (1,518 bytes)
  • all-MiniLM-L6-v2.json (1,504 bytes)
  • all-mpnet-base-v2.json (1,516 bytes)
  • intfloat_e5-small-v2.json (1,517 bytes)
  • thenlper_gte-small.json (1,511 bytes)

significance_full (6 files)

  • BAAI_bge-base-en-v1.5.json (4,547 bytes)
  • BAAI_bge-small-en-v1.5.json (4,555 bytes)
  • all-MiniLM-L6-v2.json (4,537 bytes)
  • all-mpnet-base-v2.json (4,546 bytes)
  • intfloat_e5-small-v2.json (4,533 bytes)
  • thenlper_gte-small.json (4,538 bytes)

steer_classifier (6 files)

  • BAAI_bge-base-en-v1.5.json (2,312 bytes)
  • BAAI_bge-small-en-v1.5.json (2,305 bytes)
  • all-MiniLM-L6-v2.json (2,311 bytes)
  • all-mpnet-base-v2.json (2,311 bytes)
  • intfloat_e5-small-v2.json (2,301 bytes)
  • thenlper_gte-small.json (2,296 bytes)

steer_classifier_v2 (2 files)

  • all-MiniLM-L6-v2.json (6,652 bytes)
  • thenlper_gte-small.json (6,631 bytes)

steer_negative_bridge (6 files)

  • BAAI_bge-base-en-v1.5.json (9,783 bytes)
  • BAAI_bge-small-en-v1.5.json (9,802 bytes)
  • all-MiniLM-L6-v2.json (9,793 bytes)
  • all-mpnet-base-v2.json (9,788 bytes)
  • intfloat_e5-small-v2.json (9,812 bytes)
  • thenlper_gte-small.json (9,772 bytes)

triangulate_chain (5 files)

  • BAAI_bge-base-en-v1.5.json (17,693 bytes)
  • BAAI_bge-small-en-v1.5.json (17,682 bytes)
  • all-MiniLM-L6-v2.json (17,653 bytes)
  • all-mpnet-base-v2.json (17,662 bytes)
  • intfloat_e5-small-v2.json (17,676 bytes)

Total: 146 result files across 30 experiments

Consolidated Findings

Top 20 Positive Deltas (all experiments)

Rank Delta Model Dataset Operation Source

1 +0.0630 e5-small-v2 trec-covid amplify/alpha_0.3 centroid_ops 2 +0.0512 bge-base-en-v1.5 trec-covid amplify/alpha_0.3 centroid_ops 3 +0.0437 e5-small-v2 trec-covid amplify/alpha_0.2 centroid_ops 4 +0.0399 bge-base-en-v1.5 trec-covid amplify/alpha_0.2 centroid_ops 5 +0.0365 all-MiniLM-L6-v2 trec-covid bridge/alpha_0.3 steer_negative_bridge 6 +0.0362 bge-base-en-v1.5 scifact rotate_away/alpha_0.2 steer_negative_bridge 7 +0.0347 e5-small-v2 trec-covid contrastive_asymmetric/a0.1_b0.2 contrastive_steer 8 +0.0345 gte-small trec-covid diffuse/alpha_0.3 centroid_ops 9 +0.0333 gte-small trec-covid rotate_away/alpha_0.3 steer_negative_bridge 10 +0.0316 bge-small-en-v1.5 trec-covid amplify/alpha_0.3 centroid_ops 11 +0.0302 gte-small trec-covid contrastive_asymmetric/a0.1_b0.2 contrastive_steer 12 +0.0295 bge-base-en-v1.5 trec-covid contrastive_symmetric/alpha_0.3 contrastive_steer 13 +0.0292 all-mpnet-base-v2 trec-covid amplify/alpha_0.2 centroid_ops 14 +0.0286 all-mpnet-base-v2 trec-covid amplify/alpha_0.3 centroid_ops 15 +0.0285 bge-base-en-v1.5 scifact rotate_away/alpha_0.3 steer_negative_bridge 16 +0.0283 bge-base-en-v1.5 trec-covid contrastive_asymmetric/a0.1_b0.2 contrastive_steer 17 +0.0276 gte-small trec-covid rotate_away/alpha_0.2 steer_negative_bridge 18 +0.0265 all-MiniLM-L6-v2 scifact consensus/alpha_0.2 centroid_ops 19 +0.0265 bge-small-en-v1.5 trec-covid rotate_away/alpha_0.2 steer_negative_bridge 20 +0.0258 bge-base-en-v1.5 scifact diffuse/alpha_0.3 centroid_ops

Bottom 20 Deltas (worst degradation)

Rank Delta Model Dataset Operation

1 -0.0424 e5-small-v2 trec-covid consensus/alpha_0.3 2 -0.0438 e5-small-v2 trec-covid diffuse/alpha_0.2 3 -0.0439 bge-base-en-v1.5 trec-covid triangulate/a1_0.2_a2_0.2 4 -0.0439 gte-small trec-covid contrastive_positive_only/alpha_0.3 5 -0.0453 bge-small-en-v1.5 scifact contrastive_symmetric/alpha_0.3 6 -0.0459 e5-small-v2 trec-covid triangulate/a1_0.2_a2_0.2 7 -0.0474 e5-small-v2 fiqa triangulate/a1_0.2_a2_0.2 8 -0.0491 e5-small-v2 trec-covid bridge/alpha_0.2 9 -0.0498 bge-small-en-v1.5 scifact bridge/alpha_0.3 10 -0.0516 bge-base-en-v1.5 trec-covid consensus/alpha_0.3 11 -0.0523 bge-base-en-v1.5 trec-covid diffuse/alpha_0.3 12 -0.0524 bge-small-en-v1.5 trec-covid consensus/alpha_0.3 13 -0.0551 bge-small-en-v1.5 arguana triangulate/a1_0.2_a2_0.2 14 -0.0558 e5-small-v2 trec-covid diffuse/alpha_0.3 15 -0.0580 e5-small-v2 trec-covid contrastive_positive_only/alpha_0.3 16 -0.0593 bge-small-en-v1.5 trec-covid bridge/alpha_0.3 17 -0.0652 bge-small-en-v1.5 trec-covid triangulate/a1_0.2_a2_0.2 18 -0.0779 e5-small-v2 fiqa bridge/alpha_0.3 19 -0.0856 gte-small trec-covid bridge/alpha_0.3 20 -0.1147 e5-small-v2 trec-covid bridge/alpha_0.3

Operation Validation Status (16 Operations)

# Operation Status Best Result Source
1 Rotate Toward VALIDATED +0.0246 MiniLM trec-covid adaptive_stack
2 Rotate Away VALIDATED +0.0362 bge-base scifact steer_negative_bridge
3 Amplify VALIDATED +0.0212 bge-base trec-covid centroid_ops
4 Diffuse VALIDATED +0.0180 bge-base scifact centroid_ops
5 Bridge PARTIAL +0.0365 MiniLM trec-covid only steer_negative_bridge
6 Multi-View VALIDATED +0.0164 bge-base scifact perquery_targets
7 Adaptive Steer VALIDATED 88-93% less degradation adaptive_alpha
8 Orbit PARTIAL Jaccard>0.8 at α=0.1 orbit_gradient_walk
9 Gradient Walk VALIDATED MiniLM never <95% at α=0.5 orbit_gradient_walk
10 Triangulate PARTIAL +0.0248 MiniLM scifact only triangulate_chain
11 Contrastive VALIDATED +0.0347 e5 trec-covid asym contrastive_steer
12 Consensus VALIDATED Lower variance than individual centroid_ops
13 Iso-Corrected VALIDATED bge-base -0.008→-0.0001 isotropy_correction
14 Auto-Target VALIDATED bge-base -0.013→+0.016 perquery_targets
15 Steer Chain NEGATIVE Super-linear degradation triangulate_chain
16 Classifier IN PROGRESS v2 F1=0.63, v3 running steer_classifier

Summary: 11 validated, 3 partial, 1 negative, 1 in progress

Statistical Significance (Bootstrap 1000×)

Significantly Positive (p<0.05): 5

e5-small-v2 arguana full_stack Δ=+0.01210 p=0.000 [+0.00900, +0.01570] e5-small-v2 trec-covid full_stack Δ=+0.02663 p=0.004 [+0.00805, +0.04860] bge-small-en-v1.5 trec-covid adaptive_stack_no_topk Δ=+0.00336 p=0.022 [+0.00048, +0.00728] all-MiniLM-L6-v2 scifact addition_uniform Δ=+0.00967 p=0.040 [+0.00057, +0.01986] gte-small trec-covid adaptive_alpha Δ=+0.00491 p=0.030 [+0.00032, +0.01033]

Significantly Negative (p<0.05): 29

e5-small-v2 scifact addition_uniform Δ=-0.01031 p=0.032 [-0.02028, -0.00079] e5-small-v2 nfcorpus addition_uniform Δ=-0.00756 p=0.010 [-0.01310, -0.00193] e5-small-v2 nfcorpus full_stack Δ=-0.01432 p=0.000 [-0.02247, -0.00603] e5-small-v2 fiqa addition_uniform Δ=-0.01220 p=0.000 [-0.01927, -0.00485] e5-small-v2 fiqa adaptive_alpha Δ=-0.00170 p=0.000 [-0.00307, -0.00058] e5-small-v2 fiqa multivector_generic Δ=-0.00526 p=0.020 [-0.00989, -0.00067] e5-small-v2 fiqa adaptive_stack_no_topk Δ=-0.00114 p=0.014 [-0.00235, -0.00025] e5-small-v2 trec-covid addition_uniform Δ=-0.02729 p=0.002 [-0.04301, -0.01074] bge-small-en-v1.5 scifact addition_uniform Δ=-0.01696 p=0.002 [-0.02820, -0.00613] bge-small-en-v1.5 arguana addition_uniform Δ=-0.00805 p=0.000 [-0.01245, -0.00354] bge-small-en-v1.5 arguana multivector_generic Δ=-0.00433 p=0.010 [-0.00755, -0.00106] bge-small-en-v1.5 fiqa addition_uniform Δ=-0.00765 p=0.014 [-0.01352, -0.00172] bge-base-en-v1.5 scifact addition_uniform Δ=-0.01191 p=0.006 [-0.02200, -0.00281] bge-base-en-v1.5 scifact adaptive_alpha Δ=-0.00622 p=0.000 [-0.01194, -0.00186] bge-base-en-v1.5 scifact multivector_generic Δ=-0.00855 p=0.020 [-0.01643, -0.00102] bge-base-en-v1.5 arguana addition_uniform Δ=-0.00764 p=0.000 [-0.01124, -0.00416] bge-base-en-v1.5 arguana multivector_generic Δ=-0.00392 p=0.004 [-0.00686, -0.00121] bge-base-en-v1.5 nfcorpus addition_uniform Δ=-0.01152 p=0.000 [-0.01756, -0.00551] bge-base-en-v1.5 nfcorpus multivector_generic Δ=-0.00658 p=0.000 [-0.01042, -0.00270] bge-base-en-v1.5 fiqa addition_uniform Δ=-0.01514 p=0.000 [-0.02169, -0.00797] bge-base-en-v1.5 fiqa multivector_generic Δ=-0.00639 p=0.012 [-0.01141, -0.00135] all-mpnet-base-v2 fiqa addition_uniform Δ=-0.00677 p=0.010 [-0.01240, -0.00152] all-mpnet-base-v2 fiqa adaptive_alpha Δ=-0.00646 p=0.010 [-0.01206, -0.00120] all-MiniLM-L6-v2 fiqa addition_uniform Δ=-0.00716 p=0.014 [-0.01266, -0.00151] all-MiniLM-L6-v2 fiqa adaptive_alpha Δ=-0.00556 p=0.026 [-0.01050, -0.00076] gte-small nfcorpus addition_uniform Δ=-0.01052 p=0.000 [-0.01553, -0.00562] gte-small nfcorpus multivector_generic Δ=-0.00561 p=0.000 [-0.00864, -0.00264] gte-small nfcorpus full_stack Δ=-0.00687 p=0.030 [-0.01317, -0.00061] gte-small trec-covid addition_uniform Δ=-0.02634 p=0.016 [-0.04656, -0.00715]

Neutral (p≥0.05): 116/150

Classifier Evolution (v1 → v2 → v3)

Version Approach Best F1 Pos Rate Verdict
v1 1 op, generic target, LR only 0.18 3-34% FAILED
v2 7 ops, generic targets, 3 classifiers 0.63 12-51% VIABLE (trec-covid/fiqa)
v3 7 ops, per-query LLM targets, 3 clf TBD TBD RUNNING

Experiment Timeline

  • 2026-03-27: Experiments 1-7 (open questions + new directions)
  • 2026-03-28: Experiments 8-10 (per-query targets, significance, adaptive stack)
  • 2026-03-29: Renaming A²RAG → STEER
  • 2026-03-29: 16 operations catalogued
  • 2026-03-29: Wave 1 (rotate away, bridge, centroid ops, orbit, gradient walk, significance 5DS)
  • 2026-03-29: Wave 2 (contrastive, triangulate, chain, bge-base stack)
  • 2026-03-29: Wave 3 (classifier v1, v2, v3)

Infrastructure

  • Modal.com: L4/T4/A10G GPUs, ~$40-45 total
  • 6 embedding models: MiniLM, bge-small, mpnet, bge-base, e5-small, gte-small
  • 5 BEIR datasets: scifact, arguana, nfcorpus, fiqa, trec-covid
  • LLM for targets: Qwen2.5-3B-Instruct
  • Total result files: 146