Feature/embedding result cache - #690
Open
ayalaegoz wants to merge 30 commits into
Open
Conversation
Introduce Matryoshka Representation Learning (SBERTMRL) and add HNSW+SQ8 support to the Faiss vector backend. Key changes: - New SBERTMRL embedding (gptcache/embedding/sbert_mrl.py) and factory export (gptcache/embedding/__init__.py) to produce truncated, L2-normalized embeddings (e.g. 768→256). - Extended Faiss backend (gptcache/manager/vector_data/faiss.py) with index_type (flat or hnsw_sq8), HNSW parameters, SQ8 training, tombstone-based deletions for HNSW, over-fetch filtering, and persistence of tombstones. - Vector manager updated to pass index_type and HNSW params (gptcache/manager/vector_data/manager.py). - Added/updated benchmarks for QQP and various Faiss configurations (examples/benchmark/*), and improved error handling and storage-size reporting in existing benchmark. - Fixed ONNX tokenizer usage and safe token_type_ids handling (gptcache/embedding/onnx.py). - Tests added to validate HNSW+SQ8 behavior, tombstones, rebuilds, and persistence (tests/unit_tests/manager/test_local_index.py). - Minor setup.py improvements to open files with UTF-8 encoding. These changes enable a lower-memory, faster Faiss option (HNSW+SQ8) and an MRL truncation workflow for evaluating space/latency trade-offs.
…ring them HNSW+SQ8 does not support structural deletion (IndexHNSWSQ has no reconstruct path), so vectors marked as deleted remain in the graph permanently. The previous rebuild() cleared self._tombstones, which silently allowed those deleted vectors to reappear in search results. Fix: rebuild() now skips tombstone clearance for hnsw_sq8. Tombstones are bounded by cumulative eviction count (~8 bytes/ID), so keeping them is safe even at scale. Also fix flush() to remove a stale .tombstones.npy file when the set is empty, preventing a phantom reload on a fresh index. Regression tests added for: - tombstones survive rebuild (not cleared) - deleted IDs remain filtered from search results after rebuild - tombstone file is written after rebuild+flush and survives reload Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add batch_size to Cache.import_data to support embedding functions that accept lists and return (N, dim) embeddings; adapt SBERTMRL to return (dim,) for single inputs and (N, dim) for batches. Persist embeddings in SQLStorage as float16 (reconstruct to float32 on load) to reduce disk usage ~2x. Update benchmark_qqp to ingest data in configurable batches with progress output. Also update .gitignore to exclude CLAUDE files and .env.
Introduce an exact-match processor, ONNX dynamic export script, and related unit test; update core, adapter, config, embedding, and manager modules to support exact-match/ONNX workflow. Add benchmark/example result files and scripts (including moved smoke benchmarks) and update .gitignore for docs and onnx_dynamic artifacts. This change adds instrumentation and data for evaluating exact-match performance and ONNX-based embeddings.
Introduce a new cost-aware W-TinyLFU eviction policy (CA_W_TINYLFU) implementing EWMA time-decay frequency, Count-Min Sketch + doorkeeper, and lexicographic admission that biases by estimated regeneration cost. Integrate the policy into MemoryCacheEviction factory, wire put/get semantics, and adjust benchmark logic to use eviction.get for hit detection. Add unit tests for policy internals and factory routing, plus a benchmark results JSON. This change provides a reusable, test-covered implementation to prefer expensive-to-regenerate items while preserving frequency dominance.
Introduce a Hash-DoS mitigation in the CostAwareWTinyLFU eviction policy: import random, add _HASHDOS_THRESHOLD (7.0) for EWMA freq, and implement a 1/128 random admission chance for moderately warm candidates that lose the score contest. Also change score comparison to strict greater-than so victims win ties (matching Caffeine's admit behavior). Include explanatory comments and emit evict/add flows accordingly.
Phase 1-4 hardening of the Cost-Aware W-TinyLFU eviction policy: - Adaptive window hill-climber (default off): re-tunes the window<->main boundary to maximize cost-weighted hit rate, with a two-sided gradient probe, temporal hysteresis, and settle-skip to resist noise. - Time-decay refinement: read-time EWMA discount on the frequency score (ages out stale frequency under drift) instead of a wiped accumulator. - Hash-DoS guard now uses the raw (undecayed) sketch estimate so an attacker cannot evade it by pausing to let decay shrink the signal. - Remove dead clean_size parameter/field from CostAwareWTinyLFU and stop forwarding it in MemoryCacheEviction: CA evicts one item per admission contest by construction, so the batch-evict knob never applied. - Extract _build_llm_cost() in adapter.py, de-duplicating the cost-plumbing block shared by adapt()/aadapt(). - Eviction benchmark harness (benchmark_lmsys.py) and multi-seed aggregator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the Faiss vector store with two Product-Quantization index types
alongside the existing hnsw_sq8:
- hnsw_pq: HNSW graph over PQ codes (PQ{m_pq}x8 => m_pq bytes/vector vs
dim bytes/vector for SQ8), a large storage-compression win on the index.
- hnsw_pq_refine: hnsw_pq wrapped in IndexRefineFlat, over-fetching by
k_factor and re-ranking against full-precision vectors to recover the
recall PQ alone gives up.
All hnsw_* variants share the existing tombstone deletion path (PQ has no
structural remove_ids). Adds m_pq / k_factor knobs to _create_index and the
VectorBase factory, buffered train-and-flush before the index is trained,
and unit tests + QQP benchmark coverage for both new types.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use sentence-transformers native truncate_dim/normalize_embeddings in SBERTMRL instead of hand-slicing + re-normalizing in numpy (drops the numpy import). benchmark_qqp.py gains the isolation cell I (MRL/768/Flat) and a per-cell threshold sweep that re-scores the single search pass at multiple cutoffs, so cells can be compared at matched precision rather than one global threshold. Includes the 100k/1m real-encoder frontier results. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a user-facing cost_priority knob in [0,1] that maps onto freq_weight (0 => 16, frequency dominates / max hit rate; 1 => 1, cost equal-footed / max money saved), overriding raw freq_weight when set and validating range. benchmark_lmsys.py gains --cost-priority-sweep to plot the money<->hit-rate tradeoff, plus a tokenizer disallowed_special fix. Includes the paired-seed analysis script and all drift / window / decay / cost-priority result JSONs backing the write-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Force-add the previously-ignored docs/writeup.md (results skeleton with per-section claim+evidence mapping, incl. the paired n=7 cost_priority dial analysis) and docs/walkthrough.md. They span both contributions, so they land as one docs commit rather than split across the storage/eviction ones. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a static-retrieval-mrl-en-v1 operating point (cell J) to the qqp storage benchmark and fold it into frontier.md: same MRL/256/HNSW+PQ index at 9.8x compression, but e2e p95 collapses 97.5ms -> 0.50ms by dropping the transformer forward pass, paid for in recall (-12.9pp TP at matched precision). benchmark_qqp.py gains a `model` override on the MRL encoder factory to select the static model. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a load_wildchat streamer (allenai/WildChat-1M) to benchmark_lmsys.py
as a second eviction dataset with genuine cost tiers, mirroring the LMSYS
loader. Commit the paired-seed evidence: the stationary win_z{11,15}
seeds 3-6 that complete n=7 on LMSYS, and the full WildChat replication
(bench_wildchat/win_z{11,15}_seed0-6).
Paired CA-LRU is positive 42/42 on both datasets despite very different
cost compositions (LMSYS 0.9% expensive vs WildChat 48.8%), so the
stationary cost-weighted win is not an artifact of one dataset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fill the write-up (docs/writeup.md) from skeleton to a full first-draft prose: experimental setup, eviction results (drift/stationary/isolation/ dial/latency), design, background, storage co-contribution, discussion and conclusion. Every empirical claim is traced to a paired.py-verified result. Add web-verified references (DOIs/pages/arXiv) and two figures (request-flow diagram, crossover surface) generated from the JSONs via docs/figures/make_figures.py. Figures are force-added since docs/ is otherwise gitignored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Benchmark CA_W_TINYLFU against GDSF (GreedyDual-Size-Frequency), the
classic cost-aware policy and closest prior art, in the shared harness.
- gdsf.py: pure-stdlib GDSF eviction (H = L + F*C/S, aging L := H_min)
- memory_cache.py: route GDSF/CA through put(objs, costs=costs)
- paired.py: exact Wilcoxon signed-rank + paired-t 95% CI (no scipy)
- writeup.md: four-cell GDSF head-to-head + cache-size/WildChat sweep
Result: CA ties GDSF under drift and sharp stationary skew, and beats it
on cost-weighted hit rate by +5.0pp (7/7, cs100) under flat stationary
skew via the admission filter. The flat-skew win holds across
cs in {25,50,100} and replicates on WildChat (n=5). Data committed under
bench_lmsys/ and bench_wildchat/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mple to GPT-5 Correct stationary-flat CA-vs-GDSF to +4.56pp [7/7] (from +5.05) to match the committed benchmark JSONs, and update the premium tier example from GPT-4 to GPT-5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add gdsf_headtohead() producing fig_gdsf.png, and the z15 seed5 / WildChat z11 seed5 benchmark JSONs backing the n=7 GDSF comparison. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ayalaegoz The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Collaborator
|
Welcome @ayalaegoz! It looks like this is your first PR to zilliztech/GPTCache 🎉 |
…referenced in writeup)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.