From d405e2b344de2558f9f94ac1acc9acdce4339ee5 Mon Sep 17 00:00:00 2001 From: Jim McKeeth Date: Thu, 18 Jun 2026 23:50:33 -0600 Subject: [PATCH 1/3] feature: updated alternative embedding options after testing --- EMBEDDINGS.md | 25 +++++++++-------- scripts/MTEB-RANKINGS.md | 55 +++++++++++++++++++++++++++++++++++++ scripts/find_best_models.py | 8 +++++- 3 files changed, 76 insertions(+), 12 deletions(-) create mode 100644 scripts/MTEB-RANKINGS.md diff --git a/EMBEDDINGS.md b/EMBEDDINGS.md index 42cd1dd..bccdc72 100644 --- a/EMBEDDINGS.md +++ b/EMBEDDINGS.md @@ -68,14 +68,15 @@ This option runs embedding models directly on your machine using the library. ### Recommended Models -These are based on MTEB [datasets](https://huggingface.co/datasets/mteb/results) as of 13-Jun-2026. +These are based on MTEB [datasets](https://huggingface.co/datasets/mteb/results) as of 15-Jun-2026. All listed models have been verified to work with the `sentence-transformers` provider in `cocoindex-code`. | Tier | Model | Params | Code Score | Best For | | :--- | :--- | :--- | :--- | :--- | -| **Micro** | [`Snowflake/arctic-embed-xs`](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs) | 22M | 0.67 | Old CPUs, minimal RAM usage. | -| **Small** | [`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2) | 97M | 0.80 | Modern laptops, multilingual code. | -| **Medium** | [`jinaai/jina-embeddings-v5-text-nano`](https://huggingface.co/jinaai/jina-embeddings-v5-text-nano) | 239M | **0.90** | **Performance sweet spot.** BERT-based (Fast). | -| **High** | [`geevec-ai/geevec-embeddings-1.0-lite`](https://huggingface.co/geevec-ai/geevec-embeddings-1.0-lite) | 366M | **0.92** | Maximum local accuracy (needs GPU for speed). | +| **Default** | [`Snowflake/arctic-embed-xs`](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs) | 22M | 0.67 | Default | +| **Micro** | [`lightonai/LateOn-Code-edge`](https://huggingface.co/lightonai/LateOn-Code-edge) | 17M | 0.82 | **Efficiency King.** Incredible code performance for its size. | +| **Small** | [`lightonai/LateOn-Code`](https://huggingface.co/lightonai/LateOn-Code) | 149M | 0.85 | Great balance of speed and accuracy on modern laptops. | +| **Medium** | [`microsoft/harrier-oss-v1-270m`](https://huggingface.co/microsoft/harrier-oss-v1-270m) | 270M | **0.90** | **Performance sweet spot.** High accuracy, runs well on CPUs. | +| **Multi** | [`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2) | 97M | 0.80 | Multilingual codebases (e.g. Code + Docs in different languages). | #### Other Model Options @@ -190,8 +191,8 @@ envs: ## Choosing Based on Your Content -- **Heavy Source Code**: Use **Jina v5 Nano** (Local) or **Voyage 4 Large** (Cloud). Both score >0.90 on code search benchmarks. -- **Large Documentation / Files**: Models with large context windows (8k+ tokens) like **Jina v5** (32k) or **OpenAI v3 Large** (8k). +- **Heavy Source Code**: Use **LateOn-Code** (Micro/Small) or **Harrier 270m** (Medium). Both score >0.85 on code search benchmarks. +- **Large Documentation / Files**: Models with large context windows like **Voyage 4 Large** (Cloud) or **OpenAI v3 Large** (8k). - **Multilingual Projects**: **Granite 97m** (Small Local) or **Cohere Multilingual v3** (Cloud). ### Fine-Tuning with `indexing_params` and `query_params` @@ -210,16 +211,18 @@ embedding: input_type: query ``` -**Example for Sentence-Transformers (Jina):** +**Example for Sentence-Transformers (Harrier):** ```yaml embedding: provider: sentence-transformers - model: jinaai/jina-embeddings-v5-text-nano + model: microsoft/harrier-oss-v1-270m + # Most encoder-only models don't require explicit prompts, + # but some (like Nomic or BGE) do: indexing_params: - prompt_name: retrieval.passage + prompt_name: null query_params: - prompt_name: retrieval.query + prompt_name: null ``` --- diff --git a/scripts/MTEB-RANKINGS.md b/scripts/MTEB-RANKINGS.md new file mode 100644 index 0000000..f4d12c4 --- /dev/null +++ b/scripts/MTEB-RANKINGS.md @@ -0,0 +1,55 @@ +# MTEB Model Discovery Report + +> **Data Freshness**: MTEB results dataset last updated on `2026-06-15`. + +## Top Embedding Models for Code Search + +### Tier: Micro (< 50M) + +| Model | Code Search Score | General Retrieval Score | Params (M) | +|-------------------------------------------|---------------------|---------------------------|--------------| +| lightonai/LateOn-Code-edge | 0.816549 | nan | 17 | +| lightonai/LateOn-Code-edge-pretrain | 0.791693 | nan | 16.798 | +| thenlper/gte-small | 0.781565 | 0.479423 | 33 | +| avsolatorio/GIST-small-Embedding-v0 | 0.772521 | 0.480646 | 33.36 | +| avsolatorio/NoInstruct-small-Embedding-v0 | 0.770071 | 0.488884 | 33.36 | + +### Tier: Small (< 150M) + +| Model | Code Search Score | General Retrieval Score | Params (M) | +|---------------------------------------------------|---------------------|---------------------------|--------------| +| lightonai/LateOn-Code | 0.851318 | nan | 149 | +| lightonai/LateOn-Code-pretrain | 0.832574 | nan | 149.016 | +| ibm-granite/granite-embedding-97m-multilingual-r2 | 0.799971 | 0.446515 | 97 | +| avsolatorio/GIST-Embedding-v0 | 0.78981 | 0.503411 | 109.482 | +| thenlper/gte-base | 0.789403 | 0.496155 | 109 | + +### Tier: Medium (< 500M) + +| Model | Code Search Score | General Retrieval Score | Params (M) | +|-------------------------------------------|---------------------|---------------------------|--------------| +| geevec-ai/geevec-embeddings-1.0-lite | 0.92365 | 0.53474 | 366 | +| jinaai/jina-embeddings-v5-text-nano | 0.90384 | 0.535934 | 239 | +| microsoft/harrier-oss-v1-270m | 0.89605 | 0.425505 | 270 | +| Shuu12121/CodeSearch-ModernBERT-Crow-Plus | 0.892957 | nan | 151.668 | +| codefuse-ai/F2LLM-v2-330M | 0.842182 | 0.475202 | 334 | + +### Tier: Large (> 500M) + +| Model | Code Search Score | General Retrieval Score | Params (M) | +|------------------------------------------|---------------------|---------------------------|--------------| +| voyageai/voyage-4-large | 0.97726 | nan | nan | +| voyageai/voyage-4-large (embed_dim=2048) | 0.97719 | nan | nan | +| google/gemini-embedding-2-preview | 0.972905 | nan | nan | +| microsoft/harrier-oss-v1-27b | 0.96994 | 0.483455 | 27009.3 | +| Octen/Octen-Embedding-8B-INT8 | 0.967965 | nan | 7567.3 | + +--- + +## How to Regenerate this Report + +This report was generated using the `find_best_models.py` script. To update it with the latest live data from MTEB, run: + +```bash +uv run scripts/find_best_models.py --clear-cache --output MTEB-RANKINGS.md +``` diff --git a/scripts/find_best_models.py b/scripts/find_best_models.py index 8b89fbb..ef425f0 100644 --- a/scripts/find_best_models.py +++ b/scripts/find_best_models.py @@ -163,6 +163,12 @@ def main(): "jinaai/jina-embeddings-v5-text-nano": 239, "ibm-granite/granite-embedding-97m-multilingual-r2": 97, "geevec-ai/geevec-embeddings-1.0-lite": 366, + "lightonai/LateOn-Code-edge": 17, + "lightonai/LateOn-Code": 149, + "microsoft/harrier-oss-v1-270m": 270, + "thenlper/gte-small": 33, + "thenlper/gte-base": 109, + "codefuse-ai/F2LLM-v2-330M": 334, } def categorize(size): @@ -178,7 +184,7 @@ def categorize(size): print("Analyzing top candidates to determine hardware tiers...", file=sys.stderr) results["max_score"] = results[["score_general", "score_code"]].max(axis=1) - results = results.sort_values(by="max_score", ascending=False).head(500) + results = results.sort_values(by="max_score", ascending=False).head(1000) results["size_mb"] = results["model_name"].map(known_sizes) From 4e8ecd9adfe6a6c4abe0f7f09eff85a90c5674f7 Mon Sep 17 00:00:00 2001 From: Jim McKeeth Date: Tue, 30 Jun 2026 23:47:32 -0600 Subject: [PATCH 2/3] More detailed stats, removed incompatible models, and some other minor issues. There are a number of really good options like `lightonai/LateOn-Code-edge` --- CLAUDE.md | 62 +++++++++- EMBEDDINGS.md | 43 ++++--- scripts/MTEB-RANKINGS.md | 88 ++++++++++----- scripts/find_best_models.py | 220 +++++++++++++++++++++++++++++++----- 4 files changed, 329 insertions(+), 84 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8b74612..e902766 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,4 +1,8 @@ -This is built on top of [CocoIndex v1](https://cocoindex.io/docs-v1/llms.txt). +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +This is an AST-based semantic code search tool built on top of [CocoIndex v1](https://cocoindex.io/docs-v1/llms.txt). ## Build and Test Commands @@ -6,10 +10,60 @@ This is built on top of [CocoIndex v1](https://cocoindex.io/docs-v1/llms.txt). This project uses [uv](https://docs.astral.sh/uv/) for project management. ```bash -uv run mypy . # Type check Python code -uv run pytest tests/ # Run Python tests +uv sync # Install all dev dependencies +uv run pytest tests/ # Run all tests +uv run pytest tests/test_settings.py # Run a single test file +uv run pytest tests/ -k test_name # Run tests matching a name +uv run mypy . # Type check Python code +uv run ruff check . # Lint +uv run ruff format . # Format +uv run prek run --all-files # Full CI suite (lint, format, mypy, pytest) ``` +Docker E2E tests are excluded by default (need `pytest -m docker_e2e` to run them). + + +## Architecture + +### Process Model + +The tool runs as a **long-lived background daemon** (`daemon.py`) that accepts connections over a Unix socket (Windows: named pipe). The `ccc` CLI and MCP server communicate with it via short-lived **per-request connections** (`client.py`): connect → handshake → send one request → receive response(s) → close. + +The daemon is auto-started by `client.py` when not running. It is restarted automatically when the `global_settings.yml` mtime changes (version bump or settings edit), detected via the `HandshakeResponse.global_settings_mtime_us` field. + +### Key Layers + +**`protocol.py`** — All IPC message types as `msgspec.Struct` tagged unions, serialized as msgpack. Every request type, response type, and streaming wrapper lives here. The `Request` and `Response` type aliases are the union types used by the decoder. + +**`daemon.py`** — The daemon entry point (`run_daemon()`). Holds a `ProjectRegistry` that lazily creates and caches `Project` instances keyed by project root. Each connection is handled by `handle_connection()` running as an asyncio task. Dispatches to project operations or daemon-level operations (`_dispatch()`). + +**`project.py`** — `Project` wraps a CocoIndex `Environment` + `App`. Manages an `asyncio.Lock` so only one indexing run is active at a time, and an `asyncio.Event` so search can wait for the initial index pass. + +**`indexer.py`** — Defines the CocoIndex app (`process_file`): reads files from LocalFS, detects language, applies custom chunkers or the default `RecursiveSplitter`, embeds chunks, and stores them in SQLite via `sqlite-vec`. This is the `@coco.fn(memo=True)` function that CocoIndex tracks for incremental re-indexing. + +**`shared.py`** — CocoIndex `ContextKey` definitions (`EMBEDDER`, `SQLITE_DB`, `CODEBASE_DIR`, `INDEXING_EMBED_PARAMS`, `QUERY_EMBED_PARAMS`), the `CodeChunk` dataclass (SQLite schema), and the `create_embedder()` factory. + +**`settings.py`** — YAML config loading for both global (`~/.cocoindex_code/global_settings.yml`) and per-project (`.cocoindex_code/settings.yml`) settings. Also provides all path helpers (`cocoindex_db_path`, `target_sqlite_db_path`, `resolve_db_dir`, etc.) and host-path-mapping logic for Docker. + +**`server.py`** — FastMCP wrapper that delegates `search` calls to the daemon client. Also contains the legacy `main()` entry point (`cocoindex-code` command) for backward-compatible env-var-based configuration. + +**`cli.py`** — Typer app with `ccc` subcommands (`init`, `index`, `search`, `grep`, `status`, `reset`, `doctor`, `mcp`, `daemon status/restart/stop`). CLI logic only — delegates to `client.py` or `grep.py`. + +**`grep.py`** — Structural code search (`ccc grep`) using CocoIndex's `code_match`. Runs entirely locally with no daemon or index. Matches in parallel, streams results per file. + +**`chunking.py`** — Public `Chunk` type and the `CHUNKER_REGISTRY` context key. Custom chunkers in `settings.yml` are resolved at daemon startup and injected via this context. + +### Embedding Model Flow + +The embedder is created once at daemon startup (`create_embedder()` in `shared.py`) and stored in the `ProjectRegistry`. Two param dicts (`indexing_params`, `query_params`) are threaded through CocoIndex context keys so asymmetric models (Cohere, Voyage, nomic-embed) can use different kwargs for document vs. query embedding. + +Two install flavors: `[full]` bundles `sentence-transformers` (local inference); slim uses only LiteLLM (cloud APIs). The `PacedLiteLLMEmbedder` in `litellm_embedder.py` adds rate-limiting between requests. + +### Settings and Project Discovery + +`find_project_root()` walks up from CWD looking for `.cocoindex_code/settings.yml`. Global settings at `~/.cocoindex_code/global_settings.yml` (location overridable via `COCOINDEX_CODE_DIR`). The `COCOINDEX_CODE_DB_PATH_MAPPING` env var redirects database files to a different directory (used in Docker to avoid LMDB on bind mounts). + + ## Code Conventions ### Internal vs External Modules @@ -32,7 +86,7 @@ We distinguish between **internal modules** (under packages with `_` prefix, e.g * Standard library and internal imports don't need underscore prefix * Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar) -### General principles (also covered by `/review-changes`) +### General principles - **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed. - **Specific types over `Any`.** When a value enters as a weaker form (`str`, `Any`), convert to the strong type at the earliest point. Don't propagate the weak form. diff --git a/EMBEDDINGS.md b/EMBEDDINGS.md index bccdc72..cfb63fe 100644 --- a/EMBEDDINGS.md +++ b/EMBEDDINGS.md @@ -31,7 +31,7 @@ ### Speed & Latency -- **Local Sentence-Transformers**: Typically the **fastest** option for small-to-medium models. Because it runs directly inside the `cocoindex-code` process, it avoids the network latency of Cloud APIs and the communication overhead of Local Servers (Ollama). +- **Local Sentence-Transformers**: Typically the **fastest** option for small-to-medium **encoder** models. Because it runs directly inside the `cocoindex-code` process, it avoids the network latency of Cloud APIs and the communication overhead of Local Servers (Ollama). Decoder-based models (e.g. `harrier-oss-v1`) are also in-process, but process tokens sequentially — expect 3–10× slower indexing per chunk on CPU unless you have a GPU. - **Local Servers (Ollama)**: Ideal for running **heavy models** (like `mxbai-embed-large`) on a GPU. While it has slight overhead compared to in-process execution, it is much faster than running large models on a CPU. - **Cloud APIs**: Slower per-request due to network latency, but highly parallel. Best for the initial indexing of massive repositories. @@ -45,10 +45,14 @@ In `cocoindex-code`, this matters less than you might expect due to our **Langua - **Target Size**: While respecting boundaries, it targets a chunk size of **~1,000 characters** (~300 tokens). - **Compatibility**: This hybrid approach ensures code snippets are contextually coherent while remaining small enough to fit perfectly within even the smallest 512-token context windows. -### CPU vs. GPU +### CPU vs. GPU and Architecture -- The default `xs` model is optimized for **CPUs**; you likely won't see a benefit from a GPU. -- For `medium` or `large` models, a GPU is highly recommended. If you have one, you can tell Sentence-Transformers to use it by adding `device: cuda` (or `mps` for Mac) to your `global_settings.yml`. +Indexing speed depends on **both parameter count and model architecture**: + +- **Encoder models** (BERT/ModernBERT-based, e.g. Snowflake, LateOn-Code, CodeSearch-ModernBERT): Process all tokens in parallel — fast on any modern CPU. The default `xs` and all models marked **Fast** or **Very Fast** in the table below are encoder-based. +- **Decoder models** (LLM-based, e.g. `harrier-oss-v1`): Process tokens sequentially, making them **3–10× slower** than an encoder of comparable parameter count on CPU. A GPU is strongly recommended to achieve acceptable indexing throughput with these models. + +For encoder models at medium or large sizes, a GPU will still accelerate indexing. Add `device: cuda` (or `mps` on Mac) to `global_settings.yml`. --- @@ -70,13 +74,14 @@ This option runs embedding models directly on your machine using the library. These are based on MTEB [datasets](https://huggingface.co/datasets/mteb/results) as of 15-Jun-2026. All listed models have been verified to work with the `sentence-transformers` provider in `cocoindex-code`. -| Tier | Model | Params | Code Score | Best For | -| :--- | :--- | :--- | :--- | :--- | -| **Default** | [`Snowflake/arctic-embed-xs`](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs) | 22M | 0.67 | Default | -| **Micro** | [`lightonai/LateOn-Code-edge`](https://huggingface.co/lightonai/LateOn-Code-edge) | 17M | 0.82 | **Efficiency King.** Incredible code performance for its size. | -| **Small** | [`lightonai/LateOn-Code`](https://huggingface.co/lightonai/LateOn-Code) | 149M | 0.85 | Great balance of speed and accuracy on modern laptops. | -| **Medium** | [`microsoft/harrier-oss-v1-270m`](https://huggingface.co/microsoft/harrier-oss-v1-270m) | 270M | **0.90** | **Performance sweet spot.** High accuracy, runs well on CPUs. | -| **Multi** | [`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2) | 97M | 0.80 | Multilingual codebases (e.g. Code + Docs in different languages). | +| Tier | Model | Params | Dims | Code Score | Arch | CPU Speed | Best For | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| **Default** | [`Snowflake/arctic-embed-xs`](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs) | 22M | 384 | 0.67 | Enc | Very Fast | Smallest, most compatible default | +| **Micro** | [`lightonai/LateOn-Code-edge`](https://huggingface.co/lightonai/LateOn-Code-edge) | 17M | 256 | 0.82 | Enc | Crazy Fast (231 cps) | **Efficiency King.** Incredible code performance for its size. | +| **Small** | [`lightonai/LateOn-Code`](https://huggingface.co/lightonai/LateOn-Code) | 149M | 768 | 0.85 | Enc | Fast (7 cps) | Great balance of speed and accuracy on modern laptops. | +| **Medium (Encoder)** | [`Shuu12121/CodeSearch-ModernBERT-Crow-Plus`](https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Crow-Plus) | 152M | | 0.89 | Enc | Fast | High accuracy with encoder speed; **best for CPU-only indexing at this tier.** | +| **Larger (Decoder)** | [`microsoft/harrier-oss-v1-270m`](https://huggingface.co/microsoft/harrier-oss-v1-270m) | 270M | 640 | **0.90** | *Dec* | Slow | Highest local accuracy; **GPU** *strongly recommended* for acceptable speed. | +| **Multi-Lingual** | [`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2) | 97M | 384 | 0.80 | Enc | Fast | Multilingual codebases (e.g. Code + Docs in different languages). | #### Other Model Options @@ -90,7 +95,7 @@ The default of `Snowflake/arctic-embed-xs` is a good choice in most situations, 4. Enable the **Sentence-Transformers Compatible** toggle. 5. Adjust **Model Size** to fit your hardware (e.g., `< 500M` for CPUs). - **Compatibility**: Look for **Bi-encoders** with a fixed dimension size (e.g., 384, 768, 1024). Avoid "Late Interaction" (ColBERT) or "Cross-Encoders". -- **Architecture**: **Encoder** models (BERT-based) are much faster on CPUs than **Decoder** models (LLM-based). +- **Architecture**: **Encoder** models (BERT/ModernBERT-based) are much faster on CPUs than **Decoder** models (LLM-based). A decoder model of the same parameter count can be 3–10× slower per chunk due to sequential token processing. Prefer encoders for CPU-only workloads. ### Installation & Configuration @@ -101,7 +106,7 @@ Example `global_settings.yml`: ```yaml embedding: provider: sentence-transformers - model: jinaai/jina-embeddings-v5-text-nano + model: lightonai/LateOn-Code-edge device: cpu # Use 'cuda' or 'mps' if you have a GPU ``` @@ -191,7 +196,8 @@ envs: ## Choosing Based on Your Content -- **Heavy Source Code**: Use **LateOn-Code** (Micro/Small) or **Harrier 270m** (Medium). Both score >0.85 on code search benchmarks. +- **Heavy Source Code, CPU-only**: Use **LateOn-Code** (Micro/Small) or **CodeSearch-ModernBERT-Crow-Plus** (Medium Encoder). Both are encoder-based and score >0.85 on code search benchmarks with good CPU throughput. +- **Heavy Source Code, GPU available**: **Harrier 270m** (Medium Decoder, 0.90) gives the highest local accuracy and is much faster with a GPU. - **Large Documentation / Files**: Models with large context windows like **Voyage 4 Large** (Cloud) or **OpenAI v3 Large** (8k). - **Multilingual Projects**: **Granite 97m** (Small Local) or **Cohere Multilingual v3** (Cloud). @@ -211,14 +217,15 @@ embedding: input_type: query ``` -**Example for Sentence-Transformers (Harrier):** +**Example for Sentence-Transformers (symmetric models — Harrier, CodeSearch-ModernBERT, etc.):** + +Most local models don't require asymmetric prompts. If `ccc init` doesn't auto-detect defaults +for your model, an explicit `null` disables any inherited prompt: ```yaml embedding: provider: sentence-transformers - model: microsoft/harrier-oss-v1-270m - # Most encoder-only models don't require explicit prompts, - # but some (like Nomic or BGE) do: + model: microsoft/harrier-oss-v1-270m # or Shuu12121/CodeSearch-ModernBERT-Crow-Plus indexing_params: prompt_name: null query_params: diff --git a/scripts/MTEB-RANKINGS.md b/scripts/MTEB-RANKINGS.md index f4d12c4..6298e89 100644 --- a/scripts/MTEB-RANKINGS.md +++ b/scripts/MTEB-RANKINGS.md @@ -1,55 +1,81 @@ # MTEB Model Discovery Report -> **Data Freshness**: MTEB results dataset last updated on `2026-06-15`. +> **Data Freshness**: MTEB results dataset last updated on `2026-06-23`. ## Top Embedding Models for Code Search +> **Speed note**: CPU speed is estimated from parameter count and architecture. **Encoder** models (BERT/ModernBERT-based) process tokens in parallel and are significantly faster on CPU than **Decoder** models (LLM-based), which process tokens sequentially. A decoder model of the same parameter count can be 3–10× slower on CPU. + ### Tier: Micro (< 50M) +| Model | Code Score | General Score | Params (M) | Architecture | CPU Speed | +|-------------------------------------------|--------------|-----------------|--------------|----------------|-------------| +| lightonai/LateOn-Code-edge | 0.816549 | nan | 17 | Encoder | Very Fast | +| lightonai/LateOn-Code-edge-pretrain | 0.791693 | nan | 16.798 | Encoder | Very Fast | +| thenlper/gte-small | 0.781565 | 0.479423 | 33 | Encoder | Very Fast | +| avsolatorio/GIST-small-Embedding-v0 | 0.772521 | 0.480646 | 33.36 | Encoder | Very Fast | +| avsolatorio/NoInstruct-small-Embedding-v0 | 0.770071 | 0.488884 | 33.36 | Encoder | Very Fast | +| abhinand/MedEmbed-small-v0.1 | 0.766076 | 0.52863 | 33.36 | Encoder | Very Fast | +| BAAI/bge-small-en-v1.5 | 0.75267 | 0.514409 | 33 | Encoder | Very Fast | +| Snowflake/snowflake-arctic-embed-s | 0.672949 | 0.493245 | 33 | Encoder | Very Fast | -| Model | Code Search Score | General Retrieval Score | Params (M) | -|-------------------------------------------|---------------------|---------------------------|--------------| -| lightonai/LateOn-Code-edge | 0.816549 | nan | 17 | -| lightonai/LateOn-Code-edge-pretrain | 0.791693 | nan | 16.798 | -| thenlper/gte-small | 0.781565 | 0.479423 | 33 | -| avsolatorio/GIST-small-Embedding-v0 | 0.772521 | 0.480646 | 33.36 | -| avsolatorio/NoInstruct-small-Embedding-v0 | 0.770071 | 0.488884 | 33.36 | ### Tier: Small (< 150M) +| Model | Code Score | General Score | Params (M) | Architecture | CPU Speed | +|---------------------------------------------------|--------------|-----------------|--------------|----------------|-------------| +| lightonai/LateOn-Code | 0.851318 | nan | 149 | Encoder | Fast | +| lightonai/LateOn-Code-pretrain | 0.832574 | nan | 149.016 | Encoder | Fast | +| ibm-granite/granite-embedding-97m-multilingual-r2 | 0.799971 | 0.446515 | 97 | Encoder | Fast | +| avsolatorio/GIST-Embedding-v0 | 0.78981 | 0.503411 | 109.482 | Encoder | Fast | +| thenlper/gte-base | 0.789403 | 0.496155 | 109 | Encoder | Fast | +| ibm-granite/granite-embedding-english-r2 | 0.773404 | 0.501664 | 149.014 | Unknown | Unknown | +| BAAI/bge-base-en-v1.5 | 0.767726 | 0.531966 | 109 | Encoder | Fast | +| nomic-ai/nomic-embed-text-v1.5 | 0.716379 | 0.49881 | 137 | Encoder | Fast | -| Model | Code Search Score | General Retrieval Score | Params (M) | -|---------------------------------------------------|---------------------|---------------------------|--------------| -| lightonai/LateOn-Code | 0.851318 | nan | 149 | -| lightonai/LateOn-Code-pretrain | 0.832574 | nan | 149.016 | -| ibm-granite/granite-embedding-97m-multilingual-r2 | 0.799971 | 0.446515 | 97 | -| avsolatorio/GIST-Embedding-v0 | 0.78981 | 0.503411 | 109.482 | -| thenlper/gte-base | 0.789403 | 0.496155 | 109 | ### Tier: Medium (< 500M) +| Model | Code Score | General Score | Params (M) | Architecture | CPU Speed | +|-------------------------------------------|--------------|-----------------|--------------|----------------|----------------| +| geevec-ai/geevec-embeddings-1.0-lite | 0.92365 | 0.53474 | 366 | Encoder | Moderate | +| jinaai/jina-embeddings-v5-text-nano | 0.90384 | 0.535934 | 239 | Encoder | Moderate | +| microsoft/harrier-oss-v1-270m | 0.89605 | 0.425505 | 270 | Decoder | Slow (Decoder) | +| Shuu12121/CodeSearch-ModernBERT-Crow-Plus | 0.892957 | nan | 151.668 | Encoder | Fast | +| codefuse-ai/F2LLM-v2-330M | 0.842182 | 0.475202 | 334 | Decoder | Slow (Decoder) | +| google/embeddinggemma-300m | 0.838689 | 0.459 | 302.863 | Unknown | Unknown | +| Shuu12121/NightOwl-CodeEmbedding | 0.831063 | nan | 150.779 | Unknown | Unknown | +| codefuse-ai/C2LLM-0.5B | 0.828636 | nan | 497.252 | Unknown | Unknown | -| Model | Code Search Score | General Retrieval Score | Params (M) | -|-------------------------------------------|---------------------|---------------------------|--------------| -| geevec-ai/geevec-embeddings-1.0-lite | 0.92365 | 0.53474 | 366 | -| jinaai/jina-embeddings-v5-text-nano | 0.90384 | 0.535934 | 239 | -| microsoft/harrier-oss-v1-270m | 0.89605 | 0.425505 | 270 | -| Shuu12121/CodeSearch-ModernBERT-Crow-Plus | 0.892957 | nan | 151.668 | -| codefuse-ai/F2LLM-v2-330M | 0.842182 | 0.475202 | 334 | ### Tier: Large (> 500M) +| Model | Code Score | General Score | Params (M) | Architecture | CPU Speed | +|----------------------------------|--------------|-----------------|--------------|----------------|----------------| +| microsoft/harrier-oss-v1-27b | 0.96994 | 0.483455 | 27009.3 | Decoder | Slow (Decoder) | +| Octen/Octen-Embedding-8B-INT8 | 0.967965 | nan | 7567.3 | Decoder | Slow (Decoder) | +| nvidia/llama-embed-nemotron-8b | 0.96586 | 0.51917 | 7504.92 | Unknown | Unknown | +| Octen/Octen-Embedding-4B-INT8 | 0.96369 | nan | 4022.88 | Unknown | Unknown | +| bflhc/MoD-Embedding | 0.96368 | nan | 4021.77 | Unknown | Unknown | +| Octen/Octen-Embedding-4B | 0.96236 | nan | 4021.77 | Unknown | Unknown | +| Octen/Octen-Embedding-8B | 0.9597 | 0.505307 | 7567.3 | Unknown | Unknown | +| Mira190/Euler-Legal-Embedding-V1 | 0.95635 | 0.51144 | 8188.52 | Decoder | Slow (Decoder) | -| Model | Code Search Score | General Retrieval Score | Params (M) | -|------------------------------------------|---------------------|---------------------------|--------------| -| voyageai/voyage-4-large | 0.97726 | nan | nan | -| voyageai/voyage-4-large (embed_dim=2048) | 0.97719 | nan | nan | -| google/gemini-embedding-2-preview | 0.972905 | nan | nan | -| microsoft/harrier-oss-v1-27b | 0.96994 | 0.483455 | 27009.3 | -| Octen/Octen-Embedding-8B-INT8 | 0.967965 | nan | 7567.3 | --- +## Snowflake Arctic Embed Family — Baseline Reference -## How to Regenerate this Report +These encoder-based models are included as baseline references and span the full Snowflake Arctic size range. The `xs` variant is the **default model** in `cocoindex-code`. All variants use an encoder architecture and are fast on CPU. Scores below come from the live MTEB dataset where available. + +| Model | Code Score | General Score | Params (M) | Architecture | CPU Speed | +|------------------------------------------|--------------|-----------------|--------------|----------------|-------------| +| Snowflake/snowflake-arctic-embed-xs | 0.6661 | 0.4721 | 22 | Encoder | Very Fast | +| Snowflake/snowflake-arctic-embed-s | 0.6729 | 0.4932 | 33 | Encoder | Very Fast | +| Snowflake/snowflake-arctic-embed-m | 0.7003 | 0.5197 | 109 | Encoder | Fast | +| Snowflake/snowflake-arctic-embed-l | 0.6976 | 0.5314 | 334 | Encoder | Moderate | +| Snowflake/snowflake-arctic-embed-2-m | N/A | N/A | 305 | Encoder | Moderate | +| Snowflake/snowflake-arctic-embed-2-large | N/A | N/A | 568 | Encoder | Moderate | -This report was generated using the `find_best_models.py` script. To update it with the latest live data from MTEB, run: +--- +## How to Regenerate this Report +This report was generated using the `find_best_models.py` script. To update it with the latest live data from MTEB, run: ```bash uv run scripts/find_best_models.py --clear-cache --output MTEB-RANKINGS.md ``` diff --git a/scripts/find_best_models.py b/scripts/find_best_models.py index ef425f0..724061d 100644 --- a/scripts/find_best_models.py +++ b/scripts/find_best_models.py @@ -37,36 +37,120 @@ """ import argparse +import math import os import sys from pathlib import Path import pandas as pd from datasets import load_dataset -from huggingface_hub import HfApi, dataset_info +from huggingface_hub import HfApi, ModelInfo, dataset_info from tabulate import tabulate CACHE_DIR = Path(".cocoindex_code/cache") CACHE_FILE = CACHE_DIR / "mteb_results.parquet" - -def get_model_params(model_id, api): - """Estimate model parameters from metadata or tags.""" +# Known architectures for common embedding models. +# Encoder (BERT-based) models process tokens in parallel — fast on CPU. +# Decoder (LLM-based) models process tokens sequentially — much slower on CPU, +# typically 3-10× slower than an encoder of the same parameter count. +known_architectures: dict[str, str] = { + # --- Encoder models (BERT/RoBERTa/ModernBERT-based) --- + "Snowflake/snowflake-arctic-embed-xs": "Encoder", + "Snowflake/snowflake-arctic-embed-s": "Encoder", + "Snowflake/snowflake-arctic-embed-m": "Encoder", + "Snowflake/snowflake-arctic-embed-l": "Encoder", + "Snowflake/snowflake-arctic-embed-2-m": "Encoder", + "Snowflake/snowflake-arctic-embed-2-large": "Encoder", + "lightonai/LateOn-Code-edge": "Encoder", + "lightonai/LateOn-Code-edge-pretrain": "Encoder", + "lightonai/LateOn-Code": "Encoder", + "lightonai/LateOn-Code-pretrain": "Encoder", + "ibm-granite/granite-embedding-97m-multilingual-r2": "Encoder", + "Shuu12121/CodeSearch-ModernBERT-Crow-Plus": "Encoder", + "jinaai/jina-embeddings-v5-text-nano": "Encoder", + "geevec-ai/geevec-embeddings-1.0-lite": "Encoder", + "thenlper/gte-small": "Encoder", + "thenlper/gte-base": "Encoder", + "BAAI/bge-small-en-v1.5": "Encoder", + "BAAI/bge-base-en-v1.5": "Encoder", + "BAAI/bge-large-en-v1.5": "Encoder", + "nomic-ai/nomic-embed-text-v1.5": "Encoder", + "nomic-ai/CodeRankEmbed": "Encoder", + "mixedbread-ai/mxbai-embed-large-v1": "Encoder", + "avsolatorio/GIST-small-Embedding-v0": "Encoder", + "avsolatorio/GIST-Embedding-v0": "Encoder", + "avsolatorio/NoInstruct-small-Embedding-v0": "Encoder", + "intfloat/multilingual-e5-large": "Encoder", + "intfloat/e5-small-v2": "Encoder", + "intfloat/e5-base-v2": "Encoder", + "intfloat/e5-large-v2": "Encoder", + # --- Decoder models (significantly slower on CPU without GPU) --- + "microsoft/harrier-oss-v1-270m": "Decoder", + "microsoft/harrier-oss-v1-27b": "Decoder", + "codefuse-ai/F2LLM-v2-330M": "Decoder", + "Octen/Octen-Embedding-8B-INT8": "Decoder", + # --- Cloud APIs (no local inference) --- + "voyageai/voyage-4-large": "Cloud API", + "google/gemini-embedding-2-preview": "Cloud API", +} + + +def _extract_params(info: ModelInfo) -> float | None: + """Estimate model parameters from HF model info metadata or tags.""" + if info.safetensors and info.safetensors.get("total"): + return float(info.safetensors["total"]) / 1_000_000 + for tag in info.tags or []: + if tag.endswith("B") and tag[:-1].replace(".", "").isdigit(): + return float(tag[:-1]) * 1000 + if tag.endswith("M") and tag[:-1].replace(".", "").isdigit(): + return float(tag[:-1]) + return None + + +def _extract_architecture(info: ModelInfo) -> str: + """Try to detect encoder vs decoder architecture from HF model info.""" + tags = [t.lower() for t in (info.tags or [])] + if any(t in tags for t in ["encoder-only", "bert", "roberta", "xlm-roberta"]): + return "Encoder" + if any(t in tags for t in ["decoder-only", "causal-lm", "text-generation"]): + return "Decoder" + if hasattr(info, "transformers_info") and info.transformers_info: + auto_model = getattr(info.transformers_info, "auto_model", "") or "" + if any(x in auto_model for x in ("CausalLM", "GPTNeo", "LlamaFor")): + return "Decoder" + if any(x in auto_model for x in ("MaskedLM", "BertModel", "RobertaModel")): + return "Encoder" + return "Unknown" + + +def fetch_model_metadata(model_id: str, api: HfApi) -> tuple[float | None, str]: + """Fetch a model's param count and architecture with a single HF API call.""" try: info = api.model_info(model_id) - if info.safetensors and info.safetensors.get("total"): - return info.safetensors["total"] / 1_000_000 - for tag in info.tags: - if tag.endswith("B") and tag[:-1].replace(".", "").isdigit(): - return float(tag[:-1]) * 1000 - if tag.endswith("M") and tag[:-1].replace(".", "").isdigit(): - return float(tag[:-1]) - return None except Exception: - return None - - -def main(): + return None, "Unknown" + return _extract_params(info), _extract_architecture(info) + + +def speed_label(arch: str, size_mb: float | None) -> str: + """Return a human-readable CPU indexing speed estimate.""" + if arch == "Cloud API": + return "Cloud" + if arch == "Decoder": + return "Slow (Decoder)" + if arch == "Encoder": + if size_mb is None or math.isnan(size_mb): + return "Fast (Encoder)" + if size_mb < 50: + return "Very Fast" + if size_mb < 200: + return "Fast" + return "Moderate" + return "Unknown" + + +def main() -> None: parser = argparse.ArgumentParser(description="Find best embedding models from MTEB.") parser.add_argument("--output", "-o", help="Path to save the Markdown report.") parser.add_argument( @@ -140,20 +224,24 @@ def main(): model_scores = df_retrieval.groupby("model_name")["score"].mean().reset_index() code_scores = df_code.groupby("model_name")["score"].mean().reset_index() - results = model_scores.merge( + # Keep the public-filtered results before the top-1000 head() cut, so the + # Snowflake family section below can look up scores regardless of overall rank. + all_results = model_scores.merge( code_scores, on="model_name", how="outer", suffixes=("_general", "_code") ) # Only show public models public_models = df[df["is_public"]]["model_name"].unique() - results = results[results["model_name"].isin(public_models)] + all_results = all_results[all_results["model_name"].isin(public_models)] # Hardcoded sizes for common models to speed up analysis - known_sizes = { + known_sizes: dict[str, float | None] = { "Snowflake/snowflake-arctic-embed-xs": 22, "Snowflake/snowflake-arctic-embed-s": 33, "Snowflake/snowflake-arctic-embed-m": 109, "Snowflake/snowflake-arctic-embed-l": 334, + "Snowflake/snowflake-arctic-embed-2-m": 305, + "Snowflake/snowflake-arctic-embed-2-large": 568, "mixedbread-ai/mxbai-embed-large-v1": 335, "nomic-ai/nomic-embed-text-v1.5": 137, "BAAI/bge-small-en-v1.5": 33, @@ -166,13 +254,14 @@ def main(): "lightonai/LateOn-Code-edge": 17, "lightonai/LateOn-Code": 149, "microsoft/harrier-oss-v1-270m": 270, + "Shuu12121/CodeSearch-ModernBERT-Crow-Plus": 151.668, "thenlper/gte-small": 33, "thenlper/gte-base": 109, "codefuse-ai/F2LLM-v2-330M": 334, } - def categorize(size): - if size is None: + def categorize(size: float | None) -> str: + if size is None or math.isnan(size): return "Unknown" if size < 50: return "Micro (< 50M)" @@ -183,37 +272,106 @@ def categorize(size): return "Large (> 500M)" print("Analyzing top candidates to determine hardware tiers...", file=sys.stderr) + + # Work on top 1000 for API calls (limits HF API calls for size lookups) + results = all_results.copy() results["max_score"] = results[["score_general", "score_code"]].max(axis=1) results = results.sort_values(by="max_score", ascending=False).head(1000) results["size_mb"] = results["model_name"].map(known_sizes) + results["architecture"] = results["model_name"].map(known_architectures) - # Fetch metadata for top unknown models - missing_size = results[results["size_mb"].isna()]["model_name"].tolist() - size_map = {} - for i, name in enumerate(missing_size[:100]): - size_map[name] = get_model_params(name, api) + # Fetch metadata for top unknown models. One HF API call per model covers + # both size and architecture, instead of looking each model up twice. + needs_lookup = results[results["size_mb"].isna() | results["architecture"].isna()][ + "model_name" + ].tolist() + metadata_map = {name: fetch_model_metadata(name, api) for name in needs_lookup[:100]} - results.loc[results["size_mb"].isna(), "size_mb"] = results["model_name"].map(size_map) + results.loc[results["size_mb"].isna(), "size_mb"] = results["model_name"].map( + {name: size for name, (size, _arch) in metadata_map.items()} + ) + results.loc[results["architecture"].isna(), "architecture"] = results["model_name"].map( + {name: arch for name, (_size, arch) in metadata_map.items()} + ) + results["architecture"] = results["architecture"].fillna("Unknown") results["tier"] = results["size_mb"].apply(categorize) + results["cpu_speed"] = results.apply( + lambda row: speed_label(row["architecture"], row["size_mb"]), axis=1 + ) report.append("\n## Top Embedding Models for Code Search\n") + report.append( + "> **Speed note**: CPU speed is estimated from parameter count and architecture. " + "**Encoder** models (BERT/ModernBERT-based) process tokens in parallel and are " + "significantly faster on CPU than **Decoder** models (LLM-based), which process " + "tokens sequentially. A decoder model of the same parameter count can be 3–10× " + "slower on CPU.\n" + ) tiers = ["Micro (< 50M)", "Small (< 150M)", "Medium (< 500M)", "Large (> 500M)"] for tier in tiers: report.append(f"### Tier: {tier}") tier_df = ( - results[results["tier"] == tier].sort_values(by="score_code", ascending=False).head(5) + results[results["tier"] == tier].sort_values(by="score_code", ascending=False).head(8) ) if tier_df.empty: report.append("_No models found in this tier._\n") continue - display = tier_df[["model_name", "score_code", "score_general", "size_mb"]] - display.columns = ["Model", "Code Search Score", "General Retrieval Score", "Params (M)"] + display = tier_df[ + ["model_name", "score_code", "score_general", "size_mb", "architecture", "cpu_speed"] + ].copy() + display.columns = [ + "Model", + "Code Score", + "General Score", + "Params (M)", + "Architecture", + "CPU Speed", + ] report.append(tabulate(display, headers="keys", tablefmt="github", showindex=False)) report.append("\n") + # --- Snowflake Arctic Family section --- + report.append("---") + report.append("## Snowflake Arctic Embed Family — Baseline Reference\n") + report.append( + "These encoder-based models are included as baseline references and span the full " + "Snowflake Arctic size range. The `xs` variant is the **default model** in " + "`cocoindex-code`. All variants use an encoder architecture and are fast on CPU. " + "Scores below come from the live MTEB dataset where available.\n" + ) + + snowflake_size_order = [ + ("Snowflake/snowflake-arctic-embed-xs", 22), + ("Snowflake/snowflake-arctic-embed-s", 33), + ("Snowflake/snowflake-arctic-embed-m", 109), + ("Snowflake/snowflake-arctic-embed-l", 334), + ("Snowflake/snowflake-arctic-embed-2-m", 305), + ("Snowflake/snowflake-arctic-embed-2-large", 568), + ] + + snowflake_rows = [] + for model_name, size in snowflake_size_order: + row = all_results.loc[all_results["model_name"] == model_name] + code_score = row["score_code"].values[0] if len(row) > 0 else None + general_score = row["score_general"].values[0] if len(row) > 0 else None + snowflake_rows.append( + { + "Model": model_name, + "Code Score": f"{code_score:.4f}" if pd.notna(code_score) else "N/A", + "General Score": f"{general_score:.4f}" if pd.notna(general_score) else "N/A", + "Params (M)": size, + "Architecture": "Encoder", + "CPU Speed": speed_label("Encoder", size), + } + ) + + snowflake_display = pd.DataFrame(snowflake_rows) + report.append(tabulate(snowflake_display, headers="keys", tablefmt="github", showindex=False)) + report.append("\n") + # Add regeneration instructions report.append("---") report.append("## How to Regenerate this Report") @@ -225,7 +383,7 @@ def categorize(size): report.append("uv run scripts/find_best_models.py --clear-cache --output MTEB-RANKINGS.md") report.append("```") - final_output = "\n".join(report) + final_output = "\n".join(report) + "\n" if args.output: with open(args.output, "w") as f: From 9ff1d5669c95058385fc5aa4c166fe37462f4e7f Mon Sep 17 00:00:00 2001 From: Jim McKeeth Date: Wed, 1 Jul 2026 00:03:47 -0600 Subject: [PATCH 3/3] cleaning up tiers --- EMBEDDINGS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/EMBEDDINGS.md b/EMBEDDINGS.md index cfb63fe..ba87b0a 100644 --- a/EMBEDDINGS.md +++ b/EMBEDDINGS.md @@ -79,8 +79,8 @@ These are based on MTEB [datasets](https://huggingface.co/datasets/mteb/results) | **Default** | [`Snowflake/arctic-embed-xs`](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs) | 22M | 384 | 0.67 | Enc | Very Fast | Smallest, most compatible default | | **Micro** | [`lightonai/LateOn-Code-edge`](https://huggingface.co/lightonai/LateOn-Code-edge) | 17M | 256 | 0.82 | Enc | Crazy Fast (231 cps) | **Efficiency King.** Incredible code performance for its size. | | **Small** | [`lightonai/LateOn-Code`](https://huggingface.co/lightonai/LateOn-Code) | 149M | 768 | 0.85 | Enc | Fast (7 cps) | Great balance of speed and accuracy on modern laptops. | -| **Medium (Encoder)** | [`Shuu12121/CodeSearch-ModernBERT-Crow-Plus`](https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Crow-Plus) | 152M | | 0.89 | Enc | Fast | High accuracy with encoder speed; **best for CPU-only indexing at this tier.** | -| **Larger (Decoder)** | [`microsoft/harrier-oss-v1-270m`](https://huggingface.co/microsoft/harrier-oss-v1-270m) | 270M | 640 | **0.90** | *Dec* | Slow | Highest local accuracy; **GPU** *strongly recommended* for acceptable speed. | +| **Medium** | [`Shuu12121/CodeSearch-ModernBERT-Crow-Plus`](https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Crow-Plus) | 152M | | 0.89 | Enc | Fast | High accuracy with encoder speed; **best for CPU-only indexing at this tier.** | +| **Larger** | [`microsoft/harrier-oss-v1-270m`](https://huggingface.co/microsoft/harrier-oss-v1-270m) | 270M | 640 | **0.90** | *Dec* | Slow | Highest local accuracy; **GPU** *strongly recommended* for acceptable speed. | | **Multi-Lingual** | [`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2) | 97M | 384 | 0.80 | Enc | Fast | Multilingual codebases (e.g. Code + Docs in different languages). | #### Other Model Options