Hash embedding mode should use lexical/BM25 search fallback

[vanachterjacob]

Problem

When embedding_device=hash / lexical fallback is configured, search can return low-signal vector hits instead of obvious lexical matches.

Observed query:

mempalace reindex database hnsw embeddings

The MCP/vector path returned unrelated entries such as stopword/i18n files and NUL-byte-like documents, while SQLite FTS/BM25 found relevant HNSW/reindex rows immediately.

Root Cause

Hash embeddings are a local lexical fallback, but the search path still treats them like dense semantic embeddings. That makes vector distance a weak ranking signal for some real queries. The SQLite FTS fallback also selected candidates without ORDER BY rank, so low-rowid partial matches could crowd out better candidates before BM25 reranking.

Expected Behavior

When hash/lexical embedding mode is active, search should use the SQLite BM25 path intentionally. FTS candidate selection should retrieve relevant candidates first before local BM25 reranking.

Proposed Fix

• Route MCP and CLI search to SQLite BM25 when embedding_device=hash or lexical
• Add ORDER BY rank to the SQLite FTS candidate query
• Preserve the existing vector path for normal ONNX/dense embedding mode
• Surface the fallback reason in structured search results

[vanachterjacob]

Draft fix proposed in #1381: #1381

[jphein]

@vanachterjacob — adjacent prior art on the structural fallback piece, in case useful: #1005 (open, MERGEABLE, all 6 CI green) implements the machinery you're asking for in a more general shape, then leaves the trigger conservative. Specifically it adds:

• _sqlite_fallback_and_scope — ranks the sqlite candidate pool with the same Okapi-BM25 used in _hybrid_rank and tops up the hits list when the vector path underdelivers (sparse HNSW, filter-planner failure, async-flush drift).
• available_in_scope — sqlite-authoritative scope count via paginated col.get(include=[], ...), so a query into a wing with N drawers can show that N even when vector returned 0.
• warnings: [...] in the response shape — exactly the "surface the fallback reason in structured search results" you're asking for. Each fallback emits a structured warning describing what triggered (vector_underdelivered, available > vector_hit_count, filter-planner failure).
• matched_via: "sqlite_bm25_fallback" per-hit tag so callers can tell which path each hit came from.

Today's PR triggers the fallback only when the vector path under-delivers n_results. Your ask — "use SQLite BM25 intentionally when embedding_device=hash/lexical" — is a stricter trigger that fits cleanly on top of the same machinery: in that mode you'd unconditionally route through _sqlite_fallback_and_scope instead of conditionally as a top-up. Could be either a follow-up PR or a small extension to #1005's branch (I'm easy on shape).

Re. "ORDER BY rank on the FTS candidate query" — agreed, that's a real gap. #1005 ranks via Okapi-BM25 in Python (line bm25 = _bm25_scores(query, candidate_docs) in _sqlite_fallback_and_scope), but the candidate selection before that step is just a col.get with no ranking — same shape your reporter saw. Worth fixing on either #1005's path or yours, whichever lands first.

Happy to plumb the embedding_device == "hash" trigger into #1005 if @bensig / @igorls would prefer it as one cohesive change rather than two PRs against the same code area.