perf(llm): reuse embedding context across batch inputs

[saagpatel]

Refs #44 and #34.

What changed

• Prefer cached tokenizer files and GGUF-embedded tokenizer metadata before attempting network downloads.
• Tokenize a document batch first, create one llama.cpp context sized for the longest input, and reuse it sequentially with KV-cache clearing.
• Add a real-model parity example that exits non-zero if any vector differs by more than 1e-6 from the fresh-context path.

Why

Startup tried several Hugging Face tokenizer candidates before falling back to metadata already present in the GGUF. Indexing also paid llama.cpp context creation and graph-planning costs once per chunk.

This is sequential context reuse, not packed multi-sequence batching. Packed batching was deliberately excluded because it did not preserve the existing vector-parity gate.

Impact

The sampled 32-document real-model run completed in 256 ms with one reused context versus 357 ms with fresh contexts, while remaining bitwise identical (max_abs_diff=0). Absolute timing is machine-dependent; parity is the hard gate.

Validation

• cargo fmt --check
• cargo clippy -- -D warnings
• cargo test --lib llm::tests — 43 passed
• cargo build --release --example embed_batch_parity
• embed_batch_parity <copied-lab-models> — 0/32 vectors beyond 1e-6