RAGAS prompt-sweep leaderboard, --config-dir fixes, and enforced retrieval

[swinney]

Three feature-separated commits, each with tests.

1. feat(benchmark): rank prompt variants on a RAGAS leaderboard

• scripts/benchmarking/generate_prompt_sweep.py renders one benchmarking config per prompt variant from a manifest.
• ResultHandler.build_leaderboard aggregates each variant's mean RAGAS metrics into a ranked leaderboard (with shared_context and incomplete-row handling), emitted in the dump JSON.
• Model, queries, retriever, and judge are held fixed so only the prompt varies. Docs added to docs/docs/benchmarking.md.

2. fix(cli): make archi evaluate --config-dir run multi-config sweeps

Four plumbing fixes, each surfaced by a live --config-dir run:

• config_manager: exempt services.benchmarking from the cross-config consistency check (it's the sweep axis); global + other services.* stay strict.
• templates_manager: render a distinct file per config in multi-config mode (configs sharing a top-level name no longer collide).
• config_seed: fall back to the first *.yaml when config.yaml is absent, so the chatbot-oriented seed step no longer aborts a benchmark deployment.
• templates_manager: stage every config's agent_md_file, not just the first.

3. feat(agent): enforce initial vector retrieval in CMSCompOpsAgent

• _inject_forced_retrieval prefills a completed search_vectorstore_hybrid tool round before the model's first turn, so retrieval always happens and source links populate (the model can otherwise answer from weights, leaving "Link unavailable").
• The model may still search again. Gated by services.chat_app.force_initial_retrieval (default on).

Verification

• 37 new unit tests pass; full suite green apart from one pre-existing unrelated ingestion failure.
• Pyright: zero new diagnostics vs baseline across all changed files.
• End-to-end: a live 3-variant sweep produced a real 3-row leaderboard with populated shared_context.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds prompt-sweep support to the benchmarking workflow (including a RAGAS leaderboard), fixes several archi evaluate --config-dir multi-config deployment issues, and enforces an initial vector retrieval in CMSCompOpsAgent to ensure source links populate.

Changes:

• Introduces a prompt-sweep generator script plus a leaderboard aggregator that ranks prompt variants by mean RAGAS metrics and emits results in the dump JSON.
• Fixes multi-config sweep plumbing in config rendering/staging and config-seed resolution to support --config-dir runs reliably.
• Adds a BaseReAct hook + CMSCompOpsAgent implementation to prefill a completed retrieval tool round before the model’s first turn (gated by config), with unit tests.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/test_stage_agents_multiconfig.py	Verifies multi-config benchmarking stages all agent prompt markdown files.
tests/unit/test_render_config_filename.py	Verifies per-config rendered YAML filenames are unique in multi-config mode.
tests/unit/test_prompt_sweep_leaderboard.py	Unit coverage for leaderboard aggregation (ranking, ties, incomplete metrics, shared-context drift).
tests/unit/test_prompt_sweep_dump_gating.py	Ensures dump JSON only includes leaderboard when populated.
tests/unit/test_generate_prompt_sweep.py	Tests the sweep-config generator’s output invariants and atomic failure behavior.
tests/unit/test_forced_retrieval.py	Tests forced initial retrieval injection behavior and gating/fail-open semantics.
tests/unit/test_config_seed_resolve.py	Tests config-seed fallback behavior when config.yaml is absent in multi-config renders.
tests/unit/test_config_manager_benchmarking_variation.py	Ensures cross-config consistency checking allows services.benchmarking variation only.
src/cli/tools/config_seed.py	Adds resolve_config_path to tolerate multi-config rendered directories.
src/cli/managers/templates_manager.py	Fixes multi-config staging (agents) and per-config render naming to avoid collisions.
src/cli/managers/config_manager.py	Exempts services.benchmarking from cross-config equality checks for sweeps.
src/bin/service_benchmark.py	Adds leaderboard computation + dump emission and logs a ranked summary.
src/archi/pipelines/agents/cms_comp_ops_agent.py	Implements forced initial retrieval injection using a prefilled tool round.
src/archi/pipelines/agents/base_react.py	Adds subclass hook and integrates it into agent input preparation.
scripts/benchmarking/generate_prompt_sweep.py	New script to generate one benchmarking config per prompt variant from a manifest.
docs/docs/benchmarking.md	Documentation for running prompt sweeps and interpreting the leaderboard output.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[swinney]

Fixed in 60dd655: _stage_agents now rejects two configs whose agent_md_file share a basename (they would overwrite each other when staged and when referenced by the rendered config) with a clear error instead of silently dropping one. Test: test_same_basename_different_files_is_rejected.

[swinney]

Fixed in 60dd655: render_config_target_name now loops, bumping the suffix until the name is unique against all previously-used names, so a {stem}{index} that is already taken can no longer overwrite an earlier rendered file. Test: test_repeated_collisions_stay_unique.

[Copilot]

Thanks @copilot — both findings were valid and are now fixed (commit 60dd655):

1. _render_config_target_name single-disambiguation — it now loops, bumping the suffix until the name is uniq...

Confirmed — both fixes look correct. All 16 unit tests covering the collision-hardening and agent-staging logic pass locally (commit 60dd655). The lazy-import fix for the CI collection error (7f9b419) also checks out.