prompt-pack.md

DepthBench-LC Prompt Pack

Long-Context Operating Point Suite

Version: DB-LC v1.0
Repo / CLI: depthbench
Date: March 30, 2026

This pack accompanies DepthBench-LC: Long-Context Operating Point Suite. It contains copy-paste prompts for:

1. the orchestration agent that runs the suite
2. the benchmark model under test
3. the judge model that scores outputs
4. dataset generation for exact NIAH and semantic NIAH
5. benchmark report synthesis

Use version control. Treat every prompt as a versioned benchmark artifact.

---

Prompt 1 - Orchestration prompt for Codex or another automation agent

You are running a reproducible long-context benchmark of a local or self-hosted LLM deployment.

Your objectives are:

1. Measure prompt-processing throughput versus prompt length.
2. Measure generation throughput versus actual filled KV depth.
3. Measure long-context retrieval on exact NIAH and semantic NIAH.
4. Measure structured reasoning, coherence, and instruction-following with the memo task.
5. Emit machine-readable raw artifacts plus a concise narrative report.

Non-negotiable rules:

• Do not infer deep-context generation speed from interactive agent sessions, server status lines, or allocated context windows.
• For generation-at-depth, use llama-bench with -d so the KV cache is actually prefixed to the requested depth.
• Keep prompt-processing and generation measurements separate.
• Record exact environment metadata: date, hostname, OS, kernel, driver, CUDA, engine name, engine commit, compile flags, model path, model hash if available, quantization, cache type K/V, flash attention state, offload settings, maximum context requested, and device list.
• Run one warm-up pass per lane and exclude it from reported metrics.
• Run five reported repetitions per lane unless a lane fails outright.
• Use deterministic generation settings for quality tests: temperature 0, top_p 1, fixed seed.
• Continue through partial failures. Mark failed lanes explicitly instead of stopping the entire suite.
• Save all artifacts under ./runs/<timestamp>_<model>_<profile>/.
• Never overwrite previous runs.

Inputs:

• HARDWARE_LABEL =
• ENGINE = <llama.cpp | vllm | sglang | other>
• ENGINE_COMMIT =
• MODEL_LABEL =
• MODEL_PATH =
• MODEL_MODE = <instruct/non-thinking preferred for the primary lane>
• CONTEXT_LADDER = [4096, 16384, 32768, 65536, 131072, 262144, 393216, 524288, 786432, 1048576]
• KV_CONFIGS = [ {"name":"f16_f16","ctk":"f16","ctv":"f16"}, {"name":"q8_q8","ctk":"q8_0","ctv":"q8_0"}, {"name":"q4_q4","ctk":"q4_0","ctv":"q4_0"} ]
• OPTIONAL_KV_CONFIGS = [ {"name":"q8_q4","ctk":"q8_0","ctv":"q4_0"} ]
• FLASH_ATTN = [1]
• N_GEN = 128
• REPETITIONS = 5
• NIAH_SEEDS = [11, 23, 47]
• MEMO_VARIANTS = ["IC-A-v1", "IC-B-v1", "IC-C-v1"]

If the engine or build does not support a requested KV mode, mark the lane unsupported and continue.

Task A - Environment capture

1. Create the run directory.
2. Save:
   • manifest.json
   • env.txt
   • device_list.txt
   • engine_version.txt
3. Capture command lines exactly as executed.

Task B - Capacity and throughput sweep For each supported KV configuration and flash-attention setting:

1. Determine the maximum passing context length from the context ladder.
2. Near the ceiling, refine the maximum fit with one or two extra probes if needed.
3. Run prompt-processing:
   • llama-bench -m <MODEL_PATH> -p <comma separated passing context ladder> -n 0 -r <REPETITIONS> -o jsonl ...
4. Run generation at actual filled depth:
   • llama-bench -m <MODEL_PATH> -p 0 -n <N_GEN> -d <comma separated passing context ladder> -r <REPETITIONS> -o jsonl ...
5. Save raw outputs as:
   • llama_bench_pp_<profile>.jsonl
   • llama_bench_tg_<profile>.jsonl
   • llama_bench_pp_<profile>.stderr.txt
   • llama_bench_tg_<profile>.stderr.txt
6. Extract summary tables with mean, standard deviation, coefficient of variation, and pass/fail state.

Task C - Exact NIAH

1. Generate or load the exact NIAH dataset with the supplied generator prompt and fixed seeds.
2. Evaluate across:
   • context lengths that passed capacity checks
   • insertion depths at [0.05, 0.20, 0.35, 0.50, 0.65, 0.80, 0.95]
   • seeds in NIAH_SEEDS
3. Score exact match and normalized exact match.
4. Save:
   • niah_exact_inputs.jsonl
   • niah_exact_outputs.jsonl
   • niah_exact_scores.json

Task D - Semantic NIAH

1. Generate or load the semantic NIAH dataset with the supplied generator prompt and fixed seeds.
2. Use the same context lengths and insertion depths.
3. Score exact match, normalized exact match, and judge-based semantic correctness if exact match is too brittle.
4. Save:
   • niah_semantic_inputs.jsonl
   • niah_semantic_outputs.jsonl
   • niah_semantic_scores.json

Task E - Memo-base

1. Run the memo task for each packet in MEMO_VARIANTS with no filler context.
2. Use the standard memo prompt.
3. Save the model output and then score it with the judge prompt.
4. Save:
   • memo_base_outputs.jsonl
   • memo_base_judged.jsonl

Task F - Memo-at-depth

1. For each packet in MEMO_VARIANTS, embed the packet at target depths 0.20, 0.50, and 0.80 inside natural-language filler text.
2. Use at least three context lengths that passed capacity checks, preferably including one short, one middle, and one deep context.
3. Run the same memo prompt.
4. Judge every output with the standard judge prompt.
5. Save:
   • memo_depth_inputs.jsonl
   • memo_depth_outputs.jsonl
   • memo_depth_judged.jsonl

Task G - Summaries Create:

• summary_tables.csv
• summary.md
• final_report.md

Summary requirements:

• Include maximum fit context by profile.
• Include PP(L), TG(L), PP_norm(L), TG_norm(L).
• Include NIAH exact and semantic accuracy heatmaps or tables.
• Include memo-base and memo-at-depth scores.
• Flag any profile that silently fell back to CPU or otherwise changed execution mode.
• Do not collapse everything into one number unless you also preserve the full table.

Narrative requirements for final_report.md:

1. TL;DR
2. Test configuration
3. Prompt-processing results
4. Generation-at-depth results
5. Retrieval results
6. Memo quality results
7. Key failure modes or caveats
8. Next experiments

Specific cautions:

• If the model is Nemotron-Cascade-2, prefer instruct mode for the primary standardized lane. For the Hugging Face chat template, prepend <think></think> to the assistant response when you need non-thinking mode.
• If an asymmetric K/V profile behaves unexpectedly, verify that the cache remains GPU-offloaded before treating the result as valid.
• If the primary engine is not llama.cpp, note which metrics are not strictly comparable.

Return:

• a concise console summary
• plus the path to the run directory

---

Prompt 2A - Memo task, replication mode

SYSTEM You are writing a technical memo for an infrastructure review board. Follow every structural requirement exactly. Do not use bullet points or tables. Return only the memo followed by a short self-audit.

USER Use the context packet below to write a decision memo.

Context packet version: IC-A-v1 [Paste one of the packet variants from Section 4 below.]

Write a memo with exactly these six headings in this order:

1. Situation
2. Decision Criteria
3. Candidate Path
4. Failure Mode
5. Revision
6. Recommendation

Hard constraints:

• Exactly 3 sentences per section.
• Total length 450 to 650 words.
• Exactly 8 quantitative claims.
• At least 2 explicitly uncertain estimates.
• Define 3 decision criteria in section 2 and reuse all 3 in sections 3, 5, and 6.
• Introduce one failure mode in section 4 and revisit it in sections 5 and 6.
• Overturn one assumption in section 5 and explain why.
• Do not use the words "obviously", "clearly", or "therefore".
• No bullet points or tables.
• The final sentence must contain a recommendation and a confidence percentage.

After the memo, add a short self-audit that:

• lists the 8 quantitative claims
• states the overturned assumption
• states whether all constraints were met

---

Prompt 2B - Memo task, standard mode with audit JSON

SYSTEM You are writing a technical memo for an infrastructure review board. Follow every structural requirement exactly. Do not reveal chain-of-thought. Return only the memo and the requested JSON audit object.

USER Use the context packet below to write a decision memo.

Context packet version: IC-A-v1 [Paste one of the packet variants from Section 4 below.]

Write a memo with exactly these six headings in this order:

1. Situation
2. Decision Criteria
3. Candidate Path
4. Failure Mode
5. Revision
6. Recommendation

Hard constraints:

• Exactly 3 sentences per section.
• Total length 450 to 650 words.
• Exactly 8 quantitative claims.
• Every quantitative claim must be followed by one or more supporting fact tags in parentheses, such as (F03) or (F03, F09).
• Include at least 2 explicitly uncertain estimates using the phrase uncertain estimate or uncertain range.
• Define 3 decision criteria in section 2 and reuse all 3 in sections 3, 5, and 6.
• Introduce one failure mode in section 4 and revisit it in sections 5 and 6.
• Overturn one assumption in section 5 and explain why.
• Do not use the words "obviously", "clearly", or "therefore".
• No bullet points or tables.
• The final sentence must contain a recommendation and a confidence percentage.
• Use only facts in the packet, except for explicitly labeled uncertain estimates.

After the memo, output a JSON object with exactly this top-level schema:

{
  "packet_version": "",
  "section_headings": [],
  "section_sentence_counts": [],
  "word_count": 0,
  "quantitative_claims": [
    {
      "claim_text": "",
      "fact_tags": [],
      "claim_type": "fact|uncertain_estimate"
    }
  ],
  "uncertain_estimates": [],
  "decision_criteria": [],
  "failure_mode": "",
  "overturned_assumption": "",
  "banned_words_present": [],
  "final_recommendation_sentence": "",
  "constraint_self_assessment": {
    "correct_heading_order": true,
    "three_sentences_each": true,
    "word_count_in_range": true,
    "exactly_eight_quantitative_claims": true,
    "at_least_two_uncertain_estimates": true,
    "criteria_reused_in_sections_3_5_6": true,
    "failure_mode_revisited_in_sections_5_6": true,
    "assumption_overturned_in_section_5": true,
    "no_banned_words": true,
    "no_bullets_or_tables": true,
    "final_sentence_has_recommendation_and_confidence": true
  }
}

Return only the memo and the JSON object.

---

Prompt 3 - Judge prompt for the memo task

SYSTEM You are a strict benchmark judge. Your job is to score a candidate memo against the packet and the rubric. Be literal, conservative, and explicit about failures. Do not repair the answer. Return JSON only.

USER You will receive:

1. a context packet
2. a candidate memo
3. optionally, the candidate's self-audit JSON

Evaluate the memo on both surface compliance and latent coherence.

Scoring rubric:

• format_compliance: 0 to 25
  • headings correct and in order
  • exactly 3 sentences per section
  • word count in range
  • no bullets or tables
  • banned words absent
• numerical_fidelity: 0 to 20
  • exactly 8 quantitative claims
  • claims match packet facts or are clearly marked uncertain estimates
  • fact tags, if present, are correct
• coherence_and_reuse: 0 to 20
  • 3 decision criteria defined in section 2
  • all 3 are reused in sections 3, 5, and 6
  • failure mode remains the same mechanism across sections 4, 5, and 6
• revision_quality: 0 to 15
  • a real assumption is overturned in section 5
  • the explanation is causal rather than cosmetic
• recommendation_quality: 0 to 10
  • recommendation is explicit
  • recommendation follows from the prior analysis
  • final sentence includes confidence percentage
• self_audit_quality: 0 to 10
  • candidate self-audit matches the actual memo
  • no missing or invented quantitative claims

Return JSON with exactly this schema:

{
  "packet_version": "",
  "scores": {
    "format_compliance": 0,
    "numerical_fidelity": 0,
    "coherence_and_reuse": 0,
    "revision_quality": 0,
    "recommendation_quality": 0,
    "self_audit_quality": 0,
    "total": 0
  },
  "pass_flags": {
    "headings_correct": false,
    "three_sentences_each": false,
    "word_count_in_range": false,
    "exactly_eight_quantitative_claims": false,
    "two_or_more_uncertain_estimates": false,
    "criteria_reused_in_3_5_6": false,
    "single_failure_mode_revisited": false,
    "assumption_overturned_in_5": false,
    "banned_words_absent": false,
    "final_sentence_has_recommendation_and_confidence": false
  },
  "extracted": {
    "section_headings": [],
    "section_sentence_counts": [],
    "word_count": 0,
    "quantitative_claims": [],
    "uncertain_estimates": [],
    "decision_criteria": [],
    "failure_mode": "",
    "overturned_assumption": ""
  },
  "hallucinated_or_unsupported_claims": [],
  "notes": []
}

Judge strictly. If you are uncertain, lower the score and explain why in notes.

---

Prompt 4A - Exact NIAH dataset generator

SYSTEM Generate exact-retrieval needle-in-a-haystack test items. Return JSON only.

USER Create 20 exact-retrieval items for long-context testing.

Requirements:

• Topic domain: neutral technical operations prose.
• Each item must have:
  • needle_sentence
  • question
  • answer
  • accepted_answers
  • key_type
  • value_type
• Use values that are easy to score exactly, such as:
  • 7-digit numbers
  • short alphanumeric codes
  • UUID-like identifiers
  • short noun phrases
• The question should request the answer directly.
• Keep the answer unique within the set.
• Avoid sensitive, personal, or unsafe content.
• Make the needle sentence stylistically compatible with a long technical dossier.

Return JSON:

{
  "dataset_name": "niah_exact_v1",
  "items": []
}

---

Prompt 4B - Semantic NIAH / NoLiMa-lite dataset generator

SYSTEM Generate semantic needle-in-a-haystack test items. Return JSON only.

USER Create 20 semantic-retrieval items for long-context testing.

Requirements:

• Topic domain: neutral technical operations prose.
• Each item must have:
  • needle_sentence
  • question
  • answer
  • accepted_answers
  • semantic_link_explanation
• The question and the needle sentence should have low lexical overlap.
• Avoid sharing distinctive content words between the needle and the question unless needed for grammar.
• The question must still be answerable from the needle sentence by semantic reasoning rather than literal string matching.
• Keep answers short and exactly scoreable where possible.
• Avoid sensitive, personal, or unsafe content.

Example pattern:

• Needle sentence: "Among the pilot locations, Merino recorded the lowest coolant escape rate at 0.87 liters per day."
• Question: "What leakage value belonged to the best-performing site?"
• Answer: "0.87 liters per day"

Return JSON:

{
  "dataset_name": "niah_semantic_v1",
  "items": []
}

---

Prompt 5 - Benchmark report synthesis prompt

SYSTEM You are writing a benchmark report from raw measurement files. Be precise and do not invent observations. If a conclusion is an inference, say so.

USER Use the supplied benchmark artifacts to write a concise but technically rigorous report.

Inputs available:

• manifest.json
• environment metadata
• prompt-processing summaries
• generation-at-depth summaries
• NIAH exact results
• NIAH semantic results
• memo judge results
• failure logs

Report structure:

1. TL;DR
2. Test configuration
3. Prompt-processing
4. Generation at filled depth
5. Retrieval quality
6. Memo quality
7. Caveats
8. Next experiments

Rules:

• Distinguish allocated context from filled context.
• Do not claim generation is flat unless the actual filled-depth data supports it.
• State unsupported or failed profiles clearly.
• Quote numerical deltas when helpful.
• Mention whether instruct mode or thinking mode was used.
• Mention whether results come from llama.cpp, vLLM, or another engine.
• Include one short paragraph on likely bottlenecks, but mark it as inference if not directly measured.

---

Section 4 - Standard context packets for the memo task

Packet IC-A-v1 - Hyperscale retrofit vs immersion hall

[F01] The existing air-cooled hall sustains 80 kW racks today; a direct-to-chip retrofit reaches 180 kW median and 250 kW peak. [F02] A single-phase immersion pod design supports 450 kW launch racks and 600 kW after a busway upgrade. [F03] Relative to the current hall, facility cooling power falls 22% with direct-to-chip and 48% with immersion. [F04] Expected PUE is 1.17 for the direct-to-chip retrofit and 1.05 for the immersion hall. [F05] Expected WUE is 0.18 L/kWh for direct-to-chip and 0.02 L/kWh for immersion. [F06] Immersion material-compatibility remediation costs $2.7M and adds 14 weeks. [F07] Coolant rerouting to preserve service aisles adds $3.1M to the immersion path. [F08] PFAS-free dielectric fluid costs $18/L, requires an initial fill of 140000 L, and needs 12% annual replacement. [F09] Waste-heat reuse is only economic above 55 C outlet temperature; direct-to-chip exits at 47 C, immersion exits at 58 C, and the heat-reuse value is estimated at $1.2M per year. [F10] The 24-month roadmap points to 700 kW training racks. [F11] The finance hurdle rate is 11%, and the campus electricity price is $0.082/kWh. [F12] The board requires the first production hall online within 9 months.

Packet IC-B-v1 - Colocation expansion

[F01] The program must deliver a 24 MW AI expansion inside 1200 m^2 of leased white space. [F02] Direct-to-chip racks average 200 kW with 85% floor occupancy; immersion tanks average 500 kW with 78% occupancy because of service clearances. [F03] The colocation operator charges $210 per kW-month for direct-to-chip power plus cooling, versus $165 per kW-month for immersion after a custom addendum. [F04] The immersion addendum requires an 18-month minimum term and a $1.8M upfront engineering fee. [F05] The site water-allocation cap is 15000 m^3 per year; the direct-to-chip plan uses 13800 and the immersion plan uses 1200. [F06] PFAS-free fluid inventory is 60000 L at $21/L with an 8% annual top-off rate. [F07] The annualized service-event model gives 3.1% for pump-loop leaks in direct-to-chip and 1.4% for immersion tank seal events. [F08] Mean repair window is 9 hours for direct-to-chip, 16 hours for immersion without a spare tank, and 6 hours for immersion with one spare tank. [F09] One spare immersion tank costs $0.9M. [F10] Customer demand is expected to shift to 350 kW per rack median within 18 months. [F11] The ESG requirement is WUE below 0.05 L/kWh by the next contract renewal. [F12] Missing launch by one quarter triggers a revenue penalty estimated at $4.4M.

Packet IC-C-v1 - Edge inference fleet

[F01] There are 6 regional inference sites, each capped at a 4 MW utility interconnect. [F02] Target racks are 120 to 160 kW this year and 300 kW in the next hardware cycle. [F03] A direct-to-chip retrofit can be completed in 6 weeks per site; an immersion skid takes 11 weeks per site. [F04] Relative cooling-power reduction is 28% for direct-to-chip and 44% for immersion. [F05] Expected PUE is 1.14 for direct-to-chip and 1.07 for immersion. [F06] Water costs $5.20 per m^3 in the remote regions, and seasonal water delivery is a known risk. [F07] Direct-to-chip still consumes 0.11 L/kWh of water; immersion eliminates evaporative tower water use. [F08] Technician familiarity is scored 8/10 for direct-to-chip and 4/10 for immersion; a training program raises immersion to 7/10 at a cost of $0.6M. [F09] Shipping vibration causes connector reseat events on 2.8% of moves; a sealed service frame for immersion costs $0.35M per site and cuts the event rate by half. [F10] Waste-heat reuse is unavailable at 4 sites and worth a combined $0.3M per year at the remaining 2 sites, but only for immersion. [F11] Corporate wants the fleet standardized across all 6 sites within 12 months. [F12] Internal carbon accounting uses a shadow price of $72 per ton of avoided emissions.

---

Section 5 - Optional scoring formulas

Use these in your post-processing script or notebook.

• PP_norm(L) = PP(L) / PP(4K)
• TG_norm(L) = TG(L) / TG(4K)
• Retrieval_exact = mean(exact_match over all exact NIAH items)
• Retrieval_semantic = mean(exact_match or judge-approved semantic match over semantic items)
• Memo_total = judge_total / 100
• Capacity_score = log2(L_fit / 4096) / log2(L_target / 4096) clipped to [0, 1]

Suggested optional composite for internal ranking only:

• Overall = 0.30 * Performance + 0.25 * Retrieval + 0.25 * Memo + 0.10 * Capacity + 0.10 * Stability

Do not use the composite as the only reported result. Preserve the full tables.