Part of the
qwen-benchongoing benchmark series. See the hub for the current SOTA leaderboard and a chronological index of all studies.
Head-to-head benchmark of the same model and same speculative method (dflash, num_speculative_tokens=8, drafter z-lab/Qwen3.6-27B-DFlash) on two different vLLM builds, dual NVIDIA RTX PRO 6000 Blackwell (TP=2).
| cell | Repne fork tok/s | Upstream v0.20.1 tok/s | Δ |
|---|---|---|---|
| c=1 ctx=0 | 98.9 | 85.5 | +15.6% |
| c=1 ctx=131k | 81.4 | 11.7 | +598% 🚨 |
| c=2 ctx=0 | 184.5 | 153.9 | +19.9% |
| c=2 ctx=131k | 162.7 | 22.4 | +626% 🚨 |
| c=4 ctx=0 | 358.4 | 290.9 | +23.2% |
| c=4 ctx=131k | 284.4 | 44.7 | +537% 🚨 |
Stay on Repne fork. At 131K context, upstream's dflash is essentially broken — drafter acceptance collapses from ~30% (Repne) to 1-3% (upstream), and ITL inflates 8× (11ms → 90ms per token). Even at short context where dflash is working on upstream, Repne wins +16-23%.
| Repne fork | Upstream v0.20.1 | |
|---|---|---|
| Main attention | flashinfer (full perf) |
forced to flash_attn (flashinfer rejects non-causal attn that dflash drafter needs) |
| Spec attention | flash_attn (separately set via --speculative-config.attention_backend) |
inherits main backend (no separate setting available — attention_backend field rejected with pydantic.ValidationError: Unexpected keyword argument) |
| Draft sampler | gumbel |
dropped (Repne-only) |
use_local_argmax_reduction |
true |
upstream has the field but uses default false |
| Spec accept rate at 131k (c=2) | 36.4% | 3.1% |
| ITL at 131k (c=2) | 11.2ms | 90.4ms (~8× slower per token) |
At long context the drafter on upstream barely accepts any tokens (1-3% vs Repne's 16-36%). Without the gumbel sampler and Repne's argmax reduction tricks, the upstream dflash fails to find acceptable draft sequences once attention KV gets large. The penalty is catastrophic because each rejected draft costs you the full draft+verify roundtrip with no amortization.
| Model | Qwen/Qwen3.6-27B (BF16) |
| Drafter | z-lab/Qwen3.6-27B-DFlash (8 spec tokens) |
| TP | 2 |
| max-model-len | 262144 |
| max-num-seqs | 128 |
| max-num-batched-tokens | 32768 |
| max-cudagraph-capture-size | 256 |
| GPU mem util | 0.85 |
| Spec method | dflash |
| num_speculative_tokens | 8 |
| Reasoning parser | qwen3 |
| Tool parser | qwen3_coder |
| Prefix caching | on |
| Repne fork | Upstream v0.20.1 | |
|---|---|---|
| Image | repne/vllm:latest (5e7583ca4df9, May 5 2026) |
vllm/vllm-openai:v0.20.1-cu129-ubuntu2404 (7ba11e462b5a) |
| Engine | v0.1.dev16359+ga3e24c99b.d20260505 |
v0.20.1 |
| Main attention | flashinfer |
flash_attn (forced — flashinfer doesn't support non-causal attn that dflash drafter needs) |
| Spec attention | flash_attn (separate flag) |
inherits main (no separate flag accepted) |
| Draft sampler | gumbel |
default (greedy) |
use_local_argmax_reduction |
true | false (default) |
--load-format |
instanttensor |
default |
| KV cache size | 948,399 tokens (3.62× max conc at 256K) | 927,178 tokens (3.54× max conc at 256K) |
| Boot time | 136s | 342s |
- N=1 single-run per cell, 30s sustained-decode + 10s warmup
--skip-prefill(decode-only measurement)llm_decode_bench.py v0.4.8- Run order: c=1 ctx=0 → c=1 ctx=131k → c=2 ctx=0 → c=2 ctx=131k → c=4 ctx=0 → c=4 ctx=131k
| cell | TTFT avg Repne / Upstream | TTFT p99 Repne / Upstream | ITL avg Repne / Upstream |
|---|---|---|---|
| c=1 ctx=0 | 71ms / 59ms | 80ms / 61ms | 9.85ms / 11.11ms |
| c=1 ctx=131k | 736ms / 762ms | 745ms / 762ms | 10.83ms / 86.61ms |
| c=2 ctx=0 | 165ms / 452ms | 262ms / 1139ms | 10.73ms / 12.14ms |
| c=2 ctx=131k | 1096ms / 1335ms | 1442ms / 1557ms | 11.18ms / 90.37ms |
| c=4 ctx=0 | 144ms / 182ms | 171ms / 408ms | 10.95ms / 12.56ms |
| c=4 ctx=131k | 1569ms / 2210ms | 2766ms / 2669ms | 12.12ms / 90.64ms |
| cell | Repne | Upstream |
|---|---|---|
| c=1 ctx=0 | 24.6% | 16.8% |
| c=1 ctx=131k | 16.5% | 1.2% 🚨 |
| c=2 ctx=0 | 17.2% | 23.4% |
| c=2 ctx=131k | 36.4% | 3.1% 🚨 |
| c=4 ctx=0 | 21.1% | 26.7% |
| c=4 ctx=131k | 30.0% | 2.1% 🚨 |
- N=1 single-run, no variance bands. Repne's prior morning EXP-1 baseline (N=5) showed ±2-10 tok/s std on these cells, so single-shot deltas of >10% are likely real but specific magnitudes are noisy. The 5-6× upstream long-context regression is far outside any noise band.
- Repne's c=4 ctx=0 reading of 358.4 tok/s is in line with EXP-1 morning's 332.6±6.7 (within 2σ). The earlier "regression" is probably variance, not a real defect.
- Upstream forced into a different attention configuration than Repne. This is not optional — flashinfer rejects non-causal attention with
ValueError: Selected backend AttentionBackendEnum.FLASHINFER is not valid for this configuration. Reason: ['non-causal attention not supported']. The penalty is part of running upstream as-is.
Repne fork is the only viable option for BF16+DFlash on this hardware. Upstream v0.20.1's dflash implementation is functional at short context but unusable at long context — and long context is exactly the regime where speculative decoding matters most for production agent workloads.
If you need an upstream-compatible config, FP8+MTP=3 is the way (see https://github.com/jcartu/qwen36-27b-fp8-repne-vs-upstream). MTP doesn't have the non-causal attention problem and works on flashinfer.
repne-fork/c{1,2,4}_ctx{0,131072}.json— raw bench tool outputrepne-fork/c{1,2,4}_ctx{0,131072}.log— full bench tool stdoutupstream-v0.20.1/— same layout for upstream
- FP8+MTP=3 head-to-head: https://github.com/jcartu/qwen36-27b-fp8-repne-vs-upstream
- NVFP4-MTP experiment (verdict: do not promote): https://github.com/jcartu/qwen36-27b-nvfp4-mtp-experiment
- Repne dflash newimage validation: https://github.com/jcartu/repne-dflash-newimage