The headline above is real and replays from a committed trace. The rest of this page is the evaluation that followed: random, oracle, always-drafter and learned-router baselines, three public benchmarks with answer keys, a disjoint replication of every result, and the speculative executor and semantic cache measured by replaying traces through their production code. Most of it is negative for the entropy signal, and all of it reproduces with one command from the repository.
| Dataset | Esc% | Router | Random | vs Rand | Mean-H | Learned |
|---|---|---|---|---|---|---|
| GSM8K r1answer key | 5.3 | 93.0 | 92.7 | +0.3 | 94.3 | train |
| MMLU r1answer key | 18.7 | 77.3 | 76.7 | +0.6 | 79.3 | train |
| GSM8K r2held out | 6.8 | 90.4 | 90.8 | -0.4 | 92.2 | 92.4 |
| ARC-Challengeheld out | 13.2 | 93.0 | 93.2 | -0.2 | 93.4 | 94.0 |
| MMLU r2held out | 19.0 | 78.2 | 79.2 | -1.0 | 81.6 | 82.2 |
| GSM8K, qwen drafteranswer key | 28.0 | 87.3 | 88.0 | -0.7 | 90.3 | n/a |
| Project setLLM judge | 7.7 | 98.1 | 98.1 | 0.0 | 97.9 | 98.3 |
Served accuracy in percent. Every method escalates the same share of requests as the production router at T=2.0 on that dataset. Random is the rate-matched random router. Mean-H thresholds the trace mean entropy. Learned is a 22-feature logistic model trained on the two r1 sets and tested on the r2 sets and ARC.
qwen row: drafter is qwen2.5-coder:7b via local Ollama, heavyweight gpt-4.1, T=2.0 not recalibrated. Project set is judged by gpt-4.1 against its own answer, so the heavyweight is correct by definition. Bootstrap 95% intervals for every cell are in docs/RESULTS.md sections 3 to 5, 9 and 13.
Features computed in Go and replayed in Go, so training and inference cannot disagree. Top standardised coefficients: token count +0.54, entropy std +0.47, mean entropy +0.37; the window peak goes negative (-0.30) once the others are present. The learned router costs slightly more at the same escalation rate because it prefers long, high-variance drafts.
AUROC of each per-request score as a detector of wrong drafts, bar at the point estimate, lighter band is the bootstrap 95% interval. The window peak is the exact statistic the gateway thresholds. Its interval covers 0.5 on all five answer-key samples. Mean entropy is informative on every one of them but needs the whole draft.
The pattern transfers to a different model family: window peak at chance (0.52), trace mean informative (0.73), router below random. T=2.0 escalates 28.0% of this drafter against 5.3% for gpt-4.1-nano on the same prompts; thresholds do not transfer between drafters. Ollama reports real logprobs for tokens outside the top 5, which confirms the OpenAI -9999 sentinel ceiling is an API encoding artifact.
| Trace | Trigger% | Cancel% | p50 | Cancel/1k |
|---|---|---|---|---|
| Project setn=519 | 70.5 | 96.4 | 36 ms | 680 |
| GSM8K r1n=300 | 87.7 | 98.9 | 26 ms | 867 |
| MMLU r1n=300 | 86.0 | 93.4 | 27 ms | 803 |
| GSM8K r2n=500 | 88.8 | 98.6 | 37 ms | 876 |
| ARC-Challengen=500 | 88.8 | 96.4 | 30 ms | 856 |
| MMLU r2n=500 | 86.4 | 92.6 | 29 ms | 800 |
Every recorded trace replayed through the production executor with synthetic logprobs that reproduce the recorded entropy exactly. Trigger is the share of requests that start a speculative heavyweight call at 0.8T; Cancel is the share of those calls cancelled because the draft was accepted. Head start is escalations that had a speculative call already running. Billed to completion, the cancelled calls would cost 80 to 88% of the all-heavyweight bill. Recommendation on record: speculative.enabled false.
| Trace | Hits | Wrong | NN max | p50 / p99 |
|---|---|---|---|---|
| Project setn=519 | 0(0.0%) | 0 | 0.878 | 197 / 374 ms |
| GSM8K r1n=300 | 0(0.0%) | 0 | 0.763 | 199 / 406 ms |
| MMLU r1n=300 | 3(1.0%) | 1 | 1.000 | 201 / 360 ms |
Prompts replayed in order through the production vector store with the real embedding client, inserting what the T=2.0 router would have accepted. NN max is the highest cosine any prompt reached against an earlier one. The MMLU hits are two exact duplicates and one templated near-duplicate at 0.959 that served the wrong letter.
At 0.90 the MMLU replay would hit 30 prompts and serve a wrong cached answer on 19. A second pass over the project set hits all 479 inserted prompts. Lookup is dominated by the embedding call.
| Scenario | Success | p50 | p99 |
|---|---|---|---|
confident all drafts accepted | 100% | 105.9 ms | 107.4 ms |
mixed 20% of drafts uncertain | 100% | 106.3 ms | 109.8 ms |
flaky 20% drafter 503s, 5% stalls, 2 s timeout | 100% | 107.1 ms | 2107.9 ms |
1,500 requests per scenario at 50 req/s with vegeta against a mock upstream that streams 20 SSE chunks at 5 ms each, so p50 is upstream time and gateway overhead is low single-digit milliseconds. In the flaky run every injected drafter failure was served by the heavyweight with zero 5xx responses. The 2.1 s p99 is the drafter timeout plus one heavyweight call. Config: config.loadtest.yaml.
Both fallbacks are configurable. The mock's uncertain tokens originally carried 1.58 bits, below T=2.0, so earlier load tests never exercised escalation; the mixed scenario now escalates about 21% of requests.
40 escalations caught exactly one bad draft. Random escalation at the same rate also caught one. With 11 positives and a judge that is the heavyweight itself, this set cannot distinguish routers; the answer-key sets above can.
4 categories: factual, reasoning, code generation, ambiguous/creative. LLM-as-judge evaluation. Swept T=1.0..2.5 in 0.25 steps. Top-5 renormalised entropy caps at log2(5) = 2.32 bits, so the T=2.25 and 2.50 rows are degenerate.
| T | ESC% | ACC% | COST RED% | ACCEPT | ESCAL |
|---|---|---|---|---|---|
| 1.00 | 68.0 | 100.0 | 3.6 | 166 | 353 |
| 1.25 | 48.5 | 99.8 | 24.4 | 267 | 252 |
| 1.50 | 32.6 | 99.0 | 46.4 | 350 | 169 |
| 1.75 | 15.6 | 98.3 | 73.2 | 438 | 81 |
| 2.00SEL | 7.7 | 98.1 | 82.9 | 479 | 40 |
| 2.25 | 0.8 | 97.9 | 91.9 | 515 | 4 |
| 2.50 | 0.0 | 97.9 | 92.6 | 519 | 0 |
H(X) = -SUM p(x) log2 p(x) over top-5 logprobs per token. 10-token sliding window. The router thresholds the window peak; the trace mean is the statistic that carries signal.
At T=2.0 the entropy router is 0.2 to 1.0 accuracy points below a random router escalating the same share on all three held-out sets, and level with it on the project set. The savings come from the cheap drafter being usually right, not from the entropy signal choosing what to escalate.
Drafter produces wrong answers with low entropy. 10 of 11 bad drafts pass at T=2.0 on the project set; all 11 have mean entropy under 0.5 bits. On MMLU r2 the window peak sits below chance at 0.445.
Routers restricted to the first 20 or 50 drafter tokens reach held-out AUROC 0.48 to 0.62. The information that predicts a wrong draft arrives with the answer, which removes the basis for early exit and speculative execution on these tasks.
Replayed through the executor: fires on 70 to 89% of requests, cancels 93 to 99% of those calls, median head start 26 to 37 ms. The trigger is per token, not per window, which the design doc had wrong. The under-10%-waste target is not met at any multiplier.
Cosine > 0.95, only draft-accepted responses cached. Replayed: 0 to 1% hits on unique benchmark prompts, 100% on repeats, lookup about 200 ms. At 0.90 MMLU would serve 19 wrong cached answers in 30 hits.
An 8% served-error operating point calibrated on round 1 folds held on GSM8K r2 (6.4%), held wastefully on ARC by escalating 66.8%, and missed on MMLU r2 (11.0%, where the heavyweight alone is 9.2% wrong). Risk control needs calibration data from the traffic it governs.
Every number on this page replays offline from traces committed under benchmarks/results. No API key is needed for the sweeps, the learned-router evaluation or the speculative replay; the cache replay makes embedding calls (about one cent per thousand prompts). Full commands in docs/RESULTS.md section 14.
go test -race ./... go run ./benchmarks/cmd/compare -input benchmarks/results/collected.jsonl.gz -output benchmarks/results/compare.csv go run ./benchmarks/cmd/compare -input benchmarks/results/collected_gsm8k_r2.jsonl.gz -output benchmarks/results/compare_gsm8k_r2.csv \ -models benchmarks/models/router_full_logistic.json go run ./benchmarks/cmd/replayspec -input benchmarks/results/collected_mmlu_r2.jsonl.gz OPENAI_API_KEY=... go run ./benchmarks/cmd/replaycache -input benchmarks/results/collected_mmlu.jsonl.gz