Note 002 · Speculative decoding
Speculation has a depth limit
TL;DR — Qwen3.6 ships with a built-in multi-token prediction (MTP) head: a 65th block that drafts tokens ahead so the model can verify several at once. The obvious lever — draft deeper — makes everything worse. Depth 2 decodes at 51.2 tok/s; depth 5 collapses to 17.3. Deeper speculation loses twice: its draft context evicts a layer to CPU, and its acceptance rate falls from 76% to 41%. The model card's recommended depth survived contact with measurement; the temptation to raise it did not.
How the head works
During decode, the MTP head proposes the next n tokens; the main model verifies
them in one pass. Every accepted token is a full forward pass saved. Acceptance on real
traffic measures 0.65–0.77 — roughly seven of ten guesses land, which is where the
free speed comes from.
The depth sweep
Identical prompts, fixed seed, one server restart per value, four workloads:
| Draft depth | Short | Prose | Code | 16K + reasoning | Mean tok/s |
|---|---|---|---|---|---|
| n = 2 — chosen | 55.4 | 47.2 | 54.7 | 47.4 | 51.2 |
| n = 3 | 30.1 | 28.9 | 35.1 | 29.9 | 31.0 |
| n = 4 | 25.8 | 20.7 | 26.3 | 18.3 | 22.8 |
| n = 5 | 18.6 | 15.0 | 22.3 | 13.3 | 17.3 |
Decode tok/s by --spec-draft-n-max, measured 2026-07-19. The n=2 row
is not merely fastest — it is the only row where the model still fits.
Why deeper loses twice
1 — The draft context is real memory
The head keeps a draft context per slot, and it scales with depth. At depth 3+ on a 2-slot 84K pool, that extra context blows the layout budget and llama.cpp evicts layer 0 to CPU — the exact failure documented in Note 001, with the same log signature (fused Gated-DeltaNet disabled, VRAM down, decode halved). The speed you hoped to win is spent before the first token.
2 — Deeper guesses are worse guesses
Acceptance fell from 76% to 41% on short prompts as depth rose. Each extra drafted token is conditioned on unverified guesses, so the chain breaks more often — and every broken chain wastes the verification pass. Deeper speculation buys more losing lottery tickets.
What's next
The built-in head's ceiling is its acceptance rate. A dedicated draft model is currently in training to push past it; an off-the-shelf drafter we benchmarked in July lost to the MTP head on this workload and was rejected. The rule stands until the data changes: the depth the measurements chose, not the maximum the flag allows.