Note 003 · Architecture
The economics of hybrid attention
TL;DR — Qwen3.6-27B is a hybrid: only every fourth block is full attention. The other 48 blocks are Gated-DeltaNet linear-attention blocks whose state is a fixed ~150 MiB — it does not grow with the conversation. So a token of context costs 34 KiB here versus the 136 KiB a dense 27B would pay: context is 4× cheaper by construction. That one architectural decision is what lets a single 24 GB GPU hold an 84K-token (~200-page) conversation next to the weights — and it is the quiet foundation under "free forever."
The stack, verified
The GGUF metadata says full_attention_interval = 4; the tensor names confirm it.
Only blocks 3, 7, 11 … 63 carry attn_k / attn_v weights — 16 of 64 — plus block
64, the multi-token prediction head (Note 002). The
remaining 48 blocks carry ssm_* recurrent-state tensors instead:
FIG. 1 — One attention layer in four. Blue slats grow the KV cache; the thin ones hold a fixed-size recurrent state no matter how long you talk.
What a token of context costs
Per token, only the 16 attention blocks pay KV: 16 layers × (4 KV-heads × 256 key-length, keys and values) = 32,768 elements per token.
| Cache type | Per token | 262K-token cache | Note |
|---|---|---|---|
| f16 | 64 KiB | 16.0 GiB | Uncompressed |
| q8_0 — chosen | 34 KiB | 8.5 GiB | Near-lossless |
| q4_0 | 18 KiB | 4.5 GiB | Quality degrades with depth |
| Dense 27B (hypothetical) | 136 KiB | 34.0 GiB | All 64 blocks attend |
Why q8, not q4
Halving the cache again with q4_0 would reach 128K context at full speed — but the quality cost grows with depth, which is exactly where long context matters:
| Context | Perplexity, q8_0 | Perplexity, q4_0 | Gap |
|---|---|---|---|
| 4K | 1.2423 | 1.2429 | Within noise |
| 32K | 1.0416 | 1.0434 | ~3× wider — and widening |
The trend implies a wider gap still at 128K. q8_0 at 84K avoids the question entirely.
Where the constraint moved
Because context is 4× cheaper, the KV cache stopped being the binding constraint. The fixed costs dominate — 16.7 GiB of weights, 0.86 GiB of vision encoder — and the marginal cost of context, including the speculation head's draft buffers, measures ~44 MiB per 1K tokens at two slots. That is what makes the arithmetic of Note 001 close: weights + vision + 84K of q8 cache + margin ≈ 24 GB, with decode flat from 16K to 84K (50–57 tok/s) instead of decaying.
Why this makes free possible
The expensive part of serving long conversations is memory that grows with every token. A hybrid stack caps that growth at a quarter of the dense price, so one owned machine — no fleet, no per-token vendor bill — serves 200-page conversations at interactive speed. The architecture is the subsidy. There is no venture math hiding under the $0; there is a fixed-size recurrent state.