Note 001 · Inference

Serving 84K context on a 24 GB card

JUL 2026QWEN3.6-27B · LLAMA.CPP · RTX 3090STATUS SHIPPED

TL;DR — For weeks our serving context was capped at 64K tokens: anything higher silently halved decode speed. The wall turned out to be llama.cpp's auto-fit reserving a 1,024 MiB per-device safety margin and paying for it by evicting a layer to CPU. Lowering that reserve (--fit-target 128) — not buying VRAM, not quantizing harder — raised the ceiling to 86,016 tokens (84K) with nothing given up: same speed at depth, faster when shallow, and concurrent load that the old config refused outright.

The setup

Aelius serves Qwen3.6-27B (dense, hybrid-attention, with vision and a multi-token prediction head) through llama.cpp on a single RTX 3090 — 24 GB of VRAM for everything: 16.7 GiB of weights, 0.86 GiB of vision encoder, the KV cache, and the draft buffers that speculation needs. Two server slots share one elastic KV pool (--kv-unified), so a lone heavy user can draw the whole pool while two lighter ones split it.

The wall at 64K

Raising context past 64K produced a distinctive failure: VRAM usage went down while decode speed halved. That signature matters — it isn't memory exhaustion, it's llama.cpp's auto-fit deciding the layout doesn't fit and evicting layer 0 to the CPU, which on this hybrid architecture also disables the fused Gated-DeltaNet kernel:

• W resolve_fused_ops: layer 0 is assigned to device CPU
• W resolve_fused_ops: fused Gated Delta Net (chunked) set to disabled

Losing that one fused kernel roughly halves generation on a model where 48 of 64 blocks are Gated-DeltaNet blocks.

The measurement trap

A load-only test lies. llama.cpp allocates KV at load, but speculative-decoding draft-verify and compute buffers grow during generation. Past ~80K they exceeded 24 GB and Windows' display driver silently paged VRAM to host RAM — no warning, VRAM apparently fine, decode halved anyway. Every number below is therefore a decode measurement (timings.predicted_per_second), never a "model loaded" check.

The find: the margin was the wall

llama.cpp's fit pass defaults to reserving 1,024 MiB per device (--fit-target). When its (conservative) estimate can't keep that margin, it evicts layer 0 rather than dip into it. At 80K the fit pass evicted — while the same configuration, once actually allowed to load, settled at 23.5 GB with 1.1 GB still free. The margin, not the memory, was the ceiling. Lowering the target changes only fit's decision threshold, not the real allocation:

Mean of prose + code decode, 2 slots, MTP on, vision on GPU, q8_0 KV cache. 84K is a hard ceiling: no combination of micro-batch size, draft-KV quantization, or a smaller margin buys the next 2K.

Validated under real load

Identical deep-context speed (so 84K is not silently host-paging), plus concurrency the old pool physically rejected — 2 × 34K does not fit in 64K.

What didn't work

• Smaller micro-batches (-ub 256 -b 1024) — doesn't prevent eviction at 86K+, only makes the evicted state less bad. No reason to carry it at 84K.
• Quantizing the draft KV (-ctkd/-ctvd q8_0) — saves the memory it promises and still evicts, while costing draft quality. At 2 slots these flags actually cause the eviction they're meant to avoid.
• Pinning layers (-ngl 99) — aborts at startup on this build; letting auto-fit place layers (with a sane margin) is strictly better.

The cost

Headroom. 84K leaves ~0.95 GB of VRAM free versus ~1.8 GB at 64K. On a headless box that's plenty; if the GPU also drives a display and something hungry takes a gigabyte, the driver starts host-paging and decode quietly halves — the symptom to watch for is tok/s ≈ 30. The documented fallbacks: 80K with --fit-target 256 (~1.1 GB free) or 64K (~1.8 GB free).