Engineering · Prefill attention
We cut voice-agent TTFT in half with 3.8× faster vLLM kernels
- Voice-agent prompts are mostly system instructions, tool schemas, and history, while each spoken turn adds only a few new tokens.
- Whenever a call starts or misses prefix cache, slow prefill stalls decode for every active call sharing the GPU.
- Prefill speed is therefore critical to keeping time to first token (TTFT) low, even at high concurrency.
- Replacing vLLM's prefill attention kernels makes prefill up to 3.78× faster, reaching 66.6–68.5% of the theoretical speed of light (up from 18.5% on stock vLLM).
- On reference voice-agent traffic, first-turn TTFT drops up to 43% (and warm-turn p90 up to 2.5×) with unchanged accuracy on GSM8K.
- 3.78× fasterGemma 4 12BPrefill vs. stock vLLM Triton
- 3.60× fasterGemma 4 26B-A4BPrefill vs. stock vLLM Triton
- 1.14× fasterGPT-OSS-120BPrefill vs. FlashInfer
- 400 → 240 msTTFT after 8K token prefillGemma 4 26B-A4B ingesting an ~8,000-token system prompt
Typical prefill step time cost for voice agents
1 · Why prefill matters
Voice agents are mostly system prompt, not output
- In typical voice-agent traffic, system prompts, tool schemas, and prior turns add up to 7.5K–10.5K tokens (~9K typical), while each caller turn adds roughly 64 new tokens and produces a short spoken reply.
- Continuous-batching engines cap how many tokens run in a single step (8,192 tokens here). When a cold call arrives—or a turn misses the prefix cache—its prefill consumes the step budget and holds up decode tokens for every other live call on the GPU.
- That makes prefill attention against the paged KV cache the bottleneck on both cold starts and mixed prefill-plus-decode batches.
Testbed
- GPU: 1× NVIDIA RTX PRO 6000 Blackwell (96 GB, 600 W).
- Engine: vLLM 0.32.0, BF16 weights and KV cache, prefix caching on, max 8,192 batched tokens/step, no speculative decoding.
- Baselines: vLLM Triton
unified_attentionon Gemma 4; FlashInfer prefill on GPT-OSS-120B.
Four scored step shapes
- 32 warm turns: ~64 new tokens each on 7.5K–10.5K cached tokens.
- 96 warm turns: same shape under heavy concurrent load.
- Cold 8K: one uncached 8,192-token prompt.
- Mixed step: one 4,096-token prompt chunk on 4,096 cached tokens, plus 48 warm turns.
2 · Kernel speed
3.6–3.8× faster on Gemma 4, 1.14× over FlashInfer on GPT-OSS-120B
- Every layer shape was benchmarked in isolation—called the exact way vLLM calls it—across local sliding-window and full-context attention layers.
- Gemma 4 12B (40 sliding, 8 full layers) reaches 68.5% of the card's speed of light versus 18.5% for vLLM's Triton kernel.
- Gemma 4 26B-A4B reaches 66.6% of speed of light, and GPT-OSS-120B reaches ~68% (up from FlashInfer's 60%).
Gemma 4 12B · 3.78× weighted speedup (cold 8K: 211 ms → 63 ms)
Gemma 4 26B-A4B · 3.60× weighted speedup
GPT-OSS-120B · 1.14× weighted speedup over FlashInfer (+13.9%)
Why the gap differsFlashInfer on GPT-OSS-120B already ran at roughly 60% of speed of light (leaving ~14% headroom to our ~68%); vLLM's Triton kernel on Gemma 4 ran near 19%.
3 · Layer specialization
One kernel per attention layer type, not one-size-fits-all
- Modern models mix very different attention geometries inside a single forward pass—for example, Gemma 4 12B pairs 40 local sliding-window layers with 8 full-context layers, and each layer type uses its own head count, context span, and KV layout.
- A single generic prefill kernel leaves speed on the table when forced across all of them, or causes full-context layers to miss the fast path entirely.
- Specializing our kernels for each attention layer type and its KV geometry unlocked an extra 32–36% speedup on Gemma 4 12B's full layers and 61–69% on Gemma 4 26B-A4B's full layers, while matching double-precision (FP64) reference outputs.
4 · Reference voice traffic
Two-step serving optimization: KV cache capacity + fast prefill
We replayed reference traffic from a voice-agent company on the same RTX PRO 6000—~9K-token prompts adding ~64 new tokens per turn—at two traffic levels (peak and moderate call volume).
- Step 1 · Tightly coupled KV caching: By coupling Zazu directly with the voice-agent runtime, we achieved near-perfect cache hit rates and drove single-GPU concurrency up sharply. At that density, however, the card became compute-bound on long prefills, starving new first turns.
- Step 2 · Prefill kernel optimization: Replacing the prefill attention kernels broke that compute bottleneck, clearing the queue so both cold starts and warm turns stay fast across the board.
Gemma 4 12B · 1.5× lower first-turn p50 and 1.7× lower first-turn p90 vs stock vLLM
Gemma 4 26B-A4B · 1.7× lower first-turn p50 and 1.75× lower first-turn p90
GPT-OSS-120B · Modest end-to-end gain, matching the 1.14× kernel margin
Serving noteWith FlashInfer already near speed of light, adding Zazu's prefill kernel yields a modest end-to-end shift on the same replay (first-turn p50 0.36 → 0.34 s, p99 1.24 → 1.10 s; warm-turn p99 0.85 → 0.74 s).
5 · Accuracy
Shadow checks and 12.8K-token GSM8K stay unchanged
- Kernel verification: every shipped kernel passed correctness and stress tests against a 64-bit floating-point (FP64) reference (127 test cases on Gemma 4 12B, 42 on 26B-A4B, 88 on GPT-OSS-120B).
- Live shadow checks: while serving, the engine ran the baseline kernel alongside ours on the first 24 calls per layer type and every 5,000th call thereafter, checking differences against FP64. None exceeded tolerance.
| Model | 8-shot (vLLM / Zazu) | 64-shot ~12.8K tokens (vLLM / Zazu) | FP64 suite |
|---|---|---|---|
| Gemma 4 12B | 244 / 244 | 241 / 241 | 127 FP64 tests |
| Gemma 4 26B-A4B (sliding only) | 246 / 245 | 241 / 242 | 42 FP64 tests |
| Gemma 4 26B-A4B (all layers) | 246 / 246 | 241 / 241 | 42 FP64 tests |
| GPT-OSS-120B | 236 / 235 | 236 / 238 | 88 FP64 tests |
Why outputs can differ at tiesGreedy outputs diverge from stock vLLM only at near-ties between the top two tokens due to floating-point reduction order—no detectable accuracy difference across 250 questions at either 8-shot or 12.8K-token 64-shot prompts.