Engineering · Prefill attention

We cut voice-agent TTFT in half with 3.8× faster vLLM kernels

  • Voice-agent prompts are mostly system instructions, tool schemas, and history, while each spoken turn adds only a few new tokens.
  • Whenever a call starts or misses prefix cache, slow prefill stalls decode for every active call sharing the GPU.
  • Prefill speed is therefore critical to keeping time to first token (TTFT) low, even at high concurrency.
  • Replacing vLLM's prefill attention kernels makes prefill up to 3.78× faster, reaching 66.6–68.5% of the theoretical speed of light (up from 18.5% on stock vLLM).
  • On reference voice-agent traffic, first-turn TTFT drops up to 43% (and warm-turn p90 up to 2.5×) with unchanged accuracy on GSM8K.
4 min read
  • 3.78× fasterGemma 4 12BPrefill vs. stock vLLM Triton
  • 3.60× fasterGemma 4 26B-A4BPrefill vs. stock vLLM Triton
  • 1.14× fasterGPT-OSS-120BPrefill vs. FlashInfer
  • 400 → 240 msTTFT after 8K token prefillGemma 4 26B-A4B ingesting an ~8,000-token system prompt

Typical prefill step time cost for voice agents

Base vLLM / FlashInferZazu prefill kernel
Shorter bar is faster
Gemma 4 12B18.5% → 68.5% of speed of light
Base vLLM
Zazu
3.78×faster
Gemma 4 26B-A4B18.5% → 66.6% of speed of light
Base vLLM
Zazu
3.60×faster
GPT-OSS-120B60% → 68% of speed of light
Base vLLM
Zazu
1.14×faster
Weighted across cold prompts, warm multi-turn batches, and mixed prefill-plus-decode steps on one NVIDIA RTX PRO 6000 Blackwell (vLLM 0.32.0, BF16). "Speed of light" is the theoretical hardware ceiling for the operation.

1 · Why prefill matters

Voice agents are mostly system prompt, not output

  • In typical voice-agent traffic, system prompts, tool schemas, and prior turns add up to 7.5K–10.5K tokens (~9K typical), while each caller turn adds roughly 64 new tokens and produces a short spoken reply.
  • Continuous-batching engines cap how many tokens run in a single step (8,192 tokens here). When a cold call arrives—or a turn misses the prefix cache—its prefill consumes the step budget and holds up decode tokens for every other live call on the GPU.
  • That makes prefill attention against the paged KV cache the bottleneck on both cold starts and mixed prefill-plus-decode batches.
DWG. NO. PF-1 · VOICE CALL TOKEN FOOTPRINTSCALE — NTS · SHT 01 / 01INITIAL PROMPT · ~9,000 TOKTURNSSYSTEM PROMPT · TOOLS · INITIAL STATEP1COLD ON CALL STARTCACHED KV ON TURNSDETAIL A · MARGINAL CONVERSATION TURNS~64 NEW TOK / TURNTURN 1TURN 2TURN 3TURN 4TURN 5 …FIG 01 · PROPORTIONAL TOKEN BREAKDOWNSPOKEN REPLY < PROMPT BY ~100×
Scale view of a voice call: the starting system prompt makes up nearly the entire input footprint, while each spoken turn adds only a marginal sliver.

Testbed

  • GPU: 1× NVIDIA RTX PRO 6000 Blackwell (96 GB, 600 W).
  • Engine: vLLM 0.32.0, BF16 weights and KV cache, prefix caching on, max 8,192 batched tokens/step, no speculative decoding.
  • Baselines: vLLM Triton unified_attention on Gemma 4; FlashInfer prefill on GPT-OSS-120B.

Four scored step shapes

  • 32 warm turns: ~64 new tokens each on 7.5K–10.5K cached tokens.
  • 96 warm turns: same shape under heavy concurrent load.
  • Cold 8K: one uncached 8,192-token prompt.
  • Mixed step: one 4,096-token prompt chunk on 4,096 cached tokens, plus 48 warm turns.

2 · Kernel speed

3.6–3.8× faster on Gemma 4, 1.14× over FlashInfer on GPT-OSS-120B

  • Every layer shape was benchmarked in isolation—called the exact way vLLM calls it—across local sliding-window and full-context attention layers.
  • Gemma 4 12B (40 sliding, 8 full layers) reaches 68.5% of the card's speed of light versus 18.5% for vLLM's Triton kernel.
  • Gemma 4 26B-A4B reaches 66.6% of speed of light, and GPT-OSS-120B reaches ~68% (up from FlashInfer's 60%).

Gemma 4 12B · 3.78× weighted speedup (cold 8K: 211 ms → 63 ms)

vLLM TritonZazu kernel
µs per layer per step · Lower is better
Sliding · 32 warm turns
vLLM Triton
906 µs
Zazu kernel
279 µs
3.25×
Sliding · 96 warm turns
vLLM Triton
2,458 µs
Zazu kernel
705 µs
3.49×
Sliding · Cold 8K
vLLM Triton
2,506 µs
Zazu kernel
494 µs
5.07×
Sliding · Mixed step
vLLM Triton
2,668 µs
Zazu kernel
548 µs
4.87×
Full · 32 warm turns
vLLM Triton
8,737 µs
Zazu kernel
3,790 µs
2.31×
Full · 96 warm turns
vLLM Triton
23,407 µs
Zazu kernel
9,410 µs
2.49×
Full · Cold 8K
vLLM Triton
13,903 µs
Zazu kernel
5,418 µs
2.57×
Full · Mixed step
vLLM Triton
23,563 µs
Zazu kernel
9,021 µs
2.61×
Weighted across all four step shapes (2× weight on 96 warm turns and mixed steps, 1× on 32 warm turns and cold 8K) and scaled by layer count (40 sliding, 8 full).

Gemma 4 26B-A4B · 3.60× weighted speedup

vLLM TritonZazu kernel
µs per layer per step · Lower is better
Sliding · 32 warm turns
vLLM Triton
824 µs
Zazu kernel
237 µs
3.47×
Sliding · 96 warm turns
vLLM Triton
2,204 µs
Zazu kernel
684 µs
3.22×
Sliding · Cold 8K
vLLM Triton
2,502 µs
Zazu kernel
534 µs
4.68×
Sliding · Mixed step
vLLM Triton
2,490 µs
Zazu kernel
600 µs
4.15×
Full · 32 warm turns
vLLM Triton
8,183 µs
Zazu kernel
2,940 µs
2.78×
Full · 96 warm turns
vLLM Triton
21,444 µs
Zazu kernel
7,506 µs
2.86×
Full · Cold 8K
vLLM Triton
13,847 µs
Zazu kernel
4,889 µs
2.83×
Full · Mixed step
vLLM Triton
21,753 µs
Zazu kernel
7,599 µs
2.86×

GPT-OSS-120B · 1.14× weighted speedup over FlashInfer (+13.9%)

FlashInferZazu kernel
µs per layer per step · Lower is better
Sliding · 32 warm turns
FlashInfer
45.1 µs
Zazu kernel
39.1 µs
1.15×
Sliding · 96 warm turns
FlashInfer
125.9 µs
Zazu kernel
110.5 µs
1.14×
Sliding · Cold 8K
FlashInfer
130 µs
Zazu kernel
94.1 µs
1.38×
Sliding · Mixed step
FlashInfer
123.1 µs
Zazu kernel
102.8 µs
1.20×
Full · 32 warm turns
FlashInfer
1,094 µs
Zazu kernel
1,054 µs
1.04×
Full · 96 warm turns
FlashInfer
3,526 µs
Zazu kernel
3,233 µs
1.09×
Full · Cold 8K
FlashInfer
1,824 µs
Zazu kernel
1,631 µs
1.12×
Full · Mixed step
FlashInfer
3,020 µs
Zazu kernel
2,759 µs
1.09×

Why the gap differsFlashInfer on GPT-OSS-120B already ran at roughly 60% of speed of light (leaving ~14% headroom to our ~68%); vLLM's Triton kernel on Gemma 4 ran near 19%.

3 · Layer specialization

One kernel per attention layer type, not one-size-fits-all

  • Modern models mix very different attention geometries inside a single forward pass—for example, Gemma 4 12B pairs 40 local sliding-window layers with 8 full-context layers, and each layer type uses its own head count, context span, and KV layout.
  • A single generic prefill kernel leaves speed on the table when forced across all of them, or causes full-context layers to miss the fast path entirely.
  • Specializing our kernels for each attention layer type and its KV geometry unlocked an extra 32–36% speedup on Gemma 4 12B's full layers and 61–69% on Gemma 4 26B-A4B's full layers, while matching double-precision (FP64) reference outputs.
32–36%Gemma 4 12B full-layer gain from specialization
61–69%Gemma 4 26B-A4B full-layer gain vs vLLM
40 + 8Sliding + full layers on Gemma 4 12B

4 · Reference voice traffic

Two-step serving optimization: KV cache capacity + fast prefill

We replayed reference traffic from a voice-agent company on the same RTX PRO 6000—~9K-token prompts adding ~64 new tokens per turn—at two traffic levels (peak and moderate call volume).

  • Step 1 · Tightly coupled KV caching: By coupling Zazu directly with the voice-agent runtime, we achieved near-perfect cache hit rates and drove single-GPU concurrency up sharply. At that density, however, the card became compute-bound on long prefills, starving new first turns.
  • Step 2 · Prefill kernel optimization: Replacing the prefill attention kernels broke that compute bottleneck, clearing the queue so both cold starts and warm turns stay fast across the board.

Gemma 4 12B · 1.5× lower first-turn p50 and 1.7× lower first-turn p90 vs stock vLLM

Stock vLLMZazu
First-turn TTFT in seconds · Lower is better
Peak traffic · First-turn p50 TTFT
Stock vLLM
1.28 s
Zazu
0.84 s
1.5×
Peak traffic · First-turn p90 TTFT
Stock vLLM
3.80 s
Zazu
2.20 s
1.7×
Moderate traffic · First-turn p50 TTFT
Stock vLLM
0.91 s
Zazu
0.69 s
1.3×
Moderate traffic · First-turn p90 TTFT
Stock vLLM
2.34 s
Zazu
1.61 s
1.5×
At peak traffic, Zazu beats stock vLLM across every percentile (warm-turn p50 also drops from 1.05 s to 0.46 s).

Gemma 4 26B-A4B · 1.7× lower first-turn p50 and 1.75× lower first-turn p90

Stock vLLMZazu
First-turn TTFT in seconds · Lower is better
First-turn p50 TTFT
Stock vLLM
0.40 s
Zazu
0.24 s
1.7×
First-turn p90 TTFT
Stock vLLM
0.70 s
Zazu
0.40 s
1.75×
Comparison shows Stock vLLM and Zazu with all-layers specialized prefill kernels. The only metric where stock vLLM leads is median warm-turn latency (p50: 0.08 s vs 0.10 s).

GPT-OSS-120B · Modest end-to-end gain, matching the 1.14× kernel margin

Serving noteWith FlashInfer already near speed of light, adding Zazu's prefill kernel yields a modest end-to-end shift on the same replay (first-turn p50 0.36 → 0.34 s, p99 1.24 → 1.10 s; warm-turn p99 0.85 → 0.74 s).

5 · Accuracy

Shadow checks and 12.8K-token GSM8K stay unchanged

  • Kernel verification: every shipped kernel passed correctness and stress tests against a 64-bit floating-point (FP64) reference (127 test cases on Gemma 4 12B, 42 on 26B-A4B, 88 on GPT-OSS-120B).
  • Live shadow checks: while serving, the engine ran the baseline kernel alongside ours on the first 24 calls per layer type and every 5,000th call thereafter, checking differences against FP64. None exceeded tolerance.
Model8-shot (vLLM / Zazu)64-shot ~12.8K tokens (vLLM / Zazu)FP64 suite
Gemma 4 12B244 / 244241 / 241127 FP64 tests
Gemma 4 26B-A4B (sliding only)246 / 245241 / 24242 FP64 tests
Gemma 4 26B-A4B (all layers)246 / 246241 / 24142 FP64 tests
GPT-OSS-120B236 / 235236 / 23888 FP64 tests
GSM8K greedy accuracy (correct out of 250 questions). Exact McNemar test gives p ≥ 0.73 for every pair.

Why outputs can differ at tiesGreedy outputs diverge from stock vLLM only at near-ties between the top two tokens due to floating-point reduction order—no detectable accuracy difference across 250 questions at either 8-shot or 12.8K-token 64-shot prompts.