Evaluation · Vapi side by side
Real latency on Vapi: Zazu vs the rest
- Model providers love quoting benchmark numbers that don't match up in production.
- We put six models through the same scripted call inside identical Vapi agents, reading numbers straight off Vapi's own latency dashboard.
- In real conditions, Zazu's Qwen3.5-9B was 5.3× faster than GPT-4.1 mini and 3.3× faster than Grok 4.20.
- All at the same price as GPT-4.1 mini.
The silence before each reply is normalised to Vapi's average non-LLM delay plus the agent's own model latency (see Section 2).
Hatched: Vapi's model latency from request to first token (significantly lower on Zazu than on GPT-4.1 mini for this turn).
1 · The test
Zero synthetic benchmarks: tested live on Vapi
- Why live matters: quoted numbers don't match up in production. Testing directly inside Vapi shows what users actually experience.
- One standard Vapi "Riley" clinic-scheduling agent, cloned across six LLMs. Identical system prompts, tool schemas, voice settings, and silence detection.
2 · Latency
Zazu on Vapi: 5.3× faster than GPT-4.1 mini, 3.3× faster than Grok 4.20
Grok 4.20 is Vapi's fastest built-in model1; GPT-4.1 mini is the default industry workhorse.
| Model | Mean model latency on Vapi |
|---|---|
| Qwen3.5-9B on Zazu | |
| Grok 4.20 Non-Reasoning | |
| Gemini 3.1 Flash-Lite | |
| GPT-4.1 mini | |
| GPT OSS 120B (Groq) | |
| GPT-5.6 Sol |
1 Grok 4.20 was tested as Vapi's fastest working built-in model (lower-latency models like Llama 3B were down at testing).
Grey: caller. Colour: agent. Turn window runs from 3 s before caller speech ends to 7 s after. The silence before each reply is normalised for both agents (see below). Zazu is the second call on the Oregon card; GPT-4.1 mini is from the first round.
Normalisation
- What we did: single-call recordings carry natural endpointing variance and recording lag. In the comparison players, silence was adjusted so each turn equals a consistent 0.90 s of mean non-LLM overhead on Vapi plus that agent's measured model latency.
3 · Cost
Zazu costs the same as GPT-4.1 mini, and callers wait 5.3× less
Latency against billed cost on Vapi
- GPT-4.1 mini hits $0.0011/min using cached prompts, but forces callers to wait significantly longer per turn. Zazu matches that price ($0.0011/min, $1.10 per 1,000 minutes) and delivers near-instant response times.
- Grok 4.20 and GPT OSS 120B cost $6.90–$7.60 per 1,000 minutes—about 6× more than Zazu—and both have higher latency.
- Because Vapi treats custom endpoints as bring-your-own ($0 platform surcharge), Zazu's flat pricing goes directly onto your bill with zero platform markup.
4 · Intelligence
Faster, not dumber: Qwen3.5-9B outperforms popular voice-agent models on tau-bench
| Model | $ per 1k min | Reasoning (Telecom) | Tool use (Retail) | Instruction following (Airline) |
|---|---|---|---|---|
| Qwen3.5-9B on Zazu | $1.10 | |||
| Grok 4.20 Non-Reasoning | $7.60 | |||
| Gemini 3.1 Flash-Lite | $1.90 | |||
| GPT-4.1 mini | $1.10 | |||
| GPT OSS 120B (Groq) | $6.90 |
tau-bench evaluates how well models complete realistic voice-agent tasks across three domains:
- Telecom: reason through a multi-step phone diagnostic flow to identify and resolve a customer’s issue.
- Retail: use tools and follow complex schemas to complete a customer’s request accurately.
- Airline: apply cancellation policies consistently while handling a customer’s request.
Average tau-bench score against billed cost
5 · Where it lands
Qwen3.5-9B: a smarter, faster model at the same price
Zazu · Qwen3.5-9B
- Top-tier model latency on Vapi's production telemetry.
- $1.10 per 1,000 call minutes ($0.0011/min flat).
- Strong across multi-step reasoning, tool use, and following complex instructions.
Every other option
- GPT-4.1 mini: same cost ($1.10/1k min), but 5.3× slower in conversation.
- Grok 4.20 and GPT OSS 120B: about 6× higher spend and higher latency.
6 · Appendix
The full calls
Stereo audio: simulated caller in left ear, agent audio in right ear. Unedited recordings: gaps are as recorded, including recording lag.