123456
01020304050607080910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364
01020304050607080910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364
ZAZU / INFERENCE SYSTEMSHEET 01 · REV A
Model library · Alibaba · 9B
Qwen3.5-9B
From the transcripts Top score, but it reasons out loud at the customer: policy checklists, "wait, actually", monologues up to 46,000 words. Not usable as-is.
1MiniMax-strict customer
Customer simulator: minimax/minimax-m3.1-flash (strict customer patch).
2026-10-28
openrouterfp8reasoning off
| Domain | pass^1 | Higher k | Sims | Infra errors | Agent cost / task | Transcripts |
|---|
| AirlineInstruction following | 74.0% | — | 50 | 0 | $0.0135 | View → |
| RetailTool calling | 78.1% | — | 114 | 0 | $0.0095 | View → |
| TelecomReasoning | 99.1% | — | 114 | 0 | $0.0144 | View → |
2GPT-5.2 customer
Customer simulator: gpt-5.2 (reasoning_effort=low).
2026-10-17
self-hostedbf16reasoning off
| Domain | pass^1 | Higher k | Sims | Infra errors | Agent cost / task | Transcripts |
|---|
| AirlineInstruction following | 76.0% | ^2 68.3% · ^3 64.5% · ^4 62.0% | 200 | 0 | — | View → |
| RetailTool calling | 77.2% | ^2 66.5% · ^3 59.6% · ^4 55.3% | 456 | 0 | — | View → |
| TelecomReasoning | 97.4% | ^2 94.7% · ^3 92.1% · ^4 89.5% | 456 | 0 | — | View → |
3GLM-5.3 Flash customer
Customer simulator: glm-5.3-flash (gpt-oss-120b on *_fixoss reruns).
2026-10-07
self-hostedreasoning off
| Domain | pass^1 | Higher k | Sims | Infra errors | Agent cost / task | Transcripts |
|---|
| RetailTool calling | 84.4% | — | 96 | 18 | — | View → |