Model library · Google · 12B
Gemma 4 12B
From the transcripts Solves about half the tasks, but reads its reasoning out to customers in 1,264 of 3,056 replies even with thinking barred, and its empty replies crashed 30 of 278 chats.
MiniMax-strict customer
Customer simulator: minimax/minimax-m3.1-flash (strict customer patch).
GLM-5.3 Flash customer
Customer simulator: glm-5.3-flash (gpt-oss-120b on *_fixoss reruns).
2026-10-07
self-hostedreasoning off
| Domain | pass^1 | Higher k | Sims | Infra errors | Agent cost / task | Transcripts |
|---|---|---|---|---|---|---|
| RetailTool calling | 74.4% | — | 90 | 24 | — | View → |