Model library
τ²-bench scores for every model we have run, with the transcript behind every score. τ²-bench (read "tau-squared bench") tests whether an AI model can do a customer-service job from start to finish. The model plays the agent: it gets the company's policy, tools for the company's systems and a live conversation with a simulated customer who has a goal. A task passes only if the system ends in the correct state, so a friendly reply that does the wrong thing fails.
- 86.8%
- best mean score · Alpie Seed 9B
- $0.0035
- lowest cost per task · gpt-oss-120B
- 10
- models compared
The three domains
- AirlineInstruction following50 tasksFlight bookings under a detailed fare and baggage policy. Change or cancel a reservation, add bags or passengers, upgrade a cabin, ask for compensation after a delay. The hard part is applying the rules exactly, such as refusing a change the fare does not allow.
- RetailTool calling114 tasksAn online shop. Find the customer's orders, then cancel one, return or exchange items, or change an address or payment method. Many requests combine several actions, and the policy limits what is allowed once an order has shipped or been delivered.
- TelecomReasoning114 tasksA phone provider, with a customer whose phone has no signal, no mobile data or cannot send picture messages. The agent diagnoses the fault and talks the customer through steps on their own phone, such as turning airplane mode off, while also fixing the account side.
Cost against score
Leaderboard
| Model | Mean | AirlineInstruction following | RetailTool calling | TelecomReasoning | Cost / task | Perfect cache |
|---|---|---|---|---|---|---|
| Alpie Seed 9B 2026-11-08A Qwen3.5-9B fine-tune that edges past its base and does not think out loud (no leaked reasoning in 4,565 replies), but it still reads long analyses to customers: 160 replies run past 300 words. | 86.8% | 76.0% | 85.2% | 99.1% | $0.011 * | $0.0063 * |
| Qwen3.5-9B 2026-10-28Top score, but it reasons out loud at the customer: policy checklists, "wait, actually", monologues up to 46,000 words. Not usable as-is. | 83.7% | 74.0% | 78.1% | 99.1% | $0.012 | $0.0069 |
| gpt-oss-120B 2026-10-27Quiet, policy-correct tool use, but long table-heavy replies, and it hands off to a human too readily (62% of airline chats). | 70.9% | 60.0% | 72.8% | 79.8% | $0.0035 | $0.0029 |
| LFM3-1.7B 2026-10-28Handles simple returns, but leaks stray </think> tags into replies, invents tool arguments and loops tool calls in telecom (one call 94 times). | 70.6% | 60.0% | 78.1% | 73.7% | $0.013 * | $0.0045 * |
| Grok 4.20 2026-10-27Thorough and careful, but tells customers about policy and tool names ("before I call the tool…") in long markdown replies. | 70.3% | 45.0% | 87.7% | 78.1% | $0.156 | $0.039 |
| Gemma 4 31B 2026-11-09Brief and never thinks out loud, unlike the 12B, but gives up on telecom: it hands customers to a human in 54 of 114 chats, and all 34 of its telecom failures end that way. | 69.7% | 60.0% | 79.0% | 70.2% | $0.019 | $0.0071 |
| GPT-4.1 mini 2026-10-28Calm, concise and checks identity first, but takes customers at their word on status and escalates stuck telecom cases to a human. | 60.0% | 44.0% | 65.8% | 70.2% | $0.036 | $0.012 |
| Gemini 3.1 Flash-Lite 2026-10-27Terse, but guesses customer IDs and phone numbers instead of asking, then transfers to a human; half of retail chats end that way. | 50.2% | 48.0% | 42.1% | 60.5% | $0.024 | $0.0053 |
| Gemma 4 12B 2026-11-07Solves about half the tasks, but reads its reasoning out to customers in 1,264 of 3,056 replies even with thinking barred, and its empty replies crashed 30 of 278 chats. | 48.0% | 38.0% | 55.3% | 50.9% | $0.0085 * | $0.0085 * |
| Mistral NeMo 12B 2026-11-07Cheap and brief, but invents order IDs and tool names instead of asking, then tells customers false things ("we don't have any desk lamp in stock"). | 16.3% | 12.0% | 18.4% | 18.4% | $0.0044 | $0.0030 |
pass^1 in percent: the share of tasks passed on one attempt. Every model faces the same customer simulator (minimax/minimax-m3.1-flash (strict customer patch); τ² 1.0.1, seed 300). The line under each model is our verdict from reading its transcripts, not from the score. A dash means that domain was not run or no cost was recorded. Cost per task is the agent model's spend only, at the full input rate with no cache discount, or, in the last column, with each turn reading the previous turn's prompt from cache. * priced from logged tokens at the published per-token rate. Click a column to sort.