Where does PhoneLLM Mini α2 sit on the price–intelligence curve?

1 · Baseline τ²-bench

Out of the box, PhoneLLM trails GPT-4.1

• τ²-bench has three domains that test different skills: telecom, guiding a customer through fixes on their own phone; retail, tool-heavy order changes; and airline, following a long booking policy.
• Retail looks reasonable, but telecom drops sharply against the other models.
• That pointed to a specific cause rather than a generally weak model, so we investigated the drop.

2 · One failure explains the telecom score

PhoneLLM calls tools that belong to the customer

• Telecom tests an agent guiding a customer through a problem with their phone.
• The customer holds the phone, so the benchmark gives them the actions (check the status bar, toggle airplane mode, reset the APN). The agent is meant to ask for each one and act on what the customer reports.
• PhoneLLM chose the right action but tried to perform it itself instead of guiding the customer.
• Its first phone action matched one used in a passing Qwen run of the same task in 92% of conversations.

Same task, same moment: the agent needs the phone's network status

agent:    I need you to run some checks on your
          phone. Can you please check_network_status
          on your phone and tell me what it shows?

customer: [runs check_network_status on the phone]
phone:    Airplane Mode: ON
          Cellular Connection: no_service

customer: It shows Airplane Mode is ON and no
          cellular connection.

agent:    Thanks. Please turn Airplane Mode OFF...

agent:    check_network_status()
system:   Error: Tool 'check_network_status'
          not found.
agent:    check_status_bar()
system:   Error: Tool 'check_status_bar' not found.
agent:    check_network_mode_preference()
system:   Error: ... not found.
agent:    check_apn_settings()
system:   Error: ... not found.
          ... 6 more phone functions, all "not found"
          run ends: too many errors, customer never asked

3 · The patch

PhoneLLM seems to be aggressively tuned for tool use, so a function name in its prompt reads as a tool it can call

• The agent's prompt is about 23,000 characters: the account policy, then a tech-support manual.
• The manual names the customer's phone functions, like check_network_status(), both in a list of actions the customer can take and in troubleshooting guides that say what to check.
• These functions are for the simulated customer to use, and appear in the agent's prompt only for reference. PhoneLLM called them as if they were its own tools.
• Its reasoning was right, though: in 92% of conversations, the first customer function it called was one the customer also used in a passing Qwen run of the same task.

# What the user can do on their device
Here are the actions a user is able to take on their device.
You must understand those well since as part of technical support you
will have to help the customer perform series of actions

## Fix Actions (Write/Modify)
2. **toggle_airplane_mode** - Turns Airplane Mode ON or OFF.
…
### Airplane Mode
You can check if Airplane Mode is ON by using `check_status_bar()`
or `check_network_status()`.
If it is ON, guide the user to use `toggle_airplane_mode()` to turn it OFF.

Step 1 · Label the list as the customer's, not the agent's

# What the user can do on their device
Here are the actions a user is able to take
on their device. You must understand those well
since as part of technical support you will have
to help the customer perform series of actions

1. **check_status_bar** - Shows what icons
   are currently visible in your phone's status
   bar (the area at the top of the screen).

# What the CUSTOMER can do on THEIR OWN phone
(these are NOT your tools)
The functions listed below exist only on the
customer's phone. You, the support agent, CANNOT
call any of them: they are not in your tool list,
and calling one fails. To use one, ask the customer
to run it and tell you the result.

1. **check_status_bar** [CUSTOMER'S PHONE
   ONLY, not your tool] - Shows what icons
   are currently visible in your phone's status
   bar (the area at the top of the screen).

Model reactionInvalid calls fell from 46% to 32%. Labels helped, but all 58 names were still in the prompt, and the model kept calling them.

Step 2 · Rewrite the manual so no customer function is named

• Instructions were not enough: the model still called the names it saw, so we removed them.

### Airplane Mode
You can check if Airplane Mode is ON by
using `check_status_bar()` or
`check_network_status()`.
If it is ON, guide the user to use
`toggle_airplane_mode()` to turn it OFF.

### SIM Card Issues
Use `check_sim_status()` to check the
status of the SIM card.

### Airplane Mode
Ask the customer to look at their status
bar, or to check their network status, and
tell you whether Airplane Mode is ON.
If it is ON, ask the customer to switch
Airplane Mode OFF on their phone.

### SIM Card Issues
Ask the customer to check the SIM card
status on their phone and tell you what
it says.

4 · New performance

Invalid tool calls fell by about 90%, and the pass rate became comparable to GPT-4.1 mini

• Nearly gone: invalid calls are 4.4% of calls, and 39% of tasks still contain one.

5 · Airline

Airline: PhoneLLM does what the customer asks, even when the policy says no

Airline tests whether the agent follows a long policy: which bookings can be cancelled, when compensation is due, and getting a clear yes before every change.

For example, it cancelled a basic-economy booking that the policy says cannot be cancelled, and sent a compensation certificate to a customer who did not qualify.

• We tried a few prompt changes, such as placing the relevant rule next to the decision and asking it to work out booking times with its calculator.
• None of them helped much. The model seems to be quirky when it comes to following instructions.

6 · Where it lands

A fast, low-cost peer to GPT-4.1 mini: good with tool calls, weaker at strict instruction following

Results with and without the correction (pass^4)

Prompts, patches and per-run results are published on GitHub so anyone can rerun the comparison.