Model library · 2026-10-07-tau2-ignosis-gemma-4-12b · trial 0 · ended by infrastructure_error
Retail task 24 error
Tests tool calling on order changes.
- —
- reward
- 0
- messages
- —
- agent cost
- —
- customer cost
- 0 s
- duration
Task
Reason for callYou want to cancel the grill, but if the agent asks you to confirm, you regret and want to keep it. You then want to ask which two t-shirts you have ordered in another order, and what materials are they.
Customer knowsYou are Sofia Hernandez, and you live in Seattle, WA, 98193.
Customer does not knowYou do not remember your email address
BehaviorMake everything sound very natural and make up reasons.
Full task definition
{
"id": "24",
"issues": null,
"ticket": null,
"user_tools": null,
"description": {
"notes": null,
"purpose": null,
"relevant_policies": null
},
"initial_state": null,
"user_scenario": {
"persona": null,
"instructions": {
"domain": "retail",
"known_info": "You are Sofia Hernandez, and you live in Seattle, WA, 98193.",
"unknown_info": "You do not remember your email address",
"reason_for_call": "You want to cancel the grill, but if the agent asks you to confirm, you regret and want to keep it. You then want to ask which two t-shirts you have ordered in another order, and what materials are they.",
"task_instructions": "Make everything sound very natural and make up reasons."
}
},
"required_documents": null,
"evaluation_criteria": {
"actions": [],
"reward_basis": [
"DB",
"NL_ASSERTION"
],
"nl_assertions": [
"Agent should tell the user the materials of the two t-shirts are polyester and cotton."
],
"env_assertions": null,
"communicate_info": [
"polyester",
"cotton"
]
}
}Grading
0/0 expected actions matched
Raw grading
nullConversation
AgentCustomer