Model library · 2026-10-27-tau2-gptoss120b-minimax-strict · trial 0 · ended by user_stop
Airline task 40 pass
Tests instruction following over a long booking policy.
- 1
- reward
- 12
- messages
- $0.0007
- agent cost
- $0.0003
- customer cost
- 20 s
- duration
Task
Test agent's capacity to handle a flight change.
Reason for callYou booked the flight and you want to change the passenger name on the reservation.
Customer knowsYou are Anya Garcia. Your user id is anya_garcia_5901. Your reservation id is 3RK2T9.
BehaviorYou want to change the name from Mei Lee to Mei Garcia. Be insistent and don't provide more information than necessary.
Full task definition
{
"id": "40",
"issues": null,
"ticket": null,
"user_tools": null,
"description": {
"notes": null,
"purpose": "Test agent's capacity to handle a flight change.",
"relevant_policies": null
},
"initial_state": null,
"user_scenario": {
"persona": null,
"instructions": {
"domain": "airline",
"known_info": "You are Anya Garcia.\n\nYour user id is anya_garcia_5901.\n\nYour reservation id is 3RK2T9.",
"unknown_info": null,
"reason_for_call": "You booked the flight and you want to change the passenger name on the reservation.",
"task_instructions": "You want to change the name from Mei Lee to Mei Garcia. \n\nBe insistent and don't provide more information than necessary."
}
},
"required_documents": null,
"evaluation_criteria": {
"actions": [
{
"info": null,
"name": "get_reservation_details",
"action_id": "40_0",
"arguments": {
"reservation_id": "3RK2T9"
},
"requestor": "assistant",
"compare_args": null
},
{
"info": null,
"name": "update_reservation_passengers",
"action_id": "40_1",
"arguments": {
"passengers": [
{
"dob": "1992-11-12",
"last_name": "Garcia",
"first_name": "Anya"
},
{
"dob": "1989-12-13",
"last_name": "Garcia",
"first_name": "Mei"
}
],
"reservation_id": "3RK2T9"
},
"requestor": "assistant",
"compare_args": null
}
],
"reward_basis": [
"DB",
"COMMUNICATE"
],
"nl_assertions": [
"Agent updates reservation 3RK2T9 to passenger Mei Garcia."
],
"env_assertions": null,
"communicate_info": []
}
}Grading
2/2 expected actions matched · database matches
- ✓
get_reservation_details{"reservation_id":"3RK2T9"} - ✓
update_reservation_passengers{"passengers":[{"dob":"1992-11-12","last_name":"Garcia","first_name":"Anya"},{"dob":"1989-12-13","last_name":"Garcia","first_name":"Mei"}],"reservation_id":"3RK2T9"}
Raw grading
{
"info": {
"nl": null,
"env": null,
"action": null,
"communicate": {
"note": "No communicate_info to evaluate"
}
},
"reward": 1,
"db_check": {
"db_match": true,
"db_reward": 1
},
"reward_basis": [
"DB",
"COMMUNICATE"
],
"action_checks": [
{
"action": {
"info": null,
"name": "get_reservation_details",
"action_id": "40_0",
"arguments": {
"reservation_id": "3RK2T9"
},
"requestor": "assistant",
"compare_args": null
},
"tool_type": "read",
"action_match": true,
"action_reward": 1
},
{
"action": {
"info": null,
"name": "update_reservation_passengers",
"action_id": "40_1",
"arguments": {
"passengers": [
{
"dob": "1992-11-12",
"last_name": "Garcia",
"first_name": "Anya"
},
{
"dob": "1989-12-13",
"last_name": "Garcia",
"first_name": "Mei"
}
],
"reservation_id": "3RK2T9"
},
"requestor": "assistant",
"compare_args": null
},
"tool_type": "write",
"action_match": true,
"action_reward": 1
}
],
"nl_assertions": null,
"env_assertions": [],
"reward_breakdown": {
"DB": 1,
"COMMUNICATE": 1
},
"communicate_checks": null
}Conversation
AgentCustomer
Agent · 0
Hi! How can I help you today?
Customer · 1
I need to change the passenger name on my reservation from Mei Lee to Mei Garcia.
Agent · 2
I can help update the passenger name for you. Could you please provide:
1. Your user ID (e.g., `sara_doe_496`)
2. The reservation ID you’d like to modify (e.g., `ZFA04Y`)
Once I have those, I’ll proceed with the name change.
Customer · 3
My user ID is `anya_garcia_5901`, and the reservation ID is `3RK2T9`.
Agent · 4
I’ll update the passenger information for reservation **3RK2T9** (user **anya_garcia_5901**) by changing the name **Mei Lee** to **Mei Garcia**.
Please confirm you’d like me to proceed with this change. (Reply “yes” to continue.)
Customer · 5
Yes, please proceed.
Agent · 6
get_reservation_details
{
"reservation_id": "3RK2T9"
}Tool result · 7
47 lines · click to expand
{
"reservation_id": "3RK2T9",
"user_id": "anya_garcia_5901",
"origin": "MCO",
"destination": "SFO",
"flight_type": "one_way",
"cabin": "basic_economy",
"flights": [
{
"flight_number": "HAT214",
"origin": "MCO",
"destination": "PHX",
"date": "2024-05-19",
"price": 57
},
{
"flight_number": "HAT009",
"origin": "PHX",
"destination": "SFO",
"date": "2024-05-20",
"price": 83
}
],
"passengers": [
{
"first_name": "Anya",
"last_name": "Garcia",
"dob": "1992-11-12"
},
{
"first_name": "Mei",
"last_name": "Lee",
"dob": "1989-12-13"
}
],
"payment_history": [
{
"payment_id": "gift_card_2550356",
"amount": 280
}
],
"created_at": "2024-05-02T06:02:56",
"total_baggages": 0,
"nonfree_baggages": 0,
"insurance": "no",
"status": null
}Agent · 8
update_reservation_passengers15 lines · click to expand
{
"passengers": [
{
"dob": "1992-11-12",
"last_name": "Garcia",
"first_name": "Anya"
},
{
"dob": "1989-12-13",
"last_name": "Garcia",
"first_name": "Mei"
}
],
"reservation_id": "3RK2T9"
}Tool result · 9
47 lines · click to expand
{
"reservation_id": "3RK2T9",
"user_id": "anya_garcia_5901",
"origin": "MCO",
"destination": "SFO",
"flight_type": "one_way",
"cabin": "basic_economy",
"flights": [
{
"flight_number": "HAT214",
"origin": "MCO",
"destination": "PHX",
"date": "2024-05-19",
"price": 57
},
{
"flight_number": "HAT009",
"origin": "PHX",
"destination": "SFO",
"date": "2024-05-20",
"price": 83
}
],
"passengers": [
{
"first_name": "Anya",
"last_name": "Garcia",
"dob": "1992-11-12"
},
{
"first_name": "Mei",
"last_name": "Garcia",
"dob": "1989-12-13"
}
],
"payment_history": [
{
"payment_id": "gift_card_2550356",
"amount": 280
}
],
"created_at": "2024-05-02T06:02:56",
"total_baggages": 0,
"nonfree_baggages": 0,
"insurance": "no",
"status": null
}Agent · 10
The passenger name has been successfully updated. If you need any further assistance, just let me know!
Customer · 11
###STOP###