ZAZUMODELSBLOG

Model library · 2026-11-07-tau3-gemma4-12b-minimax-strict · trial 0 · ended by infrastructure_error

Airline task 21 error

Tests instruction following over a long booking policy.

—
reward
0
messages
$0.0000
agent cost
—
customer cost
0 s
duration

Task

Test capacity of agent to reason about shortest flight option. Agent has to reason that return flights needs also to be after the outbound flight. Also not all flights have economy option. Agent also has to reason about payment amounts.

Reason for callYou want to change the return flights for your upcoming Houston to Denver trip. You want to change it to the fastest return trip (including stopover time) possible on the same day as the departure trip (May 27). If the fastest is not available, you are ok with the next fastest.

Customer knowsYour name is Sofia Kim. Your user id is sofia_kim_7287. Your Houston to Denver trip's departure date is May 27.

Customer does not knowYou don't remember your reservation id.

BehaviorYou don't care about money but want to stay in economy. You also want to add one more checked bag. You want to be sure the agent uses your gift card with the smallest balance to pay. You are reactive to the agent and will not say anything that is not asked. You are not good at math so you want the agent to calculate and decide for you. This is urgent. You want to get this done ASAP.

Full task definition
{
  "id": "21",
  "issues": null,
  "ticket": null,
  "user_tools": null,
  "description": {
    "notes": null,
    "purpose": "Test capacity of agent to reason about shortest flight option. Agent has to reason that return flights needs also to be after the outbound flight. Also not all flights have economy option. Agent also has to reason about payment amounts.",
    "relevant_policies": null
  },
  "initial_state": null,
  "user_scenario": {
    "persona": null,
    "instructions": {
      "domain": "airline",
      "known_info": "Your name is Sofia Kim.\n\nYour user id is sofia_kim_7287.\n \nYour Houston to Denver trip's departure date is May 27.",
      "unknown_info": "You don't remember your reservation id.",
      "reason_for_call": "You want to change the return flights for your upcoming Houston to Denver trip.\nYou want to change it to the fastest return trip (including stopover time) possible on the same day as the departure trip (May 27). If the fastest is not available, you are ok with the next fastest.",
      "task_instructions": "You don't care about money but want to stay in economy. \n\nYou also want to add one more checked bag. \n\nYou want to be sure the agent uses your gift card with the smallest balance to pay.\n\nYou are reactive to the agent and will not say anything that is not asked. \n\nYou are not good at math so you want the agent to calculate and decide for you. \n\nThis is urgent. You want to get this done ASAP."
    }
  },
  "required_documents": null,
  "evaluation_criteria": {
    "actions": [
      {
        "info": null,
        "name": "update_reservation_flights",
        "action_id": "21_0",
        "arguments": {
          "cabin": "economy",
          "flights": [
            {
              "date": "2024-05-27",
              "flight_number": "HAT078"
            },
            {
              "date": "2024-05-27",
              "flight_number": "HAT118"
            },
            {
              "date": "2024-05-27",
              "flight_number": "HAT290"
            },
            {
              "date": "2024-05-27",
              "flight_number": "HAT175"
            }
          ],
          "payment_id": "gift_card_6276644",
          "reservation_id": "OBUT9V"
        },
        "requestor": "assistant",
        "compare_args": null
      },
      {
        "info": null,
        "name": "update_reservation_baggages",
        "action_id": "21_1",
        "arguments": {
          "payment_id": "gift_card_6276644",
          "reservation_id": "OBUT9V",
          "total_baggages": 2,
          "nonfree_baggages": 0
        },
        "requestor": "assistant",
        "compare_args": null
      }
    ],
    "reward_basis": [
      "DB",
      "COMMUNICATE"
    ],
    "nl_assertions": [
      "Agent updates reservation OBUT9V return flights to HAT290 and HAT175 on May 27.",
      "Agent assigns payment to gift_card_6276644.",
      "Agent updates reservation OBUT9V to 2 free baggages."
    ],
    "env_assertions": null,
    "communicate_info": []
  }
}

Grading

0/0 expected actions matched

Raw grading
null

Conversation

AgentCustomer

Raw JSON →