ZAZUMODELSBLOG

Model library · Gemma 4 31B · 2026-11-09

Airline simulations

Tests instruction following over a long booking policy.

Allpassfailerror
50 simulations
TaskTrialResultEndedMessagesCost
00passuser_stop16$0.007Transcript →
10failuser_stop24$0.011Transcript →
100passuser_stop54$0.038Transcript →
110failuser_stop14$0.005Transcript →
120passuser_stop31$0.017Transcript →
130passuser_stop14$0.006Transcript →
140failuser_stop35$0.025Transcript →
150passuser_stop27$0.019Transcript →
160failuser_stop27$0.016Transcript →
170passuser_stop29$0.015Transcript →
180failuser_stop31$0.015Transcript →
190passuser_stop23$0.011Transcript →
20passuser_stop154$0.068Transcript →
200passuser_stop32$0.020Transcript →
210failuser_stop38$0.025Transcript →
220passuser_stop21$0.009Transcript →
230failuser_stop42$0.027Transcript →
240passuser_stop29$0.015Transcript →
250passuser_stop40$0.025Transcript →
260passuser_stop17$0.007Transcript →
270passuser_stop28$0.013Transcript →
280passuser_stop18$0.008Transcript →
290failuser_stop25$0.015Transcript →
30failuser_stop18$0.008Transcript →
300passuser_stop21$0.010Transcript →
310passuser_stop20$0.009Transcript →
320passuser_stop38$0.026Transcript →
330failuser_stop40$0.027Transcript →
340failuser_stop23$0.012Transcript →
350failuser_stop35$0.026Transcript →
360failuser_stop32$0.019Transcript →
370passuser_stop27$0.013Transcript →
380passuser_stop20$0.010Transcript →
390passuser_stop26$0.010Transcript →
40passuser_stop23$0.009Transcript →
400passuser_stop16$0.007Transcript →
410failuser_stop22$0.009Transcript →
420failuser_stop28$0.013Transcript →
430failuser_stop29$0.014Transcript →
440failuser_stop30$0.009Transcript →
450passuser_stop20$0.009Transcript →
460passuser_stop8$0.003Transcript →
470passuser_stop21$0.010Transcript →
480passuser_stop14$0.006Transcript →
490passuser_stop14$0.006Transcript →
50passuser_stop29$0.017Transcript →
60failuser_stop57$0.038Transcript →
70failuser_stop28$0.014Transcript →
80passuser_stop30$0.017Transcript →
90failuser_stop24$0.012Transcript →