ZAZUMODELSBLOG

Model library · Gemma 4 12B · 2026-11-07

Airline simulations

Tests instruction following over a long booking policy.

Allpassfailerror
50 simulations
TaskTrialResultEndedMessagesCost
00passuser_stop16$0.002Transcript →
10errorinfrastructure_error0$0.000Transcript →
100failuser_stop30$0.009Transcript →
110failuser_stop26$0.007Transcript →
120failmax_steps202$0.054Transcript →
130passuser_stop14$0.002Transcript →
140failuser_stop50$0.022Transcript →
150errorinfrastructure_error0$0.000Transcript →
160failmax_steps202$0.049Transcript →
170errorinfrastructure_error0$0.000Transcript →
180failuser_stop86$0.051Transcript →
190errorinfrastructure_error0$0.000Transcript →
20failuser_stop38$0.008Transcript →
200errorinfrastructure_error0$0.000Transcript →
210errorinfrastructure_error0$0.000Transcript →
220passuser_stop56$0.024Transcript →
230failuser_stop60$0.029Transcript →
240failuser_stop8$0.001Transcript →
250passuser_stop42$0.015Transcript →
260passuser_stop18$0.004Transcript →
270passuser_stop16$0.004Transcript →
280passuser_stop24$0.005Transcript →
290failuser_stop38$0.012Transcript →
30failuser_stop12$0.001Transcript →
300passuser_stop30$0.006Transcript →
310passuser_stop16$0.002Transcript →
320errorinfrastructure_error0$0.000Transcript →
330failuser_stop36$0.010Transcript →
340failuser_stop32$0.008Transcript →
350failuser_stop32$0.007Transcript →
360passuser_stop12$0.002Transcript →
370failuser_stop36$0.012Transcript →
380failuser_stop54$0.020Transcript →
390errorinfrastructure_error0$0.000Transcript →
40errorinfrastructure_error0$0.000Transcript →
400passuser_stop12$0.002Transcript →
410passuser_stop34$0.008Transcript →
420failuser_stop62$0.030Transcript →
430passuser_stop24$0.007Transcript →
440failuser_stop62$0.025Transcript →
450passuser_stop32$0.010Transcript →
460passuser_stop12$0.002Transcript →
470passuser_stop22$0.005Transcript →
480passuser_stop12$0.002Transcript →
490passuser_stop14$0.002Transcript →
50errorinfrastructure_error0$0.000Transcript →
60passuser_stop18$0.003Transcript →
70errorinfrastructure_error0$0.000Transcript →
80failmax_steps202$0.047Transcript →
90failuser_stop54$0.021Transcript →