The Metric Almost Nobody Reports
Your agent works perfectly in rehearsal. During the live demo, it takes a different path and fails the same task. That's embarrassing onstage. In production, it's a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request.
Most benchmarks hide this variability behind an average. IBM Research just published a diagnostic tool and fix that targets the gap directly: a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions on AppWorld. But it succeeded in all five runs for only 53.0% of tasks—a 24.4-point consistency gap.
Most benchmarks report the first number. IBM built a way to measure the second—and improve it.
The problem isn't capability. It's an orthogonal axis: an agent can be capable and inconsistent at the same time. You can't fix it by throwing a bigger model at the problem.
Why Pass^k Matters More Than Mean@k
Let's clarify the notation, because this is where most evaluation reports bury the lead.
Mean@k is what every leaderboard shows: run a benchmark k times, average the pass rate. If your agent scores 77%, that's Mean@k. It answers