Most agent benchmarks look at the final answer. Did the model classify the sentiment correctly? Did it transcribe the audio? Did it produce the right label? But when you're shipping software for agents to drive, the answer is only half the story.
HuggingFace just published a detailed look at benchmarking open models on real tooling, and it introduces a metric that matters more than accuracy: effort. How many turns did the agent take? How many tokens? Did it follow the happy path or debug its way through deprecated APIs?
This is the kind of granular, process-aware evaluation we need if we're serious about building software that agents can actually use.
The problem: not all successes are equal
Two agents can both solve the same sentiment classification task and both get POSITIVE (0.9999). But one writes a 40-line Python script, imports transformers, debugs a shape error, re-runs twice, and finally prints the answer. The other types transformers classify --model ... --text "..." and is done in one call.
Both reach the correct result. But they have wildly different profiles in cost, latency, token usage, and failure modes.
If your evaluation only checks the final string match, you're blind to all of this. You're also blind to whether a library change—better error messages, a new CLI, a Skill definition—actually helped agents.
What HuggingFace measured
The benchmark runs agent tasks across three "tiers" of tooling support:
- bare: just
pip install transformers, nothing else - clone: the full
transformerssource checked out in the working directory - skill: a packaged Skill with the CLI docs and task examples loaded in context
These aren't nested. The Skill doesn't contain the clone (it ships curated docs, not the source tree). Each gives the agent a different kind of help, and as the results show, models sometimes do better on clone than on skill.
Every run is its own Hugging Face Job—one per (model × revision × task)—so the whole sweep runs in parallel on identical hardware. Results and traces land in a Hugging Face Bucket. The full trace of every run is captured and shareable through the Hub's agent-traces viewer.
Large models: hold the model, vary the tool
For large, capable open models, task completion saturates near 100% on common tasks. The interesting question isn't whether they get the right answer—it's how much work it took.
The natural experiment is to fix one strong model and vary the library's revisions: successive git versions of transformers, from released tags like v5.8.0 and v5.9.0 to the specific commit that introduces the CLI and Skill.
The team wanted to know whether adding a dedicated CLI and Skill actually lightened the agents' work. For the three large models tested, the Skill commit resulted in less time spent on tasks. The median time per revision showed the Skill commit (marked as a green dot in their charts) was the fastest.
But there's a tradeoff.
The token cost of discoverability
In the clone variant, median new tokens jumped once the CLI landed in the repo. Why? The commit added the command, but it also shipped the CLI implementation and a set of cli/agentic/*.py usage examples into the repository directly.
With a full transformers checkout in front of it, roughly a third of runs went and read the new surface—the /cli/ tree and example scripts—to learn the interface before calling it. Median input tokens went from ~4k to ~6.4k.
So the commit bought the large models less time (they reach for the CLI instead of debugging Python) at the cost of more tokens (they read the code that taught them the CLI). A tradeoff worth knowing about before merging PRs.
One important caveat: this benchmark measures the worst case. Each run is a fresh agent that rediscovers the CLI from scratch, so it pays the discovery cost every time. In real usage, an agent learns the interface once and then solves task after task within the same session, amortizing that cost across many requests.
Small models: hold the revision, vary the model
Open models vary widely in size and capability. For smaller models, match %—did the final answer contain the expected result—becomes more relevant than for their larger counterparts.
The harness scores every run on multiple axes:
- match %: case-insensitive substring / regex / exact match, all explicit in the report
- median time and median tokens: new vs. cached vs. generated
- runs with error %: including a guard that flags runs which produced nothing (0 output tokens, no tool calls, no answer) so silent failures don't masquerade as zero cost
- marker adoption: tool-defined behavior markers (see below)
All of it lands in a report you can directly examine, with an overview, coverage, and results pages—all client-side.
Markers: measuring tool-specific behavior
A "marker" is a tool-defined behavior pattern you want to observe. In the transformers benchmark, markers track whether the agent used the new CLI or the legacy Python API.
This is a subtle but powerful idea. You're not just measuring pass/fail or token count—you're measuring whether the agent discovered and adopted the interface you built for it.
The team analyzed marker adoption across models and revisions. Did the Skill commit actually push agents toward the CLI? Did smaller models find it? Did the documentation structure matter?
These are the questions you can't answer with a leaderboard that only reports accuracy.
What this means for library maintainers
The post lays out two software principles that remain true in the age of agentic tooling:
If it isn't tested, then it doesn't work. If it isn't documented, then it doesn't exist.
For agents, these two are now directly tied. If you want your tool to exist for an agent, it needs to be discoverable. The API needs to be clear and the docs need to be extensive and structured so the agent has rapid access to useful files and examples.
If you want your tool to work for an agent, you should test it for agentic use.
This is the same recipe recently applied to the hf CLI, redesigned to be agent-optimized, where agents used 1.3–1.8× (and up to 6×) fewer tokens. The team wanted to know whether that kind of win generalizes and whether it could be useful for transformers as well.
Intuition said yes. But intuition isn't enough when you're adding several thousand lines of code to a widely used codebase.
The broader takeaway
Coding agents increasingly work with our software instead of us. Describe a task, and the agent picks the library, writes the calls, runs them, and debugs its own mistakes. When the library gets in the way, it will happily bypass it and rewrite the logic from scratch.
This introduces a new concept in library development: the code should not only be correct and fast, but should be designed so that an agent can drive it effectively. A clunky API or stale docs annoy human developers, but they also send the agent down a longer, more expensive path.
HuggingFace's benchmark harness is open and designed to work with any tool that can be operated from the command line. The implementation is simple: tasks × models × revisions fanned out across Hugging Face Jobs, with traces landing in Buckets.
If you're maintaining infrastructure that agents will use—and at this point, that's most infrastructure—this is the kind of evaluation that tells you whether your changes actually helped.
Not whether the agent can use your tool. Whether it wants to.