The core insight
Most foundation models optimize for general performance across benchmarks. But when you drop them into a specific agent harness—Pi, OpenCode, Hermes Agent, whatever—you're leaving performance on the table. HuggingFace's new harness adapter work demonstrates you can close that gap with reinforcement learning trained inside the production harness itself.
The numbers make the case. Take Qwen3.8-27B running in the Pi harness on Terminal-Bench 2.1: baseline score is 39% pass@1, burning 2.9K tokens per task on average. Add a LoRA adapter trained with RL inside Pi? 55% pass@1, using just 1.5K tokens. That's a 16-point accuracy gain while halving token costs. The model learns to reason less, call fewer tools, and nail it on the first try.
This matters because most agentic RL work trains against a reimplemented agent loop—verl's AgentLoop, SkyRL-Gym environments, custom rollout functions. You're teaching the model to work with a proxy of your production system. HuggingFace's approach trains on the exact tokens your deployed harness produces, captured via a proxy between the harness and vLLM. No translation layer, no abstraction mismatch.
Three adapters, three harnesses, one merged super-adapter
The team trained one LoRA per harness: Pi 1.5.2, Hermes Agent v0.22.6, and OpenCode 2.2.7. Each adapter beats the base model under its own harness on both benchmarks tested, improving both accuracy and efficiency.
Then they did something clever: merged all three LoRAs into a single "super adapter" that runs unchanged across all three harnesses. This hints at something interesting—the adaptations aren't pure overfitting to harness quirks. There's shared structure in what makes a model work better inside any agent framework.
All artifacts live in a Hugging Face collection: the adapters, GGUFs, RL tasks, and a safeagents plugin. The whole training pipeline runs on Hugging Face Jobs using TRL's AsyncGRPOTrainer, OpenEnv, and Harbor.
Building a valid RL task dataset
RL training needs three things: tasks the agent can attempt, a verifier that scores attempts, and evidence the verifier is correct. A verifier that accepts wrong answers teaches the model to fail confidently. A task the base model already solves teaches nothing.
The team built a custom dataset from three sources:
- SWE-smith (Python and Go versions): injects bugs into real repositories via LLM rewrites or mutations like flipped operators. Tests that catch each bug become the task's tests.
- SWE-Gym (packaged as Harbor tasks in TaskTrove): converts real GitHub issues and their merged fixes into tasks.
- Tasksmith (from Repo2RLEnv): generates tasks from merged pull requests, with an LLM agent writing instructions and tests that Tasksmith validates.
Tasksmith targeted gaps: they mapped skills from Terminal-Bench 2.1, SWE-bench Pro's hard subset, and DeepSWE, then chose pull requests exercising HTTP streaming, filesystem operations, serialization. From 30 merged PRs, Tasksmith produced 24 tasks across 10 Python repos.
Validation: two filters
Every task passes through openenv validate running semantic checks in fresh trials:
- Reference solution earns maximum reward
- Doing nothing scores clearly lower
- No canary scores more than doing nothing
- Replay of reference gives same reward
Each trial ran as its own Hugging Face Job, validating dozens of tasks in parallel. 858 of 1,127 Hub candidates passed. Wrong-answer controls caught real defects, including a Terminal-Bench 2.1 verifier that accepted known-wrong answers.
Then a difficulty check: Qwen3.8-27B attempted each task 4 times (8 times for a stratified sample). Of 442 scored tasks, the base model solved 151 every time and 120 never. The 171 it solved sometimes form the qualified set—only these provide learning signal for GRPO, which scores attempts relative to each other on the same task.
AsyncGRPOTrainer: scaling up efficiency
Standard GRPO alternates between generation and training: generate trajectories, train on them, repeat. This leaves compute idle.
AsyncGRPOTrainer pipelines the work. While training on batch N, it generates batch N+1. The team reports this nearly halves wall-clock time per training step. When you're burning GPU hours at 27B scale across multiple harnesses, that's real money.
What did the model actually learn?
This is the open question the team flags. The adapters clearly work—better scores, fewer tokens. But how do they achieve it?
Some of the gain comes from obvious efficiencies: less verbose reasoning, fewer tool calls, better first-time performance. But there's a deeper question about whether the model learned harness-specific tricks ("Hermes Agent doesn't need a summary after tool calls") versus more general agentic improvements ("try the simplest fix first before reasoning extensively").
The merged super-adapter working across harnesses suggests the latter. If each adapter was pure overfitting, merging them would create interference. Instead, they compose.
The production insight
Most teams building agents do this: pick a foundation model, pick or build a harness, evaluate, deploy. If performance isn't good enough, they swap the foundation model or add retrieval or tune prompts.
This work suggests another lever: use RL to teach your chosen model how to work specifically inside your chosen harness on your actual task distribution. Not "generally improve at coding" but "get better at coding in Pi doing the kinds of tasks we see."
The approach requires infrastructure—you need RL training integrated with your harness, a validated task set, compute for 27B-scale training. But the gains are substantial: 16 points of accuracy and 50% token savings isn't rounding error.
Reproducibility as first-class
One thing worth noting: the team built this as a "fully reusable recipe." The tasks, training setup, validation pipeline, and trained artifacts are all public. You can fork the approach for your own harness-model-task combination.
This feels like where agentic RL should be headed. Not just papers with numbers, but runnable pipelines that teams can adapt. The Harbor format, OpenEnv validation, and Hub-based Jobs infrastructure make this more tractable than it would've been a year ago.
Open directions
The team flags future work:
- Scaling to larger models and more diverse task distributions
- Understanding what adapter weights encode about harness structure
- Testing whether adapters transfer to new harnesses with similar designs
- Exploring whether training on multiple harnesses simultaneously beats single-harness adapters
I'd add: does this approach work for non-coding domains? The harness adapter concept should apply anywhere you have a fixed production framework and a task distribution you can sample. Customer support bots, data analysis agents, research assistants.
The efficiency gains alone make this worth exploring. If you're shipping agents at scale, you're paying for every token. Halving that cost while improving accuracy isn't a marginal win—it changes what's economically viable.