A playbook learned by the same model running the agent
IBM Research just topped the τ²-bench retail customer service leaderboard with a non-frontier model and zero weight updates. Their approach: let gpt-4.1 watch itself fail and succeed on training tasks, distill those trajectories into behavioral guidelines, then append the guidelines to the system prompt. The resulting agent hits 67.5% Pass^4 on retail tasks—meaning it solves the problem on all four independent attempts 67.5% of the time—outperforming every retail entry in the October 2026 leaderboard snapshot.
The trick isn't a new model architecture or RLHF loop. It's learned memory—a playbook extracted by the same LLM that runs the agent, applied as plain markdown in the prompt. No human annotation, no gradient descent, just gpt-4.1 analyzing its own conversational traces and writing down what worked.
This is ALTK-Evolve, and it's a sharp counterpoint to the industry's "throw a bigger model at it" reflex. The agent improvement comes from better instructions, not better weights.
What τ²-bench actually tests
τ²-bench measures customer service agents under real constraints. Each task pairs the agent with an LLM-simulated customer (powered by gpt-5.5), a set of domain tools—look up an order, issue a refund, rebook a flight—and a written policy document the agent must follow.
Success is binary and strict: the final database state has to match the expected outcome. You can't partially solve a return or mostly cancel a flight. The agent has to extract the right details from a potentially vague customer, stay within policy boundaries, and execute irreversible actions correctly.
The benchmark tests two domains: retail (40 test tasks) and airline (20 test tasks). The agent architecture is stock τ²-bench ReAct with a two-phase THINK → ACT loop. ALTK-Evolve makes exactly one change: a block of learned guidelines appended after the policy in the system prompt.
How the guidelines are extracted
ALTK-Evolve runs the agent on training tasks, collects the full conversational trajectories, and asks gpt-4.1—the same model powering the agent—to analyze successes and failures. The model distills patterns into short behavioral rules: how to authenticate a customer, when to confirm a transaction before executing it, what format to expect for user names.
Each guideline carries a support count: the number of distinct training tasks it appeared in. Higher support means the rule generalized across multiple scenarios rather than overfitting to a single edge case. After extraction, the guidelines are appended to the system prompt as a plain markdown list, unchanged, for evaluation on the held-out test set.
Here's what that looks like in practice:
## Guidelines learned from past task attempts (follow these carefully):
- Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code.
- Always confirm all intended changes with a clear summary and require explicit user approval before executing any database-updating action.
- When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing.
- Take one action per step: never combine a tool call with a user-facing message in the same turn.
Notice what's absent: no fine-tuning dataset, no RL reward model, no human labelers annotating "good" vs "bad" trajectories. The model teaches itself by reflecting on its own work.
The right dose of guidance varies by domain
IBM tested three configurations: baseline (no guidelines), sup3 (only guidelines with support ≥ 3), and lossless (every guideline retained after consolidation, support ≥ 1). Retail kept 31 high-support and 292 lossless guidelines; airline had 14 and 101 respectively.
The results split cleanly by domain.
Retail rewards more guidance
In retail, lossless dominates on both metrics. Mean accuracy climbs from 78.1% to 82.5% (+4.4 pp), and the strict Pass^4 metric—tasks solved on all four trials—jumps from 57.5% to 67.5% (+10.0 pp). The compact sup3 core barely budges the numbers.
Retail tasks appear to need the full playbook. The domain has more edge cases, more policy nuances, more ways for the conversation to diverge. A small set of high-confidence rules isn't enough.
Airline splits capability from reliability
Airline tells a different story. Lossless still leads on mean accuracy: 57.5% → 65.0% (+7.5 pp). But for Pass^4—the reliability metric—the compact sup3 core wins: 35.0% → 45.0% (+10.0 pp), while lossless only reaches 40.0%.
This echoes IBM's earlier finding that guidance delivery is a dial, not a constant. Airline tasks benefit from a tight, high-confidence ruleset that prevents variance. Too many guidelines—even correct ones—introduce noise that hurts consistency.
The takeaway: support count is a tuning knob, not a hyperparameter you set once and forget. The optimal threshold depends on domain complexity, task variance, and whether you're optimizing for average performance or worst-case reliability.
Guidelines make conversations shorter and more expensive
Adding guidelines increases cost, but not in the way you'd expect. The agent doesn't meander longer trying to satisfy more rules—it gets to the answer faster.
On tasks solved under both baseline and lossless configurations, the guideline-equipped agent takes fewer turns: retail drops from 10.59 to 10.16 assistant turns (−0.43), airline from 9.15 to 8.82 (−0.33). The agent isn't talking more; it's making better-informed decisions earlier in the conversation.
Fewer turns means lower end-to-end latency. Each turn is a sequential LLM round-trip, so cutting 0.3–0.4 turns directly reduces time-to-completion for the customer. This isn't a quality-for-speed tradeoff—both improve together.
The cost increase lands entirely on input tokens. Retail's 292-guideline lossless prompt pushes cost per conversation from $0.084 to $0.139 (+65%). Airline's smaller playbook is cheaper: $0.081 to $0.107 (+32%).
In production, most of that input cost is reclaimable with prompt caching. The guideline block is identical across every call in a session, so caching providers can amortize it heavily. You're trading a small, cacheable input overhead for higher accuracy, higher reliability, and faster conversations—a rare win on all three axes.
What this means for agent design
The frontier-model treadmill assumes agent improvement comes from better base models. ALTK-Evolve demonstrates that better instructions can outperform better weights, at least in constrained, policy-driven domains like customer service.
This approach won't generalize to every agent task. Domains without clear success/failure signals, tasks requiring novel reasoning, or scenarios where the guideline extraction itself fails—those still need model capability improvements. But for workflow automation, customer support, and repetitive decision-making under known policies, learned memory offers a practical alternative to throwing GPT-5 at the problem.
The architecture is disarmingly simple. No custom infrastructure, no new training loops, no human annotation budget. Just: run the agent, collect trajectories, ask the model what it learned, append the answer to the prompt. It's the kind of idea that feels obvious in hindsight but required someone to actually test it at scale.
IBM's earlier work applied this to coding agents; now they've shown it works for conversational customer service. The natural question is: what other structured-decision domains are sitting right in this sweet spot, waiting for someone to extract a playbook and top the leaderboard with last year's model?