The Core Tension
Give an LLM agent a realistic multi-step task—split a bill, find a song, reconcile an order across nine simulated apps—and when it fails, it usually isn't for lack of knowledge. It mis-paginates an API, resolves the wrong person, or returns a value when none was asked for.
The model knows the APIs. What it hasn't internalized is how to use them reliably.
IBM Research's ALTK-Evolve and Microsoft's ACE (Agentic Context Engineering) both tackle this the same way: turn an agent's past trajectories into reusable lessons and feed them back at inference time. No weight updates, no human labels. Just learning from your own history.
They even agree on the hard part—how to store what you've learned without collapsing it into useless summaries. But they differ dramatically on delivery, and that difference shows up as a 3–7× gap in inference cost.
What Both Systems Agree On
ACE named the failure modes first: brevity bias—optimization collapsing toward short, generic instructions—and context collapse—a model asked to rewrite its whole context each step summarizing the detail away.
Its answer: keep a rich, itemized playbook with helpful/harmful counters on every bullet, and let the model distill relevance at read time.
ALTK-Evolve reaches the same conclusion from a different angle. Every guideline keeps a support count—how many independent episodes produced it—and the system never summarizes the store down to a handful of rules. A lesson five different tasks discovered is a different object from one that appeared once, and both are worth keeping.
So on the core question—should you compress an agent's hard-won lessons into a tidy summary?—both systems say no.
Count them, don't collapse them. ACE's per-bullet counters and ALTK-Evolve's support counts are two spellings of the same idea.
Where They Part Ways
Two places matter: how the memory is built, and how it's delivered. The delivery difference is what drives the token bill.
Consolidation (How the Store Is Built)
ACE grows one comprehensive playbook through a Generator → Reflector → Curator loop, applying incremental delta updates and de-duplicating by embedding.
ALTK-Evolve clusters near-duplicate lessons and merges within a cluster, support-conserving—when several lessons merge, the survivor inherits their combined count. The store shrinks without losing the record of how much experience backs each guideline.
It also extracts typed guidelines—strategy, recovery, and optimization—with causal attribution and provenance back to the source trajectory, at subtask granularity. A lesson learned on one app can transfer to another.
Delivery (What Reaches the Model at Inference)
This is the one that decides the numbers.
ACE injects the comprehensive playbook on every step, the same way regardless of model or task.
ALTK-Evolve treats delivery as a dial, not a constant: a small fixed core of high-support guidelines, extended per task with a handful selected for the task at hand (cosine or LLM-guided, priority-weighted)—or, when a model has the headroom to use it, the full consolidated set.
The same lessons are available to both agents. The difference is that ACE always sends all of them, and ALTK-Evolve sends however many a given model can actually use.
The Numbers
On AppWorld, with the same base ReAct agent, running both systems in-house:
DeepSeek-V3.2:
- ACE: 80.4% Task Goal Completion (TGC), 634K tokens/task
- ALTK-Evolve: 89.3% TGC, 263K tokens/task
gpt-oss-120b:
- ACE: 54.8% TGC, 777K tokens/task
- ALTK-Evolve: 56.0% TGC, 116K tokens/task
On the strong model, ALTK-Evolve wins on both metrics at ~40% of ACE's inference cost.
On the weak model, the systems tie on accuracy (56.0 vs. 54.8 is within run-to-run noise) at about one-seventh the cost.
A fair word on positioning: ACE's efficiency story is about building its context cheaply. ALTK-Evolve's is on a different axis—serving it. Retrieving a few guidelines per task instead of injecting the whole playbook on every step is where the tokens go.
Why the Accuracy Gap Exists
The by-difficulty breakdown tells two different stories depending on the model.
Weak Model: Selective Beats Comprehensive
On gpt-oss-120b, ACE's full playbook has the edge on Easy (84.2% vs. 82.5%) and Medium (60.4% vs. 56.2%) tasks. There's enough of the task solved by generic instruction-following that a comprehensive prompt helps more than it distracts.
But on Hard tasks, where the model has to pick the right lesson rather than wade through all of them, curated retrieval pulls ahead: 31.8% vs. 23.8%.
That's the tier that decides the aggregate.
Strong Model: More Lessons Keep Helping
On DeepSeek-V3.2, the story flips. The stronger model absorbs ACE's full playbook well enough to edge ALTK-Evolve on Medium tasks (100.0% vs. 97.9%).
But ALTK-Evolve leads on Easy (94.7% vs. 84.2%), Hard (77.8% vs. 61.9%), and Overall (89.3% vs. 80.4%).
With more capacity to spare, more lessons—delivered the right way—keep helping instead of crowding each other out.
ALTK-Evolve gives each model its best configuration: the full consolidated set for the strong model, selective retrieval for the weaker one. A large context overwhelms a weaker model rather than helping it.
What This Means for Practitioners
If you're building agentic memory systems, the takeaway isn't "ACE bad, ALTK-Evolve good." Both systems validate the same core insight: don't compress. Keep the itemized, counted lessons.
The practical question is delivery.
If you're running a capable model with deep context windows and you can afford the tokens, injecting the whole playbook might work fine—especially if your tasks cluster around medium difficulty where generic coverage helps.
But if you're working with weaker models, tight token budgets, or tasks that punish distraction (the Hard tier), calibrated retrieval is the difference between guidance that helps and guidance that gets in the way.
The same lessons, delivered differently, bought ALTK-Evolve 85–93% cost savings at parity-or-better accuracy.
The Bigger Pattern
This isn't just about two systems. It's about a broader trend in agent architectures: the shift from monolithic context to composable memory.
Early agentic systems treated context like a knapsack—pack everything in and hope the model figures it out. That worked when contexts were short and models were improving fast enough to absorb the slack.
Now we're seeing the limits. Not every model has DeepSeek-V3.2-class context handling. Not every deployment can afford 600K+ tokens per task. And even when you can afford it, compression isn't the answer—brevity bias and context collapse are real.
The alternative is indexed, attributed, support-counted memory with retrieval tuned to the model's actual capacity. ALTK-Evolve demonstrates that this isn't a theoretical nicety—it's a 7× cost lever on real benchmarks.
Open Questions
How much to inject, and how it scales across the capability spectrum, is still open. IBM flags this as the subject of a next post, and it's the right question.
Does the optimal retrieval budget scale linearly with model size? Does it vary by task domain? How do you tune the core vs. task-specific split without overfitting to a single benchmark?
And the meta-question: if delivery calibration matters this much, should it be dynamic at inference time—per task, per model state, per step—rather than fixed at configuration?
The ALTK-Evolve library is open, so we'll find out.