Agentic memory sounds simple: distill lessons from past work, inject them back into context, and watch performance climb. More experience should mean better outcomes, right?
Not quite. IBM Research just published results from ALTK-Evolve that flip that intuition on its head. They scaled their evaluation to eight models—from a 117B parameter mixture-of-experts to frontier proprietary systems—and found something surprising: agentic memory isn't a feature you switch on. It's a dose you calibrate to the model.
The best configuration for one model can hurt another. And sometimes, the cheapest memory strategy—curated retrieval with only a few task-relevant guidelines—outperforms giving the agent everything it ever learned.
The Setup: Learning Around the Model, Not Inside It
ALTK-Evolve doesn't update model weights. It doesn't replay transcripts. Instead, it mines behavioral guidelines—strategies that worked, mistakes to avoid, edge cases—from an agent's own prior trajectories.
The loop is straightforward:
- The agent attempts tasks and produces trajectories
- ALTK-Evolve extracts guidelines from both successful and failed runs
- It consolidates those guidelines into a reusable set
- At inference time, the agent receives either the full guideline set or a task-relevant selection
No finetuning. No human annotation. The learning happens in the context window, not the parameter matrix. That's exactly why it's cheap to adopt and portable across model families.
IBM tested this on AppWorld—585 multi-step tasks across 9 simulated apps (calendars, messaging, payments). Tasks are scored two ways: TGC (Task Goal Completion—did it finish the job?) and SGC (Scenario Goal Completion—did it pass every variant of a scenario?).
The SGC metric is stricter. An agent that solves a task most of the time but fails on one edge case scores on TGC but gets zero on SGC.
Three Patterns Emerge
Across eight models, three recurring patterns appeared:
1. Strong Models with Headroom Want Everything
DeepSeek-V3.2 (671B MoE) gained +9.5 percentage points in task completion when given its full self-mined guideline set. The model had the capacity to absorb and apply every guideline, including rare edge-case lessons.
Claude Opus 4.6 and GPT-5.5 followed the same pattern. Even near the ceiling on TGC, they still gained +7.1pp and +7.2pp on SGC respectively. Memory kept paying off as long as a remaining failure mode existed to target.
2. Weaker Models Drown in Too Much Context
gpt-oss-120b (117B MoE) is the star of the counterintuitive finding. With a compact core plus a handful of task-relevant guidelines retrieved per task, it gained +16.1pp on both TGC and SGC.
When given the full guideline set instead? The gain was smaller and cost roughly 50% more tokens. Too much memory hurt both accuracy and efficiency.
3. Saturated Models Show No Measurable Gain
GLM-5 (745B MoE) saw zero improvement from memory—whether full or curated. The researchers call this the "saturated pattern" and are careful to note they're describing what they observed, not claiming a proven cause.
The model may have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. Separating those factors is ongoing work.
The Cheapest Strategy Can Also Be the Best
Here's where it gets practical. Injecting a full guideline set inflates every ReAct step's input because the guidelines are re-sent each turn.
For DeepSeek-V3.2, going from baseline to the full guideline set increased token use per task from 148K to 263K—a +78% overhead.
For gpt-oss-120b with curated retrieval? Only +5% tokens. That configuration delivered +16.1pp TGC at near-baseline cost. Better performance did not require more inference spend.
Two practical takeaways:
- Curated retrieval keeps cost near baseline. For weaker models where selection wins on accuracy, it also wins on cost—the best of both worlds.
- Prompt caching makes the full guideline set affordable. The static portion is identical across steps and can be cached, cutting effective cost substantially in production.
The researchers also note that memory doesn't blow up the reasoning loop. DeepSeek-V3.2 ran about the same number of ReAct steps with memory as without (≈18–19 on average). The added cost is input-token inflation, not longer trajectories.
What Determines Which Pattern a Model Falls Into?
It's not simply parameter count. GLM-5 (745B) showed no gain, while DeepSeek-V3.2 (671B) gained substantially.
The researchers hypothesize that benchmark headroom, context-window size, architecture, guideline quality, and task distribution all shape where a model lands. They haven't yet run controlled experiments isolating these factors.
One particularly interesting hypothesis: models with larger context windows may absorb the full guideline set more effectively, while smaller-context models benefit more from retrieval that keeps injected content compact.
The Practical Lesson
The lesson isn't to give an agent everything it has learned. It's to give it the amount of experience it can actually use.
For weak models, that means a compact core plus a few task-specific lessons—which is also the cheapest option.
For strong models with headroom, it means preserving the full guideline set, kept affordable via prompt caching.
For saturated models, it means spending no extra context until their remaining failure modes are better understood.
The gains are real—automatic, leakage-free, requiring no human annotation—but only when the dose fits the model.
What's Missing (and What's Next)
The current retrieval mechanism ranks guidelines by cosine similarity, which the researchers acknowledge doesn't perfectly predict which guidelines help a given task. A learned selector trained on outcome signal is the natural next step.
Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem they're exploring.
And these results are validated on AppWorld—a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress.
Why This Matters
We're past the "can agents learn from experience?" question. The answer is yes, under the right conditions.
The new question is how much experience, delivered how? The ALTK-Evolve results suggest that memory design needs to become model-specific. A one-size-fits-all approach leaves performance—and money—on the table.
For teams shipping agentic systems today, the implication is clear: calibrate, don't just accumulate. Test your memory strategy against your model tier. The cheapest option might also be the best.