The same set of guidelines, fed to gpt-oss-120b, lifted its task completion rate by 16.1 percentage points; fed to GLM-5, the numbers didn’t move at all. In an experiment IBM Research published on August 18, the team tested eight models and landed on one line: the right dose of memory depends on how much headroom the model itself has left.
The experiment used the AppWorld benchmark: 585 multi-step tasks, 168 at normal difficulty and 417 at high difficulty, spread across 9 simulated apps covering calendars, messaging, payments and more.
The method itself isn’t complicated. The agent first runs through the tasks once; behavioral guidelines are then distilled from the successful and failed trajectories into a reusable set, which gets fed back into the context at inference time. There are two ways to feed it in: inject the entire guideline set at every step, or keep a lean core fixed and retrieve task-relevant subsets on demand. Same guideline set, two feeding methods, compared side by side.
Three tiers of models, three optimal strategies
Across the eight models, the split was clean.
The tier with ample headroom — DeepSeek-V3.2 (671B MoE), Claude Opus 4.6 and GPT-5.5 — did best with the full guideline set injected at every step. DeepSeek-V3.2’s task completion rate rose 9.5 points and its scenario completion rate rose 16.1 points; Claude Opus 4.6 rose 4.1 and 7.1; GPT-5.5 rose 2.9 and 7.2.
The tight-capacity tier had just one model: gpt-oss-120b (117B MoE). Dumping the full set in drowned it; switching to selective retrieval lifted both metrics by 16.1 points each — the largest gain of any of the eight models.
GLM-5 (745B MoE) landed in the saturated tier. It’s already near its ceiling on this task set, so with or without memory, both metrics showed a flat 0.0-point change.
Two metrics, read separately
TGC (task goal completion) measures the share of individual tasks the agent finishes fully and correctly. SGC (scenario goal completion) is stricter: a scenario bundles several variants of the same task — different data, different phrasing, different edge cases — and only counts as passed if every variant does.
The value of separating the two is buried in the numbers: DeepSeek-V3.2’s TGC rose only 9.5 points while its SGC rose 16.1. The same pattern holds for Claude Opus 4.6 (4.1 vs. 7.1) and GPT-5.5 (2.9 vs. 7.2). Memory’s payoff lands more on “getting the same kind of task right when it’s phrased differently” than on “solving more brand-new problems outright.” For teams building production systems, the former kind of gain is far scarcer than the latter.
The cost shows up in tokens
The cost column is the most practical part of the paper. With full injection, DeepSeek-V3.2’s token spend per task rose from 148,000 to 263,000, up 78%. gpt-oss-120b, also with full injection, rose from 110,000 to 166,000, up 51%. Switching to selective retrieval, gpt-oss-120b’s spend reached only 116,000, up just 5% — in exchange for a 16.1-point gain.
Putting the two sets of numbers together for a rough cost-effectiveness read: gpt-oss-120b spends 5% more tokens to gain 16.1 points, while DeepSeek-V3.2 spends 78% more tokens to gain 9.5 points (on TGC). The former nets roughly 3.2 points per 1% token increase, the latter about 0.12 — a gap of more than twentyfold. For production services billed by the token, that gap alone is enough to turn “feed the strong model the full memory set” into a trade that doesn’t pencil out.
The learning happens outside the model
The authors add one line that marks the boundary of this method:
Learning happens around the model, not inside it.
What the guidelines change is the agent’s context environment; the model’s weights never move. The upside is that guidelines can be added or withdrawn anytime, audited, and swapped out whenever the model changes. The limit lives in the same place — every gain is bounded by the context window and inference cost, and past a certain point it backfires, as gpt-oss-120b getting drowned by the full guideline set shows.
The work comes out of IBM Research, with ten listed authors including Vatche Isahagian, Evelyn Duesterwald and Vinod Muthusamy; the code and method are published under the name ALTK-Evolve.
For teams currently adding memory to their agents, the data offers an operating order that’s more useful than the conclusion itself: first test the model’s baseline on the target task to see how much headroom is left. Low-headroom models should use selective retrieval; high-headroom models can consider the full set. Models already near the ceiling should save the tokens and put the effort elsewhere.
Sources: IBM Research and Hugging Face technical blogs, CocoLoop; the composition of AppWorld’s 585 tasks, the TGC/SGC percentage-point changes across eight models, and the token costs for each configuration have all been checked against the published experiment data.