Sample-Efficient Learning from Agent Experience

TL;DR — An agent that gets to keep its own trial-and-error history in context often improves dramatically, but the improvement vanishes the moment the context is cleared. This paper asks: can you burn that in-context gain into the weights without paying for any new environment interaction? The trick is a degenerate case of model-based RL's branched rollouts: replay the recorded history, and at each recorded step ask the experience-conditioned model for just one decision — no world model, no new rollouts — then train the experience-free model to imitate those decisions. On 749 software-engineering tasks and six text-adventure games, this retains 64.8–93.4% of the in-context gain, where naive SFT on the same trajectories retains ~0–4%, and it matches PPO/GRPO with 9.6×–57.2× fewer environment samples.

The problem: learning that evaporates

There are two very different ways an LLM agent can learn from interacting with an environment, and they occupy opposite corners of a trade-off.

Reinforcement learning writes into the weights, so what's learned persists — but it's an environment-sample glutton. That's fine when rollouts are cheap (Go self-play, unit-tested code snippets). It's fatal when a rollout means running a real experiment, waiting on CI for a large repo, or asking a human expert.

In-context learning is the opposite: astonishingly sample-efficient. Give the model its own previous failed attempts on a task — what it tried, what broke, what the tests said — and it will often solve the task on the next go. The paper's numbers make this vivid: on their curated SWE benchmark, the base model goes from 5.3% pass@1 zeroshot to 76.4% with its multi-trial history in context. But delete the context and you're back to 5.3%. Nothing was learned in any persistent sense.

The natural bridge is context distillation: train a student (no context) to match a teacher (same model, with the privileged context) — the idea behind Anthropic's early prompt-distillation work and its descendants. The catch, specific to agents: the thing you want to distill is not a static behavior on a fixed prompt, it's a trajectory distribution in an environment. To get supervision from the experience-conditioned teacher, you'd normally have to run it in the environment again — and now you've spent the environment budget you were trying to save. The paper names this problem Experience Distillation: internalize experience-induced gains with zero additional environment interaction.

Why the obvious answers fail

Naive SFT on the collected trajectories is the first thing anyone would try, and it recovers essentially nothing: 3.8% of the ICL gain on SWE, negative on the text adventures. The intuition is that the raw experience is mostly what a struggling agent does — dead ends, repeated exploration, rejected patches. Cloning that behavior teaches the model to be the agent before it learned, not after. The value of the experience is only realized when a model conditions on it and reasons about it; the trajectories themselves are not the improved policy.

A learned world model is the second obvious answer, borrowed from model-based RL: let the experience-conditioned teacher roll out inside a simulated environment (an LLM can play the environment for text tasks). But world-model errors compound over long horizons, and the appendix shows exactly how this dies in practice: their world model hallucinates a ten-point reward for an action it simultaneously says can't be executed ("You can't dance here. [Your score has just gone up by ten points.]"), and the teacher then chases that fake reward; in another rollout, the "observation" leaks policy tokens (<answer>take small black pistol</answer>) into itself. Every subsequent teacher decision conditions on a corrupted prefix.

The aha: shrink the branch to one step and the world model disappears

Model-based RL already has a partial remedy for compounding error: branched rollouts (MBPO-style) — start from a state on a real trajectory and simulate only k steps forward, so error has little room to accumulate. The paper's insight is to take the limit k=1 and notice something structural: a one-step branch contains only a teacher decision and no simulated observation. The world model has nothing to do. It's not approximated away — it's eliminated.

Concretely: you have a recorded experience \tau^{\mathrm{exp}} = (o_0, a_0, \dots, o_T), the agent's own multi-trial history on a task (o_t = observations including feedback, a_t = the model's full turn output). At every recorded prefix h_t^{\mathrm{exp}}, sample what the teacher — the same base model, but conditioned on the full experience — would do next:

then train the student to produce that decision without the experience:

The paper derives this as the local-policy-matching decomposition of the ideal trajectory-level KL objective (Section 3.1–3.2): teacher and student trajectory distributions share the same environment factors, so the KL between them reduces to per-step policy KLs — you just can't afford to evaluate those KLs along fresh teacher trajectories, so you evaluate them along the recorded ones instead. Real states, hindsight-informed actions, zero environment cost.

An ablation validates the limit is the right place to be: on five TaleSuite games, model-free 1-step distillation retains 83.8% of the ICL gain on average, a 2-step branch with one world-model observation retains 57.3%, and full 100-step world-model rollouts collapse (19.2% on the one game where they even report it).

Three implementation details do real work. The raw experience is preprocessed (g(\tau^{\mathrm{exp}}): summarized/rewritten to fit context and raise signal density). The teacher is prompted for extended reasoning before each decision — this alone lifts G_{\mathrm{ICL}} on Detective from 34.9% to 72.5%. And branch packing interleaves the teacher decisions into a single sequence (o_0, a'_0, a_0, o_1, a'_1, a_1, \dots) so one training example supervises every branch of a trajectory instead of reprocessing overlapping prefixes; it cuts teacher-generation + training time by >10× (4096 examples → 128 sequences) with no performance loss — a mild approximation, since later teacher decisions now condition on earlier ones.

The evidence

The headline metric is G_{\mathrm{ICL}} = \frac{S_m - S_{\mathrm{ZS}}}{S_{\mathrm{ICL}} - S_{\mathrm{ZS}}}: what fraction of the zeroshot→ICL gain a weight-updated model retains, evaluated without the experience in context.

Against RL, the pipeline (collect trial-and-error experience → ICL → distill) is compared to PPO on SWE and GRPO on TaleSuite, where all of the pipeline's environment cost is the experience-collection stage:

The distilled capability also transfers, modestly: on 494 held-out SWE tasks (including 278 from unseen repositories and unseen task types), pass@1 roughly doubles, 4.62% → 8.84%. And it accumulates: five collect-and-distill cycles on TaleSuite, clearing the experience context between cycles, take the mean score from 7.1 to 47.0 with only 20 total environment samples per task — a genuine (if small-scale) demonstration of iterated in-context-learn-then-consolidate as a training loop.

Two ablations are worth flagging for anyone who follows distillation methodology. First, teacher-sampled forward KL demolishes student-sampled reverse KL (on-policy distillation) here: 96.7%/98.4% vs 9.1%/0.4% G_{\mathrm{ICL}} on two games at matched compute. The explanation is clean — the experience-free student never samples the good decisions the experience induces, so grading its own trajectories can't inject them; the qualitative appendix shows OPD-trained models spiraling in "wait, no, wait…" loops. This is a useful counterpoint to the recent enthusiasm for on-policy distillation: when the teacher's advantage comes from privileged information rather than general capability, off-policy (teacher-sampled) supervision is what carries the signal. Second, sampled-token cross-entropy matches full-logit KL, so you don't need to store teacher distributions over 80k-token contexts.

There's also a lovely toy diagnostic (Appendix B.2) of why RL is sample-hungry in exactly the settings ICL shines: a 3-button "passcode" task with 18 possible sequences and binary terminal reward. An ICL agent that remembers failures finds the code in ≤18 episodes by elimination. GRPO with group size 256 — 14× the action space — never finds a passcode outside its initial policy's support: on-policy groups sample with replacement from the model's preferences, and when every trajectory in a group fails, the group-relative advantage is exactly zero and the reward gradient vanishes. Memory of failures is a fundamentally different exploration mechanism than policy-gradient sampling.

What to be skeptical about

The main-table evaluation is on the training tasks. The 51.4% SWE and 43.8 TaleSuite numbers measure whether task-specific experience got consolidated into weights — the model is re-attempting the same 749 issues (from a fresh repo state) whose histories it distilled. That's the stated problem, and it's a legitimate one (think: an agent maintaining a specific codebase), but the general-capability payoff is the much smaller OOD result (8.84% pass@1). Don't read the headline numbers as "distillation made the model 10× better at SWE in general."

The experience is curated and self-selected. For SWE, they run 8–12 independent multi-trial processes per task and keep only a trajectory that eventually reached an accepted commit after more than one trial. So the ICL context is guaranteed to contain a validated solution, and — importantly — the ICL cost accounting includes all that exploration. But it means we don't know how EPD behaves when the experience is noisy, contains no success, or contains a wrong accepted answer. The selection-triage question ("does it hold with less curated experience?") is exactly right and the paper doesn't answer it.

Unnamed in-house base models, distinct ones per domain, from a ByteDance-affiliated team. Nothing is reproducible as published, and we can't tell how much depends on the base model's ICL strength (the whole method's ceiling is what ICL delivers).

The SFT baseline is arguably a strawman — training directly on raw multi-trial histories including all the failures. A fairer intermediate would be SFT on only the successful final trials, or on rewritten/cleaned trajectories (the teacher already gets a preprocessed context g(\tau); SFT apparently doesn't). The forward-vs-reverse-KL ablation partially compensates by showing the choice of distillation direction matters enormously, but "SFT recovers 3.8%" overstates the gap to a competent behavior-cloning baseline.

Distillation isn't free, just environment-free. Matching the full 64.8% retention on SWE required teacher-generated data totaling 2.79× the 61.7M-token experience corpus (retention scales from 31.8% at 0.17×). The paper is candid that teacher-generation cost is a limitation. You're trading environment samples for inference compute — a good trade when environments are expensive, a bad one otherwise.

Why it matters, and what to read

If this holds up, it sketches an appealing agent-training loop that looks less like RL and more like sleep consolidation: learn fast and cheaply in context (episodic, ephemeral), then periodically distill the context-conditioned policy into weights via one-step hindsight relabeling of your own recorded states (consolidated, permanent), then clear the context and repeat. The continual-distillation result (five cycles, 20 rollouts/task, 7.1→47.0) is the small-scale existence proof. The mechanism itself — replay real states, query the privileged-context self for one decision, imitate — is simple enough to reimplement in an afternoon and likely to be widely adopted for any setting where trajectories are expensive: real robotics, human-in-the-loop tasks, slow CI.

Most worth your time: Section 3.2–3.3 for the one-step derivation and branch packing (the actual idea, ~2 pages), Section 4.8 for the forward-vs-reverse-KL result, and Appendix B.2 (ColorButton) for the sharpest articulation I've seen of why in-context memory of failures and on-policy exploration are different resources. The SWE case studies in Appendix C are long but genuinely trace specific repair knowledge from failed trials → teacher targets → post-distillation behavior, which is more mechanistic evidence than most papers in this space bother to provide.