Recirculation

TL;DR: Take a frozen, off-the-shelf transformer. At each token, after the forward pass, take the residual-stream activation from a deep layer and blend a small fraction of it into a shallow layer's activation for the next token's pass. No weights change, no fine-tuning. This "recirculation" turns a feedforward transformer into a genuine dynamical system that can carry state forward at a fixed layer, and on Gemma3 models it cuts perplexity substantially (8.5% training-free; 23% if you tune a tiny gating MLP while keeping the LLM frozen) and cuts GSM8k error ~21% at pass@128. The catch: prefill becomes serial, the big gains are so far mostly a Gemma phenomenon, and several headline numbers need careful unpacking.

Why depth caps state tracking

A transformer with L layers can perform at most L sequential update steps per token. That sounds like a lot until you notice that many things we care about — tracking who believes what, which meaning of a word is active, what the current world state is — are inherently sequential: the state at time t+1 is a function z_{t+1} = f(z_t, x_t) of the state at time t and the new input. An RNN or Kalman filter implements this trivially. A feedforward transformer cannot, in general: theory (Merrill & Sabharwal) says you need depth growing like \log n just to recognize regular languages of length n, and in practice trained transformers learn clever finite-horizon workarounds that eventually break.

There's a subtler failure mode the paper builds on, from Lepori et al.'s "racing thoughts" work. When Gemini reads "Fred took his fishing pole and drove to the bank," shallow layers represent bank as an ambiguous mixture of meanings; only in deep layers does context nudge it toward river edge. But the architecture is feedforward: when a follow-up question ("Will he find an ATM at this bank?") is processed, the shallow layers of later token positions can only attend to the shallow, still-ambiguous representation of bank. If the model commits to an answer using shallow-layer information, it answers as if the disambiguation never happened — and indeed models cheerfully assert the river bank has an ATM. The resolved state exists inside the network, but it lives too deep to be reused.

Lepori et al. showed a surgical fix: copy the activation from a deep layer down to a shallow layer at the critical token, then continue. That intervention cut contextualization errors by 60%. But it required knowing which token is critical and doing a hard replacement.

The trick: leak, don't replace

The paper's move is to ask: what if we do this at every token, but gently? Hard replacement everywhere would push activations wildly out of distribution — the model was never trained with feedback. But a small leak — mix in, say, 10% of the deep-layer vector — might enrich shallow representations without breaking anything.

Why should a shallow layer even be able to read a deep layer's activation without a learned adapter? Because of the residual stream. Every layer adds its contribution to a shared vector that flows to the output; by commutativity of addition, a feature direction has the same direct effect on the logits whether it was written at layer 3 or layer 30. This pressures features to stay aligned across depth — the residual stream is a shared blackboard, not a sequence of private codes. So a 1:1 feature correspondence between layers is a reasonable default, and simple vector mixing might suffice where you'd normally reach for cross-attention or a learned projection.

Recirculation is not looping

Here is the conceptual heart of the paper, and where it's easiest to confuse this with the (very popular) looped transformer. Both add a "recurrent" arrow from a deep layer back to a shallow one. The difference is when the arrow fires relative to the token clock.

In a looped transformer, the arrow stays within one token's column: you run layers 3–6, feed the output back into layer 3 of the same column, run again. That's just a deeper network with weight sharing — recurrence in depth only. State still has to climb the stack; a k-step state update needs k extra loop iterations, and you run out of budget.

In recirculation, the arrow crosses to the next token's column: layer 6's output at step t feeds layer 3 at step t{+}1. Now the same layer can hold z_t at step t and z_{t+1} at step t{+}1 — state can circulate at a fixed depth indefinitely, which is exactly the RNN property. The price is that this cannot be parallelized across the sequence: even during prefill, tokens must be processed one at a time (each step runs two stacks, which parallelize fine on modern hardware, so generation latency is nearly unchanged — but prefill becomes autoregressive).

The mechanism is one equation. Let \boldsymbol{z}_{t,t,l} be the residual stream at input step t, layer l. With source layer s (deep) and destination layer d (shallow), the destination activation used at the next step is

where \alpha \approx 0.07\text{–}0.15, \beta = 1-\alpha (Gemma3 1B) or \beta=1 (4B/12B), and f rescales the source vector to match the destination's L_2 norm — necessary because residual norms grow with depth, and without it the leak overwhelms the destination. That's it. Sweeps over (s, d) pairs give smooth, well-behaved heatmaps: for Gemma3 1B the sweet spot is source 11 → destination 4; for 4B, 18 → 9; for 12B, 35 → 16 — always deep-middle to shallow-middle, with the source 5–20 layers above the destination.

The evidence

Training-free recirculation, hyperparameters picked on a small tuning set, evaluated on ten language-modeling datasets:

Several sanity checks make this hard to dismiss as an artifact. It's not softmax sharpening: temperature tuning alone gives 8.5% on PG19, recirculation alone gives 14.2%, and they combine nearly additively to 19.6%. It's not looping in disguise: identical training-free layer sweeps with looping produce qualitatively different, mostly harmful heatmaps on Gemma3. And the token-level analysis is genuinely interesting: recirculating a single token at position t boosts log-likelihood of tokens at lag k as a power law in k with a measurable tail out to lag 256; content words (adverbs, adjectives, verbs) benefit most, function words (determiners, pronouns, numerals) least — exactly what you'd expect if the mechanism is propagating contextual state rather than generically smoothing.

Downstream: on a two-shot instruction-following task ("say first if the first word is a fruit…"), recirculation with perplexity-tuned hyperparameters cuts error ~25% (4B) and ~75% (12B). On Lepori's contextualization benchmark, 1B and 4B improve on two of three question types; 12B regresses on two of three. Standard multiple-choice benchmarks (MMLU, ARC, PiQA, etc.) show only modest movement, 6/8 slightly positive. The real signal is GSM8k with zero-shot chain-of-thought, where extended generation gives recirculation more room to help: gains at both pass@1 and pass@128, meaning both sharpening (correct answer wins more often) and expansion (correct answers enter the candidate set at all).

Adaptive recirculation: tune the valve, not the model

The natural next step: keep the LLM frozen, but learn the mixing coefficients. The winning variant trains a small 2-layer MLP that reads the (source, destination) embedding pair per token and emits vector-valued \boldsymbol{\alpha}, \boldsymbol{\beta} — per-token, per-dimension gating of the leak. Trained on just 750 documents:

That last comparison is the striking one: a tiny gating network on a frozen model matches full fine-tuning of the recirculation-augmented LLM (23.0% vs 21.6%). On GSM8k, adaptive recirculation cuts error 8.8% (pass@1) and 20.9% (pass@128) on Gemma3 4B — the abstract's "21% GSM8k" number, which, note, is a relative error reduction at pass@128 with the adaptive variant, not a 21-point accuracy jump with the training-free method.

The authors frame this as a broader methodology they call model-design affordances: instead of proposing an architectural change and paying for training runs to evaluate it, probe a frozen model to see which modifications it already accommodates, then use that as the inductive bias for light adaptation. Recirculation working out-of-the-box is the evidence that residual-stream alignment is a real, exploitable affordance.

What to be skeptical about

It's mostly a Gemma story so far. Ministral3, Pythia, Qwen3, and Phi2 all show the same qualitative heatmap structure — a robust mid-stack region where recirculation reduces perplexity — but the magnitude is <0.5% versus ~5% for Gemma3 (untuned normalization/α for the other families, to be fair; Gemma2 and Gemma4 match Gemma3). The authors speculate Gemma's Peri-LN (input and output layer norm per block) keeps late-layer contributions from being drowned out, which would matter here. Until someone gets multi-point gains on Llama- or Qwen-class models, treat the headline numbers as an existence proof, not a general recipe.

Serial prefill is a real cost. Every prefill token requires a sequential pass (two stacks in parallel each). For a 100K-token context this is prohibitive; the proposed blockwise variant (recirculate every K tokens) is unexplored. This also means the perplexity evaluations implicitly measure a regime where you'd pay a large latency tax in production.

Evaluation soft spots. The strongest adaptive downstream results in Table 2 come from tuning the gating MLP on 3,600 examples of the MMLU test split, then evaluating on the full MMLU test set — the authors flag this, and generalization to other benchmarks is the honest signal there, but it's on-balance-positive rather than decisive. The 12B model's spectacular perplexity numbers (up to 35%) come with the footnote that 12B PT is an anomalously weak language model. The 12B also regresses on the contextualization task that motivated the whole method. Hyperparameters (s, d, \alpha, normalization scheme) are family- and somewhat task-dependent, with no automatic selection procedure yet. And there are no timing measurements quantifying the prefill slowdown.

What would change if it holds. The interesting claim isn't "free perplexity points" — it's that pretrained transformers contain a latent recurrent computation you can activate by rerouting activations, because the residual stream keeps features layer-aligned. If that generalizes, it's simultaneously a cheap capability add-on (especially for long-form generation and reasoning, where the pass@128 GSM8k result suggests genuine capability expansion), a new interpretability handle (the token/PoS/lag analyses are effectively causal probes of state propagation), and a design methodology: let the frozen model vote on architecture changes before you spend a training run.

Where to spend your time: Sections 1–2 plus Figures 3–4 for the looping-vs-recirculation distinction (the paper's core intellectual content, and easy to get wrong from the abstract), Section 4.4 for the artifact controls and the elegant lag/part-of-speech analysis, and Section 4.6 for adaptive recirculation. Skim the benchmark tables; they're the weakest part.