When and How Long? The Readout-Mediator Angle in Temporal Reasoning

TL;DR: This paper takes the most common inferential move in interpretability — "a linear probe decodes feature X with near-perfect accuracy, therefore the model represents and uses X this way" — and breaks it cleanly. On calendar-date reasoning, a probe reads day-of-year from a layer's activations at R^2 = 0.996, yet deleting the probe's direction costs the model 0.6 percentage points; deleting a rank-4 subspace found by causal search at the same layer, same rank drops accuracy from 42% to 0%. The two subspaces sit at 88° from each other — statistically indistinguishable from two random subspaces. The paper names this the readout–mediator angle, gives it a closed-form null, replicates it across four model scales and three task domains, and reverse-engineers the actual circuit the probe never touches. If you read probing papers, this changes what those results are allowed to mean.

The move everyone makes

Interpretability has a workhorse: freeze a model, cache activations at some layer, fit a linear regression from those activations to a property of interest (a date, a location, a sentiment), and report the accuracy. When it's high — and it usually is — the temptation is to conclude the model "has" a representation of that property, and often further, that the model uses it.

People have long suspected this inference is leaky. Hewitt & Liang showed probes can decode random labels; Elazar et al.'s amnesic probing erased probe-found directions and measured downstream effect. But the critiques were qualitative: nobody had a single number quantifying how far the probe's direction sits from the direction the model actually computes with, nor a principled baseline for what "far" means.

This paper supplies both, and the answer is maximally bad for probes.

Two questions, two subspaces

The setup contrasts two optimization problems run at the same layer L and same rank k:

The probe asks where is the information readable? Fit a ridge regression from activations x \in \mathbb{R}^d to (\sin, \cos) of the day-of-year, take the top-k span of the weights: subspace U_P.

DAS (Distributed Alignment Search) asks where is the computation vulnerable? Parametrize a k-dimensional subspace U via QR of a trainable matrix, install a hook x \mapsto x - U^\top U x that zeroes that subspace on every forward pass, and train U (model frozen) to maximize damage to the task loss. The converged subspace U_M is the mediator.

Then measure the mean principal angle \bar\theta(U_P, U_M) = \frac{1}{k}\sum_i \arccos \sigma_i(U_P U_M^\top), where the \sigma_i are singular values of the cross-projection — the cosines of the angles between the closest-matching direction pairs.

Here's the part that requires care, and where the paper earns its "aha." In high dimensions, any two low-rank subspaces are nearly orthogonal by default. The paper proves (via the Jacobi ensemble of random matrix theory) that for random k-planes in \mathbb{R}^d, each squared cosine concentrates at k/d, so the expected angle is \arccos\sqrt{k/d}. At d = 2304 (Gemma 2 2B's residual width) and k = 2, that's 88.3°. So a large angle alone proves nothing. The finding is sharper: the measured probe–DAS angle is 88°, indistinguishable from the random-subspace null (p = 0.51–0.72 across ranks). The probe direction carries literally no more information about the model's computation than a randomly drawn plane of the same rank.

Orthogonality alone still wouldn't prove the probe is inert — an orthogonal subspace could matter through other pathways. So the paper pairs the angle with a specificity ratio \rho_k = (\text{DAS ablation drop}) / (\text{mean random ablation drop}), with a proven null scale of \rho_k^{\text{null}} \asymp d/k. On Gemma 2 2B, d/k = 576 and observed \rho_4 = 1050: DAS found genuinely directed structure. The probe sits at \rho \approx 1 — noise.

Why should this happen generically? Proposition 1 gives the clean intuition: the probe solves a Rayleigh quotient set by the data covariance — a second-moment property of activations (u_P \propto \Sigma^{-1}\mathbb{E}[xz]). The mediator maximizes ablation damage, which to first order is set by the gradient covariance of the network's output (u_M = top eigenvector of \mathbb{E}[\nabla_x f \, \nabla_x f^\top]). One is determined by the data geometry, the other by the network's sensitivity geometry, and there's no structural reason for a deep network to align them. They even verify the mechanism: a subspace built directly from output gradients sits 2.3° closer to the mediator than the null — leaning the predicted way, but diffuse (effective rank 76); DAS's optimization distills the concentrated rank-4 core.

The circuit the probe never touches

What makes this more than a geometry paper is that they open the mediator subspace and trace what actually computes "how many days between March 15 and June 22."

Boundary heads. A QK-scan over per-day mean activations finds 24 attention heads whose query–key structure implements offsets concentrated at ±30 and ±61 days — single-month and double-month steps. This is Gregorian arithmetic showing through: 30 is the minimal month length (ex-February), 61 is the most common two-month span, and \{30, 61\} greedily tiles any multi-month duration. Notably, ±7 days is absent despite weekly periodicity saturating pretraining text — the circuit reflects task structure, not corpus statistics. The same offset modes appear independently in Qwen (Monte Carlo p = 0.009). No single head matters; the top 10 together cost 17.2 pp, with the routing bottleneck at layers 11–12.

Two-stage MLPs. Decomposing what each MLP writes to the residual stream (via GemmaScope SAEs used as transcoders): layers 18–19 write calendar-position content (probe-aligned, peaking at 4.3× null), layers 20–25 write duration content (DAS-aligned, peaking at 3.2× null) — a sharp when → how long boundary. There are even month-length correction features: one at layer 22 is completely silent for February and active for every other month, patching the ≤1-day residual the ±30 hop leaves behind.

Disjoint vocabularies. Probe-aligned SAE features fire on "month of October"; DAS-aligned features at the layer-24 relay promote "months, weeks, days, years" — the answer vocabulary. The overlap of top-100 feature sets is Jaccard = 0.000, and causal attribution of probe features to the duration output is exactly zero.

Two more findings deepen the picture. First, the rank-4 mediator is a cooperative unit: ablating any single direction costs ΔNLL 0.08–0.51, but all four together cost 68.8 — a 59× super-additive interaction. This also explains why no sparse-dictionary decomposition (SAE or transcoder, any of several tried) can capture it: rank-1 steering along any direction lets the model route around the intervention through the other three dimensions. Second, a temporal-feature-analysis decomposition shows the mediator lives in the context-predictable component of activations (7× above null) — duration computation is built from accumulated context, not the current token — while the probe sits squarely in the random cloud.

How general, and how strong

The dissociation replicates at 1.5B, 2B, 7B, and 9B parameters across Gemma and Qwen families, with DAS drops of 42–51 pp against random-control drops of ≤0.3 pp — and it sharpens with scale, exactly as the d/k null predicts (wider residual streams make a fixed-rank random subspace even more inert). It holds for every probe target tried (day-of-year, month, season, quarter, day-of-week — all within 2.8° of null), for nonlinear probes (MLP, kernel ridge, random forest — all at null), and for concept-erasure methods: INLP and LEACE erasure subspaces sit within ~1.5° of the null too. Erasing what the probe finds does not erase what the model uses — a result that should worry anyone using concept erasure as a causal tool.

My favorite single result is the training-dynamics one: on untrained Pythia 1.4B (step 0), the day-of-year probe already achieves R^2 = 0.956. A random network "represents dates" by the probe's standard. The actual mechanism — boundary heads, circular geometry (37× growth in an FFT circularness index), DAS ablation effect — emerges only over training. Probes measure the linear decodability afforded by high-dimensional random features; they do not track mechanism learning at all.

The paper closes with an adversarial construction (framed as a safety-monitoring stress test, but the epistemic content stands on its own): because the two subspaces are orthogonal, you can shift the mediator's content by 71 days while independently restoring the probe's readout to within 5.7 days of correct. Mutual information between the probe's output and the mediator's state: 0.000 nats, while the probe carries 3.99 nats about the ground-truth date. The probe knows the date and knows nothing about the machine.

What to be skeptical about

The orthogonality is the default, not the surprise. By Prop 2, any two low-rank subspaces in \mathbb{R}^{2304} sit near 88°. The paper is upfront about this, but a careless reading would over-index on the angle. The real content is (a) the probe fails to beat the null at all — not by a degree — and (b) the causal asymmetry (\rho = 1050 vs. \approx 1). Relatedly, the geometry implies probes could never have passed this test unless the model happened to compute along the readable axis; the paper's contribution is making that a measurable, reportable quantity rather than a philosophical worry.

DAS finds damage by construction. The mediator is defined as the subspace whose removal maximizes loss — so "DAS ablation collapses performance" is partly tautological. What isn't tautological: the effective rank saturates at exactly 4 (rank-6 solutions scatter randomly on the Grassmannian, bracketing the dimension), five seeds converge to the same subspace (CCA > 0.94), the subspace transfers across held-out prompt sets, and it's task-specific — the duration drop is 12.6× the average effect on 240 non-calendar prompts, with arithmetic and counting completely unaffected. Still, the honest supplements show DAS ablation removes ~19% of activation norm (vs. 4% for random) and induces measurable distribution shift (JS divergence ~10× random) on unrelated prompts, even though top-1 outputs are preserved. The mediator is not surgically clean.

Scope. The deep circuit analysis is one task family on small models (1.5–9B). The spatial and arithmetic extensions use tiny prompt sets (n = 60 and 25) and are explicitly preliminary. Clean accuracies are low (42–51%), so there's headroom for the dissociation to look different on tasks a model actually does well. And there's an oddity worth noting: the probe's R^2 peaks at layer 1 of Gemma 2 2B, while attribution patching localizes the causal action at layer 24 — the layer-1 DAS ablation works because the residual stream propagates the mediator forward. The angle is at null at every layer, so the result doesn't depend on the layer choice, but it's a reminder that "where the probe reads best" and "where the computation happens" are dissociated in depth as well as direction.

Why it matters

If this holds, the standard sentence "the model represents X (probe R^2 = 0.99)" becomes ambiguous in a way that matters: readable-from and computed-with are, in the generic case, orthogonal subspaces, and the untrained-network result shows probes can't even distinguish learned structure from random-feature decodability. The paper's practical prescription — report \bar\theta against the Haar null and the specificity ratio \rho_k alongside any probe result — is cheap, has closed-form baselines, and I'd expect it to get adopted (or at least demanded by reviewers). It also lands a real blow on concept-erasure and probe-based monitoring pipelines, which inherit the exact same blind spot.

Where to spend your time: Section 3 (the three propositions — short, and the Rayleigh-quotient vs. gradient-covariance argument is the conceptual core), then Section 5 for the circuit trace, which is one of the more complete reverse-engineering efforts I've seen at this scale. Among the supplements, S11 (placing nine interpretability tools on a single readout-to-mediator ruler) and S49 (why sparse dictionaries structurally cannot capture a cooperative rank-4 mechanism) are the ones with ideas that generalize beyond this task.