Scaling Inherently Interpretable Language Models

TL;DR: Guide Labs trained an 8B diffusion language model, Steerling-8B, where interpretability is not something you excavate after training but a set of constraints baked into the data, architecture, and loss. Every output logit decomposes exactly into contributions from ~34,000 human-named concepts plus a residual; input attribution uses the [MASK] token as an in-distribution "absence" baseline; and training data attribution comes from an index over the model's own representations. The headline empirical claim: across three orders of magnitude of IsoFLOP sweeps, the interpretability machinery costs a small fixed offset in compute-optimal scaling rather than a growing tax — and the interpretability metrics themselves improve with scale, predictably enough to extrapolate from small models to 8B. The capability story is more modest than the abstract suggests, but the paper is unusually honest about its own failures, and the scaling-law framing of interpretability is genuinely new.

The premise being tested

The standard workflow in interpretability is archaeology: train the most capable model you can, freeze it, then dig through it with probes, sparse autoencoders, gradient attributions, or influence functions. The paper opens with a crisp diagnosis of why this keeps failing, and it's worth internalizing because it motivates everything else.

Each post-hoc method assumes a structure that training never created. A linear probe tells you a concept is decodable from activations, not that the model uses it — probes recover information from randomized embeddings just fine. An SAE feature reconstructs activations, but SAEs trained with different seeds find different features, and random SAE baselines match trained ones on causal-editing metrics. Perturbation attributions (SHAP, occlusion) replace tokens with zero vectors or padding — states the model has never seen, so the measured effect is extrapolation behavior, not the token's role. And chain-of-thought text is simply not coupled to the computation that produced the answer.

The paper distills these into two conditions an attribution must satisfy to be faithful: agreement (the attribution's scores predict the effect of an intervention before you run it) and validity (the interventions live in a family the model was actually trained on). Post-hoc methods violate one or both. The proposal: instead of hoping these conditions emerge, enforce them by construction — which means modifying data, architecture, objective, and loss simultaneously.

The mechanism

Three pieces do the work.

Piece 1: A concept-annotated corpus (Atlas). No concept library existed at pretraining scale, so they built one. Sample 6.6M documents across web, code, math, and academic text; chunk into ~150–256 token spans; prompt Mistral-24B to emit ~500M free-form tags; embed and k-means-cluster 14M unique tags at k=80{,}000; filter incoherent clusters; label survivors with an LLM; deduplicate via graph community detection. Result: 33,732 canonical concepts, each with a name, description, and characteristic words. A trained annotator (Qwen3-Embedding-0.6B encoder + classification heads) then labels the full 1.5T-token corpus at chunk level — about 11 billion chunks.

The obvious worry is that an LLM will assign fluent-sounding names to statistically arbitrary clusters. They ran a blind two-phase human study: annotators shown only a concept's "lifted words" (tokens with high P(w \mid c)/P(w)) wrote their own labels, then blindly rated candidate labels. Atlas's LLM-generated labels scored 3.98/5 versus 3.50 for independent human labels, both far above distractors (2.08–2.72). Caveat lodged in their own data: only 55% of lifted-word lists were judged clearly coherent concepts, and 17% were flagged as noise. The library is real but not uniformly clean.

Piece 2: An additive concept bottleneck. Between the transformer backbone and the LM head, the hidden state h is decomposed as

where \hat{k} = \sum_i k_i K_i is a weighted sum of learned embeddings for the 33,732 known concepts (k_i are sigmoid activations from a small head), \hat{u} is the same construction over ~100K unknown concepts (a low-rank-factorized channel that absorbs whatever the library misses), and \varepsilon is the residual. Only \bar{h} reaches the linear LM head, so every logit splits exactly:

No approximation — concept attribution is just reading off the forward pass. And because each concept embedding K_c is a parameter the model actually computes with, steering is a closed-form edit: inject \gamma \, K_c/\|K_c\| into hidden states to amplify a concept, or subtract its logit contribution (ReLU-gated, to avoid accidentally promoting anti-aligned tokens) to suppress it.

Piece 3: A diffusion backbone with a trained absence baseline. They abandon autoregression for masked diffusion, with a twist they call Causal Diffusion: attention is bidirectional within fixed blocks (64 tokens) and causal across blocks. This retains diffusion's parallel, any-order generation within a block while recovering autoregressive-style KV caching across blocks — and, unlike Block Diffusion, drops the duplicated clean-sequence copy during training, halving that cost. The interpretability payoff of diffusion: the model is trained constantly to predict tokens replaced by [MASK], so [MASK] becomes a learned representation of "no information here." Integrated Gradients from the mask embedding to the real embedding then traverses states the model was trained on — the validity condition that zero-baseline attribution in AR models violates.

Auxiliary losses tie it together: a chunk-level concept BCE (via OR-aggregation over token activations, since labels say a concept appears somewhere in a chunk), a reconstruction loss driving \varepsilon \to 0, and an HSIC-style independence penalty decorrelating \hat{k} from \hat{u} so the unknown channel can't smuggle in duplicate copies of the known concepts.

Does interpretability scale?

This is the paper's most interesting contribution: treating interpretability as a quantity you can fit scaling laws to. They run IsoFLOP sweeps (Chinchilla-style parabola fits over four compute slices, ~10^{19}–3 \times 10^{20} FLOPs) for four families: AR, causal diffusion (CDLM), and each with the concept module.

On capability: the AR baseline gives \alpha_P = 0.528 (compute-optimal parameter exponent), in line with Chinchilla's 0.49 and DeepSeek's 0.524 — a sanity check that their setup reproduces known results. Adding the concept module shifts exponents by small per-backbone offsets (CDLM: \alpha_P 0.632→0.503; AR: 0.528→0.621) and lowers the fitted irreducible loss, though they candidly note the smallest +Concept models are distorted by a fixed-size concept library (89% of parameters at 10M scale, 4% at 8B, <1% at frontier scale), which biases the fits. The cleanest number: extrapolating the joint Chinchilla fit from small models predicts Steerling-8B's validation loss to within 0.11 nats at 7.6 \times 10^{22} FLOPs — where a naive log-linear extrapolation misses by 0.47.

On interpretability: all four metrics — concept prediction loss, known/unknown independence, concept contribution (fraction of logit mass routed through concepts vs. residual), and known-concept alignment (LLM-judged match between a concept's top tokens and its name) — improve monotonically with compute on both backbones, fit with irreducible-loss power laws. Three of four extrapolate to the 8B model within tight bounds (concept contribution predicted 0.915, actual 0.876; alignment predicted 3.87, actual 3.77 on a 1–5 scale); the fourth (independence) comes in better than predicted. A nice second-order finding from the joint (P, D) fits: concept accuracy and disentanglement are parameter-driven (near-vertical contours), while alignment is token-driven — so no single compute split optimizes all four at once.

The capability price, honestly

The abstract says Steerling-8B is "competitive with open peers trained on 2–16× more compute" and "within ~10% of their average." Here are the actual numbers:

Steerling beats LLaMA2-7B and DeepSeek-7B (the ~2× compute peers) but sits 8–16 points below LLaMA3, Gemma, LLaDA, and OLMo 2. "Within ~10%" is doing heavy lifting. The fair reading: the concept module itself isn't the bottleneck (the scaling laws support that), but the total system — 1.35T tokens, a diffusion objective, a first-time training recipe — trails well-tuned modern baselines. Notably LLaDA-8B, a plain masked diffusion model, hits 65.9 MMLU vs. Steerling's 46.4, so diffusion per se isn't the excuse either.

Section 9.4 is the most refreshing part of the paper: a frank post-mortem of pretraining. The moving-Gaussian masking curriculum, validated at 1B scale, spent the back third of training above 50% masking and degraded MMLU and WinoGrande from their mid-run peaks. The concept heads entangled late in training as teacher forcing annealed. The corpus starved math and code (~100B of 1.2T tokens). And — most damning for the thesis — the pretrained model did not respond to steering: injecting concept directions it had never seen during training was an out-of-distribution perturbation, and prompting beat steering on their own benchmark. Mid-training (150B tokens with uniform masking, a code-augmented mix, a two-term independence loss, and four interleaved phases of explicit steering training on token-level concept labels) fixed most of this: MMLU +17 points, GSM8K +30, and steering finally works:

The steering lesson generalizes and is worth remembering: an intervention interface only works if the model was trained under it. This is exactly their own "validity" condition, and they initially violated it themselves.

What to make of it

What's genuinely new here is not any single component — concept bottlenecks (Koh et al. 2020), masked-baseline attribution, and CBGMs all predate this — but the demonstration that the full recipe survives pretraining scale, and that interpretability admits scaling-law treatment at all. If interpretability metrics follow power laws in compute and extrapolate from cheap sweeps, then "will this model be understandable at 100B?" becomes an empirical planning question rather than a hope. That framing could stick even if this particular architecture doesn't.

Reasons for skepticism, beyond the benchmark gap:

• The interpretability metrics are the training objectives (three of four are literally held-out versions of the losses). "Interpretability improves with scale" partly means "the auxiliary losses go down with scale," which is expected. The exception — known concept alignment, which is not directly supervised — is the most interesting curve, but it's scored by an LLM judge (Mistral-24B) on a 1–5 scale.
• Semantic faithfulness is not certified. Concept contribution of 0.876 means ~12% of logit mass still flows through the uninterpreted residual, and the paper's own definition of concept leakage (a concept variable encoding more than its card says) is acknowledged but not systematically measured on Steerling.
• Exponent comparisons are confounded by the fixed concept library at small scale; the authors say so, but it means "small fixed offset, not a growing tax" rests substantially on one 8B extrapolation point.
• Training data attribution is retrieval, not influence — the paper is explicit that it claims similarity in the model's own geometry, not counterfactual training influence. Useful, but weaker than the phrase "trace to training data" suggests.
• This is a company team evaluating its own product model; the human study is a 100-concept stratified pilot, not a library-wide audit.

If the result holds and transfers, the practical implication is significant: a fixed, human-auditable explanation interface — same concepts, same absence baseline, same decomposition across all runs — that costs ~4% of parameters at 8B and under 1% at frontier scale, with attribution as a forward-pass readout and steering as a parameter edit. Their proposed extensions (reward objectives expressed over concepts, agents whose decisions decompose into monitorable concepts mid-trajectory) are speculative but follow naturally from the architecture.

Where to spend your time: Section 8 (scaling laws) is the core evidence and the methodological contribution; Section 9.4 (pretraining lessons) is the most candid and instructive read; Section 3 is worth skimming for the faithfulness formalism, which is a clean vocabulary for arguing about any interpretability method, post-hoc or not.