Smooth Scaling Laws Hide Stepwise Token Learning

TL;DR: The smooth power-law loss curves that govern LLM scaling are, at the microscopic level, not smooth at all. Track the loss of individual contextualized tokens through pretraining and you find that most of them sit on a plateau, drop sharply over a localized window, and plateau again — a step, well fit by a sigmoid. The shape of that step is nearly the same for every token; the only thing that differs is when it happens. The distribution of these "learning times" is heavy-tailed, and convolving it with the shared step shape quantitatively reconstructs the loss derivative along training steps, data scale, and model scale — on industrial MoE pretraining runs up to ~6B parameters and 300B tokens, not toy models. The signal is also actionable: reweighting training samples by when their tokens become learnable speeds validation-loss reduction by 11%.

The puzzle: why should loss be a power law at all?

Scaling laws — L = k\,a^{-\alpha} + E in model size, data, or compute — are so reliable that entire training runs are budgeted around them. But their origin has remained oddly under-explained. Why should averaging cross-entropy over trillions of predictions produce something this regular?

The leading story is data-distributional: natural language contains a long tail of patterns, from "period follows sentence-final word" down to "this variable was defined 800 tokens ago." If pattern frequency is Zipfian and the model masters patterns roughly in frequency order, a power-law learning curve falls out. Hutter's 2021 analysis and Michaud et al.'s "quantization hypothesis" (discrete skill quanta learned in order of use frequency) both make versions of this argument — but on toy models and synthetic data. Nobody had checked, at token granularity, in a real large-scale pretraining run, whether this is actually what happens.

This paper does that check, and the answer is a clean yes, with a mechanism you can write in one line.

The aha: radioactive decay for tokens

Here's the analogy that makes the whole paper click. An individual radioactive atom doesn't decay gradually — it's intact, then suddenly it isn't. Yet a lump of uranium loses mass along a perfectly smooth exponential, because the distribution of decay times across atoms is smooth. The macroscopic curve tells you almost nothing about individual atoms; it tells you about the ensemble of event times.

The claim here is that token learning works the same way. A specific contextualized token — this word, in this context, in the validation set — is essentially unpredictable for a while, then over a relatively short training window the model acquires whatever regularity is needed, and the token's loss steps down to a new plateau where it stays. The aggregate loss is smooth for the same reason the uranium curve is smooth: millions of asynchronous step events, each localized, with a smooth distribution of event times. And where an exponential distribution of decay times gives you exponential mass loss, a heavy-tailed distribution of learning times gives you a power law.

The mechanism, concretely

The authors set up three competing hypotheses for where the power law comes from: (A) global optimization dynamics — LR schedules, noise; (B) individual token losses themselves decaying as power laws; (C) different tokens being learned at different times. An observational study kills A and B quickly: the same token instance shows nearly identical descent timing and shape across independent seeds (so it's not optimizer noise), and individual trajectories are plateau–step–plateau, not power laws.

That motivates fitting each validation token's loss trajectory \ell_i(a) (along axis a — training step T, data scale D, or model scale M) with a sigmoid:

where \tau_i is the learning time (center of the drop), \lambda_i the sharpness, and \Delta_i the total loss reduction. Differentiating gives each token a learning pulse — a localized \mathrm{sech}^2 bump centered at \tau_i, integrating to \Delta_i. The negative loss derivative of the whole model is then a sum of pulses.

The key move is factoring this sum into two objects: the average pulse shape g(a), obtained by shifting every pulse so its center sits at zero and averaging, and the learning-time spectrum p(\tau), the empirical density of pulse centers. If the aligned pulses all share roughly the same shape, the sum becomes a convolution:

And since g is narrow relative to how p varies, the convolution barely smooths anything: the shape of the loss derivative is essentially the shape of p(\tau). The power law lives in when tokens are learned, not in how any of them is learned.

Two empirical facts make this factorization legitimate rather than wishful. First, on synthetic data, the fitted shape parameters (\Delta_i, \lambda_i) show no strong systematic drift with \tau_i — late-learned tokens aren't shaped differently from early ones. Second, an SVD of the matrix of center-aligned pulses is strongly low-rank:

The evidence

Step axis. A large MoE model trained on hundreds of billions of tokens from an industrial corpus (Chinese, English, math, reasoning, books, papers, code), with token-level validation losses recorded at dense checkpoints. Four checks (Fig. 3 of the paper): the averaged sigmoid fits recover the empirical validation loss curve; pulses grouped into 40 learning-time bins collapse to nearly the same shape after alignment; the spectrum p(\tau) is heavy-tailed and roughly power-law; and the reconstruction (p*g)(a) overlays the empirical loss derivative.

Data and model axes. Here the "axis" isn't a single training run but a frontier: a 50-run sweep with token budgets from 1B to 35B (data axis, small MoE, full warmup-constant-decay schedules, recipe-tuned per budget), and a dense-model sweep at fixed 10B tokens with model scale measured in non-embedding FLOPs per token rather than parameter count (a sensible choice for MoE-adjacent comparisons). Fitting each validation token's loss as a function of D or M and redoing the decomposition, the same picture reappears: heavy-tailed spectrum, shared pulse, reconstruction matching the frontier's loss derivative. That the mechanism transfers from optimization time to these classical scaling axes is the paper's most striking result — a token has a "learning time" in model scale too: below some capacity it's unpredictable, above it, learned.

The synthetic control — the closest thing to a causal test. On the Mano modular-arithmetic task (answer-only supervision, difficulty = expression depth L), the authors set the difficulty distribution. With uniform difficulty over L \in \{1,\dots,10\}, the aggregate loss is not a power law — it decays roughly linearly — while still showing strict easy-to-hard learning order and clean sigmoid token trajectories. Switch the sampling distribution to \propto L^{-2} with everything else fixed, and the pulse-spectrum reconstruction continues to hold under the new curve shape. This is the piece that elevates the story above curve-fitting: change the distribution of learning times, and the macroscopic loss shape changes accordingly. Power laws are not an inevitable artifact of SGD; they're inherited from the data.

A unification bonus (Appendix A.2). Mean learning time recapitulates familiar curriculum intuitions: books/web text learned earliest, math and knowledge mid, reasoning/QA/code latest; high-frequency tokens early, rare tokens late; tokens recoverable from short contexts early, tokens needing >256-token context late. The suggestion is that domain heuristics, frequency-based curricula, and n-gram-order arguments are all shadows of one underlying variable, \tau.

Making it actionable: 11% faster

If loss reduction at step t comes from tokens whose \tau_i \approx t, you should be able to accelerate a training phase by feeding it the samples whose learning events land there.

They train a small MoE reference model, estimate \tau_i for 1B candidate tokens, score each training sample by the fraction of its tokens whose learning events fall in steps 2000–3800, split into 10 buckets, upweight the top and downweight the bottom, and resume training from step 2000. The buckets show cleanly ordered loss behavior (the score means something), training loss falls faster despite a higher irreducible floor (they've removed easy, already-learned material), and validation loss reaches targets 11% faster than the original distribution.

What to make of it, and what to be skeptical of

The honest framing: this is not a fundamentally new hypothesis. It is the first token-level, real-data, industrial-scale validation of the difficulty-spectrum / quantization family of explanations (Hutter, Michaud et al., Cagnetta et al.), plus a decomposition clean enough to be operational. That's genuinely valuable — the gap between "toy model with Zipfian sparse parities" and "6B MoE on 300B mixed-corpus tokens" was exactly where doubt lived.

Caveats worth holding onto:

Descriptive vs. mechanistic. The sigmoid is explicitly a fitting device ("we use the sigmoid not to claim exact logistic dynamics"), and real token trajectories are noisy and non-monotone — the fits allow ascending trajectories and forgetting. A four-parameter sigmoid can absorb a lot; the reconstruction matching L'(a) is partly guaranteed once the fits are good, since summing the fits recovers the loss by construction. The non-trivial claims are the two that break that circularity: pulse shapes are shared across learning times (the SVD result), and the synthetic intervention where changing the difficulty distribution changes the macroscopic curve shape. Those are the load-bearing results, and they're solid — but the synthetic causal test is on modular arithmetic with answer-only supervision, the cleanest possible setting. On real language, the story remains a very good decomposition with strong circumstantial support, not a demonstrated mechanism. Notably, the paper doesn't explain why p(\tau) is power-law on real data beyond a hand-wavy compositional-hierarchy argument in Appendix A.1.

The 11% number. It's an 11% acceleration of validation-loss reduction on one target interval (steps 2000–3800), at small-MoE scale, requiring a full reference run plus dense checkpointing and per-token trajectory fitting — expensive machinery. There's no comparison against cheaper data-selection baselines (loss-based selection, perplexity filtering, RHO-style excess-loss weighting), so we can't tell whether learning-time targeting beats simpler proxies for the same compute. The higher training-loss floor also hints the method is spending its budget on a narrower slice of the distribution; whether this compounds across a full run or borrows from later progress is untested.

Opacity. "Large scaled MoE" and "small scaled MoE" are never given exact parameter counts in the text; this reads like an industrial lab (the corpus and scale suggest so) publishing what it can. Validation tokens are drawn from the training distribution, so how the spectrum behaves out-of-distribution is unknown.

If the picture holds, the practical implication is a shift in how to think about data curation: not "what data is good" but "what data is learnable now" — curriculum design as shaping p(\tau) directly, with domain mixing, frequency filtering, and difficulty staging all becoming special cases of one measurable quantity. It also reframes scaling-law extrapolation: the exponent is a property of the data's learning-time tail, which suggests both a route to predicting exponents from corpus structure and a warning that interventions on the data mix should be expected to bend the law.

Where to spend your time: Section 3.3 for the pulse/spectrum factorization (it's short and the whole paper in miniature), Figure 3 for the step-axis evidence, and Appendix A.3 — the synthetic experiments are the most rigorous part of the paper and the closest it comes to causality, especially the uniform-difficulty run where the power law disappears.