A law of comparative judgment.

TL;DR — In 1927 Thurstone showed that you can build a genuine interval scale of any subjective quantity — beauty, seriousness of a crime, quality of a sentence — from nothing but binary "A vs B" judgments, by modeling each judgment as the difference of two Gaussian random variables. That equation, in its most degenerate special case, is now the loss function of every Bradley–Terry reward model in RLHF. The interesting part is what current practice threw away: Thurstone's full model has per-item variances and correlated judgment errors, and those are exactly the pieces modern reward modeling is missing.

The idea as Thurstone had it

Psychophysics in the 1920s had a good story for stimuli with a physical scale: Weber and Fechner related just-noticeable differences in weight or brightness to physical magnitude. But most of what people judge — the elegance of a handwriting sample, the gravity of an offense — has no physical magnitude at all. The prevailing view was that such things could be ranked but not measured.

Thurstone's move was to locate the scale inside the noise. When a stimulus i is presented, it evokes what he called a discriminal process: a value on an internal psychological continuum, drawn from a distribution with mean \mu_i and standard deviation \sigma_i (the discriminal dispersion). The mean is the item's true scale value; the dispersion is how inconsistently that particular item is experienced. When a judge compares i and j, they sample once from each distribution and report whichever sample lands higher. So

where \Phi is the standard normal CDF and \rho_{ij} is the correlation between the two discriminal processes on a given trial (judgments made moments apart, by the same person, in the same context, are not independent).

This is the full "law of comparative judgment" — what Thurstone called Case I. Because it is hopelessly underdetermined from pairwise win rates alone, he defined a ladder of simplifications, ending in Case V: all dispersions equal, all correlations zero. Then

and the scale values are recoverable, up to affine transformation, by inverting empirical win rates through \Phi^{-1} and solving a linear system. Replace the probit with a logistic and you have Bradley–Terry (1952) — empirically almost indistinguishable. Replace it again and you have Elo, and the reward-model loss -\log \sigma\!\big(r(y_w) - r(y_l)\big) used to train InstructGPT and essentially every RLHF pipeline since.

The conceptual jump deserves emphasis: the noise is not a nuisance, it is the ruler. If two items are never confused, you learn only their order; the rate of confusion is what pins down interval distances. This is exactly why an RLHF reward model outputs a scalar with meaningful magnitudes rather than a mere ranking.

Why it could not scale in 1927

Three walls, all quantitative.

Computation. Fitting Case V for n stimuli means inverting \Phi for each of n(n-1)/2 empirical proportions (via printed normal tables) and solving a least-squares system by hand or mechanical calculator. Thurstone and his students did this for n \approx 20 — roughly 190 cells, days of clerical work. At n = 100 you have ~5,000 cells; at n = 10{,}000, ~50 million. And Case V was the only feasible case: Cases I–III, with free per-item dispersions and correlations, are nonlinear estimation problems with more parameters than a hand computation could ever identify. Thurstone wrote them down and then, of necessity, never fit them. The richest part of the theory was stillborn for computational reasons.

Data. Each proportion needs enough judgments to be a stable estimate — say 100+ per pair. Thurstone-era studies used a few hundred undergraduates filling out paper forms; total judgment counts in the tens of thousands were heroic. There was no mechanism to collect the millions of comparisons needed for large stimulus sets, let alone comparisons over open-ended generated items.

No consumer for the output. Even with a fitted scale, in 1927 there was nothing to do with a scalar utility function except publish it. The idea that a learned scalar could steer an optimization process — that the scale is a training signal — required optimization machinery that didn't exist. So the method lived on inside psychometrics (attitude scaling, signal detection theory grew out of the same Gaussian-observer picture) and, via Elo, inside chess.

What changed

Everything on the list. Maximum-likelihood fits of Thurstone/BT models over millions of parameters are trivial on modern hardware; more importantly, the scale values no longer need to be free parameters per item — they are amortized by a neural network r_\theta(x, y) that generalizes to unseen items. That single move dissolves Thurstone's data-per-item problem: you don't need 100 judgments on each item, you need enough judgments overall to fit the shared function. Crowdsourcing and product telemetry made millions of comparisons routine. And RLHF (Christiano et al. 2017 → Stiennon et al. 2020 → Ouyang et al. 2022) supplied the consumer: the fitted scale is the reward that a policy is optimized against. Chatbot Arena is, quite literally, a planet-scale Thurstone Case V experiment with an Elo/BT fit on top.

The vindication is total — and oddly partial. The field adopted the 1927 paper's worst case. Case V's assumptions are plainly false for LLM outputs: a bland, safe response is a low-dispersion stimulus; a creative or borderline response is high-dispersion; two responses to the same prompt share context and annotator, so their evaluation errors are correlated. Homoscedastic BT compresses all of this into a single temperature, which forces the model to explain annotator disagreement by shrinking reward gaps — conflating "these are equally good" with "judges are inconsistent about this one."

A serious 2026 revival

Take Thurstone Case III (free per-item dispersions, and reintroduce correlation where it's cheap) as a drop-in replacement for the BT reward model:

• Reuse from the paper: the full generative story — per-item mean \mu, per-item dispersion \sigma, correlation \rho between simultaneous judgments.
• Replace: free parameters with two heads on a shared LM backbone, r_\theta (mean) and s_\theta (log-dispersion); add a per-annotator reliability scalar \gamma_a (an IRT-style discrimination parameter) and a shared-context correlation term. Amortization is what makes Case III identifiable where it wasn't in 1927.
• Downstream uses that BT cannot provide: (1) variance-penalized policy optimization — optimize r - \lambda s so the policy is pushed away from high-dispersion regions, a principled brake on reward hacking; (2) active preference collection — query pairs where predicted dispersion, not just reward gap, is high, targeting the label budget where labels carry information; (3) disagreement decomposition — separating "genuinely tied" from "contentious," which matters for pluralistic alignment.
• Evaluation: not reward-model accuracy (heteroscedastic models can lose raw pairwise accuracy while being better calibrated) but downstream policy quality and, especially, overoptimization curves in the style of Gao et al. (2023): does the Thurstonian penalty flatten the late-stage decline where policies exploit reward-model error?

Already tried, descendants, open questions

Partially tried. TrueSkill (Herbrich et al., 2006) is a genuinely Thurstonian system — Gaussian skill beliefs with per-player variance — deployed at Xbox scale, and is the clearest proof that the richer model works when the compute exists. In LLM-land there is a growing literature on reward-model uncertainty (ensembles, distributional and quantile reward models), on annotator disagreement, and on critiques of the BT assumption itself; several recent papers propose heteroscedastic or judge-aware preference losses. I'm confident the direction is active; I'm not confident any single variant has become standard, and to my knowledge none is the default in a major RLHF stack. DPO and its descendants inherit the BT likelihood wholesale, so a Thurstonian DPO — where the implicit reward gap is scaled by predicted dispersion — is a natural, and I believe still underexplored, follow-up.

What Thurstone cannot fix: any model of the form "scalar per item plus noise" implies stochastic transitivity in the population, and real human preferences (and mixtures of annotators) violate it. Descendants that drop the scalar entirely — general preference models, Nash learning from human feedback — are the escape route when the 1927 assumption of a one-dimensional continuum fails, which is the genuinely open modeling question underneath all of RLHF.

Where to read it

The original: Thurstone, A law of comparative judgment, Psychological Review 34 (1927), doi:10.1037/h0070288 — short, readable, and the five Cases are laid out exactly as described. Read alongside: Bradley & Bradley–Terry (1952) for the logistic twin; Herbrich et al., TrueSkill (2006) for the scaled Thurstonian system; Ouyang et al. (2022) for the Case V loss in production; and Gao, Schulman & Hilton (2023) on reward-model overoptimization — the failure mode that per-item dispersion, sitting unused in a 1927 paper, is best positioned to address.