Brain alignment in deep neural networks emerges early and independently of object classification

Only the paper's abstract was available to me, so this explainer builds the context and stakes around the claims stated there; specific methods, numbers beyond the abstract, and figures from the paper itself are not reproduced.

TL;DR: The central empirical pillar of "deep nets are good models of the visual brain" is that trained CNNs predict neural responses well. This paper tracks that alignment densely across training for three architectures against human fMRI, human EEG, and macaque electrophysiology, and finds that most of it is already there at random initialization, that it saturates within 1–5 epochs while classification accuracy is still climbing toward 75%, and that per-kernel lesioning shows a unit's contribution to brain alignment is essentially uncorrelated with its contribution to classification. If this holds, brain–model alignment is mostly telling you about the statistics of the visual world (and convolutional inductive biases), not about the brain and the network converging on the same recognition algorithm.

The claim being challenged

The modern love affair between visual neuroscience and deep learning started around 2014, when Yamins, DiCarlo, and colleagues showed that CNNs trained on ImageNet predicted responses in macaque IT cortex better than any hand-designed model, and—crucially—that networks with higher classification accuracy tended to be more brain-aligned. That correlation launched an entire research program: Brain-Score leaderboards, the "goal-driven modeling" framework, and the interpretation that the ventral visual stream and ImageNet-trained CNNs are two solutions to the same optimization problem, converging on similar representations because they solve the same task.

That interpretation has always had a crack in it. The same networks that predict IT responses also exhibit texture bias where humans use shape (Geirhos et al.), collapse under perturbations humans shrug off, and are fooled by adversarial noise invisible to us. So the field has lived with a tension: models that supposedly share our representations don't share our behavior. This paper attacks the tension head-on by asking a question that's surprisingly rarely answered with dense measurement: when during training does brain alignment arise, and does it track the thing we assumed it reflects—classification competence?

What "brain alignment" measures, mechanically

To feel the force of the result, it helps to be concrete about the measurement. The standard pipeline: show the same images to a brain (fMRI voxels, EEG electrodes over time, or spiking neurons in macaque cortex) and to a network. Extract the network's layer activations for each image. Then either fit an encoding model—a regularized linear regression from network features to neural responses, scored by predictivity on held-out images—or compare representational geometries: build each system's representational dissimilarity matrix (how differently it responds to each pair of images) and correlate the matrices (RSA). Both families of methods answer roughly the same question: does the network organize images the way the brain does?

The critical, easily-forgotten property of these metrics is that they are satisfied by any feature basis that captures the covariance structure the brain also captures. They do not require that the features are used the same way, computed the same way, or useful for the same task. That gap between "shares a representational geometry" and "shares a computation" is exactly where this paper drives its wedge.

The experiment: watch alignment and accuracy grow up together

The design is simple and that's its strength. Train three convolutional architectures from scratch, and checkpoint them densely—including epoch zero, the randomly initialized network. At each checkpoint, measure two things:

1. Classification accuracy on the training task (which ends up around 75% top-1, consistent with standard ImageNet-scale training).
2. Brain alignment against three very different kinds of neural data: human fMRI (spatial, slow, whole ventral stream), human EEG (temporal dynamics of visual processing), and macaque electrophysiology (single-neuron/population spiking, the gold standard for IT predictivity).

If the goal-driven story is right, the two curves should rise together: alignment should be low at initialization (the network knows nothing) and should climb as the network acquires the representations that support recognition.

That is not what happens.

Three details in the abstract sharpen the picture. First, the effect is not metric- or species-specific: it appears against fMRI, EEG, and macaque spiking data alike, which rules out an artifact of any one recording modality. Second, alignment isn't merely "present early"—it's largely present at initialization, when the network classifies at chance. Random convolutional filters, passed over natural images, already produce feature spaces that predict brain responses to a substantial degree. Third, the plateau comes absurdly early: one to five epochs, i.e., one to five passes over the data, while the network still has the vast majority of its accuracy gains ahead of it.

The lesioning analysis: the sharper knife

The training-trajectory result alone admits a comeback: maybe the alignment-relevant features are learned early because they're the foundation of classification—edges and textures first, categories later—so alignment and classification could still be causally intertwined, just on different schedules.

The lesioning analysis is designed to close that escape route. The logic: take a trained (or partially trained) network and ablate individual kernels—zero them out one at a time—and measure two numbers per kernel: how much its removal hurts classification accuracy, and how much its removal hurts brain alignment. If the network's brain-like representations are the substrate of its recognition ability, these two importance scores should correlate: the units carrying the brain-aligned structure should be the units doing the classifying.

The abstract reports they are "essentially uncorrelated... throughout training." The features that make the network look like a brain and the features that make it recognize objects are, at the level of individual units, different features.

Why random networks are already brain-aligned: the reinterpretation

Here is the intuition the paper builds toward. Think of a random CNN as a bank of thousands of random filters convolved over an image at multiple scales, with rectification and pooling. This is not a blank slate—it's very close to a classical random-projection / scattering-transform feature extractor. Random projections approximately preserve the geometry of their input space (Johnson–Lindenstrauss), and the convolutional + pooling structure imposes locality, translation tolerance, and a coarse spatial-frequency decomposition. So the representational geometry of a random CNN is, to first order, the geometry of natural images themselves, viewed through a biologically sensible architectural prism.

The brain also encodes natural images. Two systems that both encode the covariance structure of the same environment will look aligned under geometry-based metrics—regardless of what either system does with that structure downstream. In this reading, the celebrated brain-predictivity of CNNs was never primarily evidence of a shared recognition algorithm. It was evidence, mostly, that both systems are pointed at the same world, plus a modest bonus from the first few epochs of training (which likely handle low-level gain/statistics adaptation—the fast, generic part of learning) and from convolutional inductive biases that happen to echo the retinotopic, hierarchical layout of visual cortex.

An analogy that I think genuinely fits: two students' notebooks from the same lecture will look strikingly similar—same topics, same order, same emphasized formulas—even if one student goes on to ace the exam and the other fails. The notebooks' similarity reflects the shared lecture (the visual environment), not shared understanding (the recognition computation). Alignment metrics have been grading the notebooks.

How this sits with prior evidence

This isn't a bolt from the blue—it's the strongest and most direct version of a suspicion that has been accumulating. It has long been known that untrained CNNs predict early visual cortex (V1) reasonably well, and several groups have reported that untrained networks capture a surprising fraction of the trained-network Brain-Score. The trained-vs-untrained gap was always largest in higher areas like IT, which is where the goal-driven story staked its claim. What's new here, per the abstract, is (a) dense temporal resolution over training, showing the gap closes within 1–5 epochs rather than accumulating alongside task competence; (b) triangulation across fMRI, EEG, and monkey electrophysiology; and (c) the lesioning result, which converts a correlational observation into a mechanistic dissociation at the unit level. Also relevant: within modern architectures, classification accuracy and Brain-Score have already decoupled at the top of the accuracy range (better ImageNet models stopped becoming better brain models years ago). This paper says the coupling was weak from the very first gradient step.

It also dovetails with the behavioral critique. If alignment lives in features orthogonal to the classification-critical ones, there's no paradox in a network that predicts IT responses well but has alien texture-biased, adversarially fragile behavior. The two facts were about different parts of the network all along.

What to be skeptical about

Reading only the abstract, several things need checking in the full text before updating hard:

Noise ceilings and effect sizes. "Largely present at initialization" and "changes only modestly" are quantitative claims that live or die on the numbers. If random-init alignment is, say, 70% of the noise ceiling and training adds the remaining 30%, that remaining 30% could still be the scientifically interesting part—the part specific to higher-order representation. The literature consistently finds the training-induced gain concentrates in IT/higher visual areas; if the paper's "largely present" is driven by early visual cortex while IT still shows a meaningful trained–untrained gap, the goal-driven story survives in weakened form for exactly the regions it always cared about.

Metric dependence. Encoding models with generous linear readouts are famously permissive—a big random feature bank gives regression a lot to work with, and predictivity can reflect the regression's power rather than the representation's specificity. Whether the result holds under stricter metrics (RSA without fitting, CKA, or constrained readouts) matters a lot.

Lesioning granularity. Single-kernel lesions measure marginal, not joint, importance. Distributed representations can make every individual unit look unimportant for classification (redundancy) while population-level structure carries both alignment and recognition. A null correlation at the single-kernel level is suggestive but not decisive; group-lesion or subspace-lesion analyses would be more convincing.

Architecture scope. Three convolutional networks. Vision transformers—now the alignment leaders on several benchmarks and lacking the convolutional prior—are the natural stress test. If ViTs show low alignment at init that grows substantially with training, the "it's mostly architecture + image statistics" story would need revising; if they show the same pattern, it strengthens dramatically.

Stimulus set. Alignment measured on natural images shared across brain and model datasets inherits natural-image covariance by construction. Controlled or synthetic stimulus sets that break natural statistics would isolate what's environment versus computation.

What changes if it holds

The practical consequence is blunt: brain-predictivity scores, as currently computed, are weak evidence about mechanism. A high Brain-Score would tell you a model ingests natural images through a vaguely cortex-shaped funnel—not that it recognizes objects the way you do. That reframes a decade of "CNNs are the best models of the ventral stream" claims and puts pressure on the field to develop metrics that specifically reward shared computation: behavioral error-pattern consistency, robustness profiles, causal/perturbation-matched predictions, and alignment measured on stimuli engineered to decorrelate environmental structure from task-relevant structure.

There's a constructive flip side. If most alignment comes from environment statistics plus architecture, then the residual—the part that training does add, in the regions and epochs where it adds it—becomes the precious signal. Isolating and maximizing that residual is a better-posed modeling target than the raw score ever was. And for NeuroAI broadly, the result is a caution that applies beyond vision: language-model/brain alignment results, which also rely on linear predictivity of responses to shared naturalistic stimuli, deserve the same untrained-baseline, dense-trajectory scrutiny.

When the full text is available, go straight to the lesioning analysis and the per-region breakdown of the training trajectories (especially whatever they show for IT and category-selective fMRI regions versus early visual cortex)—that's where the strong version of the claim either survives or quietly becomes "alignment in early visual areas is cheap," which we already knew.