Issue 35 · Pick 10 AI / ML ✓ read
The Emergent Symbolic Structure of Artificial Neural Networks
TL;DR: This paper claims that the continuous vector representations inside neural networks—from tiny GRUs up to 20B-parameter LLMs—implicitly encode discrete symbolic structure, specifically Tensor Product Representations (TPRs): sums of (symbol ⊗ position) outer products passed through one linear map. The evidence is unusually strong for an interpretability paper: the authors replace the network's entire encoding process with a closed-form TPR equation and the network keeps working (often within a couple of percent of its original accuracy), they perform algebraic surgery on hidden states that changes model behavior exactly as the symbolic analysis predicts (average 0.903 accuracy across 31 intervention types on GPT-OSS), and they show the fitted structure generalizes to role-filler combinations never seen during fitting—which rules out the boring "lookup table" explanation. If this holds up, it's a serious data point in the 40-year symbols-vs-vectors debate, and a candidate foundation for interpretability's unit-of-analysis problem.
The oldest fight in cognitive science, now with falsifiable stakes
Since Fodor and Pylyshyn (1988), there's been a standing argument: intelligence looks symbolic (syntax trees, logical formulas, algebra), but brains and neural networks compute with continuous vectors. Either symbols are eliminable folklore, or any network that behaves symbolically must secretly implement a symbol system. LLMs doing arithmetic and code made this concrete: how does a stack of matrix multiplies represent that concatenate(x, y) ≠ concatenate(y, x)?
The standard tool for peering into representations—the Linear Representation Hypothesis, the basis of probing and sparse autoencoders—says a hidden state is a sum of concept vectors: e(\text{cats chase dogs}) = e(\text{cats}) + e(\text{chase}) + e(\text{dogs}). But addition is commutative, so this assigns the same vector to dogs chase cats. This is the binding problem: how does a vector encode not just which elements are present, but which element occupies which position? Most current interpretability methods have no principled answer.
The proposed answer: Tensor Product Representations
TPRs (Smolensky, 1990) are an old proposal for how to design networks that encode structure. Decompose a structure into fillers (the elements) and roles (their positions). Give each filler a vector f_i and each role a vector r_i. Bind each pair with an outer product f_i \otimes r_i—a matrix that encodes "this filler in this role"—and sum:
The affine map W, b (making this a linearly-transformed TPR) just re-shapes and rotates the space; it changes nothing structurally but lets the same geometry hide inside any coordinate system. The multiplicative binding is what fixes commutativity: f_{\text{cats}} \otimes r_{\text{subj}} + f_{\text{dogs}} \otimes r_{\text{obj}} \neq f_{\text{dogs}} \otimes r_{\text{subj}} + f_{\text{cats}} \otimes r_{\text{obj}}, even though the aggregation is still a sum. And because "the whole is literally the sum of the parts," you can do algebra on structures: subtract one bound pair, add another, and you've edited the structure.
The hypothesis in this paper is the interesting inversion: nobody built TPRs into these networks, but perhaps standard networks trained by SGD converge to TPR-structured representations on their own.
DISCOVER: replace the encoder with an equation
The test, called DISCOVER, is refreshingly demanding compared to probing. You don't ask "can I decode X from the hidden state?"—you ask "can I regenerate the entire hidden state from a symbolic equation, and does the rest of the network still work when I do?"
- Take a trained target network. Pick the hidden vectors E you want to explain.
- Hypothesize a role scheme (e.g., "each token's role is its left-to-right position," or "its position in the dependency parse").
- Train a TPR model—free parameters are just the filler embeddings, role embeddings, and W, b—to minimize \|E - E^{\text{TPR}}\|^2.
- Swap E^{\text{TPR}} in for E and run the rest of the original network. Approximation accuracy = fraction of test inputs where the network still produces the fully correct output.
Crucially, this can fail. A TPR with a fixed role scheme is a rigid bilinear function; it forces algebraic identities on the vector space (e.g., \text{TPR}(\texttt{QMZ}) - \text{TPR}(\texttt{QS}) + \text{TPR}(\texttt{VS}) = \text{TPR}(\texttt{VMZ}) under left-to-right roles). And indeed it often does fail: on GRUs trained to reverse lists, a left-to-right role scheme approximates poorly, a bag-of-words scheme (the "no structure" null hypothesis) fails badly everywhere order matters—but a bidirectional scheme (each letter's role is its position from both ends) hits ≥99.98% approximation accuracy across all 10 training reruns. Across 12 architecture × task combinations (MLPs, GRUs, Transformers, and a "bottleneck Transformer" forced to pack a whole sequence into one vector; copy, reverse, interleave), bidirectional TPRs approximate everything—worst case 0.973, all others >0.99. Nicely, when the task doesn't require structure (alphabetical sorting), the structureless bag-of-words scheme suddenly works, and role schemes with fewer parameters routinely beat schemes with 35× more roles—so this isn't just capacity chasing (Appendix N).
Scaling to LLMs, and replacing everything
For seven LLMs (Gemma-3-27B, GPT-2-XL, GPT-OSS-20B, Pythia-12B, Qwen3-14B, OLMo-2-13B, Llama-3.1-8B), the authors first exploit a known trick: the hidden state at a sentence-final period acts as a summary of the whole preceding sentence (a trained "period-unpacking" decoder can reconstruct it, perfectly for subject-verb-object sentences). DISCOVER approximates these period encodings well with bidirectional roles—and, interestingly, linear-order roles beat syntactic roles here, suggesting the period summary is mostly a positional record, not a parse.
The heavyweight experiment (Section 6, ~3,000 GPU-hours) drops the intermediary entirely. For GPT-OSS-20B doing arithmetic, syllogisms, Python code execution, and three syntax transformations, they train one DISCOVER model per layer (25 layers) and replace every input-token representation at every layer with its TPR reconstruction, then let the LLM generate. The winning role scheme, task-specific (all), hypothesizes that each token's hidden state contains bound pairs for itself and all preceding tokens, with roles like subj_noun-object_adj ("the subject noun, as it appears inside the object adjective's representation"). Under this full-substitution regime, the LLM's task accuracy drops by at most 2.36% (arithmetic; less elsewhere) relative to running on its own representations. "Self-only" schemes (one binding per token) do markedly worse—so these hidden states really do superimpose many bound pairs, and the bindings track task structure, not just linear position.
Symbolic algebra on hidden states
Because TPRs are sums, editing is subtraction and addition: E' = E - W(f_{\text{clever}} \otimes r_{\text{obj\_adj}}) + W(f_{\text{clever}} \otimes r_{\text{subj\_adj}}). Apply that (across all affected tokens and layers) and GPT-OSS, given The spy helped the clever poet, behaves as if it read The clever spy helped the poet. Across 31 intervention types spanning all six tasks, average accuracy is 0.903. Note these edits touch every coordinate of the vector by a precise amount—no assumption of axis-aligned features.
Two sub-results deserve attention. First, editing only the target token's own representation suffices for filler changes but fails for role changes: token identity is local, structure is distributed. Second, an elegant controlled comparison (Appendix J) crosses edit vectors between intervention pairs that make the same linear-position change but different structural changes; the edits only work on their structurally matched inputs, so the interventions manipulate abstract structure, not token position. (And since GPT-OSS uses RoPE, there are no positional vectors in the residual stream to piggyback on—the positional roles being edited are genuinely emergent.)
The lookup-table objection, and the killer control
A skeptic's best remaining move: maybe the network stores each role-filler pair as an atomic concept (cats-as-subject), and DISCOVER just learned a fancy dictionary. Section 8 addresses this head-on. Withhold certain pairs during DISCOVER fitting (e.g., scientist never appears as a subject), then test on inputs containing them. If binding is atomic, no method can beat placing the n unseen fillers randomly into the n unseen slots—accuracy 1/n!. DISCOVER beats this baseline substantially in nearly every setting (all GPT-OSS tasks except arithmetic; all LLM period encodings; most letter models). A white-box sanity check (Appendix M) confirms the logic: DISCOVER generalizes on a frozen encoder built from real TPRs, and completely fails to generalize on a frozen encoder with atomic pair embeddings—even though both are approximated equally well in-distribution. So the target networks must compose roles and fillers systematically.
The strangest result: the approximation beats the original
For complex sentences, the period-unpacking decoders—trained on real LLM encodings—perform better when fed DISCOVER's TPR reconstructions than when fed the encodings they were trained on.
The authors' reading, which I find persuasive: the LLM's encoding is a TPR, but a noisy one. The decoder learns to rely on the TPR structure; DISCOVER renders that structure noise-free, so downstream performance improves. This is exactly the prediction of Smolensky's old "limitivist" position—networks approach symbol systems in the limit but realize them only approximately—as opposed to both eliminativism ("symbols are folklore") and strict implementationalism ("networks must exactly implement symbol systems"). The 0.903 intervention accuracy tells the same story: mostly symbolic, not exactly symbolic. Whether the residual deviation is noise or is doing useful work (the fuzzy, statistical side of language that pure symbol systems famously miss) is left open, and it's arguably the most interesting question the paper raises.
What to be skeptical about
- Supervised role schemes on templatic data. Humans hand-designed the roles, and the stimuli are narrow: fixed arithmetic templates, CFG-generated sentences with single-token words, restricted syllogism formats. This is deliberate (isolate how structure is encoded, given known what), but it means nothing here shows TPR structure for open-ended natural text, where the right roles are unknown and possibly not discrete. The paper is candid that fully systematic tasks were chosen precisely because a symbolic program could solve them.
- The superset problem. A successful DISCOVER fit shows the TPR captures everything the network encodes, not that the network encodes everything in the TPR. A richer role scheme can degenerately emulate a simpler one, so "task-specific roles win" is weaker evidence than it sounds—though the structure-sensitivity interventions partially rescue the claim.
- One LLM for the flagship result. The full-replacement and intervention results are GPT-OSS-20B only (compute constraints). The seven-LLM evidence is the softer period-encoding version.
- Representations, not mechanism. The paper explicitly does not explain how attention and MLPs produce TPR-like vectors (there's no tensor product in the architecture), nor how downstream layers consume them. Their own analogy: characterizing the feather, not how it grows.
- Tension with compositional-generalization failures. Networks with compositional representations still generalize compositionally poorly in the behavioral literature (SCAN, COGS). Section 9.5's proposed reconciliation—representations are compositional only over encountered pairs—is plausible but untested for these architectures.
Why it matters, and what to read
If networks broadly converge to role-filler binding with multiplicative structure, that's a direct challenge to the additive-features assumption underlying sparse autoencoders and most probing—it says the right unit of analysis is f \otimes r, not c. It also hands interpretability a rare tool: closed-form, algebraically editable descriptions of hidden states, with causal teeth. And it suggests a concrete, testable hypothesis for neuroscience: look for TPR geometry in neural population recordings during structured tasks.
If you read two things: Section 6 (the full-representation replacement in GPT-OSS, including the role-scheme design in Table 2, which is where the real conceptual work lives) and Section 7 plus Appendix J (the interventions and the structure-vs-linear-order control). Section 9.1's limitivism discussion is the best five minutes of theory in the paper.