Holographic reduced representations

TL;DR — In 1995, Tony Plate showed that circular convolution can bind two n-dimensional vectors into a single n-dimensional vector that can later be approximately unbound, so arbitrarily nested symbolic structure — roles, fillers, sequences, trees — fits in a fixed-width vector with no dimensional blowup. It was elegant and unusable: at the few-hundred-dimensional vector sizes and MFLOP-scale machines of the day, the crosstalk noise swamped the signal, and nothing needed it anyway. Today, when the dominant cost in LLMs is a KV cache that grows linearly with context and attention that grows quadratically, a fixed-width, compositional, decodable memory is exactly the missing primitive — and modern linear-attention math turns out to be Plate's idea with the compression step left out.

The problem Plate was solving

The connectionist wars of the late 1980s had a specific technical battlefield: how do you represent structure — "John loves Mary," not just "John," "loves," "Mary" — in a vector? Fodor and Pylyshyn's famous 1988 critique argued neural nets couldn't do systematic composition at all.

Smolensky's answer (1990) was the tensor product: bind a role vector r to a filler vector f by taking the outer product r \otimes f. It works, and it's exactly compositional. But it's dimensionally explosive: binding two n-vectors gives n^2 numbers, and nesting a structure inside another gives n^3, n^4, and so on. Representation size depends on structure depth, which is fatal for a fixed neural substrate — and, we'd now say, for a fixed-width residual stream.

Plate's move: don't store the full outer product. Compress it back to n dimensions by summing along the wrapped diagonals. That operation has a name — circular convolution:

Bind role to filler with \circledast; the result is the same size as the inputs, so bindings can be nested indefinitely. Superpose multiple bindings by plain addition to get a memory trace:

Why it works, and what it costs

If the components of x are drawn i.i.d. \mathcal{N}(0, 1/n), then correlating with an approximate inverse — the involution x^\dagger, defined by x^\dagger_j = x_{(-j) \bmod n} — undoes the binding:

The "≈" is the whole story. Unbinding returns the true filler plus zero-mean crosstalk noise from everything else superposed in the trace; the signal-to-noise ratio scales roughly like \sqrt{n/k} for k items stored in n dimensions. So the output is a noisy pointer that must be cleaned up by nearest-neighbor lookup against an item memory (a codebook of known vectors). Plate analyzed the capacity: the number of bindings you can reliably store and decode grows roughly linearly in n, with logarithmic dependence on codebook size.

Two more properties made this more than a trick. First, via the convolution theorem, \mathcal{F}(x \circledast y) = \mathcal{F}(x) \odot \mathcal{F}(y): binding is elementwise complex multiplication in the Fourier domain, computable in O(n \log n) — or O(n) if you just live in the frequency domain, where inverses become exact phase conjugation. Second, binding is randomizing: x \circledast y is nearly orthogonal to both x and y, so bound structures don't interfere with their constituents. Similarity is preserved in a graded way — structures sharing fillers are somewhat similar — which Plate exploited for analogy retrieval. The "holographic" name is earned: like Gabor's holograms and the 1960s–70s correlographic associative memories (Longuet-Higgins, Willshaw), every element of the trace carries a little of every stored association.

Why it couldn't work in 1995

Put numbers on it. Plate's simulations used vectors of a few hundred to a couple thousand dimensions — that was what a workstation delivering on the order of 10^7–10^8 FLOPs sustained could sweep experiments over. At n \approx 512, the \sqrt{n/k} crosstalk means you can hold maybe a handful to a dozen bindings before decoding becomes unreliable, and every level of nesting compounds the noise multiplicatively. HRRs at 1995 scale were a demonstration, not an engine.

The cleanup memory was the second wall. Every unbinding needs a nearest-neighbor search over the item codebook. With no fast approximate-NN algorithms, no vector hardware, and codebooks that had to stay tiny, cleanup was both the bottleneck and the fragility: a slightly-too-noisy pointer snaps to the wrong symbol and the error cascades through the nested structure.

Third, and decisive: no customer. There was no application in 1995 that needed compositional structure in fixed-width vectors, because nothing downstream consumed vectors at all. Symbolic AI didn't want approximate vectors; neural nets of the day were too small to want symbols. HRRs solved tomorrow's problem.

What changed

Compute, obviously: a modern GPU does elementwise complex multiplies on 100k-dimensional vectors in microseconds, and at n = 10^4–10^5 the crosstalk math turns friendly — hundreds of superposed bindings decode reliably. The field that grew around this regime is now called vector symbolic architectures or hyperdimensional computing (Kanerva's 2009 synthesis is the standard entry point), and HRR is one of its founding algebras, alongside Kanerva's binary spatter codes and Gayler's MAP. Neuromorphic groups have adopted VSAs as a natural programming model for chips like Loihi.

The customer arrived too. Everything in deep learning is now a fixed-width vector, and the systemic pain point is precisely what HRR addresses: storing structured content in bounded space. The sharpest connection is to linear attention. A linear-attention layer maintains a fast-weight matrix M = \sum_i v_i k_i^\top and reads out with M q — Schlag and Schmidhuber made this "fast weight programmer" reading explicit in 2021. But \sum_i v_i k_i^\top is Smolensky's tensor-product memory, dimensional blowup and all (n^2 state). Plate's paper is, in modern terms, the observation that you can compress that fast-weight matrix to a single n-vector, m = \sum_i k_i \circledast v_i, and still unbind: k_j^\dagger \circledast m \approx v_j. Linear attention is HRR with the compression step skipped; HRR is linear attention with n\times less state. Nobody has seriously run that trade-off at LLM scale.

What has already been tried

The idea did not die; it went underground and resurfaced in pieces:

• Eliasmith's Semantic Pointer Architecture built HRRs into spiking neurons; Spaun (2012), the ~2.5-million-neuron functional brain model, uses circular-convolution binding as its core symbolic mechanism. This is the strongest existing vindication.
• Holographic embeddings (HolE), Nickel et al. 2016, used circular correlation to score knowledge-graph triples — competitive at the time, and later shown essentially equivalent to complex-valued embeddings (ComplEx), which is the Fourier-domain view of HRR.
• Hrrformer (Alam et al., ~2023) replaced attention with HRR binding/unbinding, reporting strong long-sequence results at much lower cost; earlier work by the same group differentiated through HRR operations for multi-label learning.
• TPR-based models (Schlag, Smolensky et al.) pursued the uncompressed tensor-product branch inside transformers for interpretable reasoning.
• Mechanistic-interpretability work increasingly describes what trained transformers do in binding language — key-value association, superposition, near-orthogonal role vectors — suggesting networks partially rediscover the HRR toolkit. (I'd state this as a reading of the literature, not a settled result.)

What has not been done, to my knowledge, is the head-to-head that matters: HRR-style fixed-width memory versus full attention on retrieval and multi-hop reasoning at matched compute, inside a real LLM.

A serious 2026 revival

The experiment writes itself. Give an LLM a fixed-width episodic memory: a single hypervector m \in \mathbb{R}^n (or \mathbb{C}^{n} in the Fourier/FHRR parameterization, where unbinding is exact phase conjugation — use this; it removes one noise source Plate had to fight).

• Write: for each chunk of context, produce a key k_i (position, entity, or learned role) and a content vector v_i; update m \leftarrow \gamma m + k_i \circledast v_i with a learned decay \gamma.
• Read: the model emits a query, unbinds q^\dagger \circledast m, and the noisy result goes through cleanup.
• Cleanup, modernized: this is where 2026 beats 1995 decisively. Plate needed nearest-neighbor against a fixed codebook. An LLM can use a learned denoiser — a small network, or even a couple of diffusion-style refinement steps — mapping noisy pointers back onto the manifold of valid content vectors. Cleanup was the fragile hand-built component; make it the trained component.
• Keep from Plate: the binding algebra, the involution-based unbinding, the capacity analysis as a design guide (n sized so that expected items k keeps \sqrt{n/k} comfortably above the denoiser's threshold), and the nesting discipline for multi-hop structure ("A's location's owner" as chained unbinding).
• Replace: random i.i.d. vectors with learned embeddings constrained to keep the algebra healthy (unit-magnitude Fourier components); hand-built cleanup with the denoiser; single trace with a small bank of traces if capacity binds.

Benchmark against a long-context transformer and a modern SSM at matched FLOPs on needle-in-haystack retrieval, multi-hop QA, and entity tracking. The prediction worth testing: HRR memory loses on verbatim recall (a 10^4-dim vector cannot losslessly hold 10^5 tokens — information theory is not optional) but wins decisively on structured recall per unit of state, because it stores relations, not transcripts. Honest open questions: whether gradients flow well through bind-superpose-unbind chains at depth; whether learned keys collapse the near-orthogonality the capacity results assume; and where the crossover with attention sits.

Where to read it

The paper is Plate, "Holographic Reduced Representations," IEEE Transactions on Neural Networks, 1995 — doi.org/10.1109/72.377968 (bibliographic details verified). It is long and unusually careful; the capacity appendices reward reading. Alongside it: Plate's 2003 book Holographic Reduced Representation for the full treatment; Smolensky 1990 for the problem HRR compresses away; Kanerva 2009 ("Hyperdimensional Computing") for the field it seeded; Eliasmith's How to Build a Brain (2013) for the neural implementation; and Schlag & Schmidhuber's "Linear Transformers Are Secretly Fast Weight Programmers" (2021) to see modern architectures walking backward into Plate's front door.