Eye Movements and Vision

TL;DR — In 1967 Alfred Yarbus published scanpath recordings showing that where a person looks at a picture is dictated by the question they are trying to answer: same image, different task, radically different fixation sequence. Vision, he showed, is task-conditioned sampling, not uniform image processing. Every frontier VLM today does the opposite — it tokenizes every patch of every frame at uniform resolution and pays quadratic attention cost for the privilege. Yarbus's result is a sixty-year-old argument that this is the wrong architecture, and the economics of video understanding have finally made the argument urgent.

What Yarbus actually did

Yarbus, working in Moscow, built the best eye tracker of his era: small suction devices ("caps") attached directly to the anesthetized sclera of the eyeball, carrying a tiny mirror that deflected a light beam onto photographic film. Heroic for the subject, but precise — good enough to resolve individual saccades and fixations over minutes of viewing.

Two results from the book matter here.

The famous one: he showed observers Repin's painting An Unexpected Visitor for several minutes at a time, each time with a different instruction — free viewing, estimate the family's wealth, give the ages of the people, remember the positions of people and objects, guess how long the visitor had been away. The recorded scanpaths are strikingly different across instructions. "Estimate wealth" produces fixations on clothing and furnishings; "give the ages" produces fixations almost exclusively on faces; "how long away" produces repeated shuttling between faces, apparently reading the reunion. The retinal input available is identical; the sampling policy is set entirely by the task.

The less famous one, arguably deeper: using the same suction-cap apparatus, Yarbus stabilized images on the retina so that eye movements no longer changed the input. Within a second or two the stabilized image fades to nothing. Vision does not merely benefit from movement; it requires it. The eye is not a camera with a jitter problem — the jitter is part of the computation.

The quantitative backdrop makes the point sharper. The human retina has roughly 100 million photoreceptors but only about 1 million optic-nerve fibers, and high-acuity vision covers only the fovea — roughly 1–2 degrees of a ~180-degree field, well under 1% of the visual field by area. The visual system compensates by making about 3–4 saccades per second, pointing that tiny high-resolution window wherever the current task needs it. Human vision is a hard-attention system with a learned, query-conditioned glimpse policy. Yarbus recorded the policy.

Why nothing could be built on it in 1967

Three walls, each independently fatal.

Measurement. The suction-cap method required topical anesthetic and physical attachment to the eyeball. Sessions were short, subjects few, and data lived on photographic film that had to be traced by hand. There was no path from Yarbus's rig to a dataset of any size — collecting even thousands of task-labeled scanpaths was unthinkable.

Framework. In 1967 there was no computational language for what Yarbus had found. Machine perception meant template matching and early pattern recognition on static arrays; reinforcement learning as a formal framework for sequential decision-making under a reward was more than a decade away, and POMDPs — the natural formalism for "choose where to sense next given what you know" — were only being formalized in the operations-research literature around then, nowhere near perception. There was no way to write down a learned fixation policy, let alone train one.

Compute. Even ignoring the framework problem, the arithmetic was hopeless. A top scientific machine of the mid-1960s delivered roughly 10^5–10^6 operations per second. A single 512×512 grayscale image is ~2.6×10⁵ pixels; one pass of even trivial per-pixel processing consumed seconds. Simulating a closed loop of "glimpse → update belief → choose next glimpse" over images was out of the question. Today a single GPU delivers on the order of 10^{14}–10^{15} operations per second — roughly nine to ten orders of magnitude more — and the loop is cheap.

What changed — and why the pressure is now economic

Eye tracking became a commodity: infrared trackers cost hundreds of dollars, webcam-based tracking is nearly free, and gaze datasets with explicit task labels exist at scale — COCO-Search18 (visual search fixations), EGTEA and Ego4D-adjacent egocentric gaze data, and many others. The measurement wall is gone.

The framework wall fell with deep RL. Mnih et al.'s Recurrent Attention Model (2014) is essentially Yarbus formalized: a recurrent network that emits glimpse locations, receives a foveated crop, and is trained with REINFORCE. It worked on MNIST-scale problems and then largely lost to soft attention and ViTs, because when compute is cheap relative to your images, "look at everything" is simpler than learning where to look.

That tradeoff has now inverted, and this is the real story. Modern VLMs tokenize video uniformly: a typical frame becomes a few hundred visual tokens, so a minute of video at even 1 fps is on the order of 10^4 tokens before a single word of the question is processed — and self-attention cost grows as O(n^2) in token count n. Long-video understanding is bottlenecked not by model capacity but by the sheer waste of encoding pixels the query does not care about. Yarbus's observers answered "how old are the people?" by sampling perhaps a few dozen foveal patches. The mismatch between that and uniform dense tokenization is two to three orders of magnitude, and it is now a line item on every inference bill.

A serious 2026 revival

The design writes itself once you take the book literally: the fixation policy must be query-conditioned, not a saliency map.

1. Peripheral pass. Encode the whole video at aggressively low resolution and low frame rate — the "periphery." Cheap, a few tokens per frame. This is the analogue of the low-acuity visual field that guides saccade targeting.
2. Fixation policy. A small policy network takes the query embedding plus the peripheral tokens and emits a sequence of spatiotemporal glimpses: (frame range, image region, resolution). Train it with RL against task reward minus a token budget penalty, or differentiably with Gumbel/straight-through relaxations over a patch lattice.
3. Foveal tokenization. Only the glimpsed regions are encoded at full resolution and appended to the context. The LLM can request further glimpses — saccades — interleaved with reasoning, exactly like Yarbus's observers shuttling between faces.
4. Human scanpaths as prior and benchmark. Pretrain the policy by behavior cloning on task-labeled gaze data, then fine-tune with RL. Evaluate not just accuracy-per-token but scanpath similarity to humans given the same question — a genuinely novel interpretability probe: does the model's policy fixate faces when asked about ages?

Reuse from Yarbus: the core claim (policy is a function of the query), the fovea/periphery split, and the experimental paradigm itself as an evaluation. Replace: film with token streams, the human oculomotor system with a learned glimpse policy, and "one fovea" with the freedom to make several glimpses in parallel — silicon owes biology no serial bottleneck.

The target is credible: 10–100× fewer video tokens at equal accuracy on question-answering over long video, because the information relevant to a typical query genuinely occupies that small a fraction of the pixels.

Has it been tried? What's still open?

Pieces exist; the full loop does not. Active vision was articulated by Bajcsy ("active perception," 1988) and Ballard ("animate vision," 1991), explicitly citing the Yarbus tradition. Saliency models (Itti & Koch 1998; DeepGaze) predict free-viewing fixations well but are task-blind — precisely the part Yarbus said matters most. Hard-attention glimpse networks worked at small scale and stalled. Foveated rendering is standard in VR, driven by the same bandwidth economics. In today's VLMs, token merging and pruning (ToMe and successors), query-aware token selection, and learned resamplers (Perceiver-style, Flamingo's) all reduce tokens, but almost none close the loop: a sequential, query-conditioned policy that decides what to encode next based on what it has seen so far remains rare at frontier scale.

One honest caveat on the science: the strong reading of Yarbus — "the task can be decoded from the scanpath" — was challenged by Greene, Liu & Wolfe (2012), who failed to classify task from fixations above chance; Borji & Itti (2014) and later work recovered above-chance decoding with better features and more data. The consensus is that task shapes scanpaths robustly but noisily, subject- and image-dependently. That is fine for the engineering program — the policy needs to be useful, not human-identical — but it cautions against treating human scanpaths as ground truth rather than as a prior.

Open questions: whether glimpse policies trained with RL scale stably inside frontier VLMs or need differentiable relaxations; whether a fixed compute budget is better spent on a smarter policy or a denser periphery; and whether Yarbus's second result — stabilized-image fading, i.e., the necessity of change for perception — has a machine analogue (predictive-coding-flavored encoders that only tokenize what the peripheral model failed to predict).

Where to read it

The book is Eye Movements and Vision, Plenum Press, 1967 (translated from the Russian), DOI 10.1007/978-1-4899-5379-7; the Repin scanpaths are in the chapter on eye movements during perception of complex objects. Read alongside: Ballard, "Animate vision" (1991) for the computational manifesto; Mnih et al., "Recurrent Models of Visual Attention" (2014) for the first deep-learning instantiation; Greene et al. (2012) and Borji & Itti (2014) for the decoding debate; and Hayhoe & Ballard, "Eye movements in natural behavior" (2005) for what task-driven gaze looks like outside the lab — which is, if anything, even more Yarbusian than the paintings.