DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

TL;DR: Take a pretrained vision-language model, write eye-fixation coordinates as zero-padded two-digit text tokens, and LoRA-fine-tune it to predict human scanpaths as ordinary next-token prediction. That's the whole method — and it beats every specialized gaze architecture by a wide margin: 2.18 bits/fixation of Information Gain on MIT1003 versus 1.49 for DeepGaze III, even when DeepGaze III is given a bigger vision encoder from the same model family. The likelihood you optimize during training turns out to be exactly the field's standard evaluation metric, and because conditioning is just prompting, the same model can be steered by viewer identity, search targets, or fixation durations — enabling counterfactual "experiments" on gaze behavior that would be impossible to run on real eyeballs.

The problem: predicting where eyes go next

When a person looks at an image, their eyes don't sweep smoothly — they jump in saccades between fixation points, two to four times per second. A scanpath is that ordered sequence of fixations. Static saliency prediction (where do people look, averaged over time?) was essentially solved by deep pretrained features a decade ago. But predicting the sequence — where you'll look next, given where you've looked so far — has remained the domain of bespoke architectures.

The state of the art before this paper, DeepGaze III, illustrates the standard recipe: a pretrained vision backbone for spatial features, bolted onto hand-designed dynamic machinery — separate branches encoding fixation history, explicit mechanisms inspired by known oculomotor phenomena like inhibition of return (the tendency not to revisit recent fixations). More recent work swapped the hand-crafted dynamics for diffusion models or neural point processes, but still trained the sequential machinery from scratch on gaze data alone.

The authors point out the asymmetry: everyone uses pretrained priors for space, but nobody uses pretrained priors for sequence. Meanwhile, VLMs have sequential priors trained on billions of tokens of visually grounded text. Why not use them?

The reframing: gaze as a language

The mechanism is almost embarrassingly simple. A fixation at position (x, y) is encoded as percentage coordinates on a 100 \times 100 grid, written as zero-padded two-digit integers: (50, 48). A scanpath becomes a Python-style list of tuples in a text prompt: "Analyze this image and predict a human eye movement scanpath during free viewing for 3 seconds... Output ONLY a Python list of tuples." The model — InternVL3.5-8B with LoRA adapters on the LLM's MLP layers — is trained with plain cross-entropy on ground-truth scanpaths.

Two details make this rigorous rather than a hack, and they're the parts worth internalizing.

Uniform tokenization. If "7" and "73" tokenize to different numbers of tokens, likelihood comparisons between coordinates are confounded by string length, not spatial probability. Zero-padding to exactly two digits — and verifying that the tokenizer splits every coordinate into identical single-digit tokens — guarantees every fixation costs exactly four tokens. One remaining leak: the VLM's vocabulary has tens of thousands of non-digit tokens that soak up probability mass. The fix is to restrict and renormalize over the ten digits at each position, \log\hat{P}(t) = \log P(t) - \log\sum_{d\in\{0..9\}} P(d), yielding exact spatial densities.

The training loss is the evaluation metric. The field's gold-standard metric is Information Gain: the model's log-likelihood of held-out human fixations, in bits, above a center-bias baseline (the fixed spatial prior capturing that people look near image centers). With uniform tokenization, the autoregressive chain rule gives you the exact log-likelihood of any scanpath in one teacher-forced pass — four forward steps per fixation. Next-token prediction literally maximizes IG. Contrast this with diffusion or point-process scanpath models, which can't compute exact densities and fall back on heuristic sequence-alignment metrics (ScanMatch, MultiMatch) whose parameter choices, the appendix shows, can flip model rankings entirely.

There's one genuine engineering wrinkle: for metrics needing the full spatial map P(x,y) over all 10,000 grid cells (e.g., AUC), naive scoring takes 10,000 forward passes per fixation. The authors exploit the shared-prefix structure: decompose P(x,y) = P(x_1)P(x_2|x_1)P(y_1|x_1,x_2)P(y_2|x_1,x_2,y_1) and evaluate a four-level tree with KV-caching — 1 + 10 + 100 + 1000 = 1{,}111 passes for the exact map. Rapid IG evaluation on the MIT validation set takes under a minute with vLLM.

The evidence, and the control that matters

The headline numbers, from Table 1 (IG in bits/fixation; center bias = 0 by definition):

The obvious objection — "you're just using a bigger, better vision encoder" — is the one the selection note asked to verify, and the paper handles it well. The authors retrained DeepGaze III with InternViT-6B, an encoder larger and more capable than the InternViT-300M inside their own 8B VLM. Matched-backbone DeepGaze III: 1.49 bits on MIT1003. With InternViT-300M: 1.45 bits. So the vision encoder is worth ~0.04 bits, and the LLM's contribution is ~0.7 bits. As a second control, two LLaVA variants sharing an identical vision encoder but different LLM sizes diverge substantially in IG. The gain genuinely comes from the language model's sequential and semantic priors, not the features.

The most provocative result is Figure 1 of the paper: across four VLM families (LLaVA-Next, Gemma 3, InternVL 3.5, Qwen3.5) and nine scales, IG after LoRA tuning correlates with MMMU score — a general multimodal reasoning benchmark — at Spearman \rho = 0.93. Raw parameter count doesn't explain it: SmolVLM2 stays flat at ~0.85 bits from 256M to 2.2B parameters, while InternVL3.5-4B hits 2 bits, beating LLaVA-13B's 1.6. The representations that support broad visual reasoning are apparently the same ones needed to predict where humans look. That's a striking empirical bridge to the cognitive-science view that visual exploration is driven by high-level semantic understanding, not low-level salience.

The leave-one-dataset-out row is quietly the most impressive: DeepGaze3.5-VL, never trained on the target dataset, matches or beats DeepGaze III's in-distribution performance on four of five datasets (e.g., 1.94 vs. 1.49 on MIT1003). And a rank-1 LoRA adapter tuned on ~1,000 scanpaths (<7% of the training set) closes over 60% of the remaining gap to full in-distribution training.

What the model actually learned about sequences

A skeptic could still say: maybe the VLM just learned a better static saliency map, and the "sequence modeling" is decoration. Section 3.3 dismantles this cleanly with three ablated models trained on deliberately corrupted histories:

The spatial-only model (trained with shuffled cross-subject histories, so it can't use sequential structure) starts strong at ~2.7 bits but decays to ~0.25 bits by fixation 12 — image content drives early orienting, then runs dry. A history-invariant model (knows only the fixation index, not the individual's path) adds a stable 0.3–0.6 bits: population-level "phase of exploration" information. The full model, given the observer's true history, holds total predictability roughly constant deep into the scanpath. The complementary shuffling experiment agrees: shuffled ground-truth fixations start near ground-truth surprise but diverge for later fixations. Late-stage gaze cannot be explained as draws from any static density — the model has learned genuinely stateful dynamics, including whatever inhibition-of-return and coverage strategies humans use, without any of that being architecturally imposed.

Conditioning: flexibility, mostly cheap capability rather than big IG

The selection note asked how much the conditioning flexibility improves fit versus just adding capacity. The honest answer from the numbers: the fit improvements from conditioning are real but modest; the capability is the point.

• Fixation duration (appended as a third value per tuple, millisecond bins): improves spatial IG from 2.03→2.12 on MIT1003, 2.15→2.30 on COCO-FreeView, 2.67→2.84 on DAEMONS. Roughly +0.1–0.17 bits.
• Subject identity (an anonymized ID in the prompt, with 10% masked to a generic token during training): +0.09 bits/fixation on average under leave-subject-out cross-validation — about 5% of total IG, consistent with the literature's finding that image content dominates and individual differences are a small systematic component.
• Visual search is where conditioning shines, because the task actually changes behavior:

The search result has two nice wrinkles. First, the task effect is almost entirely mediated by target presence: when the target is absent, IG drops to 2.29 bits, near free-viewing levels — the model has learned that fruitless search looks like exploration. Second, search generalizes zero-shot to object categories never seen in gaze training (bottle, chair, knife: 3.0–3.3 bits), because the target is just a word and the VLM already knows what a knife looks like. No specialized architecture can do that.

In-silico oculomotor experiments

The most conceptually interesting section is the counterfactual intervention. Because duration is just a conditioning token, you can hold the image, fixation history, and weights fixed and sweep the duration of the previous fixation from 50 ms to 600 ms, asking: how does the predicted distribution over the next fixation change?

You can't run this experiment on humans — you can't force a free-viewing fixation to last 50 ms without destroying the "free" part. The model's answer recovers two known oculomotor regimes from data alone. Conditioned on short durations (50–100 ms), predicted mass concentrates sharply near the previous fixation with some long-range jumps — the signature of "ambient" processing and pre-programmed saccades (saccadic momentum). As duration increases past 250 ms, spatial entropy rises monotonically from 9.56 bits (at 50 ms) to 10.68 bits (at 600 ms), with a pronounced +0.68-bit step between 250 and 400 ms — deliberate, "focal" saccade planning with broader options. The duration-agnostic model's entropy (9.80 bits) sits exactly where the median-duration regime predicts. This also explains the asymmetric value of duration conditioning: short fixations select a low-entropy regime and thus yield the biggest IG boost.

Nothing about ambient/focal modes or parallel saccade programming was built in. The model absorbed these regularities from the statistics of gaze data, and the prompting interface makes them queryable. The authors frame this as an in-silico sandbox for cognitive science — hypotheses generated here could feed mechanistic models like SceneWalk, the way DeepGaze's saliency insights previously did.

What to be skeptical about

The controls are unusually good for this genre, but a few things deserve caution. The matched-backbone comparison retrains DeepGaze III's encoder but keeps its architecture; one could argue no amount of encoder swapping tests whether a from-scratch transformer of comparable size to the 8B LLM would close the gap — the SmolVLM2 flatline and the MMMU correlation suggest pretraining quality is what matters, but that's correlational. Validation sets are small (270 scanpaths on MIT1003), and the images in these public eye-tracking datasets (MIT1003, COCO) may well appear in VLM pretraining corpora — the LODO and zero-shot-target results mitigate but don't eliminate contamination concerns for the semantic side. The 100×100 grid caps spatial resolution at 1% of image extent, coarser than eye-tracker precision. AUC differences vs. DeepGaze III are small (0.94 vs. 0.91–0.93), which the authors reasonably attribute to AUC saturating — but it means the win is specifically in probabilistic calibration, which IG rewards and their training objective directly optimizes. And exact full-map computation still costs 1,111 forward passes per fixation (~2 GPU-hours for AUC on 270 scanpaths), though they sketch a fix: adding all 10K coordinate pairs as atomic vocabulary tokens for single-pass density maps.

The interventional claims are the softest part: "recovering known phenomena" is validation against the literature, not new discovery, and the model remains a behavioral simulator, not a mechanistic explanation — the authors say so themselves.

Why this matters

This is a clean instance of the Bitter Lesson in a field that had resisted it: decades of inhibition-of-return modules, branching spatial/temporal pathways, and hand-coded oculomotor constraints, outperformed by "write the coordinates as text and fine-tune." But the deeper takeaway is the IG-equals-cross-entropy alignment: by making the training objective identical to the field's exact evaluation metric, the paper turns scanpath modeling into ordinary sequence modeling, with all the scaling behavior, prompting flexibility, and tooling that entails. And the \rho = 0.93 correlation between MMMU and gaze predictability is a result cognitive scientists should sit with — it says predicting human attention and general multimodal reasoning draw on the same representations.

Most worth your time: Section 3.3 (the spatial/sequential decomposition — the cleanest evidence the model learned dynamics, not just a better saliency map) and Section 3.4's counterfactual duration sweep. If you're evaluating the headline claim, the matched-backbone paragraphs in Section 3.2 are short and decisive.