In-Context Collapse in Vision-Language Models and How to Mitigate it?

TL;DR: Feeding a vision-language model more correctly-labeled in-context examples can destroy accuracy it already had — Qwen2-VL-2B drops from 94% to 43% on 4-way CIFAR after a single demonstration, and some models fall below chance while still emitting perfectly well-formed labels. The paper shows this "in-context collapse" is graded across models (including Claude Sonnet 4.5), dissociates from the ability to actually learn a novel rule in context, and — the interesting part — is causally localizable: a tiny LoRA adapter on the connector and early/mid layers removes the collapse and restores genuine in-context learning, while an equal-capacity adapter on the late layers fails and can even make things worse. A single such adapter, trained once on one synthetic task, transfers collapse-resistance to unseen tasks.

The setup: many-shot ICL is supposed to be a free lunch

In text LLMs, the folk wisdom is that in-context learning improves roughly monotonically with the number of demonstrations — hundreds or thousands of shots can rival fine-tuning. VLMs inherit this promise with higher stakes: image labels are expensive, visual domains drift after deployment, and prompting is weight-free, so it can't cause catastrophic forgetting by construction. If accuracy grows with shot count, showing a VLM more labeled examples is nearly free adaptation.

The paper's opening observation is that for many VLMs it is the opposite. Define a K-shot episode as a prompt interleaving K (image, label) pairs before a query image, and let A(K) be balanced accuracy as a function of shot count. For a large fraction of an 11-model open panel (0.5B–11B parameters, spanning MLP-projector, pixel-shuffle, and cross-attention connectors), A(K) decreases, sometimes catastrophically:

Three details make this more than "prompting is finicky":

It's not a formatting failure. Qwen2-VL-2B's malformed-output fraction stays at zero across all K while its accuracy halves — the model keeps emitting valid labels, just wrong ones. (Two models, LLaVA-OV-0.5B and SmolVLM-500M, do degenerate into garbage at high K; the paper carefully separates these two failure modes and anchors all mechanistic claims on the clean case.)

The damage is front-loaded. For the clean-collapse models, 86–100% of the total accuracy drop happens at K = 0 \to 1. One demonstration knocks a confident zero-shot classifier off its prior, and the curve never recovers — at K = 64 (prompts exceeding 10^5 tokens) Qwen2-VL-2B still sits at 0.38. This kills any "long-context degradation" story: the collapse is triggered by the presence of demonstrations, not by their accumulation.

Where the accuracy goes is legible. The lost predictions don't scatter randomly — they overwhelmingly copy the label of the most recent demonstration. Qwen2-VL-2B's copy rate jumps from 0 at K = 0 to 0.83 at K = 1, exactly where accuracy craters. And a shuffled-labels control gives nearly identical accuracy to correct labels, so the model isn't reading demonstration content at all. The collapsed model has substituted "answer like the last example" for "look at the query."

The collapse reaches frontier scale: Claude Sonnet 4.5 drops from 0.88 to 0.12 on CIFAR-4 at K = 4 (Nova Lite/Pro and Haiku 4.5 are robust). It also reproduces, in attenuated form, on real VQA benchmarks: Qwen2-VL-2B degrades on VQAv2 (0.70→0.62) and TextVQA while its 7B sibling — same architecture, same recipe, different scale — holds steady.

The dissociation: surviving demonstrations ≠ learning from them

Here is the framing move that organizes the whole paper. "Does ICL work?" is actually two questions, and the paper separates them with a clever verbalizer trick. In the standard condition, classes get their natural names ("cat"), so a correct answer could come from pretraining priors or from the demonstrations. In the remap condition, classes get arbitrary tokens ("blorp") that carry zero pretraining signal — above-chance accuracy can only come from binding the in-prompt mapping. This gives two independent axes: robustness (does standard accuracy survive added demonstrations?) and learning (does remap accuracy rise above chance?).

These axes come apart in every combination. SmolVLM-2B is perfectly robust but cannot learn — remap stays pinned at chance no matter how many shots you give it; it just rides its priors. Qwen2-VL-2B collapses and cannot learn. Only Qwen2-VL-7B and Qwen2.5-VL-7B are both robust and genuine learners (Qwen2-VL-7B reaches 0.98 remap accuracy on shapes at 16 shots). Most strikingly, Claude Sonnet 4.5 collapses on natural labels while simultaneously learning arbitrary remapped labels (0.50–0.89 remap at K = 8) — its failure is specifically about robustness to prior-conflicting demonstrations, not about in-context learning per se.

The two capabilities even have opposite temporal signatures: collapse strikes at the first shot; genuine learning is absent at low K and builds slowly (Qwen2-VL-7B shows no remap gain through K = 2, then climbs steadily). The very demonstrations that destroy one model's prior are what a learner is only beginning to exploit.

The aha: the collapse lives at the integration interface, not the readout

A VLM factors into a vision encoder, a connector (projecting visual features into the LM's embedding space), and a language model stack. Prior interpretability work established that visual information becomes linearly readable only in deep layers. The paper's central hypothesis distinguishes reading from binding: the early/mid layers plus the connector form an integration locus \mathcal{I} that must bind visual demonstrations to their in-prompt labels; the late third is a readout locus \mathcal{R} that maps an already-formed decision to tokens. The collapse, they claim, is a deficit of the integration computation.

K demos + query

Parameter-matched control: early vs. late regions differ by <10% of adapter modules, so the rescue asymmetry reflects location, not capacity. Bonus dissociation: durable weight-based consolidation works best at 𝓡, not 𝓘.

The test is a parameter-matched lesion-and-rescue. Attach a rank-8 LoRA restricted to one depth region of a frozen collapse-prone model (Qwen2-VL-2B), train only the adapter on remap episodes — where the in-context labels are arbitrary tokens, so the adapter cannot memorize a label→answer shortcut and can only succeed by learning to read the demonstrations — and re-measure the remap learning curve. Crucially, regions are capped to matched module counts (70 vs. 63 modules for early vs. late), so any difference reflects location, not capacity.

Three things stand out. The gradient is monotone front-to-back. The smallest target (the connector, one projection block, ~1% of parameters) is the most consistent rescue site, beating the all-layer adapter — strong evidence for a compact locus at the modality bottleneck. And the late adapter isn't merely useless: it drives remap from 0.39 down to 0.08, below chance, suggesting capacity placed at the readout during many-shot prompts actively entrenches the recency-copy heuristic. The same integration-over-readout ordering replicates on Qwen2.5-VL-3B and, more noisily, on the pixel-shuffle InternVL3-2B — two connector families, two backbone lineages. The rescue is also K-specific: at K \le 2 intervention regions are indistinguishable from baseline; the gap opens at K = 4 and is maximal at K = 8–16, exactly the demonstration-overload regime. This refines the interpretability picture nicely: late layers read out visual decisions (as prior logit-lens work found) but do not integrate demonstrations — readability and manipulability come apart.

From diagnosis to remedy: the one-shot "vaccine"

If the deficit is a compact, localized computation, you might be able to install it once and reuse it. The paper trains a single integration-locus adapter on one synthetic task (procedurally generated shapes with remapped labels — contamination-free, so the adapter is forced to learn the "use the in-prompt mapping" procedure, not any specific content), freezes it, and evaluates on task families the adapter never saw:

The locus ablation is the strongest single piece of evidence in the paper: re-targeting the same-capacity vaccine to the late layers doesn't just fail to transfer — it drives even the trained shapes task to 0.00. Meanwhile a training-free "inject" baseline (extracting a task vector from the model's own forward pass and adding it at the integration locus, the multimodal analogue of function vectors) leaves the collapse-prone model at chance — informative, because a patch that only relocates an existing computation cannot help if the computation is absent. The collapse is a missing capability, not a misplaced one.

A final, unexpected dissociation: the layers best for in-context integration are not the layers best for durable weight-based learning. On a 5-task class-incremental CIFAR-100 stream, sequential LoRA at the late layers achieves higher average accuracy (0.717 vs. 0.668) and less forgetting (−0.292 vs. −0.344 BWT) than the integration circuit, with half the parameters. Fast, prompt-driven adaptation and slow, weight-based consolidation live in different parts of the network — a concrete targeting rule for parameter-efficient continual learning, and a pleasing echo of complementary-learning-systems ideas.

How much to believe, and what to watch

The behavioral phenomenon looks solid and is easy to reproduce (code is released; all white-box experiments run on a single 24GB GPU). The dissociation framing — robustness and learning as independent axes measured via the remap verbalizer — is genuinely useful and will likely be borrowed by others evaluating multimodal ICL. Note that "many-shot ICL degrades in VLMs" is not itself new (VL-ICL Bench and others reported it behaviorally); the new claims are the below-chance severity in fluent models, the two-axis decomposition, and above all the causal localization.

Reasons for caution:

The "rescue" involves training. The vaccine is gradient descent on ICL episodes — meta-training, essentially. What licenses the causal reading is the asymmetry under matched capacity: if any 1%-of-parameters LoRA could install demonstration-use, the late adapter should too, and it catastrophically doesn't. That's good evidence, but "equal module count and rank" isn't the same as "equal effective capacity" — early and late layers may differ in trainability for reasons (gradient paths, feature abstraction level) orthogonal to where the collapse "lives." The below-chance late result argues the late layers genuinely can't implement the binding computation, which supports the paper's reading, but a skeptic could still frame this as "where is ICL learnable" rather than "where was ICL broken."

The causal claim is one-directional. The paper is admirably honest here: the reverse experiment — inducing collapse in a robust model by lesioning the same circuit — is weak and diffuse. Collapse-proneness looks localized; collapse-immunity looks distributed. So the claim is best read as "prone → immune is a surgical edit," not a full circuit-level account.

Scale and statistics. All mechanistic work is on 2–3B open models; the frontier evidence is 24–40 queries per cell, with standard-condition curves single-seed. The Sonnet 4.5 collapse (0.88→0.12) is striking but thin. Panel cells use 25–40 queries, so individual numbers carry meaningful noise (visible in Table 1's occasional oddities). Whether the same low-rank edit repairs frontier models is untestable through APIs.

An alternative reading of the first-shot drop. 86–100% of the collapse landing on K = 0 \to 1 is consistent with "integration overload," but also with a blunter story: the demonstration-formatted prompt is simply out-of-distribution relative to the model's instruction tuning, and the model falls into a pattern-completion mode (hence the 0.83 copy rate). The paper would say these are the same thing described at different levels — the alignment recipe determines whether demonstrations engage integration or trigger copying — and its own evidence (regime membership tracks recipe, not scale or connector) supports that. But "collapse" connotes accumulation, and the data actually show a step-function mode switch.

If the result holds, the practical implications are real: many-shot prompting on VLMs needs per-model validation (zero-shot skill predicts nothing about ICL behavior — a practitioner who validates on Qwen2-VL-7B and deploys on the 2B silently loses half their accuracy), the copy rate is a free, label-less canary for the failure, and a cheap offline adapter can immunize open models. The most provocative idea is in the future-work list: predicting collapse-proneness directly from weight signatures (e.g., conditioning of the connector-to-early-layer Jacobian) and pre-vaccinating at release.

Where to spend your reading time: Section 4.2.4 and Tables 3, 5, and 7 — the lesion-and-rescue with its capacity control, the vaccine locus ablation (late-only → 0.00), and the K-specificity curves. That cluster is the paper's evidentiary core; the rest of the paper is characterization around it.