Vision-language models for chest radiography do not always need the image

TL;DR: Take a chest-X-ray VQA benchmark, and instead of just scoring answers, intervene on the image: swap in a different patient's scan with the same label, black out the region a radiologist marked as the evidence, or black out an equally-sized irrelevant region. If the answer doesn't move, the model wasn't reading the image. Applied to nine systems, this audit finds that three (including a specialist medical VLM) ignore the image entirely, one 119B multimodal model is causally unstable, and the five that do use the image ground only ~18–25% of their correct answers in patient-specific pixels. A text-only LLM with no image access lands within 5.7 accuracy points of the best multimodal system — and is statistically indistinguishable from a board-certified radiologist's accuracy on the same questions, while grounding at exactly zero. The uncomfortable conclusion: reported near-expert accuracy on chest radiography benchmarks is substantially a measure of prior-to-dataset fit, not radiology.

Why accuracy can't tell you if a model looked

Here's the trap. A yes/no question like "Is cardiomegaly present in this chest X-ray?" carries information in the question itself. Finding names have base rates. Benchmarks have label statistics. Reports have linguistic regularities that leak into training corpora. A language model that has absorbed all of this can answer "credibly" without ever consuming a pixel — and on a leaderboard, it looks identical to a model that actually examined the cardiac silhouette.

This isn't a hypothetical. Shortcut learning is a documented failure mode of medical imaging classifiers (scanner artifacts, race-correlated features, acquisition signals), and prior work has shown medical VLMs collapse under negated questions or on image pairs constructed so language priors can't help. But those are adversarial probes: they show accuracy overstates reliability by breaking the model on weird inputs. What nobody had done is ask, for an ordinary correct answer to a standard question, whether that answer causally depended on the image — and bound what text-alone and vision-alone can achieve on the exact same items. Saliency maps don't do this; they describe where a model attends, not whether the output would change if the evidence vanished.

The fix is old-fashioned causal reasoning: don't observe, intervene.

The causal triad

The probe set is 2,575 yes/no decisions built from three MIMIC-CXR-ecosystem sources: MS-CXR (which crucially provides radiologist-drawn bounding boxes localizing the evidence for eight findings), globally-labeled MIMIC-CXR studies, and ReXErr report-error sentences. Every case is run through one fixed pipeline under four image conditions, with the question held constant:

Ignores image: CGR = 0, UAR = 100, IS = 100. Unstable: IS < 70. Uses image: CGR > 0, IS ≥ 90.

The three metrics, read jointly:

• CGR (causal grounding rate): among answers correct on the original image, the fraction that flip when the radiologist-marked evidence region is occluded. A model reading that evidence should flip often.
• UAR (unrelated-image answer rate): the fraction of correct answers preserved when the image is swapped for a different patient's radiograph with the same label. A model using patient-specific evidence should drop below 100; a model answering from the question alone sits at exactly 100.
• IS (irrelevant-mask stability): the fraction of answers preserved when an equal-area black rectangle covers an irrelevant region. This is the negative control — it separates "sensitive to the evidence" from "sensitive to any occlusion."

The elegance is that this is purely behavioral and input-side: it works identically on GPT-5 (no weights) and on a linear probe over frozen RAD-DINO features. And the categories it produces are defined without reference to accuracy, which is what makes the punchline meaningful.

What the audit found

Nine systems: four general-purpose multimodal (Gemma-4-26B, Qwen3-VL-32B, Mistral-Small-4-119B, GPT-5), two medical multimodal specialists (MedGemma-1.5-4B, LLaVA-Med-7B), two text-only baselines that never receive an image (MedGemma-27B-text, DeepSeek-R1-7B), and a vision-only RAD-DINO linear probe.

Three systems land at CGR = 0.0, UAR = 100.0, IS = 100.0 — no image edit changes any answer, ever. Two are the text-only baselines, which is by construction. The third is LLaVA-Med-7B, a specialist medical VLM, which turns out to be a degenerate always-yes classifier (sensitivity 99.9, specificity 0.0). Mistral-Small-4-119B pairs a nominally high CGR of 40.0 with IS of only 56.0: it flips its answer about as readily when you mask a random corner as when you mask the evidence, so its "grounding" is just occlusion-flakiness. The remaining five — Gemma-4-26B, GPT-5, Qwen3-VL-32B, MedGemma-1.5-4B, and the RAD-DINO probe — show CGR from 6.4 to 33.5 with IS above 90: genuine but partial image use.

Now overlay accuracy, and the decoupling is stark:

The headline comparisons: the text-only MedGemma-27B, which literally never sees an image, is within 5.7 ± 1.3 points of the best multimodal system and significantly beats both specialist medical VLMs on shared cases (+3.7 over MedGemma-1.5-4B, +5.9 over LLaVA-Med). The 119B multimodal Mistral is statistically indistinguishable from the 7B text-only DeepSeek-R1 (Δ = −1.9, p = 0.219). Accuracy shows no trend with parameter count.

Even for the five image users, image use is thin and structured. Only 17.9–24.7% of their correct answers are image-contingent — they flip under the same-label swap; the rest are reachable from label-aligned priors given any compatible image. And grounding is sparse across findings: atelectasis and lung opacity register CGR = 0 for nearly every model, while cardiomegaly, consolidation, edema, pleural effusion, and pneumonia carry almost all the signal (Gemma-4-26B hits 63.2 on pneumonia; MedGemma-1.5-4B hits 69.7 on edema). Model rankings invert across findings. Most clinically pointed: every image user grounds better on posteroanterior than anteroposterior views (Gemma: 73.0 vs 25.1) — and AP is the portable, supine acquisition on the sickest patients, exactly where automated triage matters most.

The categories are robust: they replicate on CheXpert (UAR ranking transfers at Spearman ρ = 0.931; all three image-ignorers re-register UAR = 100), at 512px resolution, and across prompt phrasings. A telling detail from the CheXpert transfer: the multimodal advantage grows out-of-domain (5.7 → 8.6 points), consistent with MIMIC-specific text priors that don't transfer while image-driven accuracy does.

The radiologist comparison, and why confidence gating fails

Two board-certified radiologists took the same audit on an 80-case sub-sample. The reference reader scored 81.3 accuracy with CGR 23.1. The text-only MedGemma-27B was statistically indistinguishable from her on accuracy (+2.5, p = 0.746) while grounding at zero (−25.0 vs. reader, p = 0.001). Meanwhile the uses-image models grounded at reader-comparable rates — MedGemma-1.5-4B actually exceeded the reader's CGR (+27.7, p = 0.006). Note the radiologist's own CGR ceiling of ~23: much diagnostic evidence is diffuse or inferable from context, so occlusion-flipping is a conservative measure even for experts. This calibrates the metric — the image users aren't deficient relative to humans; the concern is entirely the systems that never look.

The confidence result is the piece with immediate deployment implications. Stratifying answers into grounded-correct, ungrounded-correct, and incorrect: the four image-using VLMs report dramatically higher confidence on grounded-correct answers (Gemma: 97.5 vs 45.6; GPT-5: 100.0 vs 48.6). So for models that use the image, a low-confidence correct answer is likely a coincidental prior-hit. But the signal vanishes or inverts everywhere else — LLaVA-Med reports ~97–99 confidence on everything including its errors, RAD-DINO is most confident when wrong, and Mistral's confidence-based AUROC is below chance (45.0). Confidence gating is uninformative or anti-calibrated for exactly the models that most need a guardrail, and calibration error is 31–47 across the board (versus the <5 usually deemed acceptable).

What to make of it, and what to be skeptical of

If this holds, the practical upshot is that "near-expert accuracy on chest-X-ray VQA" should stop being read as evidence of visual reasoning, and intervention-based grounding audits — which require only input access, no weights — should sit inside evaluation pipelines for any clinical-deployment claim. The audit recipe is cheap and general: it needs a benchmark with localized evidence annotations (MS-CXR-style boxes) plus label-matched swap candidates, both of which exist or can be built for other modalities.

The honest caveats, which the paper itself is unusually forthcoming about:

CGR is a conservative lower bound. A model reading global image features that survive local occlusion scores low despite genuinely using the image — that's precisely RAD-DINO's signature (CGR 6.4, but it collapses to 1.0 on cardiomegaly, a finding it classifies well from global structure). The joint triad mitigates but doesn't dissolve the global-vs-absent ambiguity.

The ground truth for grounding is itself noisy. Only ~48% of the MS-CXR boxes fully contained the queried evidence per the radiologist's own validation. Restricting CGR to fully-valid boxes raises the best system only from 33.2 to 43.9 and leaves the image-ignorers at 0.0, so the decoupling survives — but per-finding numbers should be read loosely. The human reference is also thin: two readers, 80 cases, κ = 0.224, and one reader registered CGR = 0 herself, so the "matches a radiologist" claim rests largely on a single reference reader and a failure-to-reject, not an equivalence test.

The task is binary finding-presence, not report generation — the models' actual clinical use — and finding-name questions carry exploitable base rates. Part of what the text-only baselines exploit is the benchmark's own statistical structure, which is simultaneously the phenomenon being documented and a reason not to over-read the absolute accuracies. Black-rectangle occlusion is also off-distribution; semantically realistic counterfactuals (generatively removing a finding) would be a cleaner intervention. And the assignments are snapshot- and protocol-specific: chain-of-thought, few-shot, or fine-tuning could change how much a model looks.

The selection note asked whether this is specific to prior-heavy datasets — partly yes, and the paper effectively concedes this: what CGR/UAR measure transfers only where localized evidence annotations exist, and the finding-name-prior exploit is strongest exactly in yes/no chest-radiograph QA. But the framing — accuracy and grounding are separable, measurable properties, and the field has been reporting the first while implying the second — travels.

Where to spend your time in the paper: the Methods subsection "Interventional conditions and behavioral metrics" (the formal triad, ~one page, fully specifies the audit), Table 1 plus Figure 2 for the categorization, and the limitations paragraph of the Discussion, which is one of the more candid self-critiques you'll read and doubles as a roadmap for building a better version of this audit.