Priors Over Pixels

Fluent is not the same as grounded

Medical VLMs are increasingly proposed for tasks that are discriminative rather than descriptive: confirm that a structure is there, flag a segmentation that has gone wrong, check a report against the scan. Those tasks need something report generation never tests — the ability to say no.

A model that confirms an organ it cannot see is not reading the image; it is reciting anatomy. We call this systematic over-affirmation of expected-but-absent structures present-bias, and it is invisible to any evaluation that only asks whether the model names what is there.

Why we alter no pixels

The usual way to test grounding is to break the image: erase the organ, void the region, paste in a counterfactual, and see whether the model notices. That introduces a confound. An edited region is itself an out-of-distribution artifact, so a model's reaction may be to the visual discontinuity rather than to the absence. In CT it is worse than usual — voiding voxels produces intensities and boundaries that no scanner would ever produce.

Our protocol never touches a pixel. Every input is a genuine, unaltered axial slice; the absences are real ones, taken from the anatomy itself. If a model fails to reject an organ here, the failure cannot be blamed on the edit, because there is no edit.

Data and ground truth

We use the BTCV / Synapse multi-organ abdominal CT collection: 30 labelled volumes with integer organ-label maps. Slices are reoriented to a canonical orientation and rendered under a single soft-tissue window (width 400, level 40 HU), grayscale replicated across three RGB channels.

Presence is decided per organ per slice by a hysteretic voxel-count rule that guards against partial-volume flicker: an organ counts as present at ≥ 50 voxels and as absent only below 5 voxels. Everything between is excluded, so every scored negative is an unambiguous absence. A stratified sample of 30 adversarial negatives was hand-verified to contain zero voxels of the probed organ.

The probe comprises 3,639 positive and 3,951 negative items per sampling strategy and output format, over 30 cases and 13 organs. Slices are drawn from organ-boundary transition, mid-range and out-of-range strata, so axial position alone is never a reliable cue.

Grading the prior: POPE for volumetric anatomy

Not every absent organ is equally tempting. Asking about an organ that rarely appears on that kind of slice is an easy negative; asking about one that almost always accompanies the organs actually visible is a hard one. We adapt the POPE polling protocol to 3D anatomy to turn that intuition into a dial. From the label volumes we precompute slice-level presence frequencies and a pairwise co-occurrence matrix, then select two absent organs per slice by one of three strategies:

• Random — drawn uniformly over the absent organs.
• Popular — the two absent organs with the highest slice frequency overall.
• Adversarial — the two highest in co-occurrence with the organs actually present on that slice.

For the adversarial setting, co-occurrence is conditioned only on spatially localised organs. The near-ubiquitous vessels — aorta, inferior vena cava, portal and splenic vein — are excluded, because their high slice prevalence collapses conditional co-occurrence back toward marginal frequency and degenerates the adversarial setting into the popular one.

The same candidate items are reused across the three output formats — free-text, verdict-first JSON and reasoning-first JSON — so format comparisons are paired. Responses parse into present, absent and not-addressed, and not-addressed items are reported separately rather than silently scored as absent.

Result: specificity collapses, sensitivity does not

Turning the dial produces a clean monotone collapse. Specificity falls from random to popular to adversarial in every output format, with non-overlapping case-clustered confidence intervals at the extremes — while sensitivity stays flat between about 0.89 and 0.95, confirming that the sampling strategy changes only the difficulty of the negatives, not the positives.

MedGemma-27B, zero-shot with greedy decoding. All rates carry patient-clustered (case-level) bootstrap 95% CIs, resampling cases rather than slices.

Under adversarial verdict-first output the model rejects the most anatomically expected absent organ 4.7% of the time. That asymmetry — near-ceiling affirmation, floor-level rejection — rather than overall accuracy, is the signature of prior-driven answering: the affirmative response is preserved and the failure lands specifically on absence.

Is it just missing the boundary by a few slices?

A charitable reading would be that the model is over-extrapolating: the organ is almost in frame, a slice or two away, and the model rounds up. We test that directly by fitting a logistic regression of correct rejection on the z-distance in millimetres from each absent probe to the organ's nearest true-presence slice.

• Free-text is distance-invariant — slope +0.002 log-odds/mm, 95% CI [−0.005, +0.008], including zero. The model fabricates the organ regardless of how far the slice is from where it actually occurs.
• Both JSON formats do rise with distance — verdict-first +0.027 [.020, .037], reasoning-first +0.024 [.018, .032] — so a boundary over-extrapolation component is real, on top of the prior.

But it cannot carry the extreme failures. Far out of range (≥ 40 mm), and on organs absent from the entire volume — where verdict-first specificity is 0.000 — the model still affirms presence. Present-bias persists where no nearby boundary exists to extrapolate from.

The finding that matters most: format does not fix grounding, it licenses abstention

Reasoning-first output — where the model writes an observation and a rationale before committing to a presence map — looks like the fix. Under standard binary scoring it recovers a large amount of specificity, and for the general-domain model it looks like a solution: 0.642 against the medical models' floor.

It is not grounding. The format lets the model hedge — "the spleen is not clearly visible" — and binary scoring, following the surface verdict, records every hedge as a correct rejection. Hedging is absent from free-text and verdict-first output by construction, and pervasive here.

We detect it with a deliberately inclusive lexical rule over the response (predominantly the observation and reasoning fields), flagging cues such as not clear, not clearly, obscured, cannot confirm, no definite, possibly, appears. Flagged items leave both the numerator and the denominator of confident-answer specificity. Because the rule is inclusive it can only shrink the confident set, which makes the resulting estimate conservative — and a single unambiguous cue (not clearly) accounts for roughly 90% of the general-domain model's flagged items, so the result does not hinge on the broader cue list.

Reasoning-first output, adversarial negatives. Binary specificity scores the surface verdict; confident-answer specificity excludes hedged responses from both numerator and denominator. Abstention is the hedge rate over all adversarial queries.

The general-domain model's apparent 0.642 rests on the 6.6% of queries it actually answers. Restricted to those, it scores 0.069 — squarely in the same low band as the medical models. When it is required to commit, reasoning-first prompting has converted confident fabrication into abstention; it has not produced visual grounding.

Abstention is not worthless — a model that signals uncertainty instead of fabricating is safer to deploy. But it is a different property from grounding and should not be scored as though it were the same one. Confident-answer specificity, reported alongside an explicit abstention rate, is what measures grounding.

It is not the rendering, the scale, or the medical fine-tuning

Under forced-choice free-text output, where abstention is unavailable, all five models remain present-biased across every sampling strategy.

Free-text specificity under forced choice. Higher is better; an always-present model scores 0.000.

Medical fine-tuning mitigates it — it does not cause it

Gemma3-4B and MedGemma-4B are a controlled pair: identical architecture, identical scale, identical vision encoder, differing only in medical fine-tuning. On adversarial negatives the base model is worse (0.143 vs 0.245). The bias originates in the general-domain VLM, and medical specialisation reduces it. What specialisation changes is the failure mode under uncertainty — confident fabrication versus abstention — not whether the bias is there.

Scale does not help, and on free output makes it worse

Within the MedGemma family, the 27B affirms more absent organs than the 4B (0.102 vs 0.245 adversarial free-text specificity). Capacity is not the bottleneck.

Rendering

Re-running matched items under MedGemma's native triple-window false-colour encoding instead of our single-window grayscale leaves the result unchanged: adversarial free-text specificity 0.102 → 0.112, with parsed verdicts agreeing on 96% of negative items. The renders are not byte-identical, so we claim verdict concordance and specificity invariance rather than identity — and notably, the ~4% of verdicts that differ flip from absent under grayscale to present under the native render, i.e. toward more affirmation.

Architecture

LLaVA-Med is a lower-capability boundary case: it saturates into an always-present regime (adversarial free-text specificity 0.000) and produces essentially no parseable structured output — clean-resolution below 0.01 for both JSON formats, against ≥ 0.995 for every other model. The two-regime z-distance structure replicates across all models that commit, so the effect is not a property of one model family.

The control that settles it: take the image away

If the verdict comes from the image, removing the image should change it. We re-queried adversarial negatives on a stratified subsample (n ≈ 976 per cell) with no image token at all.

Removing the image drives free-text specificity down in both models — MedGemma-27B 0.115 → 0.054, Qwen 0.173 → 0.043, with non-overlapping case-clustered CIs — while sensitivity rises. With nothing to look at, both still affirm the anatomically expected absent organ.

A uniform blank-gray field behaves differently: specificity goes up (0.257 and 0.322), so a blank image is not a no-information control but an absence-biasing one. The contrast is the useful part — the models clearly do respond to image content, and yet on adversarial negatives the prior still overrides the pixels. Abstention, meanwhile, barely moves when the image is removed (Qwen reasoning 0.945 → 0.846), which makes the hedging itself largely image-independent.

What it means, and what it does not

The direct caution is for quality assurance. A VLM asked to verify that an organ is present, or to flag a segmentation that has drifted, will on anatomically plausible absent structures tend to confirm the expected organ rather than reject it — and structured output, usually adopted precisely to make such systems reliable, does not fix that and can mask it. Negative-control and confident-answer evaluation should come before any such deployment.

The limits are worth stating plainly. This is abdominal CT, one collection, one anatomical-presence task; whether present-bias generalises to other modalities, body regions or finer-grained findings such as lesions is untested. The abstention measure is a lexical heuristic — it can miss non-lexical hedging and over-flag confident sentences containing cue words — so we report it as a diagnostic rather than a calibrated metric, and the headline collapse under POPE sampling is computed from surface verdicts and does not depend on it. The base-vs-medical ablation is a single architecture pair at 4B. And the protocol measures grounding behaviourally: it shows that the prior can dominate the image evidence, not the mechanism by which it does so.

References

1. A. Nazarov and A. Mayer, "Priors Over Pixels: Present-Bias in Organ-Presence Grounding for Medical VLMs," SAFER workshop, MICCAI, 2026.
2. B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein, "MICCAI multi-atlas labeling beyond the cranial vault — workshop and challenge," Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault Workshop Challenge, vol. 5, p. 12, 2015.
3. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, "Evaluating object hallucination in large vision-language models," arXiv:2305.10355, 2023.
4. N. Pham and M. Schott, "H-POPE: Hierarchical polling-based probing evaluation of hallucinations in large vision-language models," arXiv:2411.04077, 2026.
5. A. Sellergren, S. Kazemzadeh, T. Jaroensri, et al., "MedGemma technical report," arXiv:2507.05201, 2026.
6. Gemma Team, "Gemma 3 technical report," arXiv:2503.19786, 2025.
7. C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, "LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day," NeurIPS, 2023.
8. P. Wang, S. Bai, S. Tan, et al., "Qwen2-VL: enhancing vision-language model's perception of the world at any resolution," arXiv:2409.12191, 2024.
9. Z. R. Tam, C. K. Wu, Y. L. Tsai, C. Y. Lin, H. yi Lee, and Y. N. Chen, "Let me speak freely? A study on the impact of format restrictions on performance of large language models," arXiv:2408.02442, 2024.
10. K. il Lee, M. Kim, S. Yoon, M. Kim, D. Lee, H. Koh, and K. Jung, "VLind-Bench: measuring language priors in large vision-language models," arXiv:2406.08702, 2025.
11. H. Wang and Z. Li, "HallucXR: benchmarking and mitigating hallucinations in medical vision-language models for chest radiograph interpretation," arXiv:2605.20469, 2026.
12. U. Khan, U. Nawaz, L. D. M. S. S. Teja, N. Saeed, M. Bilal, Y. Xie, M. Yaqub, and M. H. Khan, "MedObvious: exposing the medical Moravec's paradox in VLMs via clinical triage," arXiv:2603.23501, 2026.