Answer or Defer? Testing VLMs' Stated Confidence on Brain MRI

Abstract

Vision-language models can report confidence alongside an answer, but that number is useful for selective prediction only if it ranks reliable answers above errors. This work audits stated confidence from eight open-weight VLMs on 3,888 brain MRI slices drawn from post-treatment glioma volumes and healthy volunteers.

We evaluate coverage at fixed error budgets, separate answer-level confidence bias from within-answer signal, test sensitivity to population composition, and measure whether confidence tracks lesion area. The results show that high stated confidence often reflects which answer a model gives more than whether that answer is correct, limiting confidence-only answer-or-defer policies.

TL;DR

Selective prediction

Thresholds are judged by both accepted-slice coverage and empirical error, not confidence alone.

Confidence structure

Answer preference can dominate the within-answer signal that should separate correct from incorrect cases.

Clinical deferral

A high score is not treated as permission to automate; unsupported cases remain deferred for review.

Evaluation Design

BraTS supplies glioma-positive and glioma-negative slices; IXI supplies a healthy comparison population. Each model returns an answer and stated confidence, then the same outputs are examined through four complementary questions rather than a single aggregate accuracy score.

Study pipeline and four research questions — **Answer-or-defer evaluation.** Q1 measures attainable coverage at 5%, 10%, and 20% error; Q2 decomposes answer-level and within-answer confidence; Q3 evaluates case-mix dependence; and Q4 tests whether confidence responds to lesion size.

Key Findings

7 / 8

models retain no nonempty BraTS subset at 5% error

24%

best retained coverage at a 10% error budget

0.969 / 0.970

Gemma-4 confidence on true vs. false positive answers

Case mix changes the story

Gates that look safe on healthy brains can favor models that are highly specific but miss many tumor slices.

Recalibration is not enough

Strictly monotone recalibration cannot improve selective coverage because it does not change confidence ranking.

Qwen model gives the same high confidence to correct and incorrect brain MRI answers — **High confidence does not imply correctness.** The same model reports 0.95 confidence for healthy, lesion-negative, small-lesion, and large-lesion slices while producing both false positives, a false negative, and a correct positive answer.

Why Stated Confidence Fails

The failure is structured rather than random. Confidence is strongly tied to the answer a model produces, shifts with the evaluation population, and remains nearly unchanged across lesion sizes even when detection sensitivity changes substantially.

Mean stated confidence by true-positive, false-positive, true-negative, and false-negative outcome — **Confidence follows the answer more than correctness.** For each model, true and false positives receive nearly the same confidence, as do true and false negatives. The larger separation is between positive and negative answers.

Confidence-correctness correlation changes sign between BraTS and IXI populations — **The association reverses across populations.** Every model changes sign between the BraTS glioma cohort and IXI healthy volunteers. Because IXI contains only negative reference labels, its sign mainly reflects which answer is stated with greater confidence—not a transferable measure of reliability.

Sensitivity increases with in-plane lesion area for eight VLMs on BraTS positive slices — **Detection improves with lesion size, but confidence barely responds.** Sensitivity rises steeply across in-plane lesion-area bins for most models, while mean stated confidence changes little. Confidence therefore gives limited warning on the small lesions that are most likely to be missed.

Reproducibility

The repository contains the complete evaluation pipeline: slice-level reference construction, resumable inference, response parsing, selective-risk analysis, covariance decomposition, case-mix experiments, subject-clustered bootstrap, and paper table/figure regeneration. Medical images and raw model transcripts are not redistributed.

git clone https://github.com/amir-sbg/VLM-Answer-or-Defer.git
cd VLM-Answer-or-Defer

conda env create -f environment.yml
conda activate deferral
export PYTHONPATH="$PWD"

make test
make paper

BibTeX

@inproceedings{sabbaghziarani2027answer,
  title     = {Answer or Defer? Testing {VLMs}' Stated Confidence on Brain {MRI}},
  author    = {Sabbaghziarani, Amir and Plis, Sergey},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year      = {2027},
  note      = {Under review}
}