SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems

Abstract: Combining multiple Vision-LLMs (VLMs) can enhance multimodal reasoning and robustness, but aggregating heterogeneous models' outputs amplifies uncertainty and increases the risk of hallucinations. We propose SCoOP (Semantic-Consistent Opinion Pooling), a training-free uncertainty quantification (UQ) framework multi-VLM systems through uncertainty-weighted linear opinion pooling. Unlike prior UQ methods designed for single models, SCoOP explicitly measures collective, system-level uncertainty across multiple VLMs, enabling effective hallucination detection and abstention for highly uncertain samples. On ScienceQA, SCoOP achieves an AUROC of 0.866 for hallucination detection, outperforming baselines (0.732-0.757) by approximately 10-13%. For abstention, it attains an AURAC of 0.907, exceeding baselines (0.818-0.840) by 7-9%. Despite these gains, SCoOP introduces only microsecond-level aggregation overhead relative to the baselines, which is trivial compared to typical VLM inference time (on the order of seconds). These results demonstrate that SCoOP provides an efficient and principled mechanism for uncertainty-aware aggregation, advancing the reliability of multimodal AI systems.

• The paper introduces SCoOP, a training-free method that projects heterogeneous VLM outputs into a shared answer space and combines them with inverse-entropy weights to estimate system-level uncertainty.
• SCoOP improves hallucination detection and abstention on ScienceQA, reaching 0.866 AUROC and 0.907 AURAC versus 0.732–0.757 AUROC and 0.818–0.840 AURAC for heuristic baselines.
• The method adds only microsecond-scale aggregation overhead, remains stable as ensemble size changes, and preserves useful uncertainty signals even when all constituent VLMs perform poorly, but remains untested on free-form VQA.

SCoOP (Semantic-Consistent Opinion Pooling) is a training-free uncertainty quantification (UQ) framework that estimates system-level uncertainty for multi-VLM systems by aggregating heterogeneous models' probabilistic outputs through uncertainty-weighted linear opinion pooling (2603.23853). The paper addresses a gap the authors identify as unaddressed in prior literature: no existing UQ method explicitly targets hallucination detection at the level of an entire multi-VLM system, rather than a single model. The work is positioned between single-model UQ techniques and reliability methods for autonomous multi-agent systems.

Motivation and positioning

The authors motivate SCoOP from two observations. First, hallucination remains a central obstacle to deploying VLMs in safety-critical settings such as medical diagnosis and multi-robot collaboration, and UQ has become a primary mechanism for detecting it. Second, while multi-VLM aggregation can improve multimodal reasoning, improper aggregation can amplify individual model errors and propagate hallucinations; uncertainty must therefore be quantified not only per model but for the system as a whole.

Existing UQ approaches are judged impractical for this setting: token-level methods require logit access that commercial providers may restrict; self-verbalized confidence typically requires fine-tuning or calibration data; conformal prediction requires calibration sets. Consistency-based methods such as Semantic Entropy (Kossen et al., 2024) and VL-Uncertainty (Zhang et al., 2024) avoid these requirements but operate on single models. The closest related works—UQ-Merge [(Padhi et al., 30 Apr 2025)-adjacent ACL 2025 work] and Consensus Entropy—either do not quantify overall system uncertainty or evaluate only OCR tasks without assessing hallucination detection or abstention. SCoOP claims to fill this specific gap across broader VQA task types.

Method

SCoOP treats each VLM as a probabilistic expert over a shared hypothesis space of multiple-choice options $\Theta$. For each input $x=(I,q)$, each model $M_k$ generates $N$ sampled responses at temperature $T=1.0$; responses are mapped to options via regular-expression extraction following Semantic Entropy's semantic clustering practice, yielding an empirical probability vector $\mathbf{p}_k$ over $\Theta$. This step resolves the key obstacle that classical linear opinion pooling assumes identical class spaces across experts: heterogeneous VLMs are projected onto a unified option space, with absent options assigned zero probability.

Each model's Shannon entropy $H_k$ serves as its uncertainty estimate, and its confidence is defined as the inverse entropy $1/H_k$, normalized across models to produce pooling weights $w_k$. The aggregated distribution is:

$x=(I,q)$0

The final prediction $x=(I,q)$1 is the argmax of $x=(I,q)$2, and system-level uncertainty is the normalized Shannon entropy $x=(I,q)$3, ensuring scale-invariance across questions with different numbers of options. Ties are broken by the lowest-entropy individual model, then by fixed ordering. The authors acknowledge that entropy does not perfectly align with accuracy, citing prior calibration studies, but rely on the repeatedly observed positive correlation between high entropy and hallucination.

Experimental setup

Evaluation covers three benchmarks—ScienceQA (952 samples), MMMU (825 questions), and MMBench (950 samples)—using sixteen open-weight VLMs from five families (LLaVA-v1.6, Gemma-3, InternVL3, DeepSeek-VL2, Qwen2.5-VL) spanning four parameter tiers (2–4B to 72–78B), yielding 52 unique system combinations for $x=(I,q)$4. Because no prior method produces system-level uncertainty, baselines are two heuristic training-free aggregators built on Semantic Entropy: Naive Selection (pick the lowest-entropy model) and Majority Voting (entropy of the vote distribution). Metrics are AUROC for hallucination detection, AURAC for abstention, and E2E-Latency@p50.

Results

Headline performance: On ScienceQA with extra-large 3-VLM systems, SCoOP achieves AUROC 0.866 versus 0.757 (Majority Voting) and 0.732 (Naive Selection)—gains of 10.9% and 13.4%. AURAC is 0.907 versus 0.840 and 0.818. Notably, both baselines fall below the mean AUROC/AURAC of the three individual models evaluated separately (0.789 / 0.853), whereas SCoOP exceeds them by 7.7% and 5.4%. This is a strong claim: naive aggregation actively degrades uncertainty quality relative to doing nothing, while SCoOP's weighting preserves informative disagreement rather than diluting it.

Efficiency: Aggregation overhead is microsecond-scale (0.88–12.76 µs slower than Majority Voting), against total latencies of roughly 1–3.3 seconds dominated by per-model sampling. The latency cost of improved uncertainty is negligible.

Scaling behavior (RQ1–RQ2): Across all four parameter scales, SCoOP leads on both metrics, outperforming Majority Voting by 8.9–10.9% (AUROC). Across system sizes, SCoOP maintains near-flat performance (AUROC varies ≤0.6%, AURAC ≤1.3% across $x=(I,q)$5), averaging 0.824 AUROC and 0.876 AURAC over 26 configurations. In contrast, Majority Voting improves with $x=(I,q)$6 while Naive Selection degrades—suggesting SCoOP's benefit does not depend on careful system sizing.

Accuracy (RQ3): SCoOP consistently beats the average single-model accuracy (gains of 0.9–5.3%) but is generally second to Majority Voting on raw accuracy, with Naive Selection edging ahead at the extra-large scale. The paper concedes this directly: SCoOP's primary value is uncertainty quality, not accuracy maximization.

Low-accuracy regimes (RQ4): When all constituent VLMs score below 50% accuracy, baselines degrade toward non-informative (AUROC ≈ 0.5), while SCoOP retains signal: 0.608 on MMBench (4-VLM) and 0.667 on MMMU (4-VLM), beating baselines by 7.5–9.1%. This is arguably the most practically significant result, indicating usable uncertainty even when every member model is unreliable.

Limitations and open questions

The paper states plainly that although aggregation itself is cheap, total inference cost is dominated by per-model sampling ($x=(I,q)$7 samples per model), which scales with both $x=(I,q)$8 and $x=(I,q)$9. Evaluation is restricted to multiple-choice question answering; free-form VQA, where semantic clustering of open-ended generations is substantially harder, is left untested. The reliance on inverse-entropy weighting inherits the known imperfect correlation between entropy and correctness, and no alternative weighting schemes (e.g., learned or calibrated weights) are ablated. Baselines are heuristic rather than principled competitors, since no prior system-level UQ method exists—a comparison structure that strengthens the reported margins but leaves open how SCoOP compares against more sophisticated ensemble UQ techniques adapted to this setting.

Conclusion

SCoOP provides a simple, training-free extension of classical linear opinion pooling to heterogeneous VLM ensembles, delivering system-level uncertainty estimates that materially outperform heuristic aggregation on hallucination detection (AUROC 0.866 vs. 0.732–0.757) and abstention (AURAC 0.907 vs. 0.818–0.840) at negligible computational cost. Its robustness in low-accuracy regimes and stability across system sizes support its use as a reliability layer for multi-VLM deployments, though validation beyond multiple-choice formats remains an open question.