LVLM-VAR: Risk-Aware LVLM Evaluation

Updated 10 July 2026

LVLM-VAR is a framework that assesses LVLM reliability by evaluating variance across languages, decoding-stage hallucination signals, and bias amplification.
It leverages benchmarks like MM-JudgeBench and methods such as TruthPrInt to measure multilingual robustness and mitigate object hallucination.
Drawing an analogy with Value at Risk, LVLM-VAR highlights that average accuracy alone can mask critical tail failures and instability in multimodal systems.

Searching arXiv for “LVLM-VAR” and closely related LVLM / VaR papers to ground the article. “LVLM-VAR” (Editor’s term) denotes a variance- and risk-aware perspective on large vision-LLMs (LVLMs) in which reliability is assessed not only by mean benchmark accuracy but also by cross-lingual instability, decoding-stage hallucination signals, and bias amplification under distributional shift. In the recent literature, this perspective is instantiated most directly by MM-JudgeBench, which evaluates multilingual and multimodal judge models across 25 languages, and by TruthPrInt, which mitigates object hallucination through latent truthful-guided pre-intervention during decoding. These results suggest a shift from English-centric or output-only evaluation toward multilingual variance analysis, per-token truthfulness estimation, and test-time control of adverse generation trajectories (Laskar et al., 21 Apr 2026, Duan et al., 13 Mar 2025).

1. Problem setting and conceptual scope

Two reliability problems dominate the present formulation of LVLM-VAR. The first concerns LVLM judges: automatic evaluators such as reward models are described as central to the alignment and evaluation of LVLMs, yet they are almost exclusively assessed on English-centric benchmarks. MM-JudgeBench addresses this by introducing the first large-scale benchmark for multilingual and multimodal judge model evaluation, with over 60K pairwise preference instances spanning 25 typologically diverse languages. Its two subsets are a multilingual extension of VL-RewardBench for general vision-language preference judgments and a chart-centric visual-text reasoning subset derived from OpenCQA. A separate 100K-sample multilingual training set, M-MM-RewardBench, is released for domain adaptation and is disjoint from evaluation data (Laskar et al., 21 Apr 2026).

The second problem concerns object hallucination (OH) in captioning and VQA-style LVLM generation. TruthPrInt treats OH as a trustworthy challenge in which the model names objects, attributes, or events that are not actually present in the image. The work emphasizes that hallucinations are not merely isolated wrong tokens; they are linked to latent decoding states that precede hallucinated object tokens. Because object tokens are only a small fraction of all generated tokens, the paper argues that an effective detector must be high-specificity, since false positives unnecessarily block normal generation and degrade caption quality (Duan et al., 13 Mar 2025).

Taken together, these two lines of work suggest that LVLM reliability has at least three technically distinct dimensions: variance across languages, bias under evaluation protocol perturbations, and truthfulness of token-level decoding dynamics. The common methodological theme is that average performance is insufficient when failure modes are sparse, state-dependent, or hidden by English-only reporting.

2. Benchmark construction for multilingual variance analysis

MM-JudgeBench is constructed by translating two existing English-centric multimodal evaluation datasets with Gemini-3-Pro into 25 typologically diverse languages, comprising 24 non-English languages + English. The language set spans Latin, Cyrillic, Arabic/Hebrew, Devanagari, and CJK scripts, and explicitly adds Bengali and Kazakh to the 23 languages used in prior multilingual reward benchmarking. Translation quality is checked with LaBSE and CometKiwi, and a filtering threshold of 0.75 is used, with manual inspection and back-translation for low-score cases. On VL-RewardBench, the reported translation quality is LaBSE 0.91 / CometKiwi 0.85 for Gemini-3-Pro, matching Gemini-2.5-Pro and exceeding Flash and Flash-Lite variants (Laskar et al., 21 Apr 2026).

Component	Origin	Role
M-VL-RewardBench	VL-RewardBench	General vision-language preference evaluation
M-OpenCQA	OpenCQA	Chart-centric visual-text reasoning
M-MM-RewardBench	MM-RewardBench	100K-sample multilingual training set for domain adaptation

The evaluation protocol asks an LVLM judge to choose the better answer from a pair. The primary metric is pairwise accuracy. The benchmark also measures position bias by presenting each pair in original and reversed order, length bias, instruction-following / JSON format adherence, and reasoning quality via LaBSE similarity for generated rationales. The tables report Avg and Var across languages, where lower Var indicates better multilingual robustness. The benchmark therefore operationalizes variance as a first-class evaluation target rather than a by-product of mean accuracy.

This design is important because it separates multilingual robustness from raw capability. A model can have high Avg and still exhibit unfavorable Var, and a model family can scale in average accuracy without achieving language invariance. That benchmark logic is a central component of the LVLM-VAR viewpoint.

3. Empirical regularities in multilingual robustness

The MM-JudgeBench study evaluates 22 LVLMs, including 7 proprietary and 15 open-source models. A central finding is that English-only evaluation hides large multilingual instability. Even strong models show meaningful drops in some languages; English is often the best-performing language, and Kazakh is repeatedly the hardest and often the worst-performing one. The paper further concludes that model size and architecture are poor predictors of multilingual robustness (Laskar et al., 21 Apr 2026).

Representative results on M-VL-RewardBench illustrate the separation between average performance and cross-lingual stability:

Model	Avg	Var
GPT-5	81.3	0.2
GPT-5-Mini	78.1	0.4
GPT-5-Nano	73.2	1.2
Gemini-2.5-Flash-Lite	40.8	2.6
Qwen3-VL-32B	68.8	3.3
LLaVA-Critic-7B	49.5	3.4

These numbers show that high average accuracy does not imply multilingual stability. The paper gives a concrete example in which a judge can choose the correct response in English but the wrong one after translation into French. Among open models, Qwen3-VL is identified as the strongest and most consistent family, with Qwen3-VL-32B best among open models on M-VL-RewardBench, yet even Qwen3 is not perfectly invariant across languages. InternVL-3.5 is competitive but less stable, Gemma-3 is generally weaker and has larger variance, and LLaVA-Critic-7B, despite reward-model-oriented training, performs poorly in multilingual settings and also has weak instruction-following behavior.

Bias analyses reinforce the variance story. Average English versus non-English positional bias amplification is reported as 11.03 vs 14.44 for Qwen3-VL, 15.79 vs 31.18 for InternVL-3.5, and 5.11 vs 9.95 for Gemma-3. Thus, biases amplify outside English. On chart reasoning, M-OpenCQA is often better and more stable than M-VL-RewardBench, but the same high-level trends remain: Qwen3-VL-32B is best among open models, English is usually best, and Kazakh is often worst. Because M-OpenCQA has no gold labels, the study uses GPT-5 as a reference judge and validates it with humans, reporting 93.5% human agreement on a 300-sample check.

Within the LVLM-VAR framing, these findings imply that multilingual robustness is not reducible to scale, that benchmark averages can conceal operational instability, and that language-conditioned bias is itself a major failure mode.

4. Latent truthfulness and hallucination subspaces

TruthPrInt approaches LVLM reliability from the internal-state side. Its first major result is that LVLM internal states, especially hidden states, are meaningful per-token indicators of hallucination. For a model $\mathcal{M}$ with parameters $\bm\theta$ , image $x$ , prompt $p$ , and generated token $z_i$ , token generation is written as

$z_i = p_{\bm\theta}(\cdot|x, z_{<i}, p),$

and the hidden state at layer $l$ is denoted

$h_{z_i}^{l} = \mathcal{M}^{l}(x, p, z_{<i+1};\bm\theta).$

The detector is trained not on the hidden state of the hallucinated object token itself but on the hidden state of the previous token, which is intended to provide an early-warning signal before the object token is emitted. A token is marked hallucinated if its object is absent from the image reference description (Duan et al., 13 Mar 2025).

Empirically, the detector’s overall accuracy is not impressive, but its specificity is very high. The paper reports a likelihood ratio for positive results

$LR^+ = TPR/FPR$

of nearly 20 at $FPR=0.01$ . This matters because OH detection is highly imbalanced: only about 5.6% of generated tokens are object tokens in MiniGPT-4 captions, and only around 10% of those are hallucinated. In this regime, high specificity is more valuable than balanced accuracy, because false alarms degrade otherwise valid captions.

The detector is a 3-layer MLP with architecture $\bm\theta$ 0, trained with binary cross-entropy, 30 epochs, batch size 512, learning rate 0.001, Adam, and 80%/20% train-validation splits. The paper reports that the best layer is typically a middle layer; layer 16 is used by default, and ablations show that middle layers encode more truthfulness information than early or late layers.

A second major claim is that hallucination-related information is shared across models in a common latent subspace, implying generic truthful directions shared by various LVLMs. Raw hidden-state transfer across models or datasets is weak, but transfer improves after projection into an aligned subspace. This motivates ComnHallu, an unsupervised domain adaptation procedure that constructs source and target latent subspaces and then aligns them. In effect, the work proposes that hallucination and truthfulness are not only output-level properties but also latent geometric structures that can be detected and transferred.

5. TruthPrInt decoding and mitigation performance

TruthPrInt is a pre-intervention method rather than a post hoc filter. The decoding process first generates normally while monitoring hidden states with the detector $\bm\theta$ 1. If hallucination is flagged, the algorithm continues until sentence completion, then backtracks to an earlier position with the lowest top confidence and restarts generation from that point. During the final decode, when the hidden state at position $\bm\theta$ 2 is flagged, the method uses the second-best candidate instead of the top candidate. The paper characterizes this as a detector-guided reranking and backtracking decoder that replaces risky token choices with more truthful alternatives (Duan et al., 13 Mar 2025).

The motivation for backtracking is supported by an analysis of Preceding Minimum Confidence (PMC): hallucinated objects show a characteristic low-confidence predecessor pattern, indicating that uncertain tokens often precede hallucinated object mentions. This shifts the intervention point from the hallucinated object itself to the earlier token that likely triggered the trajectory toward hallucination.

Evaluation covers COCO CHAIR, POPE, offline POPE (OPOPE), and qualitative analysis on LLaVA-Bench. CHAIR is measured by

$\bm\theta$ 3

and

$\bm\theta$ 4

POPE and offline POPE are reported with Precision and $\bm\theta$ 5 using $\bm\theta$ 6. In the practical co-transferring setup, training is done on CC-Sbu-Align hidden states from MiniGPT-4, thresholds are chosen to hit a target FPR on validation, and testing is done on other LVLMs using hidden states from COCO 2014 val images. Default hyperparameters are $\bm\theta$ 7, $\bm\theta$ 8, layer $\bm\theta$ 9, and $x$ 0. Experiments are repeated three times on 500 randomly selected images, averaging results and reporting standard deviations.

Quantitatively, TruthPrInt is reported as consistently the best or near-best method. On the three main LVLMs, it improves over HALC by about 12%–14% on CHAIR $x$ 1 and more than 2% on CHAIR $x$ 2, while also improving BLEU by roughly 2%. On MiniGPT-4, it achieves 16.87 CHAIR $x$ 3, 7.53 CHAIR $x$ 4, and 17.21 BLEU, compared with 21.40 / 8.99 / 14.81 for HALC. On LLaVA-1.5, it reports 10.33 / 3.87 / 19.79 versus 15.20 / 5.30 / 15.69. On mPLUG-Owl2, it obtains 11.13 / 5.27 / 18.82 versus 15.60 / 5.77 / 15.45. On offline POPE, it gives the best average Precision and $x$ 5, with 61.03 Precision and 61.18 $x$ 6. In transfer to non-Llama backbones, it reduces CHAIR $x$ 7 / CHAIR $x$ 8 to 6.2 / 3.4 on Qwen2-VL-7B-Instruct and to 3.2 / 3.0 on InternVL-2.5-8B, both better than Greedy, Beam, and HALC.

Ablations show that increasing the traceback budget $x$ 9 from 1 to 3 improves CHAIR and POPE performance, with diminishing returns thereafter. Smaller $p$ 0 yields stronger truthfulness but reduces diversity because more decode paths are rejected. The efficiency analysis reports runtime close to Greedy generation and much lighter than methods that depend on large auxiliary models such as CLIP or external LLMs.

6. Risk interpretation and the “VaR” analogy

In a separate literature, VaR denotes Value at Risk, a risk measure widely implemented by financial institutions. One line of work studies implied correlation from VaR by reversing the usual VaR calculation: instead of assuming a correlation and computing portfolio VaR, it infers the correlation between asset price changes from observed VaR values. For a two-asset portfolio, the central formula is

$p$ 1

Using S&P 500 and FTSE 100 data from January 1, 1995 to December 31, 2003, the study finds that implied correlation is not constant and tends to be higher for left-tail crashes than for right-tail booms. For an equally weighted portfolio at the highest daily probability level, implied correlation is 0.516 for long positions and 0.261 for short positions; the corresponding weekly example is 0.744 versus 0.358. The reported ordinary Pearson correlation is about 0.42, so VaR-implied correlation can lie above it in left-tail events and below it in right-tail events (Cotter et al., 2011).

A second financial result studies direct VaR forecasting with a long memory and non-linear realized volatility model called RNN-HAR. The model extends the HAR framework by integrating a simple RNN with tanh activation and estimating the target quantile directly through the quantile score

$p$ 2

Estimation uses loss-based generalized Bayesian inference with the asymmetric Laplace working model and Sequential Monte Carlo. Across 31 market indices from 2000 to 2022, the model achieves the best QS in more than 77% of markets at $p$ 3, the best VRate in more than 90% of markets at $p$ 4 and $p$ 5, and the fewest DQ rejections among competitors (Peiris et al., 2024).

A plausible implication for the use of “LVLM-VAR” as an Editor’s term is methodological rather than substantive. The financial VaR literature shows that averages and constant-parameter summaries can hide tail dependence, state dependence, and calibration errors. The LVLM results reviewed above show an analogous phenomenon in multimodal systems: average English accuracy can hide cross-lingual degradation, and output-level correctness can hide latent hallucination risk. Under this interpretation, LVLM-VAR refers to a research program in which LVLM reliability is studied with the same attention to adverse tails, instability, and conditional dependence that VaR-based risk analysis brings to financial data.