---
title: 'LVLM-VAR: Risk-Aware LVLM Evaluation'
url: https://www.emergentmind.com/topics/lvlm-var
type: topic
---

# LVLM-VAR: Risk-Aware LVLM Evaluation

Searching arXiv for “LVLM-VAR” and closely related LVLM / VaR papers to ground the article.
“LVLM-VAR” (*Editor’s term*) denotes a variance- and risk-aware perspective on large vision-language models (LVLMs) in which reliability is assessed not only by mean benchmark accuracy but also by cross-lingual instability, decoding-stage hallucination signals, and bias amplification under distributional shift. In the recent literature, this perspective is instantiated most directly by MM-JudgeBench, which evaluates multilingual and multimodal judge models across 25 languages, and by TruthPrInt, which mitigates object hallucination through latent truthful-guided pre-intervention during decoding. These results suggest a shift from English-centric or output-only evaluation toward multilingual variance analysis, per-token truthfulness estimation, and test-time control of adverse generation trajectories [2604.19405][2503.10602].

## 1. Problem setting and conceptual scope

Two reliability problems dominate the present formulation of LVLM-VAR. The first concerns **LVLM judges**: automatic evaluators such as reward models are described as central to the alignment and evaluation of LVLMs, yet they are almost exclusively assessed on English-centric benchmarks. MM-JudgeBench addresses this by introducing the first large-scale benchmark for multilingual and multimodal judge model evaluation, with over 60K pairwise preference instances spanning 25 typologically diverse languages. Its two subsets are a multilingual extension of VL-RewardBench for general vision-language preference judgments and a chart-centric visual-text reasoning subset derived from OpenCQA. A separate 100K-sample multilingual training set, M-MM-RewardBench, is released for domain adaptation and is disjoint from evaluation data [2604.19405].

The second problem concerns **object hallucination (OH)** in captioning and VQA-style LVLM generation. TruthPrInt treats OH as a trustworthy challenge in which the model names objects, attributes, or events that are not actually present in the image. The work emphasizes that hallucinations are not merely isolated wrong tokens; they are linked to latent decoding states that precede hallucinated object tokens. Because object tokens are only a small fraction of all generated tokens, the paper argues that an effective detector must be **high-specificity**, since false positives unnecessarily block normal generation and degrade caption quality [2503.10602].

Taken together, these two lines of work suggest that LVLM reliability has at least three technically distinct dimensions: **variance across languages**, **bias under evaluation protocol perturbations**, and **truthfulness of token-level decoding dynamics**. The common methodological theme is that average performance is insufficient when failure modes are sparse, state-dependent, or hidden by English-only reporting.

## 2. Benchmark construction for multilingual variance analysis

MM-JudgeBench is constructed by translating two existing English-centric multimodal evaluation datasets with **Gemini-3-Pro** into **25 typologically diverse languages**, comprising **24 non-English languages + English**. The language set spans Latin, Cyrillic, Arabic/Hebrew, Devanagari, and CJK scripts, and explicitly adds **Bengali** and **Kazakh** to the 23 languages used in prior multilingual reward benchmarking. Translation quality is checked with **LaBSE** and **CometKiwi**, and a filtering threshold of **0.75** is used, with manual inspection and back-translation for low-score cases. On VL-RewardBench, the reported translation quality is **LaBSE 0.91 / CometKiwi 0.85** for Gemini-3-Pro, matching Gemini-2.5-Pro and exceeding Flash and Flash-Lite variants [2604.19405].

| Component | Origin | Role |
|---|---|---|
| M-VL-RewardBench | VL-RewardBench | General vision-language preference evaluation |
| M-OpenCQA | OpenCQA | Chart-centric visual-text reasoning |
| M-MM-RewardBench | MM-RewardBench | 100K-sample multilingual training set for domain adaptation |

The evaluation protocol asks an LVLM judge to choose the better answer from a pair. The primary metric is **pairwise accuracy**. The benchmark also measures **position bias** by presenting each pair in original and reversed order, **length bias**, **instruction-following / JSON format adherence**, and **reasoning quality** via **LaBSE similarity** for generated rationales. The tables report **Avg** and **Var** across languages, where lower **Var** indicates better multilingual robustness. The benchmark therefore operationalizes variance as a first-class evaluation target rather than a by-product of mean accuracy.

This design is important because it separates multilingual robustness from raw capability. A model can have high **Avg** and still exhibit unfavorable **Var**, and a model family can scale in average accuracy without achieving language invariance. That benchmark logic is a central component of the LVLM-VAR viewpoint.

## 3. Empirical regularities in multilingual robustness

The MM-JudgeBench study evaluates **22 LVLMs**, including **7 proprietary** and **15 open-source** models. A central finding is that **English-only evaluation hides large multilingual instability**. Even strong models show meaningful drops in some languages; **English is often the best-performing language**, and **Kazakh** is repeatedly the hardest and often the worst-performing one. The paper further concludes that **model size and architecture are poor predictors of multilingual robustness** [2604.19405].

Representative results on **M-VL-RewardBench** illustrate the separation between average performance and cross-lingual stability:

| Model | Avg | Var |
|---|---:|---:|
| GPT-5 | 81.3 | 0.2 |
| GPT-5-Mini | 78.1 | 0.4 |
| GPT-5-Nano | 73.2 | 1.2 |
| Gemini-2.5-Flash-Lite | 40.8 | 2.6 |
| Qwen3-VL-32B | 68.8 | 3.3 |
| LLaVA-Critic-7B | 49.5 | 3.4 |

These numbers show that high average accuracy does not imply multilingual stability. The paper gives a concrete example in which a judge can choose the correct response in English but the wrong one after translation into French. Among open models, **Qwen3-VL** is identified as the strongest and most consistent family, with **Qwen3-VL-32B** best among open models on M-VL-RewardBench, yet even Qwen3 is not perfectly invariant across languages. **InternVL-3.5** is competitive but less stable, **Gemma-3** is generally weaker and has larger variance, and **LLaVA-Critic-7B**, despite reward-model-oriented training, performs poorly in multilingual settings and also has weak instruction-following behavior.

Bias analyses reinforce the variance story. Average English versus non-English positional bias amplification is reported as **11.03 vs 14.44** for **Qwen3-VL**, **15.79 vs 31.18** for **InternVL-3.5**, and **5.11 vs 9.95** for **Gemma-3**. Thus, biases amplify outside English. On chart reasoning, **M-OpenCQA** is often better and more stable than M-VL-RewardBench, but the same high-level trends remain: **Qwen3-VL-32B** is best among open models, **English** is usually best, and **Kazakh** is often worst. Because M-OpenCQA has no gold labels, the study uses **GPT-5** as a reference judge and validates it with humans, reporting **93.5%** human agreement on a **300-sample** check.

Within the LVLM-VAR framing, these findings imply that multilingual robustness is not reducible to scale, that benchmark averages can conceal operational instability, and that language-conditioned bias is itself a major failure mode.

## 4. Latent truthfulness and hallucination subspaces

TruthPrInt approaches LVLM reliability from the internal-state side. Its first major result is that LVLM **internal states, especially hidden states, are meaningful per-token indicators of hallucination**. For a model $\mathcal{M}$ with parameters $\bm\theta$, image $x$, prompt $p$, and generated token $z_i$, token generation is written as
$$
z_i = p_{\bm\theta}(\cdot|x, z_{<i}, p),
$$
and the hidden state at layer $l$ is denoted
$$
h_{z_i}^{l} = \mathcal{M}^{l}(x, p, z_{<i+1};\bm\theta).
$$
The detector is trained not on the hidden state of the hallucinated object token itself but on the **hidden state of the previous token**, which is intended to provide an early-warning signal before the object token is emitted. A token is marked hallucinated if its object is absent from the image reference description [2503.10602].

Empirically, the detector’s **overall accuracy is not impressive**, but its **specificity is very high**. The paper reports a likelihood ratio for positive results
$$
LR^+ = TPR/FPR
$$
of **nearly 20 at $FPR=0.01$**. This matters because OH detection is highly imbalanced: only about **5.6%** of generated tokens are object tokens in MiniGPT-4 captions, and only around **10%** of those are hallucinated. In this regime, high specificity is more valuable than balanced accuracy, because false alarms degrade otherwise valid captions.

The detector is a **3-layer MLP** with architecture **$(4096,128)\rightarrow(128,64)\rightarrow(64,1)$**, trained with **binary cross-entropy**, **30 epochs**, **batch size 512**, **learning rate 0.001**, **Adam**, and **80%/20% train-validation splits**. The paper reports that the best layer is typically a **middle layer**; **layer 16** is used by default, and ablations show that middle layers encode more truthfulness information than early or late layers.

A second major claim is that hallucination-related information is **shared across models in a common latent subspace**, implying **generic truthful directions shared by various LVLMs**. Raw hidden-state transfer across models or datasets is weak, but transfer improves after projection into an aligned subspace. This motivates **ComnHallu**, an unsupervised domain adaptation procedure that constructs source and target latent subspaces and then aligns them. In effect, the work proposes that hallucination and truthfulness are not only output-level properties but also latent geometric structures that can be detected and transferred.

## 5. TruthPrInt decoding and mitigation performance

TruthPrInt is a **pre-intervention** method rather than a post hoc filter. The decoding process first generates normally while monitoring hidden states with the detector $\mathcal{G}$. If hallucination is flagged, the algorithm continues until sentence completion, then backtracks to an earlier position with the **lowest top confidence** and restarts generation from that point. During the final decode, when the hidden state at position $i-1$ is flagged, the method uses the **second-best candidate** instead of the top candidate. The paper characterizes this as a detector-guided reranking and backtracking decoder that replaces risky token choices with more truthful alternatives [2503.10602].

The motivation for backtracking is supported by an analysis of **Preceding Minimum Confidence (PMC)**: hallucinated objects show a characteristic low-confidence predecessor pattern, indicating that uncertain tokens often precede hallucinated object mentions. This shifts the intervention point from the hallucinated object itself to the earlier token that likely triggered the trajectory toward hallucination.

Evaluation covers **COCO CHAIR**, **POPE**, **offline POPE (OPOPE)**, and qualitative analysis on **LLaVA-Bench**. CHAIR is measured by
$$
\text{CHAIR}_S = \frac{|\text{sentences with hallucinated objects}|}{|\text{all sentences}|}
$$
and
$$
\text{CHAIR}_I = \frac{|\text{hallucinated objects}|}{|\text{all objects mentioned}|}.
$$
POPE and offline POPE are reported with **Precision** and $F_\beta$ using $\beta=0.1$. In the practical co-transferring setup, training is done on CC-Sbu-Align hidden states from **MiniGPT-4**, thresholds are chosen to hit a target FPR on validation, and testing is done on other LVLMs using hidden states from **COCO 2014 val** images. Default hyperparameters are **$\tau=0.4$**, **$d'=64$**, **layer $l=16$**, and **$\mathcal{N}_B=5$**. Experiments are repeated **three times** on **500 randomly selected images**, averaging results and reporting standard deviations.

Quantitatively, TruthPrInt is reported as consistently the best or near-best method. On the three main LVLMs, it improves over **HALC** by about **12%–14% on CHAIR$_S$** and more than **2% on CHAIR$_I$**, while also improving **BLEU** by roughly **2%**. On **MiniGPT-4**, it achieves **16.87 CHAIR$_S$**, **7.53 CHAIR$_I$**, and **17.21 BLEU**, compared with **21.40 / 8.99 / 14.81** for HALC. On **LLaVA-1.5**, it reports **10.33 / 3.87 / 19.79** versus **15.20 / 5.30 / 15.69**. On **mPLUG-Owl2**, it obtains **11.13 / 5.27 / 18.82** versus **15.60 / 5.77 / 15.45**. On **offline POPE**, it gives the best average **Precision** and **$F_\beta$**, with **61.03 Precision** and **61.18 $F_\beta$**. In transfer to non-Llama backbones, it reduces **CHAIR$_S$ / CHAIR$_I$** to **6.2 / 3.4** on **Qwen2-VL-7B-Instruct** and to **3.2 / 3.0** on **InternVL-2.5-8B**, both better than **Greedy**, **Beam**, and **HALC**.

Ablations show that increasing the traceback budget **$\mathcal{N}_B$** from **1** to **3** improves CHAIR and POPE performance, with diminishing returns thereafter. Smaller **$\tau$** yields stronger truthfulness but reduces diversity because more decode paths are rejected. The efficiency analysis reports runtime close to **Greedy** generation and much lighter than methods that depend on large auxiliary models such as **CLIP** or external **LLMs**.

## 6. Risk interpretation and the “VaR” analogy

In a separate literature, **VaR** denotes **Value at Risk**, a risk measure widely implemented by financial institutions. One line of work studies **implied correlation from VaR** by reversing the usual VaR calculation: instead of assuming a correlation and computing portfolio VaR, it infers the correlation between asset price changes from observed VaR values. For a two-asset portfolio, the central formula is
$$
\rho_{12}=\frac{(VaR_{port})^2-x_1^2\cdot (VaR_1)^2-x_2^2\cdot (VaR_2)^2}{2\cdot x_1\cdot x_2\cdot VaR_1\cdot VaR_2}.
$$
Using **S&P 500** and **FTSE 100** data from **January 1, 1995 to December 31, 2003**, the study finds that implied correlation is **not constant** and tends to be higher for **left-tail crashes** than for **right-tail booms**. For an equally weighted portfolio at the highest daily probability level, implied correlation is **0.516** for **long positions** and **0.261** for **short positions**; the corresponding weekly example is **0.744** versus **0.358**. The reported ordinary Pearson correlation is about **0.42**, so VaR-implied correlation can lie above it in left-tail events and below it in right-tail events [1103.5655].

A second financial result studies **direct VaR forecasting** with a **long memory and non-linear realized volatility model** called **RNN-HAR**. The model extends the **HAR** framework by integrating a **simple RNN** with **tanh** activation and estimating the target quantile directly through the **quantile score**
$$
\textup{QS}_{t} = \left ( y_{t}-\textup{VaR}_{t}^{\alpha}\right )\left ( \alpha-I(y_{t}<\textup{VaR}_{t}^{\alpha})\right ).
$$
Estimation uses **loss-based generalized Bayesian inference** with the **asymmetric Laplace** working model and **Sequential Monte Carlo**. Across **31 market indices** from **2000 to 2022**, the model achieves the best **QS** in **more than 77%** of markets at **$\alpha=1\%$**, the best **VRate** in **more than 90%** of markets at **$\alpha=1\%$** and **$\alpha=2.5\%$**, and the fewest **DQ** rejections among competitors [2408.13588].

A plausible implication for the use of “LVLM-VAR” as an *Editor’s term* is methodological rather than substantive. The financial VaR literature shows that averages and constant-parameter summaries can hide **tail dependence**, **state dependence**, and **calibration errors**. The LVLM results reviewed above show an analogous phenomenon in multimodal systems: average English accuracy can hide cross-lingual degradation, and output-level correctness can hide latent hallucination risk. Under this interpretation, LVLM-VAR refers to a research program in which LVLM reliability is studied with the same attention to adverse tails, instability, and conditional dependence that VaR-based risk analysis brings to financial data.

Source: https://www.emergentmind.com/topics/lvlm-var