DoveBench: Diverse LLM & Multimodal Evaluation

• DoveBench is a set of large-scale benchmarks that evaluate LLM behavior across prompt sensitivity, cultural value alignment, and multimodal video translation.
• The prompt sensitivity benchmark uses a factorial design over five dimensions generating over 250 million responses to quantify accuracy variance and stability.
• The cultural and multimodal benchmarks rigorously assess model performance using metrics like Unbalanced Optimal Transport, BLEU, and SubER for actionable insights.

DoveBench refers to multiple distinct large-scale benchmarks for evaluating LLM behavior in diverse contexts. Notably, "DoveBench" is used for (1) fine-grained prompt sensitivity assessment of LLMs, (2) culturally-grounded open-ended evaluation of value alignment, and (3) multimodal video subtitling and translation benchmarking. Each instantiation employs rigorous methodology, enabling principled analysis of model capabilities and limitations in specific domains.

1. Multi-Dimensional LLM Prompt Sensitivity Benchmark

DoveBench, as introduced by the SLAB NLP group, is a comprehensive dataset and evaluation suite that exposes LLMs to a factorial design of prompt variations to quantify their output sensitivity (Habba et al., 3 Mar 2025). The benchmark covers 7,800 unique instances drawn from 78 widely used multiple-choice domains (including MMLU, ARC, RACE), each subjected to thousands of systematically perturbed prompt presentations.

Structural Overview

Dimension	# Variations	Types/Examples
Enumerator	6	A, a, 1, I, symbols, Greek
Separator	7	space, newline, ", ", "; ", "
Choice Ordering	7	original, by length ↑/↓, alphabet ↑/↓, correct first, last
Instruction Phrasing	13	Paraphrastic instruction templates
Demonstrations	2	Zero-shot, five-shot

Total prompt variants per instance reach 7,644, yielding approximately 60 million prompt-model pairs per foundation model, for a total of over 250 million model responses across five open-weight families (Llama 1B/3B/8B, OLMoE 7B, Mistral 7B).

Methodology

A deterministic linearization function $T(x, d_1, ..., d_n) \to p$ produces each prompt $p$ from a base input $x$ and a selection along the $n=5$ intent-preserving dimensions (enumerator, separator, choice ordering, instruction phrasing, demonstrations). Automatic correctness is assigned using semantic-similarity to human answers, and full input/output schemas are logged including token-level log-probs and metadata.

Prompt sensitivity is operationalized via:

• Per-dimension accuracy variance
• Divergence of the original prompt from the mean, measured as $(Acc_{original} - \mu)/\sigma$
• "Stability score": area under the accuracy/sample-size curve for different prompt-selection routines

Key Results

• Substantial prompt sensitivity is quantifiable: e.g., OLMoE’s HellaSwag accuracy ranges from 1% to 99% across variants; Mistral 7B shows up to 8% accuracy swings with instruction phrasing changes.
• Marginal-wise prompt tuning (dimension-wise optimization) almost matches regression-optimized full-prompt selection and is more sample-efficient at small scales.
• Five-shot demonstrations reduce but do not eliminate sensitivity (variance narrows by about 20 points, yet >15 points remain on hard domains).
• Some instances are universally solved or failed, regardless of prompt; these are candidates for robustness studies.

Usage

DoveBench facilitates robust LLM evaluation, supporting reproducibility, extension to new languages and domains, and error analysis focused on content (not formatting artifacts). All data, code, and tools are public at https://slab-nlp.github.io/DOVE.

2. Distributional Open-Ended Cultural Value Alignment (DOVE)

DOVE (Dataset Of Variation Evaluation), also referenced as DoveBench, is a framework and dataset for assessing the extent to which LLM-generated open-ended content aligns with latent cultural values of human communities (Lee et al., 16 Mar 2026). Unlike knowledge-based or multiple-choice value benchmarks, DOVE directly compares the distributions of structured value codes in human-authored versus model-authored texts.

Codebook and Mapping Methodology

• Human and LLM documents (10,676 total; four cultures: China, Japan, Korea, US) are mapped into a codebook of $K\approx 213$ discrete "value codes" using rate-distortion variational optimization:

$$\mathcal{C}^* = \arg\min_{\mathcal{C}} \mathbb{E}_{\hat{p}(x)}\bigl[\mathbb{E}_{q_\omega(s|x,\mathcal{C})}[ -\log\,p_\phi(x|s,\mathcal{C}) ]\bigr] - \beta_1 H_q(s|x,\mathcal{C}) + \beta_2 H_q(s|\mathcal{C})$$

• Value expressions are LLM-extracted and soft-assigned to codebook entries, producing per-document code histograms.

Alignment Metric

• Alignment is computed by measuring Unbalanced Optimal Transport (UOT) distance between human and model value-code distributions, with semantic-geometric weighting:

$$\mathcal{D}_\mathrm{UOT}(\hat{p}^g, p_\theta) = \min_{\pi\geq 0} \sum_{i,j}\biggl[D_{ij} \pi_{ij} + \epsilon \pi_{ij}(\log \pi_{ij} - 1)\biggr] + \gamma \mathrm{KL}(\pi\mathbf{1} \| \mathbf{a}^g) + \gamma \mathrm{KL}(\pi^\top\mathbf{1} \| \mathbf{a}^\theta)$$

• The debiased UOT is converted to an alignment score $$r = (0.1 - \widehat{\mathcal{D}_\mathrm{UOT}}) \times 10$$.

Empirical Results

• DOVE achieves the highest predictive validity among existing benchmarks: mean correlation with downstream (real-world) cultural tasks is +31.56% (compared to +23.56% for CDEval, +16.2% for WVS, and negative for others).
• Strong construct validity is recorded via role-playing (value priming), multitrait-multimethod diagnostics, and template invariance (Cronbach’s $\alpha=0.9830$).
• The method is stable with as few as 300 topics per culture, supporting efficient scaling.

Limitations

• Coverage is restricted to CN, JP, KR, and US cultures, with data predominantly from internet-active populations.
• The codebook treats each country as a single culture; intra-national plurality is an open direction.
• Ongoing codebook adaptation is needed for new topics or value expressions.

3. DoveBench for Video Subtitling and Multimodal Translation

DoveBench, within the context of the ViDove system, denotes a large-scale benchmark for evaluating automatic video subtitling and translation with multimodal context (Lu et al., 9 Jul 2025). It consists of 17.23 hours of annotated video content (50 videos, Counter-Strike and StarCraft 2 genres), with 16,968 millisecond-aligned subtitle segments translated from English speech to simplified Chinese.

Dataset Structure

Property	CS (Counter-Strike)	SC2 (StarCraft 2)	Total
Videos	23	27	50
Duration (hours)	4.97	12.27	17.23
Subtitle segments	-	-	16,968
Total words	-	-	189,157
Avg. segment length	-	-	11.1 w

Subtitle annotation was performed by expert subtitle groups, with translations preserving technical, colloquial, and speaker intent content.

Task Definitions and Metrics

• Automatic Subtitling: Speech-to-text transcript generation with precise temporal alignment.
• Machine Translation: English transcript to Chinese subtitle translation.
• Multimodal Alignment: Evaluation of subtitle accuracy with respect to audio and video cues.

Evaluation protocols use:

• BLEU (n-gram overlap, uniform weights)
• SubER (Subtitle Error Rate)
• BLEURT (neural text similarity)
• SubSONAR (Subtitle Sound-And-Reading metric)

Baseline Results

System	BLEU ↑	BLEURT ↑	SubER ↓	SubSONAR ↑
Gemini-2.5-Flash	8.11	17.21	103.46	0.31
Qwen-2.5-Omni	14.60	13.83	108.94	0.39
VideoCaptioner	12.65	14.62	85.75	0.41
Whisper + DelTA	18.26	12.30	86.83	0.28
ViDove	23.51	19.55	73.38	0.39

Ablations reveal the specific impact of domain memory, visual cues, and post-editing agents on translation performance.

Usage

DoveBench serves as the primary, public benchmark for automatic subtitling and translation in esports and gaming contexts. The dataset and codebase are distributed under an open research license at https://github.com/pigeonai-org/ViDove. Users are advised to verify dataset license details in the repository.

4. Comparative Significance and Methodological Advances

DoveBench, across its instantiations, advances evaluation rigor in several respects:

• Prompt Sensitivity: Provides a definitive empirical basis for rejecting single-prompt evaluations in LLM benchmarking, demonstrating magnitude and structure of sensitivity in common workloads (Habba et al., 3 Mar 2025).
• Value Alignment: Establishes the first distributional, codebook-grounded, open-ended evaluation protocol for cultural value orientation, with demonstrably higher validity and reliability than prior benchmarks (Lee et al., 16 Mar 2026).
• Multimodal Translation: Delivers large-scale, high-quality ground truth for research on automatic subtitling, especially in challenging gaming contexts (Lu et al., 9 Jul 2025).

The separation of concerns in each DoveBench implementation—format robustness, value-signal fidelity, and multimodal grounding—reflects emerging demands for transparency and coverage in LLM evaluation.

5. Limitations, Open Issues, and Future Directions

• Cultural Benchmarks: Current coverage is limited to select cultures; extension to broader sociolinguistic groups and intra-country variation remains necessary (Lee et al., 16 Mar 2026).
• Prompt Evaluation: While marginal-wise prompt selection offers sample efficiency, residual sensitivity suggests inherent model brittleness unmitigated by current prompt engineering (Habba et al., 3 Mar 2025).
• Subtitling Datasets: The legal status of some DoveBench media assets is not exhaustively reported; careful review is recommended prior to large-scale or commercial use (Lu et al., 9 Jul 2025).

Continued development, community participation, and methodological refinements (e.g., codebook adaptation, uncertainty quantification via stored log-probs, finer segmentation) are explicitly encouraged in DoveBench releases.

6. Public Access and Community Integration

All DoveBench variants are distributed under open or research licenses, with codebases, web viewers, and extension tools made available for academic and industrial integration. Resources are hosted at:

• https://slab-nlp.github.io/DOVE (multi-dimensional LLM prompt evaluation)
• https://github.com/pigeonai-org/ViDove (video subtitling, translation)

Researchers are encouraged to utilize instance-level analytics, prompt optimization routines, and distributional metrics for their respective tasks. The open structure is intended to promote critical assessment of LLM robustness, fairness, and value alignment in real-world deployments.