LIBERO Simulation Benchmark

• LIBERO is a standardized evaluation benchmark for assessing large vision-language models across tasks like image-text retrieval, visual QA, and linguistic robustness.
• It measures cross-modal alignment and multilingual generalization using metrics such as Recall@k and accuracy percentages derived from diverse, established datasets.
• The benchmark emphasizes systematic experimental control and direct comparability by analyzing failure modes and the impact of scaling, distillation, and pretraining strategies.

The LIBERO Simulation Benchmark is a standardized evaluation suite for assessing large vision-LLMs (VLMs) and multimodal LLMs (MLLMs) on a broad spectrum of vision-language tasks. The benchmark focuses on two central axes: robustness of retrieval and question answering across linguistic and visual complexity, and the impact of model scaling, knowledge distillation, and pretraining objectives in both monolingual and multilingual settings. It collects data from established VLM families (including CLIP, SigLIP2, and their variants) and emphasizes systematic experimental control, direct comparability, and fine-grained analysis of failure modes.

1. Benchmark Scope and Design Principles

LIBERO encompasses a carefully selected set of vision-language tasks to probe the cross-modal alignment, semantic robustness, and linguistic generalization of VLMs. The tasks are constructed to expose model limitations in:

• Text-to-Image and Image-to-Text Retrieval: Evaluating recall at top ranks ($\text{Recall}@k$) for matching images and natural language captions, both in-domain and out-of-domain.
• Multilingual Vision-Language Understanding: Assessing performance drop-off when extending models from English-only to multilingual settings using datasets with diverse languages and caption noise.
• Visual Question Answering (VQA): Testing the model’s ability to answer compositional and factual questions about images, requiring both vision and language reasoning.
• Fine-Grained Linguistic Robustness: Measuring invariance to paraphrasing and sensitivity to targeted semantic flips (object, color, count).

Evaluation protocols apply strict data splits and standardized metrics to ensure comparability and reproducibility across VLM families and scales (Sriratanawilai et al., 30 Oct 2025, Lee, 17 Nov 2025).

2. Core Tasks and Evaluation Datasets

LIBERO aggregates multiple core tasks from established open-source datasets:

Task	Dataset(s)	Metric(s)
Image-Text Retrieval	Multi30k, COCO, WIT, xFlickr, XM3600	Recall@1,5,10
Visual QA	CVQA, ALM-Bench	Accuracy (%)
Linguistic Robustness	MS-COCO (w/ LGIP)	$E_{\mathrm{inv}}$, $E_{\mathrm{sens}}$, PR

Text-to-Image (T2I) and Image-to-Text (I2T) retrieval are measured using recall at various thresholds (Recall@1, Recall@5, Recall@10); Visual Question Answering employs percent accuracy on multiple-choice and open-ended benchmarks. The Language-Guided Invariance Probing (LGIP) task uses specially constructed parap