---
title: 'PISA Benchmark: Multimodal AI Evaluation'
url: https://www.emergentmind.com/topics/pisa-benchmark
type: topic
---

# PISA Benchmark: Multimodal AI Evaluation

Searching arXiv for the benchmark papers to ground the article and verify the cited records.
to=arxiv_search.search  大发时时彩是json code  blocks ਨਹੀਂ?
to=arxiv_search.search 平台直属json code  大发分分彩
{"query":"PISA-Bench multilingual multimodal metric evaluation vision-language models arXiv 2510.24792", "max_results": 5}
to=arxiv_search.search  大发时时彩开奖ിയjson code 
to=arxiv_search.search {"query":"PISA-Bench multilingual multimodal metric evaluation vision-language models", "max_results": 5}
The label **PISA benchmark** appears in several distinct forms in the cited literature. At its source is the OECD **Programme for International Student Assessment (PISA)**, a triennial survey of 15-year-old students’ competencies in reading, mathematics, and science in over 80 countries; around that source, multiple AI evaluation suites have been constructed, including **mmPISA-bench** for multilingual LLM reasoning, **PISA-Bench** for multilingual multimodal VLM evaluation, and other acronymically related benchmarks such as **PiSA-Bench** for 3D MLLMs, **PisaBench** for physical accuracy in video generation, and **PISA** for saliency detection [2606.07069], [2510.24792], [2503.10529], [2503.09595], [1505.03227]. In the cited literature, the term therefore denotes not one canonical benchmark but a cluster of evaluation frameworks that share a concern with controlled assessment, annotation quality, and fine-grained diagnosis.

## 1. OECD PISA as the reference framework

The OECD’s Programme for International Student Assessment is a triennial survey that measures 15-year-old students’ competencies in reading, mathematics, and science in over 80 countries. Its items are subjected to rigorous localization, translation, and validation procedures so that difficulty and meaning remain consistent across languages and cultures [2606.07069]. This institutional framework is the direct source for both educational-statistical analyses and later AI benchmarks derived from public PISA items.

Two statistical problems recur in research using PISA as a benchmark of national performance. The first is **sample selection bias**. Because PISA samples students still in school at age 15, cross-country comparisons can be distorted by survival bias. A quantile-selection model formalizes this by introducing a latent score $Y^{*}$, an enrollment indicator $S \in \{0,1\}$, and an observed score $Y$ equal to $Y^{*}$ only when $S=1$; under stochastic dominance, the latent quantile function is bounded by observed quantiles, and under a parametric beta-cost selection rule it becomes point identified [2309.10642]. In the PISA 2018 application, correcting for selection bias shifted several country rankings, including Canada from official rank 12 to corrected rank 19 in mathematics, Slovenia from rank 14 to rank 8, Germany from 20 to 12, and the U.K. from 19 to 28 [2309.10642].

The second problem is **measurement non-invariance**, operationalized as **Differential Item Functioning (DIF)** in a multigroup IRT model. In the PISA 2022 analysis, student proficiencies are modeled as country-specific Gaussian latent traits, while item responses use a logistic form with common discrimination $a_j$, shared intercept $d_j$, and country-item DIF effects $\gamma_{jk}$. A two-step estimator is proposed: a baseline MML fit, followed by an $\ell_1$-projection that enforces an identifiability constraint without requiring reference groups or anchor items [2505.16608]. On PISA 2022 mathematics, the analysis used $N \approx 277{,}000$ students in 37 OECD countries on $J=169$ binary items; about 12% of country-item pairs had $|\hat\gamma_{jk}|>0.05$, and seven countries changed rank by at least two positions relative to the official RMSD-anchoring procedure [2505.16608].

| Benchmark name | Domain | Core unit |
|---|---|---|
| OECD PISA | Educational assessment | Student test items in reading, mathematics, and science |
| mmPISA-bench | Multilingual LLM reasoning | 25 MCQs in 43 languages with two translation types |
| PISA-Bench | Multilingual multimodal VLM evaluation | 122 image-text MCQs in 6 languages |
| PiSA-Bench | 3D MLLM evaluation | 240 point-cloud → caption pairs across 6 aspects |
| PisaBench–Real / PisaBench–Sim | Physics-aware video generation | Real slow-motion videos and synthetic Kubric videos |
| PISA | Saliency detection | Pixel-level saliency maps on multiple image datasets |

This coexistence of educational, multimodal, 3D, generative-video, and saliency meanings suggests that acronym disambiguation is necessary whenever “PISA benchmark” is cited.

## 2. mmPISA-bench: multilingual reasoning with official and machine translations

**mmPISA-bench** is a compact multilingual reasoning benchmark derived from public OECD PISA items and built to test whether LLMs “reason equally well” across languages [2606.07069]. It uses 25 hand-picked text-only multiple-choice questions: 11 from **PISA 2022 math** and 14 from **PISA 2018 reading**. The selection criteria were that an item be exclusively text-based, in MCQ format, and available in a broad set of official PISA translations [2606.07069].

Each question is provided in **43 officially human-translated languages** and paired with a **machine-translated counterpart**, yielding a total of **2,150 data points**:
$$
25\ \text{questions} \times 43\ \text{languages} \times 2\ \text{translation types} = 2{,}150.
$$
For every question-language pair, the benchmark stores the context and stem, four answer options, and the correct key for both translation types [2606.07069].

The evaluation protocol is **zero-shot**. Two proprietary LLM families are tested under multiple reasoning-effort settings, including no reasoning, low/medium/high chain-of-thought, and a “double-prompt” repetition in the no-reasoning case. Accuracy is defined conventionally as
$$
\text{Accuracy}=\frac{\#\text{correct answers}}{\#\text{total questions}}.
$$
The benchmark additionally records per-language variance, human-versus-machine translation differences, reasoning length in output tokens, and a tokenization “premium,” defined as the ratio of input tokens in a non-English language to English [2606.07069].

The reported results show high cross-lingual robustness but nontrivial cost heterogeneity. Claude achieved **92.3%** average accuracy at a total cost of **\$198.1**, while GPT achieved **89.5%** at **\$81.5**. Under high chain-of-thought, Claude reached **96.6%** and GPT **95.7%**; under no reasoning, they scored **87.5%** and **78.9%**, respectively [2606.07069]. Machine-translated questions yielded equal or slightly higher accuracy than official human translations: for Claude at high effort, **96.6%** on human versus **97.4%** on machine; for GPT, **95.7%** versus **96.1%** [2606.07069].

A common concern is that synthetic translation necessarily degrades multilingual reasoning evaluation. On this benchmark, that concern is not supported: token usage differences between translation types are reported as negligible, below 5% for both input and output, and machine-translated items did not reduce accuracy [2606.07069]. At the same time, tokenization artifacts mattered: languages with higher tokenization premiums, such as Thai and Greek, were simultaneously more expensive and less accurate, and accuracy–cost correlation was negative for both model families [2606.07069].

## 3. PISA-Bench for multilingual and multimodal VLM evaluation

**PISA-Bench** extends the PISA-derived paradigm from text-only reasoning to **vision-language models**. It is built from publicly available OECD PISA tests from **2012 and earlier**, filtered for items that contain both text and an associated image or figure, are self-contained, and do not depend on preceding tasks [2510.24792]. After manual filtering for clarity, completeness, and multimodality, **122 English examples** remained.

Each example was decomposed into four fields: **Instruction**, **Image**, **Question**, and **Answer Options**. When original multiple-choice options were missing, they were generated via GPT-4o and rephrased into a four-option MCQ format. Two independent annotators checked every English item against five criteria, including faithfulness to original intent, absence of answer leakage, plausibility of distractors, and grammatical correctness. The result was exactly **122 high-quality, human-verified English items** [2510.24792].

The corpus was translated in parallel into **Spanish, German, Chinese, French, and Italian**, producing a fully parallel benchmark in **six languages**. Translation prompts preserved all units, symbols, numbers, and abbreviations verbatim and forbade any reordering of answer options, so the correct answer remained correct. Automatic validation used **WMT23 COMET-KIWI** and **GEMBA-MQM**, while native speakers reviewed 50 random items per language; the human verification found no critical errors, and error-free rates ranged from **76–88%** [2510.24792].

Items are assigned to four question-type categories: **Spatial & Geometric Reasoning**, **Quantitative Reasoning**, **Graph & Pattern Analysis**, and **Text & Diagram Understanding**. Evaluation is zero-shot and free-form: models receive the instruction, question text, and image, and generate an answer string. GPT-4 acts as judge via semantic matching, and the principal metric is accuracy over the **122** examples:
$$
\mathrm{Acc}_{\mathrm{overall}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}(\hat y_i=y_i), \qquad N=122.
$$
Per-category accuracy and a contamination test comparing text-only with text-plus-image performance are also reported [2510.24792].

The results show a marked dependence on parameter scale and language. Small models below **20B** parameters averaged below **55%** accuracy on English, while larger open-source models above **20B** reached up to approximately **67%**. Proprietary GPT-4o achieved **71.0%** on English [2510.24792]. Multilingual degradation was systematic: **10/12** multilingual models dropped by **–1.4 to –8.4 percentage points** outside English. Spatial and geometric reasoning remained a consistent failure mode; for GPT-4o, error rates in this category stayed relatively high at approximately **40%** [2510.24792].

A possible misconception is that such benchmark accuracy could largely reflect text memorization. The contamination test argues against that interpretation: for example, Qwen2.5-VL-7B-Instruct dropped from **61.1%** with images to **33.6%** without images, and Qwen2-VL-7B-Instruct dropped from **54.0%** to **35.4%** [2510.24792]. This suggests that image-conditioned reasoning is materially contributing to performance.

## 4. PiSA-Bench for 3D multimodal large language models

**PiSA-Bench** belongs to a different research line and acronym expansion. It was introduced together with **PiSA-Engine (Point-Self-Augmented-Engine)** in work on 3D understanding with large models [2503.10529]. The benchmark responds to limitations in earlier 3D benchmarks such as Objaverse, Cap3D, and ScanObjectNN, which are described as using short, coarse captions and limited object categories, thereby making it difficult to localize which components of 3D MLLM behavior succeed or fail [2503.10529].

The benchmark has three stated design goals: to provide multi-aspect, richly detailed captions that capture true **3D spatial semantics**; to cover a broad but controlled set of **40 held-out objects** spanning common household, industrial, and natural categories; and to supply classification labels with **synonyms and subclass groupings** so that open-vocabulary evaluation does not unfairly penalize near-synonyms such as “mug” and “cup” [2503.10529].

Each benchmark item is annotated along **six orthogonal dimensions**: **Description**, **Color**, **Shape (Geometry)**, **Count**, **Spatial Relations**, and **Usage (Function)**. The dataset contains **240 point-cloud → caption pairs**, corresponding to **6 aspects × 40 objects**. The objects were selected from Objaverse but held out of all training, with the paper noting **3,000 reserved objects**, of which **40** form PiSA-Bench. For each object, **3–5** synonyms or subclass names are generated via GPT-4 and human-verified [2503.10529].

PiSA-Bench supports two tasks. In **zero-shot 3D object captioning**, models generate a free-form caption from the native point cloud. Evaluation combines automated metrics—**BLEU-1, ROUGE-L, METEOR, CIDEr**—with a **GPT-4o multi-aspect scoring** protocol that awards **0–100** on each of the six aspects and averages them, plus a human evaluation in which each distinct correct attribute receives one point and near-matches receive **0.25, 0.5, or 0.75** credit [2503.10529]. In **generative 3D object classification**, a natural-language answer is judged correct if any synonym or subclass in the label set matches, with **Top-1 Accuracy** and **Top-K Accuracy** defined in the usual way [2503.10529].

Reported zero-shot results on PiSA-Bench show iterative gains for the PiSA training framework. In generative classification, **PointLLM-7B** scored **47.50%**, **PointLLM-PiSA-7B** scored **61.25%**, and **PointLLM-PiSA³-7B** reached **63.75%**. In 3D object captioning using GPT-4o average over six aspects, the baseline scored **38.12%**, **PointLLM-PiSA-7B** scored **43.70%**, **PointLLM-PiSA²-7B** scored **46.41%**, and **PointLLM-PiSA³-7B** scored **46.45%** [2503.10529].

The benchmark’s evaluation recipe is explicit: researchers should generate captions and classification answers for the 40 point clouds, use the provided GPT-4o prompts for scoring, optionally conduct human evaluation with the shuffled-group protocol, and ensure that PiSA-Bench objects are never seen during training. Missing spatial relations or usage information is described as heavily penalized [2503.10529].

## 5. PisaBench for physical accuracy in video generation

A third meaning appears in **“PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop”**, where **PISA** stands for **Physics-Informed Simulation and Alignment** and the associated benchmark is a diagnostic suite for free-fall dynamics in video generation [2503.09595]. The target phenomenon is object free-fall under gravity, optionally with simple collision against a pile of objects.

The benchmark has two components. **PisaBench–Real** contains **361 slow-motion videos** recorded at **120 fps** with cell phones mounted on tripods and cropped to **1:1**. Scenes include indoor and outdoor environments, with **1–6 objects** suspended by an invisible wire and released at frame 1; each clip is annotated with a caption of the form “\<object description\> falls.” and with segmentation masks from **SAM-2** [2503.09595]. **PisaBench–Sim** is a synthetic dataset rendered in **Kubric** with **PyBullet physics @240 Hz** and Blender rendering. It uses standard Earth gravity, a physics step of $\Delta t = 1/240\ \mathrm{s}$, **2-second** videos rendered as **32 frames at 16 fps**, and Google Scanned Objects split into **930 train objects** and **103 test objects** [2503.09595].

The evaluation metrics are geometric and motion-centric. They include **Trajectory $L_2$**, defined as mean 2D centroid error across frames; **Chamfer Distance** between generated and ground-truth foreground-pixel sets; **Intersection-over-Union** over binary masks; and, on simulated data only, **drop-time error**, where the physical reference time is
$$
t_{\mathrm{drop}}=\sqrt{2Y_0/g}.
$$
These metrics explicitly target motion fidelity, shape fidelity, and object permanence rather than texture or semantics [2503.09595].

The experimental protocol first performs zero-shot evaluation of **eight off-the-shelf models**, then studies post-training on Open-Sora v1.2. **Supervised Fine-Tuning (PSFT)** uses the same rectified-flow objective at batch size **128**, learning rate **1e−4**, for **5,000** steps on **5,000** simulated videos. **Object Reward Optimization (ORO)** then optimizes
$$
J(\theta)=\mathbb{E}_{(x,c)\sim D,\ x'\sim p_\theta(\cdot|c)}[R(x',x)]
$$
with rewards based on segmentation IoU, optical flow from **RAFT**, and depth from **DepthAnyV2** [2503.09595].

The main findings are diagnostic. All zero-shot models are reported to fail at realistic free-fall, with typical errors including stuck objects, floating in midair, wrong shapes, implausible trajectories, and hallucinated new objects. After only **5k** simulated samples, PSFT improves Open-Sora on both real and synthetic evaluation; on real data, **Trajectory $L_2$** drops from approximately **0.15** to **0.076**, **Chamfer** from approximately **0.45** to **0.188**, and **IoU** rises from approximately **0.06** to **0.14** [2503.09595]. ORO further improves different aspects depending on the reward. At the same time, generalization degrades outside the training ranges for height and depth, and the learned conditional drop-time distribution remains statistically misaligned, with **KS test $p<0.05$ in all 50 tests** [2503.09595].

## 6. PISA as a saliency benchmark and the problem of acronymic overload

An earlier and entirely unrelated usage is **“PISA: Pixelwise Image Saliency by Aggregating Complementary Appearance Contrast Measures with Edge-Preserving Coherence”**, where PISA denotes both a saliency algorithm and a benchmark suite for evaluating pixel-accurate saliency detection [1505.03227]. The benchmark combines several public datasets—**ASD (MSRA-1000)**, **SOD**, **SED1**, **ECSSD**, and **PASCAL-1500**—and introduces the **TCD** commodity-image dataset with **800** images [1505.03227].

Ground truth throughout is a binary pixel mask, produced by drawing object outlines and filling the interior. Because the method is unsupervised and bottom-up, there are **no predefined train/val/test splits**; all images are used for evaluation [1505.03227]. The principal metrics are **Precision** and **Recall** across thresholds, **adaptive $F$-measure** with threshold
$$
T=\frac{2}{WH}\sum_{x,y}S(x,y),
$$
and **MAE**, the mean absolute pixelwise deviation between normalized saliency map and ground-truth mask [1505.03227].

The algorithm underlying the benchmark aggregates two complementary appearance contrasts—**color contrast** in Lab-space histograms and **structure contrast** in orientation–magnitude histograms—within a **pixel-wise adaptive observation region**. These are modulated by spatial priors favoring image center and boundary exclusion, combined into an unnormalized saliency confidence
$$
\hat f(p)=U^c(p)D^c(p)+U^g(p)D^g(p),
$$
then discretized into **$L=24$** saliency levels and optimized via a multi-label MRF solved with **cost-volume filtering** rather than graph cuts or belief propagation [1505.03227]. A faster variant, **F-PISA**, uses gradient-driven subsampling and joint-bilateral upsampling.

Quantitatively, the benchmark reports strong performance across six datasets. On **ECSSD**, PISA leads with **PR-area ≈ 0.95**, **$F_{0.3} ≈ 0.88$**, and **MAE ≈ 0.05**; on **PASCAL-1500**, it attains **$F_{0.3} ≈ 0.55$** and **MAE ≈ 0.20**; on **TCD**, it achieves the highest PR curve, **$F_{0.3} ≈ 0.83$**, and **MAE ≈ 0.10** [1505.03227]. Failure cases are tied to violations of the center and boundary priors, such as salient objects located at the border or background regions occupying the image center [1505.03227].

Taken together, these usages show that “PISA benchmark” is not a single standardized artifact. In the cited literature it may refer to an educational assessment framework, a multilingual text benchmark, a multilingual multimodal VLM benchmark, a 3D point-language benchmark, a physics benchmark for video diffusion, or a saliency benchmark. A plausible implication is that any technical discussion of “PISA” should state the expansion of the acronym and the target modality explicitly before reporting results.

Source: https://www.emergentmind.com/topics/pisa-benchmark