PISA Benchmark: Multimodal AI Evaluation

• PISA benchmark is a collection of evaluation frameworks based on OECD assessments, addressing educational and AI testing across different modalities.
• It includes variants like mmPISA-bench for multilingual reasoning, PISA-Bench for multimodal vision-language tasks, PiSA-Bench for 3D model evaluation, and benchmarks for video and saliency.
• The frameworks employ rigorous protocols to analyze translation quality, sample selection bias, differential item functioning, and physics-aware video generation accuracy.

Searching arXiv for the benchmark papers to ground the article and verify the cited records. to=arxiv_search.search 大发时时彩是json code blocks ਨਹੀਂ? to=arxiv_search.search 平台直属json code 大发分分彩 {"query":"PISA-Bench multilingual multimodal metric evaluation vision-LLMs arXiv (Haller et al., 27 Oct 2025)", "max_results": 5} to=arxiv_search.search 大发时时彩开奖ിയjson code to=arxiv_search.search {"query":"PISA-Bench multilingual multimodal metric evaluation vision-LLMs", "max_results": 5} The label PISA benchmark appears in several distinct forms in the cited literature. At its source is the OECD Programme for International Student Assessment (PISA), a triennial survey of 15-year-old students’ competencies in reading, mathematics, and science in over 80 countries; around that source, multiple AI evaluation suites have been constructed, including mmPISA-bench for multilingual LLM reasoning, PISA-Bench for multilingual multimodal VLM evaluation, and other acronymically related benchmarks such as PiSA-Bench for 3D MLLMs, PisaBench for physical accuracy in video generation, and PISA for saliency detection (Sapenov et al., 5 Jun 2026, Haller et al., 27 Oct 2025, Guo et al., 13 Mar 2025, Li et al., 12 Mar 2025, Wang et al., 2015). In the cited literature, the term therefore denotes not one canonical benchmark but a cluster of evaluation frameworks that share a concern with controlled assessment, annotation quality, and fine-grained diagnosis.

1. OECD PISA as the reference framework

The OECD’s Programme for International Student Assessment is a triennial survey that measures 15-year-old students’ competencies in reading, mathematics, and science in over 80 countries. Its items are subjected to rigorous localization, translation, and validation procedures so that difficulty and meaning remain consistent across languages and cultures (Sapenov et al., 5 Jun 2026). This institutional framework is the direct source for both educational-statistical analyses and later AI benchmarks derived from public PISA items.

Two statistical problems recur in research using PISA as a benchmark of national performance. The first is sample selection bias. Because PISA samples students still in school at age 15, cross-country comparisons can be distorted by survival bias. A quantile-selection model formalizes this by introducing a latent score $Y^{*}$, an enrollment indicator $S \in \{0,1\}$, and an observed score $Y$ equal to $Y^{*}$ only when $S=1$; under stochastic dominance, the latent quantile function is bounded by observed quantiles, and under a parametric beta-cost selection rule it becomes point identified (Boussim, 2023). In the PISA 2018 application, correcting for selection bias shifted several country rankings, including Canada from official rank 12 to corrected rank 19 in mathematics, Slovenia from rank 14 to rank 8, Germany from 20 to 12, and the U.K. from 19 to 28 (Boussim, 2023).

The second problem is measurement non-invariance, operationalized as Differential Item Functioning (DIF) in a multigroup IRT model. In the PISA 2022 analysis, student proficiencies are modeled as country-specific Gaussian latent traits, while item responses use a logistic form with common discrimination $a_j$, shared intercept $d_j$, and country-item DIF effects $\gamma_{jk}$. A two-step estimator is proposed: a baseline MML fit, followed by an $\ell_1$-projection that enforces an identifiability constraint without requiring reference groups or anchor items (Ouyang et al., 22 May 2025). On PISA 2022 mathematics, the analysis used $N \approx 277{,}000$ students in 37 OECD countries on $S \in \{0,1\}$0 binary items; about 12% of country-item pairs had $S \in \{0,1\}$1, and seven countries changed rank by at least two positions relative to the official RMSD-anchoring procedure (Ouyang et al., 22 May 2025).

Benchmark name	Domain	Core unit
OECD PISA	Educational assessment	Student test items in reading, mathematics, and science
mmPISA-bench	Multilingual LLM reasoning	25 MCQs in 43 languages with two translation types
PISA-Bench	Multilingual multimodal VLM evaluation	122 image-text MCQs in 6 languages
PiSA-Bench	3D MLLM evaluation	240 point-cloud → caption pairs across 6 aspects
PisaBench–Real / PisaBench–Sim	Physics-aware video generation	Real slow-motion videos and synthetic Kubric videos
PISA	Saliency detection	Pixel-level saliency maps on multiple image datasets

This coexistence of educational, multimodal, 3D, generative-video, and saliency meanings suggests that acronym disambiguation is necessary whenever “PISA benchmark” is cited.

2. mmPISA-bench: multilingual reasoning with official and machine translations

mmPISA-bench is a compact multilingual reasoning benchmark derived from public OECD PISA items and built to test whether LLMs “reason equally well” across languages (Sapenov et al., 5 Jun 2026). It uses 25 hand-picked text-only multiple-choice questions: 11 from PISA 2022 math and 14 from PISA 2018 reading. The selection criteria were that an item be exclusively text-based, in MCQ format, and available in a broad set of official PISA translations (Sapenov et al., 5 Jun 2026).

Each question is provided in 43 officially human-translated languages and paired with a machine-translated counterpart, yielding a total of 2,150 data points:

$S \in \{0,1\}$2

For every question-language pair, the benchmark stores the context and stem, four answer options, and the correct key for both translation types (Sapenov et al., 5 Jun 2026).

The evaluation protocol is zero-shot. Two proprietary LLM families are tested under multiple reasoning-effort settings, including no reasoning, low/medium/high chain-of-thought, and a “double-prompt” repetition in the no-reasoning case. Accuracy is defined conventionally as

$S \in \{0,1\}$3

The benchmark additionally records per-language variance, human-versus-machine translation differences, reasoning length in output tokens, and a tokenization “premium,” defined as the ratio of input tokens in a non-English language to English (Sapenov et al., 5 Jun 2026).

The reported results show high cross-lingual robustness but nontrivial cost heterogeneity. Claude achieved 92.3% average accuracy at a total cost of \$S \in \{0,1\}$481.5. Under high chain-of-thought, Claude reached 96.6% and GPT 95.7%; under no reasoning, they scored 87.5% and 78.9%, respectively (Sapenov et al., 5 Jun 2026). Machine-translated questions yielded equal or slightly higher accuracy than official human translations: for Claude at high effort, 96.6% on human versus 97.4% on machine; for GPT, 95.7% versus 96.1% (Sapenov et al., 5 Jun 2026).

A common concern is that synthetic translation necessarily degrades multilingual reasoning evaluation. On this benchmark, that concern is not supported: token usage differences between translation types are reported as negligible, below 5% for both input and output, and machine-translated items did not reduce accuracy (Sapenov et al., 5 Jun 2026). At the same time, tokenization artifacts mattered: languages with higher tokenization premiums, such as Thai and Greek, were simultaneously more expensive and less accurate, and accuracy–cost correlation was negative for both model families (Sapenov et al., 5 Jun 2026).

3. PISA-Bench for multilingual and multimodal VLM evaluation

PISA-Bench extends the PISA-derived paradigm from text-only reasoning to vision-LLMs. It is built from publicly available OECD PISA tests from 2012 and earlier, filtered for items that contain both text and an associated image or figure, are self-contained, and do not depend on preceding tasks (Haller et al., 27 Oct 2025). After manual filtering for clarity, completeness, and multimodality, 122 English examples remained.

Each example was decomposed into four fields: Instruction, Image, Question, and Answer Options. When original multiple-choice options were missing, they were generated via GPT-4o and rephrased into a four-option MCQ format. Two independent annotators checked every English item against five criteria, including faithfulness to original intent, absence of answer leakage, plausibility of distractors, and grammatical correctness. The result was exactly 122 high-quality, human-verified English items (Haller et al., 27 Oct 2025).

The corpus was translated in parallel into Spanish, German, Chinese, French, and Italian, producing a fully parallel benchmark in six languages. Translation prompts preserved all units, symbols, numbers, and abbreviations verbatim and forbade any reordering of answer options, so the correct answer remained correct. Automatic validation used WMT23 COMET-KIWI and GEMBA-MQM, while native speakers reviewed 50 random items per language; the human verification found no critical errors, and error-free rates ranged from 76–88% (Haller et al., 27 Oct 2025).

Items are assigned to four question-type categories: Spatial & Geometric Reasoning, Quantitative Reasoning, Graph & Pattern Analysis, and Text & Diagram Understanding. Evaluation is zero-shot and free-form: models receive the instruction, question text, and image, and generate an answer string. GPT-4 acts as judge via semantic matching, and the principal metric is accuracy over the 122 examples:

$S \in \{0,1\}$5

Per-category accuracy and a contamination test comparing text-only with text-plus-image performance are also reported (Haller et al., 27 Oct 2025).

The results show a marked dependence on parameter scale and language. Small models below 20B parameters averaged below 55% accuracy on English, while larger open-source models above 20B reached up to approximately 67%. Proprietary GPT-4o achieved 71.0% on English (Haller et al., 27 Oct 2025). Multilingual degradation was systematic: 10/12 multilingual models dropped by –1.4 to –8.4 percentage points outside English. Spatial and geometric reasoning remained a consistent failure mode; for GPT-4o, error rates in this category stayed relatively high at approximately 40% (Haller et al., 27 Oct 2025).

A possible misconception is that such benchmark accuracy could largely reflect text memorization. The contamination test argues against that interpretation: for example, Qwen2.5-VL-7B-Instruct dropped from 61.1% with images to 33.6% without images, and Qwen2-VL-7B-Instruct dropped from 54.0% to 35.4% (Haller et al., 27 Oct 2025). This suggests that image-conditioned reasoning is materially contributing to performance.

4. PiSA-Bench for 3D multimodal LLMs

PiSA-Bench belongs to a different research line and acronym expansion. It was introduced together with PiSA-Engine (Point-Self-Augmented-Engine) in work on 3D understanding with large models (Guo et al., 13 Mar 2025). The benchmark responds to limitations in earlier 3D benchmarks such as Objaverse, Cap3D, and ScanObjectNN, which are described as using short, coarse captions and limited object categories, thereby making it difficult to localize which components of 3D MLLM behavior succeed or fail (Guo et al., 13 Mar 2025).

The benchmark has three stated design goals: to provide multi-aspect, richly detailed captions that capture true 3D spatial semantics; to cover a broad but controlled set of 40 held-out objects spanning common household, industrial, and natural categories; and to supply classification labels with synonyms and subclass groupings so that open-vocabulary evaluation does not unfairly penalize near-synonyms such as “mug” and “cup” (Guo et al., 13 Mar 2025).

Each benchmark item is annotated along six orthogonal dimensions: Description, Color, Shape (Geometry), Count, Spatial Relations, and Usage (Function). The dataset contains 240 point-cloud → caption pairs, corresponding to 6 aspects × 40 objects. The objects were selected from Objaverse but held out of all training, with the paper noting 3,000 reserved objects, of which 40 form PiSA-Bench. For each object, 3–5 synonyms or subclass names are generated via GPT-4 and human-verified (Guo et al., 13 Mar 2025).

PiSA-Bench supports two tasks. In zero-shot 3D object captioning, models generate a free-form caption from the native point cloud. Evaluation combines automated metrics—BLEU-1, ROUGE-L, METEOR, CIDEr—with a GPT-4o multi-aspect scoring protocol that awards 0–100 on each of the six aspects and averages them, plus a human evaluation in which each distinct correct attribute receives one point and near-matches receive 0.25, 0.5, or 0.75 credit (Guo et al., 13 Mar 2025). In generative 3D object classification, a natural-language answer is judged correct if any synonym or subclass in the label set matches, with Top-1 Accuracy and Top-K Accuracy defined in the usual way (Guo et al., 13 Mar 2025).

Reported zero-shot results on PiSA-Bench show iterative gains for the PiSA training framework. In generative classification, PointLLM-7B scored 47.50%, PointLLM-PiSA-7B scored 61.25%, and PointLLM-PiSA³-7B reached 63.75%. In 3D object captioning using GPT-4o average over six aspects, the baseline scored 38.12%, PointLLM-PiSA-7B scored 43.70%, PointLLM-PiSA²-7B scored 46.41%, and PointLLM-PiSA³-7B scored 46.45% (Guo et al., 13 Mar 2025).

The benchmark’s evaluation recipe is explicit: researchers should generate captions and classification answers for the 40 point clouds, use the provided GPT-4o prompts for scoring, optionally conduct human evaluation with the shuffled-group protocol, and ensure that PiSA-Bench objects are never seen during training. Missing spatial relations or usage information is described as heavily penalized (Guo et al., 13 Mar 2025).

5. PisaBench for physical accuracy in video generation

A third meaning appears in “PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop”, where PISA stands for Physics-Informed Simulation and Alignment and the associated benchmark is a diagnostic suite for free-fall dynamics in video generation (Li et al., 12 Mar 2025). The target phenomenon is object free-fall under gravity, optionally with simple collision against a pile of objects.

The benchmark has two components. PisaBench–Real contains 361 slow-motion videos recorded at 120 fps with cell phones mounted on tripods and cropped to 1:1. Scenes include indoor and outdoor environments, with 1–6 objects suspended by an invisible wire and released at frame 1; each clip is annotated with a caption of the form “<object description> falls.” and with segmentation masks from SAM-2 (Li et al., 12 Mar 2025). PisaBench–Sim is a synthetic dataset rendered in Kubric with PyBullet physics @240 Hz and Blender rendering. It uses standard Earth gravity, a physics step of $S \in \{0,1\}$6, 2-second videos rendered as 32 frames at 16 fps, and Google Scanned Objects split into 930 train objects and 103 test objects (Li et al., 12 Mar 2025).

The evaluation metrics are geometric and motion-centric. They include Trajectory $S \in \{0,1\}$7, defined as mean 2D centroid error across frames; Chamfer Distance between generated and ground-truth foreground-pixel sets; Intersection-over-Union over binary masks; and, on simulated data only, drop-time error, where the physical reference time is

$S \in \{0,1\}$8

These metrics explicitly target motion fidelity, shape fidelity, and object permanence rather than texture or semantics (Li et al., 12 Mar 2025).

The experimental protocol first performs zero-shot evaluation of eight off-the-shelf models, then studies post-training on Open-Sora v1.2. Supervised Fine-Tuning (PSFT) uses the same rectified-flow objective at batch size 128, learning rate 1e−4, for 5,000 steps on 5,000 simulated videos. Object Reward Optimization (ORO) then optimizes

$S \in \{0,1\}$9

with rewards based on segmentation IoU, optical flow from RAFT, and depth from DepthAnyV2 (Li et al., 12 Mar 2025).

The main findings are diagnostic. All zero-shot models are reported to fail at realistic free-fall, with typical errors including stuck objects, floating in midair, wrong shapes, implausible trajectories, and hallucinated new objects. After only 5k simulated samples, PSFT improves Open-Sora on both real and synthetic evaluation; on real data, Trajectory $Y$0 drops from approximately 0.15 to 0.076, Chamfer from approximately 0.45 to 0.188, and IoU rises from approximately 0.06 to 0.14 (Li et al., 12 Mar 2025). ORO further improves different aspects depending on the reward. At the same time, generalization degrades outside the training ranges for height and depth, and the learned conditional drop-time distribution remains statistically misaligned, with KS test $Y$1 in all 50 tests (Li et al., 12 Mar 2025).

6. PISA as a saliency benchmark and the problem of acronymic overload

An earlier and entirely unrelated usage is “PISA: Pixelwise Image Saliency by Aggregating Complementary Appearance Contrast Measures with Edge-Preserving Coherence”, where PISA denotes both a saliency algorithm and a benchmark suite for evaluating pixel-accurate saliency detection (Wang et al., 2015). The benchmark combines several public datasets—ASD (MSRA-1000), SOD, SED1, ECSSD, and PASCAL-1500—and introduces the TCD commodity-image dataset with 800 images (Wang et al., 2015).

Ground truth throughout is a binary pixel mask, produced by drawing object outlines and filling the interior. Because the method is unsupervised and bottom-up, there are no predefined train/val/test splits; all images are used for evaluation (Wang et al., 2015). The principal metrics are Precision and Recall across thresholds, adaptive $Y$2-measure with threshold

$Y$3

and MAE, the mean absolute pixelwise deviation between normalized saliency map and ground-truth mask (Wang et al., 2015).

The algorithm underlying the benchmark aggregates two complementary appearance contrasts—color contrast in Lab-space histograms and structure contrast in orientation–magnitude histograms—within a pixel-wise adaptive observation region. These are modulated by spatial priors favoring image center and boundary exclusion, combined into an unnormalized saliency confidence

$Y$4

then discretized into $Y$5 saliency levels and optimized via a multi-label MRF solved with cost-volume filtering rather than graph cuts or belief propagation (Wang et al., 2015). A faster variant, F-PISA, uses gradient-driven subsampling and joint-bilateral upsampling.

Quantitatively, the benchmark reports strong performance across six datasets. On ECSSD, PISA leads with PR-area ≈ 0.95, $Y$6, and MAE ≈ 0.05; on PASCAL-1500, it attains $Y$7 and MAE ≈ 0.20; on TCD, it achieves the highest PR curve, $Y$8, and MAE ≈ 0.10 (Wang et al., 2015). Failure cases are tied to violations of the center and boundary priors, such as salient objects located at the border or background regions occupying the image center (Wang et al., 2015).

Taken together, these usages show that “PISA benchmark” is not a single standardized artifact. In the cited literature it may refer to an educational assessment framework, a multilingual text benchmark, a multilingual multimodal VLM benchmark, a 3D point-language benchmark, a physics benchmark for video diffusion, or a saliency benchmark. A plausible implication is that any technical discussion of “PISA” should state the expansion of the acronym and the target modality explicitly before reporting results.