---
title: 'ConservationBench: Video Invariance Benchmark'
url: https://www.emergentmind.com/topics/conservationbench
type: topic
---

# ConservationBench: Video Invariance Benchmark

Searching arXiv for the benchmark and related conservation-law papers to ground the article.
ConservationBench is a video-based benchmark for evaluating whether vision-language models (VLMs) can reason about **physical transformation invariance**, defined as whether a physical quantity remains unchanged even when its appearance changes over time. It was introduced to test a capability that existing VLM benchmarks mostly do not isolate: integrating sequential visual evidence across a transformation and deciding whether a property \(q\) satisfies \(q(\text{initial}) = q(\text{final})\). The benchmark spans four quantitative properties—**number, length, volume, and size**—with paired conserving and non-conserving scenarios, yielding **384 total videos**, **23,040 total evaluation trials**, and an evaluation of **112 VLMs** [2603.07109]. In a broader methodological sense, its emphasis on conservation aligns with a wider literature in which conservation is treated as a structural invariant whose violation materially degrades inference, prediction, or physical fidelity [2302.11002].

## 1. Conceptual scope

ConservationBench formalizes conservation in the **Piagetian** sense: a quantity remains invariant under transformation despite changes in appearance. The benchmark asks whether a model can determine, from a sequence of images or frames, whether a target property is conserved across a dynamic scene. Its core target is not static recognition, object counting in still images, or generic multimodal question answering, but **transformation-invariant representation** under sequential visual change [2603.07109].

This scope differentiates the benchmark from prior VLM evaluation along several axes. It uses **videos / frame sequences** rather than single images; it pairs each conservation case with a **matched non-conserving control**; it explicitly varies prompting, frame count, and frame extraction strategy; and it treats conservation as a reasoning problem over continuous transformation rather than as outcome prediction or descriptive inference. The benchmark therefore isolates whether models can bind a verbal notion of invariance to temporally extended visual evidence rather than merely exploit static cues or language priors [2603.07109].

The four benchmarked properties are divided into two classes. **Transformation-helpful** tasks are **number** and **length**, where the answer can often be inferred from initial and final states. **Transformation-mandatory** tasks are **volume** and **size**, where the transformation process itself must be understood because final appearance alone is insufficient. This division is central to the benchmark’s design because it separates cases in which sequential evidence is useful from cases in which it is indispensable [2603.07109].

## 2. Dataset construction and paired scenario design

The benchmark contains **192 conservation videos** and **192 matched non-conserving control videos**, for **384 total videos**. The control videos are explicitly described as **counterfactuals**: they preserve the same general environment, objects, and irrelevant visual context while changing the target quantity during the transformation. This paired design is meant to prevent success by a degenerate “always conserved” strategy and to force discrimination between invariance and violation under matched conditions [2603.07109].

The benchmarked properties and paired scenarios are organized as follows:

| Property | Conserving scenario | Matched non-conserving control |
|---|---|---|
| Number | Rows of identical coins are spread apart, no coins added/removed | One coin is added while spreading |
| Length | A straw is repositioned without changing length | A straw is altered in actual length |
| Volume | Liquid is poured into a different-shaped container, volume constant | Some liquid is left behind |
| Size | Playdough reshaped, same mass/material | Part of the playdough is left in the hand |

To reduce shortcut learning, the benchmark counterbalances task-irrelevant features such as **object type**, **mapping shift**, **distance spread**, **direction**, **number of objects**, **color**, **container shape**, and **transformation action**. This makes the conserving/non-conserving distinction depend on the target physical quantity rather than on nuisance correlations [2603.07109].

A human study was conducted on a representative subset of **864 questions**, with humans achieving **98.35% accuracy**. This result is used to establish that the benchmark is cognitively meaningful and straightforward for humans even though it is difficult for current VLMs [2603.07109].

## 3. Task format and evaluation protocol

Each benchmark instance is posed as a **three-way multiple-choice question** over frames extracted from the underlying video. An example given for number conservation is: *“Is the number of coins in the upper row the same as in the lower row in the final image?”* with answer options distinguishing lower-row excess, upper-row excess, or equality. Because the format is three-way multiple choice, the benchmark’s nominal chance level is **\(33.3\%\)** [2603.07109].

The full evaluation grid comes from a factorial design over prompting, temporal resolution, and frame extraction. The benchmark tests **4 prompt types**—**Direct Question**, **Sequential**, **CoT**, and **Continuous**—together with **5 frame counts**—**3-frame**, **5-frame**, **7-frame**, **9-frame**, and **16-frame**—and **3 extraction methods**—**Uniform sampling**, **Human-based selection**, and **Model-based selection** via **SEVILA / BLIP-2 localizer**. Multiplying **384 videos** by these **60 conditions** yields **23,040 total evaluation trials** [2603.07109].

The evaluation protocol also defines a stricter paired criterion. Under **strict pairwise scoring**, a model is counted as correct only if it answers **both** the conservation item and its matched non-conserving control correctly. The paper identifies the corresponding pairwise chance level as **\(11.1\%\)**. This strict metric is more diagnostic than ordinary accuracy because it suppresses gains obtained by a generic invariance bias [2603.07109].

For statistical analysis, the benchmark uses **repeated-measures ANOVA** with model as the unit of analysis, accuracies averaged across nuisance factors, and **Bonferroni-corrected pairwise comparisons**. This makes the prompt, frame-count, and sampling analyses explicitly comparative rather than anecdotal [2603.07109].

## 4. Empirical results across 112 vision-language models

Across **112 VLMs**, reported accuracies range from about **20% to 69%**, with most models only slightly above **chance = 33.3%**. This produces a large gap relative to the human result of **98.35%**. The best average accuracies reported are approximately **gemini-2.5-pro: 69.11%**, **doubao-seed-1.6-vision: 64.71%**, **gpt-5: 63.65%**, and **claude-sonnet-4-5: 60.41%**; most other models cluster in the mid-50s or lower [2603.07109].

A central quantitative finding is the **negative correlation** between performance on conserving tasks and matched non-conserving controls, reported as
\[
r = -0.510,\quad n = 112.
\]
This indicates that models improving on conservation items often deteriorate on controls, which the paper interprets as evidence of a bias toward answering that the quantity is conserved. The pattern holds at the domain level as well: in **number**, **length**, **size**, and **volume**, models generally perform better on conservation cases than on matched non-conserving controls [2603.07109].

The strict paired evaluation is still more severe. Under this criterion, **82/112 models (73.2%)** score **below 10% strict accuracy**, and only **three models** exceed chance: **gemini-2.5-pro**, **doubao-seed-1.6-vision**, and **claude-sonnet-4-5**. This result is presented as one of the strongest indications that most current models do not reliably distinguish invariance from change under matched visual conditions [2603.07109].

Scaling yields little relief. Model size shows almost **no relationship** with conservation accuracy, with
\[
R^2 = 0.019,
\]
and only a modest relationship with non-conserving control accuracy,
\[
R^2 = 0.239.
\]
Within the benchmark’s reported results, conservation reasoning therefore does not emerge simply from increasing parameter count [2603.07109].

## 5. Failure modes and diagnostic analyses

The benchmark includes several controls intended to determine whether failure arises from language priors, visual grounding, temporal resolution, or prompting. One striking result is the presence of a strong **textual prior favoring invariance**. In **Empty Image Control**, where images are replaced with white or content-free images, models often still answer “Conserve”; for example, **85.7%** of cases with a true conservation answer of “Conserve” remain “Conserve.” In **Text Control**, where no visual input is given, a similar but weaker pattern appears, with about **73.7%** maintaining “Conserve” in conservation settings [2603.07109].

The diagnostic implication is not that models understand the visual transformations, but almost the reverse. The paper reports that models can perform **better with empty images than with real visual input**, suggesting that the correct response in many cases is driven more by a language prior than by grounded visual inference. Real visual content often disrupts rather than stabilizes that heuristic, which the paper interprets as a **visual grounding failure** in dynamic scenes [2603.07109].

Prompt engineering does not resolve the problem. For **number and length**, prompt type matters statistically,
\[
F(3,333)=18.28,\ p<0.001,
\]
with **CoT** performing significantly worse than **Continuous**, **Sequential**, and **Direct**. For **volume and size**, prompt type is not significant,
\[
F(3,333)=2.00,\ p=0.114.
\]
Thus, explicit step-by-step prompting does not reliably induce the missing transformation reasoning and can even degrade it [2603.07109].

Increasing temporal resolution also fails to produce a reliable gain. For **number and length**, frame count shows no effect,
\[
F(4,444)=0.98,\ p=0.416.
\]
For **volume and size**, there is a nominal effect,
\[
F(4,444)=2.66,\ p=0.032,
\]
but only one pairwise difference is significant and the pattern is not monotonic. Frame extraction strategy is similarly limited: for **volume and size**,
\[
F(2,222)=8.75,\ p=0.0002,
\]
with **Uniform sampling** outperforming both human-selected and model-selected frames. The paper’s interpretation is that curated frame selection may emphasize misleading static cues rather than support the tracking of transformation [2603.07109].

## 6. Position within conservation-centered benchmarking

ConservationBench belongs to a broader research pattern in which **conservation** is used as a benchmark criterion rather than treated only as a modeling preference. In scientific machine learning, ProbConserv argues that for conservation laws it is insufficient merely to “inform” a model through a residual penalty; the model must respect an integral quantity exactly or nearly exactly because conservation is a structural physical invariant that strongly affects downstream predictions, especially for discontinuous or shock-like solutions [2302.11002]. In numerical wave scattering, a flux-conservation law is proposed as a benchmark for any theoretical or numerical solution of SH-wave response in a sedimentary basin, with the explicit warning that significant violation makes a proposed solution suspect, even though the condition is necessary rather than sufficient [1911.11425].

A similar benchmark logic appears in kinetic theory and neural-operator research. The LightningBoltz work presents a multidisciplinary benchmark suite for a conservative spectral solver of the nonlinear Boltzmann equation, emphasizing exact discrete conservation of **mass**, **momentum**, and **energy** while comparing against analytic predictions and established solvers across plasma, gas, and atomic-plasma settings [2208.05428]. The **Exterior-Embedded Conservation Framework (ECF)** evaluates conservation-law-constrained PDE benchmarks by enforcing the conserved Fourier mode exactly and reports that the correction drives conservation error to around \(10^{-6}\) to \(10^{-7}\) while often improving RMSE [2511.16573].

This suggests a common design principle across otherwise unrelated areas: conservation-based benchmarks tend to expose failures that are not visible under ordinary accuracy metrics alone. In the VLM setting, the exposed weakness is the inability to maintain **transformation-invariant representations** across dynamic scenes [2603.07109]. In PDE learning, spectral solvers, and conservative numerical discretization, the exposed weakness is drift in physically constrained modes or violation of exact balance laws [2302.11002; 2208.05428; 2511.16573]. The shared benchmark function is therefore diagnostic rather than merely comparative.

## 7. Significance and limitations

The principal significance of ConservationBench is that it isolates a narrow but foundational capability: deciding whether a physically meaningful quantity remains invariant under transformation when the model must integrate sequential visual evidence. The benchmark shows that current VLMs often possess the **right verbal prior**—that quantities frequently should remain unchanged—but fail to bind that prior to visual evidence over time. The empirical signature is systematic: near-chance overall performance, pronounced asymmetry between conservation and matched non-conservation controls, and extremely weak strict paired accuracy for most models [2603.07109].

The benchmark also establishes several negative results. Higher temporal resolution does not reliably improve performance; better prompting does not repair the failure; curated or model-based frame selection does not help and can hurt; and model scale shows almost no relationship with conservation accuracy. These results narrow the likely explanation away from superficial prompt or sampling issues and toward a more structural deficiency in dynamic multimodal representation [2603.07109].

At the same time, the benchmark’s scope is specific. It evaluates four properties—**number, length, volume, and size**—under matched conserving and non-conserving transformations. A plausible implication is that it functions less as a general physical-reasoning benchmark than as a controlled probe of one specific competence: grounded invariance reasoning in dynamic scenes. Within that scope, its conclusion is unambiguous: current VLMs do not reliably track whether physical quantities persist—or fail to persist—through transformation, and therefore do not yet provide robust transformation-invariant physical reasoning in the sense operationalized by ConservationBench [2603.07109].

Source: https://www.emergentmind.com/topics/conservationbench