ConservationBench: Video Invariance Benchmark
- ConservationBench is a video-based benchmark that evaluates whether vision-language models can determine if a physical quantity remains unchanged during dynamic transformations.
- It systematically tests four properties—number, length, volume, and size—using 384 videos and 23,040 evaluation trials with paired conserving and non-conserving scenarios.
- Empirical results reveal that most models perform near-chance accuracy with a pronounced bias from language priors, exposing a significant visual grounding failure in dynamic settings.
Searching arXiv for the benchmark and related conservation-law papers to ground the article. ConservationBench is a video-based benchmark for evaluating whether vision-LLMs (VLMs) can reason about physical transformation invariance, defined as whether a physical quantity remains unchanged even when its appearance changes over time. It was introduced to test a capability that existing VLM benchmarks mostly do not isolate: integrating sequential visual evidence across a transformation and deciding whether a property satisfies . The benchmark spans four quantitative properties—number, length, volume, and size—with paired conserving and non-conserving scenarios, yielding 384 total videos, 23,040 total evaluation trials, and an evaluation of 112 VLMs (Luo et al., 7 Mar 2026). In a broader methodological sense, its emphasis on conservation aligns with a wider literature in which conservation is treated as a structural invariant whose violation materially degrades inference, prediction, or physical fidelity (Hansen et al., 2023).
1. Conceptual scope
ConservationBench formalizes conservation in the Piagetian sense: a quantity remains invariant under transformation despite changes in appearance. The benchmark asks whether a model can determine, from a sequence of images or frames, whether a target property is conserved across a dynamic scene. Its core target is not static recognition, object counting in still images, or generic multimodal question answering, but transformation-invariant representation under sequential visual change (Luo et al., 7 Mar 2026).
This scope differentiates the benchmark from prior VLM evaluation along several axes. It uses videos / frame sequences rather than single images; it pairs each conservation case with a matched non-conserving control; it explicitly varies prompting, frame count, and frame extraction strategy; and it treats conservation as a reasoning problem over continuous transformation rather than as outcome prediction or descriptive inference. The benchmark therefore isolates whether models can bind a verbal notion of invariance to temporally extended visual evidence rather than merely exploit static cues or language priors (Luo et al., 7 Mar 2026).
The four benchmarked properties are divided into two classes. Transformation-helpful tasks are number and length, where the answer can often be inferred from initial and final states. Transformation-mandatory tasks are volume and size, where the transformation process itself must be understood because final appearance alone is insufficient. This division is central to the benchmark’s design because it separates cases in which sequential evidence is useful from cases in which it is indispensable (Luo et al., 7 Mar 2026).
2. Dataset construction and paired scenario design
The benchmark contains 192 conservation videos and 192 matched non-conserving control videos, for 384 total videos. The control videos are explicitly described as counterfactuals: they preserve the same general environment, objects, and irrelevant visual context while changing the target quantity during the transformation. This paired design is meant to prevent success by a degenerate “always conserved” strategy and to force discrimination between invariance and violation under matched conditions (Luo et al., 7 Mar 2026).
The benchmarked properties and paired scenarios are organized as follows:
| Property | Conserving scenario | Matched non-conserving control |
|---|---|---|
| Number | Rows of identical coins are spread apart, no coins added/removed | One coin is added while spreading |
| Length | A straw is repositioned without changing length | A straw is altered in actual length |
| Volume | Liquid is poured into a different-shaped container, volume constant | Some liquid is left behind |
| Size | Playdough reshaped, same mass/material | Part of the playdough is left in the hand |
To reduce shortcut learning, the benchmark counterbalances task-irrelevant features such as object type, mapping shift, distance spread, direction, number of objects, color, container shape, and transformation action. This makes the conserving/non-conserving distinction depend on the target physical quantity rather than on nuisance correlations (Luo et al., 7 Mar 2026).
A human study was conducted on a representative subset of 864 questions, with humans achieving 98.35% accuracy. This result is used to establish that the benchmark is cognitively meaningful and straightforward for humans even though it is difficult for current VLMs (Luo et al., 7 Mar 2026).
3. Task format and evaluation protocol
Each benchmark instance is posed as a three-way multiple-choice question over frames extracted from the underlying video. An example given for number conservation is: “Is the number of coins in the upper row the same as in the lower row in the final image?” with answer options distinguishing lower-row excess, upper-row excess, or equality. Because the format is three-way multiple choice, the benchmark’s nominal chance level is (Luo et al., 7 Mar 2026).
The full evaluation grid comes from a factorial design over prompting, temporal resolution, and frame extraction. The benchmark tests 4 prompt types—Direct Question, Sequential, CoT, and Continuous—together with 5 frame counts—3-frame, 5-frame, 7-frame, 9-frame, and 16-frame—and 3 extraction methods—Uniform sampling, Human-based selection, and Model-based selection via SEVILA / BLIP-2 localizer. Multiplying 384 videos by these 60 conditions yields 23,040 total evaluation trials (Luo et al., 7 Mar 2026).
The evaluation protocol also defines a stricter paired criterion. Under strict pairwise scoring, a model is counted as correct only if it answers both the conservation item and its matched non-conserving control correctly. The paper identifies the corresponding pairwise chance level as . This strict metric is more diagnostic than ordinary accuracy because it suppresses gains obtained by a generic invariance bias (Luo et al., 7 Mar 2026).
For statistical analysis, the benchmark uses repeated-measures ANOVA with model as the unit of analysis, accuracies averaged across nuisance factors, and Bonferroni-corrected pairwise comparisons. This makes the prompt, frame-count, and sampling analyses explicitly comparative rather than anecdotal (Luo et al., 7 Mar 2026).
4. Empirical results across 112 vision-LLMs
Across 112 VLMs, reported accuracies range from about 20% to 69%, with most models only slightly above chance = 33.3%. This produces a large gap relative to the human result of 98.35%. The best average accuracies reported are approximately gemini-2.5-pro: 69.11%, doubao-seed-1.6-vision: 64.71%, gpt-5: 63.65%, and claude-sonnet-4-5: 60.41%; most other models cluster in the mid-50s or lower (Luo et al., 7 Mar 2026).
A central quantitative finding is the negative correlation between performance on conserving tasks and matched non-conserving controls, reported as
This indicates that models improving on conservation items often deteriorate on controls, which the paper interprets as evidence of a bias toward answering that the quantity is conserved. The pattern holds at the domain level as well: in number, length, size, and volume, models generally perform better on conservation cases than on matched non-conserving controls (Luo et al., 7 Mar 2026).
The strict paired evaluation is still more severe. Under this criterion, 82/112 models (73.2%) score below 10% strict accuracy, and only three models exceed chance: gemini-2.5-pro, doubao-seed-1.6-vision, and claude-sonnet-4-5. This result is presented as one of the strongest indications that most current models do not reliably distinguish invariance from change under matched visual conditions (Luo et al., 7 Mar 2026).
Scaling yields little relief. Model size shows almost no relationship with conservation accuracy, with
and only a modest relationship with non-conserving control accuracy,
Within the benchmark’s reported results, conservation reasoning therefore does not emerge simply from increasing parameter count (Luo et al., 7 Mar 2026).
5. Failure modes and diagnostic analyses
The benchmark includes several controls intended to determine whether failure arises from language priors, visual grounding, temporal resolution, or prompting. One striking result is the presence of a strong textual prior favoring invariance. In Empty Image Control, where images are replaced with white or content-free images, models often still answer “Conserve”; for example, 85.7% of cases with a true conservation answer of “Conserve” remain “Conserve.” In Text Control, where no visual input is given, a similar but weaker pattern appears, with about 73.7% maintaining “Conserve” in conservation settings (Luo et al., 7 Mar 2026).
The diagnostic implication is not that models understand the visual transformations, but almost the reverse. The paper reports that models can perform better with empty images than with real visual input, suggesting that the correct response in many cases is driven more by a language prior than by grounded visual inference. Real visual content often disrupts rather than stabilizes that heuristic, which the paper interprets as a visual grounding failure in dynamic scenes (Luo et al., 7 Mar 2026).
Prompt engineering does not resolve the problem. For number and length, prompt type matters statistically,
with CoT performing significantly worse than Continuous, Sequential, and Direct. For volume and size, prompt type is not significant,
Thus, explicit step-by-step prompting does not reliably induce the missing transformation reasoning and can even degrade it (Luo et al., 7 Mar 2026).
Increasing temporal resolution also fails to produce a reliable gain. For number and length, frame count shows no effect,
For volume and size, there is a nominal effect,
0
but only one pairwise difference is significant and the pattern is not monotonic. Frame extraction strategy is similarly limited: for volume and size,
1
with Uniform sampling outperforming both human-selected and model-selected frames. The paper’s interpretation is that curated frame selection may emphasize misleading static cues rather than support the tracking of transformation (Luo et al., 7 Mar 2026).
6. Position within conservation-centered benchmarking
ConservationBench belongs to a broader research pattern in which conservation is used as a benchmark criterion rather than treated only as a modeling preference. In scientific machine learning, ProbConserv argues that for conservation laws it is insufficient merely to “inform” a model through a residual penalty; the model must respect an integral quantity exactly or nearly exactly because conservation is a structural physical invariant that strongly affects downstream predictions, especially for discontinuous or shock-like solutions (Hansen et al., 2023). In numerical wave scattering, a flux-conservation law is proposed as a benchmark for any theoretical or numerical solution of SH-wave response in a sedimentary basin, with the explicit warning that significant violation makes a proposed solution suspect, even though the condition is necessary rather than sufficient (Wirgin, 2019).
A similar benchmark logic appears in kinetic theory and neural-operator research. The LightningBoltz work presents a multidisciplinary benchmark suite for a conservative spectral solver of the nonlinear Boltzmann equation, emphasizing exact discrete conservation of mass, momentum, and energy while comparing against analytic predictions and established solvers across plasma, gas, and atomic-plasma settings (Wilkie et al., 2022). The Exterior-Embedded Conservation Framework (ECF) evaluates conservation-law-constrained PDE benchmarks by enforcing the conserved Fourier mode exactly and reports that the correction drives conservation error to around 2 to 3 while often improving RMSE (Dong et al., 20 Nov 2025).
This suggests a common design principle across otherwise unrelated areas: conservation-based benchmarks tend to expose failures that are not visible under ordinary accuracy metrics alone. In the VLM setting, the exposed weakness is the inability to maintain transformation-invariant representations across dynamic scenes (Luo et al., 7 Mar 2026). In PDE learning, spectral solvers, and conservative numerical discretization, the exposed weakness is drift in physically constrained modes or violation of exact balance laws (Hansen et al., 2023, Wilkie et al., 2022, Dong et al., 20 Nov 2025). The shared benchmark function is therefore diagnostic rather than merely comparative.
7. Significance and limitations
The principal significance of ConservationBench is that it isolates a narrow but foundational capability: deciding whether a physically meaningful quantity remains invariant under transformation when the model must integrate sequential visual evidence. The benchmark shows that current VLMs often possess the right verbal prior—that quantities frequently should remain unchanged—but fail to bind that prior to visual evidence over time. The empirical signature is systematic: near-chance overall performance, pronounced asymmetry between conservation and matched non-conservation controls, and extremely weak strict paired accuracy for most models (Luo et al., 7 Mar 2026).
The benchmark also establishes several negative results. Higher temporal resolution does not reliably improve performance; better prompting does not repair the failure; curated or model-based frame selection does not help and can hurt; and model scale shows almost no relationship with conservation accuracy. These results narrow the likely explanation away from superficial prompt or sampling issues and toward a more structural deficiency in dynamic multimodal representation (Luo et al., 7 Mar 2026).
At the same time, the benchmark’s scope is specific. It evaluates four properties—number, length, volume, and size—under matched conserving and non-conserving transformations. A plausible implication is that it functions less as a general physical-reasoning benchmark than as a controlled probe of one specific competence: grounded invariance reasoning in dynamic scenes. Within that scope, its conclusion is unambiguous: current VLMs do not reliably track whether physical quantities persist—or fail to persist—through transformation, and therefore do not yet provide robust transformation-invariant physical reasoning in the sense operationalized by ConservationBench (Luo et al., 7 Mar 2026).