OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models

Abstract: Vision-LLMs (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is N-th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude, from small numbers to extreme cases up to 300; (ii) arrangement complexity, from single loops to maze-like paths; and (iii) object count. The benchmark provides 39,000 question-answer pairs, each annotated with a ground-truth reasoning trajectory and balanced across difficulty levels for controlled large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured stepwise traces of the counting process and provides an open evaluation toolkit that measures both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench provides a reproducible benchmark and diagnostic framework for developing VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io/

• The paper introduces OrdinalBench, a synthetic benchmark with 2,600 images, 39,000 questions, multiple scene layouts, ordinal ranges, and step-level reasoning traces to test procedural visual counting.
• The paper finds that current VLMs struggle sharply with maze traversal, large ordinals, and skip counting, with the best 2D Single-Loop accuracy reaching only 32.83% and GPT-5 scoring 11.04% on 2D Maze-Loop tasks.
• The paper shows that trace-level metrics expose failures hidden by final-answer accuracy, motivating structured state representations, procedural training curricula, and schema-constrained outputs for more reliable multimodal reasoning.

Motivation and problem statement

Vision-LLMs (VLMs) achieve strong results on standard multimodal benchmarks, yet they continue to fail on elementary visual reasoning tasks such as object enumeration and spatial relations. The authors of OrdinalBench (2603.07786) argue that one such failure mode—ordinal number understanding, i.e., the ability to track relative positions under a traversal rule and generalize to large indices—is systematically under-evaluated. Existing benchmarks either target compositional reasoning (CLEVR-style tasks), cardinal counting (TallyQA, HowManyQA), or static spatial reasoning (CAPTURe, NumCLIP); none is designed to probe whether a model can execute a counting procedure step by step over a visual input. OrdinalBench fills this gap by framing ordinal understanding not as memorized ordinal vocabulary but as procedural reasoning: given an image, a textual traversal rule, a starting object counted as 1st, a target ordinal $N$, and an optional stride $k$, the model must identify the $N$-th object along the path.

The task has practical relevance for user-interface automation (clicking the $N$-th icon after menu navigation) and embodied robotics (grasping the $N$-th part along a conveyor), where attributes alone are insufficient to disambiguate targets.

Benchmark design

OrdinalBench follows the CLEVR diagnostic philosophy: synthetic scenes with unambiguous identifiers isolate reasoning from recognition confounders such as lighting and occlusion. Difficulty is controlled along three independent axes:

• Arrangement complexity: Level 1 (Single-Loop, a closed loop) versus Level 2 (Maze-Loop, an algorithmically generated loop with no dead ends, traversed by a deterministic rule: move forward if free; otherwise turn to the preferred side, falling back to the opposite side).
• Object count: 5/10/20 objects for Single-Loop; $7\times7$, $11\times11$, $21\times21$ grids for Maze-Loop.
• Ordinal magnitude: Within ($2 \le N \le S$, where $S$ is object count or grid size), Exceed ($k$0), and Large Scale ($k$1).

A skip-counting condition with stride $k$2 additionally requires algorithmic state management beyond simple instruction following. The released public_test split contains 2,600 images and 39,000 QA pairs across three task categories (2D Single-Loop, 2D Maze-Loop, 3D Single-Loop built on CLEVR-Ref+ infrastructure), balanced across all difficulty combinations. Each QA pair is annotated with a ground-truth reasoning trace recording per-step states (identifier, position, count, ordinal position).

Evaluation protocol

Models must output both a final prediction $k$3 and a structured JSON trace $k$4, enabling four metrics:

Metric	What it measures
Acc@N	Final-answer correctness
nLCP	Normalized length of the longest correct prefix of the predicted trace
STA	Per-step agreement between predicted and ground-truth traces
Cov.	Fraction of samples yielding at least one valid trace step

This protocol distinguishes models that reach the correct answer through faithful execution from those that guess, and localizes where the counting process breaks down.

Experimental findings

Seven VLMs were evaluated zero-shot with temperature 0.0 and standardized prompts: GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL (7B/32B/72B), InternVL3.5 (8B/14B), and Molmo (7B/72B). The headline results are stark:

• On 2D Single-Loop, the best model (Qwen2.5-VL-32B) reaches only 32.83% Acc@N against an 11.7% chance level.
• On 3D Single-Loop, most models degrade substantially; only Qwen2.5-VL-72B holds at 31.33%.
• On 2D Maze-Loop, even GPT-5 achieves just 11.04% Acc@N (chance ≈ 2.5%), indicating that maze traversal is near-unsolvable for current VLMs.
• The Molmo family performs at or below chance on all tasks, largely due to low Trace Coverage (70–80% versus ~100% for others), reflecting failures in basic output formatting and instruction following.

Three degradation patterns emerge consistently across models. First, skip counting is catastrophic: Qwen2.5-VL-7B drops from 42.62% Acc@N at stride 1 to 11.80% at stride 2 on 2D Single-Loop, with nLCP values collapsing to the 5–6% range—evidence that models cannot reliably execute algorithmic instructions requiring internal state updates. Second, large ordinals cause sharp decline: under Large Scale conditions ($k$5) on Maze-Loop, many models fall below 3% Acc@N, indicating limited sustained attention for long-distance counting. Third, scene clutter compounds the problem: average accuracy drops to single digits with 20 objects in 3D scenes, approaching the 5% chance level.

Notably, open-source Qwen2.5-VL models outperform GPT-5 and Gemini 2.5 Flash Lite on the simpler tasks, suggesting these weaknesses are architecture- and training-general rather than specific to any single model family.

Discussion

The authors attribute the observed failures to two mechanisms. The language bottleneck hypothesis holds that VLMs convert visual input into linguistic representations, which lose fidelity for hard-to-verbalize structures like mazes; consistent failures on simple branching decisions suggest models do not form stable internal representations of global spatial topology. The lack of procedural knowledge hypothesis holds that models possess declarative knowledge but cannot execute algorithms—managing and updating internal state such as "which ordinal are we currently counting"—a reading directly supported by the stride $k$6 collapse across all models.

These diagnoses motivate concrete development directions: explicit structured state representations to bypass the language bottleneck, training curricula emphasizing step-level procedural execution from short to long horizons, schema-constrained generation to reduce parse failures, and adoption of trace-level metrics alongside final accuracy.

Limitations and open questions

The paper concedes two principal limitations. The benchmark is fully synthetic by design, so it excludes real-image complexity (lighting, occlusion, texture diversity) where recognition and reasoning are intertwined; extending evaluation to natural images remains open. Second, the evaluation covers static images only, leaving temporal ordering in dynamic environments (video, robotics) unaddressed. A further implicit limitation is that zero-shot prompting with a fixed template may underestimate models that could perform better with tool use, code execution, or chain-of-thought scaffolding—the paper does not test whether such interventions close the gap, which is a natural question left open.

Conclusion

OrdinalBench provides a controlled, scalable diagnostic testbed for ordinal number understanding in VLMs, combining systematic difficulty axes, skip-counting conditions, ground-truth reasoning traces, and process-level metrics (nLCP, STA, Cov.). Its central empirical finding—that state-of-the-art VLMs, including GPT-5, exhibit severe degradation under large ordinals, complex paths, and non-unit strides, often approaching chance level—establishes procedural sequential reasoning as a measurable and currently unsolved weakness shared across model families.