SMART-840++ Multimodal Dataset

• SMART-840++ dataset is a large-scale, programmatically generated benchmark comprising 825 synthetic math puzzles to test vision-language reasoning in controlled settings.
• It employs 55 distinct puzzle formats with five difficulty levels and three variants per format to rigorously avoid spurious correlations.
• Integrated with the WISE multi-agent debate framework, the dataset facilitates systematic evaluation of model generalization and multimodal reasoning robustness.

The SMART-840++ dataset is a programmatically generated, large-scale multimodal benchmark designed to facilitate controlled evaluation of vision-and-language reasoning in multi-agent debate (MAD) frameworks. It was introduced in the context of WISE ("Weighted Iterative Society-of-Experts"), a modular MAD architecture targeting robust, grade-scaled assessment of LLM and MLLM agents on synthetic math puzzle tasks (Cherian et al., 2 Dec 2025).

1. Dataset Construction and Scope

SMART-840++ was created as a systematic extension of the original SMART-840—an established benchmark consisting of 840 children’s math puzzles with visual and textual components, six grade-pair splits, and five-way multiple-choice answers. The SMART-840++ expansion is motivated by the need for a larger suite of compositional, vision-text problems with algorithmically controlled difficulty.

Key construction methodology:

• Problem types: 55 distinct puzzle formats (spanning pattern recognition, counting, spatial transformation, visual arithmetic, multi-object matching, etc.)
• Difficulty control: 5 mesh/difficulty levels per type, scaling visual complexity (e.g., number of subregions to count, degree of symmetry, object occlusion).
• Variants: For each (type, level), three distinct variants are generated to further probe generalization.
• Total size: 55 × 5 × 3 = 825 unique, programmatically generated multimodal question instances.

Instances are generated to rigorously avoid spurious correlations between visual cues and answers, and to enforce grade-appropriate reasoning difficulty. Both question generation and answer sets are fully synthetic: for multiple-choice, problems include one correct and four distractor options per instance.

2. Benchmarking Protocol and Intended Usage

SMART-840++ was introduced specifically to test the robustness and generalization of MAD protocols across a spectrum of vision-language complexity. Each instance consists of:

• A rendered image containing structured visual elements (e.g., geometric arrangements, spatially distributed tokens, colored objects)
• An accompanying natural language question targeting compositional or spatial reasoning
• For most tasks, five multiple-choice answer options (A–E); for select tasks, free-form answers

The dataset is used in an iterative debate setting:

• Solver agents (LLMs or MLLMs) receive the image, question, and choices, and must explain their reasoning and select an answer.
• Critic/Reflector agents evaluate solver outputs, provide weights and feedback.
• An external orchestrator manages debate rounds, feedback, and consensus.

Difficulty scaling allows analysis of model/debate protocol performance across a controlled spectrum of visual-textual reasoning demands, from simple counting to advanced combinations of spatial, numeric, and logical constraints.

3. Integration Within the WISE MAD Framework

In (Cherian et al., 2 Dec 2025), SMART-840++ serves as a stress test for the WISE framework, which partitions agents into Solvers and Reflectors and aggregates their outputs using a weighted, Dawid–Skene-style two-dimensional EM procedure. The dataset’s grade- and type-controlled composition is critical for:

• Evaluating multimodal generalization
• Measuring incremental improvements attributable to MAD aggregation, self-correction, and heterogeneous agent pooling
• Testing robustness against agent redundancy, noise, and debate protocol variations

Each debate is capped at a maximum of 8 rounds (average ≈2–3 rounds in practice), and performance is measured as the majority-vote (or exact match) accuracy per instance.

4. Quantitative Results

The most pertinent findings concerning SMART-840++ include:

Method	Best Accuracy (%)
Best single model	18.2
WISE (best config)	26.9

WISE, in its optimized configuration, improves accuracy on SMART-840++ by +8.7% over the best single-model baseline (Cherian et al., 2 Dec 2025). This demonstrates both the difficulty of the dataset (absolute accuracy remains low for all models) and the nontrivial challenge posed by vision-language reasoning under controlled compositional and visual complexity.

5. Relation to Other Benchmarks and Broader Context

SMART-840++ complements datasets such as VisualPuzzles and EvoChart-QA by offering:

• Fine-grained difficulty control across problem types for rigorous scalability analysis
• Sufficient size for differentiating high-capacity MAD architectures
• Inherent resistance to overfitting due to on-the-fly instance generation and high visual variability

Unlike ad hoc or purely language-driven benchmarks, SMART-840++ is specifically engineered to push the limits of LLM/MLLM generalization, multimodal perception, and their interactions with algorithmic debate protocols. The controlled mesh sizes and type splits allow fine-grained ablations (e.g., model scaling, MAD round tuning, agent heterogeneity) and systematic gauge of cross-round learning or ensemble benefits.

6. Access and Future Directions

SMART-840++ was introduced and used exclusively in (Cherian et al., 2 Dec 2025) and is currently not referenced as an independent open-source benchmark outside that context. The programmatic generation methodology and split structure are detailed in the paper’s supplementary material, enabling reproduction and extension. Ongoing research may augment it with:

• New problem types or fine-grained distractor generation
• Additional language modalities
• More realistic visual renderings or naturalistic data augmentations

A plausible implication is that as more robust multimodal MAD architectures emerge, SMART-840++ will become an essential diagnostic for vision–language reasoning under adversarial, high-complexity, and artificially controlled conditions.

---

References:

• WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate (Cherian et al., 2 Dec 2025)