Bongard-RWR+: Scalable Abstract Visual Reasoning

• Bongard-RWR+ is a benchmark of 5,400 generated Bongard problems that test fine-grained abstract visual reasoning using real-world-like images.
• It employs a four-stage pipeline—Describe, Augment, Render, and Review—to ensure the images faithfully represent subtle abstract concepts.
• Empirical results show that current vision-language models struggle with precise relational discrimination and free-form concept description.

Bongard-RWR+ is a Bongard Problems benchmark composed of 5,400 instances that represent original Bongard Problem abstract concepts using real-world-like images generated via a vision LLM pipeline. It was introduced to preserve the fine-grained abstract concepts targeted by Bongard-RWR while overcoming the latter’s scale limitation of 60 problems. In this benchmark, the central task remains Bongard-style abstract visual reasoning: infer the latent rule that separates two sets of six images and, depending on the formulation, classify test images or verbalize the underlying concept (Pawlonka et al., 16 Aug 2025).

1. Origins, scope, and motivation

Bongard Problems (BPs) are classically defined by two sets of six images, one on the left and one on the right, separated by an underlying rule. They test few-shot concept learning from a handful of examples, contextual visual interpretation, and joint vision-language reasoning. Earlier benchmark families exposed complementary but incomplete aspects of this challenge. The original synthetic BPs were manually designed black-and-white drawings that foreground abstraction but are visually far from natural-image perception. Real-world variants such as Bongard-HOI and Bongard-OpenWorld use natural images, but the represented concepts are often high-level and coarse. Bongard-RWR narrowed this gap by using real-world images to represent the abstract concepts of classical BPs, but its manual construction restricted it to 60 problems (Pawlonka et al., 16 Aug 2025).

Bongard-RWR+ was created to retain the abstractness of Bongard-RWR while scaling the benchmark. Its explicit design goal is to keep the fine-grained structural relations characteristic of classical Bongard reasoning—such as contour, rotation, count, angle, similarity, branching, size, position, and shape—while moving away from both synthetic line drawings and coarse real-world semantics. The benchmark therefore occupies a distinct position: the images are generated, the concepts are abstract, and the intended difficulty lies in recovering subtle relational structure rather than obvious object or scene categories (Pawlonka et al., 16 Aug 2025).

This positioning also reflects the motivation behind Bongard-RWR itself. That earlier dataset was introduced to test whether poor multimodal performance on synthetic BPs arose from the synthetic and geometric image domain or from more general abstract visual reasoning limitations. Its results suggested that weak performance was “not due to domain specificity but rather reflects their general AVR limitations.” Bongard-RWR+ extends that line of inquiry by making evaluation more robust at scale (Małkiński et al., 2024).

2. Construction pipeline and data generation

The generation pipeline has four stages: Describe, Augment, Render, and Review. The starting point is the manually curated Bongard-RWR source material. From this base, the authors generate new concept-faithful images while attempting to avoid drift into the opposite concept class (Pawlonka et al., 16 Aug 2025).

For each image $L_i$ or $R_i$ in a Bongard-RWR problem, Pixtral-12B is used as an image-to-text (I2T) model. The paper formalizes this as

$$\text{Describe}(L_i, C_L) = (\mathcal{L}^{+}_i, \mathcal{L}^{-}_i), \qquad \text{Describe}(R_i, C_R) = (\mathcal{R}^{+}_i, \mathcal{R}^{-}_i),$$

where the positive descriptions are intended to be faithful to the image and concept, and the negative descriptions specify what should not appear. The negative prompt is used to steer image generation away from the opposite concept.

Each positive caption is then expanded into $N = 15$ alternative descriptions with a text-to-text (T2T) model: $$\text{Augment}(\mathcal{L}^{+}_i, C_L) = \{\mathcal{L}^{+}_{i,j}\}_{j=1}^{N},$$ and analogously for the right side. These augmented descriptions are meant to preserve the same concept while varying surface details and scene composition.

The augmented descriptions are passed, together with their corresponding negative prompts, to Flux.1-dev: $$\text{Render}(\mathcal{L}^{+}_{i,j}, \mathcal{L}^{-}_{i}) = L_{i,j}, \qquad \text{Render}(\mathcal{R}^{+}_{i,j}, \mathcal{R}^{-}_{i}) = R_{i,j}.$$ The rendered outputs are 512×512 images.

All generated images are then manually reviewed. Images are discarded if they fail to reflect the intended concept or if they introduce elements associated with the opposite side. This manual verification is central to the benchmark’s construction because the paper explicitly notes that some concepts are difficult for current text-to-image systems to render faithfully. One cited example is “inside figures of the second order,” which was excluded because it could not be generated reliably enough (Pawlonka et al., 16 Aug 2025).

The final scale follows directly from this pipeline. Bongard-RWR+ is built from 54 Bongard-RWR source matrices, and each source matrix yields 100 new BP instances. The resulting benchmark contains 5,400 Bongard problems and 49 concepts. The paper additionally notes that 6 source matrices were discarded because suitable generated images could not be produced reliably (Pawlonka et al., 16 Aug 2025).

3. Formal structure, subset selection, and benchmark variants

A Bongard-RWR+ instance is defined as

$\mathcal{BP} = (L, R, T, C),$

where $L = \{L_1, \ldots, L_P\}$ and $R = \{R_1, \ldots, R_P\}$ are the left and right context images, $T = \{T_L, T_R\}$ are two test images, and $R_i$0 is the underlying concept description. In the main setting, $R_i$1, so each side contains six context images plus one test image per side (Pawlonka et al., 16 Aug 2025).

The benchmark does not simply sample images arbitrarily from the generated pool. Instead, the paper uses ViT-L/14 embeddings to maximize within-side visual diversity. Candidate $R_i$2-element subsets are embedded, and the construction algorithm chooses the subset with minimum maximum pairwise cosine similarity. An image-removal step can then discard one redundant image after each subset selection to increase variety. This process is repeated until it yields $R_i$3 diverse subsets per side. Since there are 10 left-side subsets and 10 right-side subsets, the Cartesian product produces

$R_i$4

Bongard matrices per source BP (Pawlonka et al., 16 Aug 2025).

Several benchmark variants are defined. Bongard-RWR+/GS is a grayscale version in which all images are converted to grayscale. Bongard-RWR+/LP disables the image-removal step; the paper associates this with fewer unique images, lower average intra-side embedding similarity, and greater image reuse. This LP variant is instantiated for

$R_i$5

to study the effect of the number of demonstrations. The paper also introduces supervised-learning-oriented splits: Bongard-RWR+/TVT with 12,150 matrices partitioned into 7,290 train, 1,215 validation, and 3,645 test, and TVT-Large with 86,400 matrices partitioned into 51,840 train, 8,640 validation, and 25,920 test (Pawlonka et al., 16 Aug 2025).

These design choices matter because the benchmark is intended to test concept recovery rather than memorization of a small hand-curated set. A plausible implication is that diversity-based subset selection makes the benchmark less sensitive to accidental image overlap and more sensitive to actual concept induction.

4. Task formulations and evaluation protocol

Bongard-RWR+ defines six task settings, ordered by increasing difficulty. Image-to-Side (I1S) asks for classification of a single test image as LEFT or RIGHT. Images-to-Sides (I2S) requires joint classification of two test images, one from each class. Description-to-Side (D1S) and Descriptions-to-Sides (D2S) first convert the visual inputs into captions and then classify one or two test descriptions. Concept Selection (CS) is a multiclass formulation in which the correct concept $R_i$6 must be selected from a candidate set

$R_i$7

with $R_i$8. Concept Generation (CG) requires free-form natural-language description of the concept (Pawlonka et al., 16 Aug 2025).

The evaluation covers four open-access VLMs: InternVL2.5 78B, Qwen2-VL 72B-Instruct, LLaVA-Next 110B, and MiniCPM-o 2.6 8B. For text-only reasoning in D1S and D2S, the paper additionally evaluates DeepSeek-R1 70B. It also includes a Similarity Classifier (SC) baseline and supervised models in the TVT setting. All models are run at decoding temperature 0.5, and structured JSON outputs are enforced via Outlines (Pawlonka et al., 16 Aug 2025).

The CG task is evaluated with multiple text-generation metrics: BLEU—specifically

$R_i$9

—along with METEOR, ROUGE-L, CIDEr, and BERTScore precision, recall, and F1. The supervised baselines in the TVT setting are MLP, WReN, and SNAIL (Pawlonka et al., 16 Aug 2025).

The benchmark formulations are significant because they separate several subproblems that are often conflated in Bongard evaluation: direct visual-side assignment, caption-mediated classification, constrained concept retrieval, and unconstrained concept verbalization. This suggests that Bongard-RWR+ is structured not only as a dataset but also as an evaluation suite for disentangling different failure modes in multimodal reasoning.

5. Empirical results and observed model behavior

The principal empirical result is that current VLMs remain close to chance on the binary Bongard-style classification settings. In the main results table, I1S and I2S are near random for all evaluated VLMs. The reported scores are:

• InternVL2.5: I1S 0.50, I2S 0.39, D1S 0.57, D2S 0.49
• Qwen2-VL: I1S 0.49, I2S 0.44, D1S 0.58, D2S 0.42
• LLaVA-Next: I1S 0.50, I2S 0.50, D1S 0.54, D2S 0.43
• MiniCPM-o 2.6: I1S 0.48, I2S 0.45, D1S 0.51, D2S 0.41
• DeepSeek-R1: D1S 0.57, D2S 0.56
• Similarity Classifier: I1S 0.52, I2S 0.54, D1S 0.49, D2S 0.50 (Pawlonka et al., 16 Aug 2025)

The paper’s interpretation is direct: on raw image-based Bongard classification, VLMs struggle heavily, and the simple Similarity Classifier often matches or exceeds them. The particularly weak I2S results indicate that jointly reasoning over two test images remains difficult.

The Concept Selection task provides a cleaner discriminative view. InternVL2.5 is the strongest model overall, reaching 91% at $$\text{Describe}(L_i, C_L) = (\mathcal{L}^{+}_i, \mathcal{L}^{-}_i), \qquad \text{Describe}(R_i, C_R) = (\mathcal{R}^{+}_i, \mathcal{R}^{-}_i),$$0 but dropping to 57% at $$\text{Describe}(L_i, C_L) = (\mathcal{L}^{+}_i, \mathcal{L}^{-}_i), \qquad \text{Describe}(R_i, C_R) = (\mathcal{R}^{+}_i, \mathcal{R}^{-}_i),$$1. MiniCPM-o and LLaVA-Next are much weaker at large $$\text{Describe}(L_i, C_L) = (\mathcal{L}^{+}_i, \mathcal{L}^{-}_i), \qquad \text{Describe}(R_i, C_R) = (\mathcal{R}^{+}_i, \mathcal{R}^{-}_i),$$2, around 19% for $$\text{Describe}(L_i, C_L) = (\mathcal{L}^{+}_i, \mathcal{L}^{-}_i), \qquad \text{Describe}(R_i, C_R) = (\mathcal{R}^{+}_i, \mathcal{R}^{-}_i),$$3. The pattern indicates that models can often reject obviously wrong concepts when the distractor set is small, but fail to identify the correct fine-grained rule as the candidate set becomes more competitive (Pawlonka et al., 16 Aug 2025).

Concept-group analysis divides the benchmark into 9 semantic categories: Size, Position, Count, Branching, Similarity, Contour, Shape, Rotation, and Angle. Performance is relatively better on Size, Shape, Count, and some Branching concepts, and relatively poor on Contour, Rotation, and Angle. The paper attributes the harder categories to requirements for subtle geometric comparisons, precise spatial relationships, and detailed structural sensitivity (Pawlonka et al., 16 Aug 2025).

The paper also reports several controlled comparisons. Increasing model size generally improves performance, but not enough to resolve the hardest concept-selection settings. On Bongard-RWR+/GS, performance is similar or sometimes better than with color, especially for lower-performing models, consistent with the benchmark’s origin in black-and-white synthetic concepts. Comparison between Bongard-RWR and Bongard-RWR+/LP shows similar performance trends, which the paper interprets as evidence that the generated images are functionally equivalent to real images for this AVR setting. In the open-ended Concept Generation task, scores are generally low across all models; MiniCPM-o 2.6 is often best or near-best in image-based CG, while Qwen2-VL tends to lead in several text-based CG metrics, but with modest margins. Finally, the supervised baselines MLP, WReN, and SNAIL achieve roughly 0.48–0.51 test accuracy, indicating that Bongard-RWR+ is difficult not only for VLMs but also for standard supervised AVR models (Pawlonka et al., 16 Aug 2025).

6. Relation to prior Bongard benchmarks, misconceptions, and limitations

Bongard-RWR+ is best understood as the large-scale successor to Bongard-RWR, not as a replacement for the full spectrum of Bongard benchmarks. Synthetic BPs remain the canonical source of abstract concepts. Bongard-HOI and Bongard-OpenWorld probe performance on real-world imagery but with more high-level and coarse semantics. Bongard-RWR established the more demanding bridge from synthetic abstraction to real-world imagery by translating synthetic concepts into real-world images; Bongard-RWR+ preserves that conceptual agenda while scaling the dataset through generation and manual verification (Małkiński et al., 2024).

The benchmark also stands apart from solver-oriented Bongard research. For example, a separate line of work reformulates classical BPs as a reinforcement learning / contextual bandit problem over image pairs, with a skip action interpreted as a counterfactual intervention and a Siamese/contrastive feature extractor for low-level representation learning. That work is methodologically about solving and interpreting BPs, whereas Bongard-RWR+ is primarily a benchmark for evaluating fine-grained abstract visual reasoning under real-world-like image statistics (Youssef et al., 2022).

A recurrent misconception concerns the acronym RWR. In the Bongard literature, RWR refers to Real-World Representations, as in Bongard-RWR and Bongard-RWR+. This is unrelated to the Regression-With-Residuals (RWR) method used in causal inference and public-policy analysis; the overlap is nominal rather than conceptual (Balgi et al., 2022).

The limitations reported for Bongard-RWR+ are concrete. The pipeline still requires manual verification, so the dataset is not fully automatic. Some concepts remain too difficult for current text-to-image systems to render faithfully and were therefore excluded. Only selected VLMs are evaluated, and the paper explicitly calls for broader community benchmarking. Within those constraints, the central conclusion is stable: present VLMs can often recognize coarse or global properties, but they struggle with the fine-grained abstract visual reasoning demanded by Bongard-RWR+, especially when the task requires precise relational discrimination or free-form concept description (Pawlonka et al., 16 Aug 2025).