---
title: 'Bongard-RWR+: Scalable Abstract Visual Reasoning'
url: https://www.emergentmind.com/topics/bongard-rwr
type: topic
---

# Bongard-RWR+: Scalable Abstract Visual Reasoning

Bongard-RWR+ is a Bongard Problems benchmark composed of **5,400 instances** that represent original Bongard Problem abstract concepts using **real-world-like images** generated via a **vision language model pipeline**. It was introduced to preserve the **fine-grained abstract concepts** targeted by Bongard-RWR while overcoming the latter’s scale limitation of **60 problems**. In this benchmark, the central task remains Bongard-style abstract visual reasoning: infer the latent rule that separates two sets of six images and, depending on the formulation, classify test images or verbalize the underlying concept [2508.12026].

## 1. Origins, scope, and motivation

Bongard Problems (BPs) are classically defined by **two sets of six images**, one on the left and one on the right, separated by an underlying rule. They test **few-shot concept learning** from a handful of examples, **contextual visual interpretation**, and **joint vision-language reasoning**. Earlier benchmark families exposed complementary but incomplete aspects of this challenge. The original synthetic BPs were manually designed **black-and-white drawings** that foreground abstraction but are visually far from natural-image perception. Real-world variants such as **Bongard-HOI** and **Bongard-OpenWorld** use natural images, but the represented concepts are often **high-level and coarse**. **Bongard-RWR** narrowed this gap by using **real-world images** to represent the **abstract concepts** of classical BPs, but its manual construction restricted it to **60 problems** [2508.12026].

Bongard-RWR+ was created to retain the abstractness of Bongard-RWR while scaling the benchmark. Its explicit design goal is to keep the **fine-grained structural relations** characteristic of classical Bongard reasoning—such as **contour**, **rotation**, **count**, **angle**, **similarity**, **branching**, **size**, **position**, and **shape**—while moving away from both synthetic line drawings and coarse real-world semantics. The benchmark therefore occupies a distinct position: the images are **generated**, the concepts are **abstract**, and the intended difficulty lies in recovering subtle relational structure rather than obvious object or scene categories [2508.12026].

This positioning also reflects the motivation behind Bongard-RWR itself. That earlier dataset was introduced to test whether poor multimodal performance on synthetic BPs arose from the **synthetic and geometric** image domain or from more general **abstract visual reasoning** limitations. Its results suggested that weak performance was **“not due to domain specificity but rather reflects their general AVR limitations.”** Bongard-RWR+ extends that line of inquiry by making evaluation more robust at scale [2411.01173].

## 2. Construction pipeline and data generation

The generation pipeline has four stages: **Describe**, **Augment**, **Render**, and **Review**. The starting point is the manually curated Bongard-RWR source material. From this base, the authors generate new concept-faithful images while attempting to avoid drift into the opposite concept class [2508.12026].

For each image \(L_i\) or \(R_i\) in a Bongard-RWR problem, **Pixtral-12B** is used as an **image-to-text (I2T)** model. The paper formalizes this as
\[
\text{Describe}(L_i, C_L) = (\mathcal{L}^{+}_i, \mathcal{L}^{-}_i), \qquad \text{Describe}(R_i, C_R) = (\mathcal{R}^{+}_i, \mathcal{R}^{-}_i),
\]
where the positive descriptions are intended to be faithful to the image and concept, and the negative descriptions specify what should not appear. The negative prompt is used to steer image generation away from the opposite concept.

Each positive caption is then expanded into \(N = 15\) alternative descriptions with a **text-to-text (T2T)** model:
\[
\text{Augment}(\mathcal{L}^{+}_i, C_L) = \{\mathcal{L}^{+}_{i,j}\}_{j=1}^{N},
\]
and analogously for the right side. These augmented descriptions are meant to preserve the same concept while varying surface details and scene composition.

The augmented descriptions are passed, together with their corresponding negative prompts, to **Flux.1-dev**:
\[
\text{Render}(\mathcal{L}^{+}_{i,j}, \mathcal{L}^{-}_{i}) = L_{i,j}, \qquad \text{Render}(\mathcal{R}^{+}_{i,j}, \mathcal{R}^{-}_{i}) = R_{i,j}.
\]
The rendered outputs are **512×512** images.

All generated images are then **manually reviewed**. Images are discarded if they fail to reflect the intended concept or if they introduce elements associated with the opposite side. This manual verification is central to the benchmark’s construction because the paper explicitly notes that some concepts are difficult for current text-to-image systems to render faithfully. One cited example is **“inside figures of the second order,”** which was excluded because it could not be generated reliably enough [2508.12026].

The final scale follows directly from this pipeline. Bongard-RWR+ is built from **54 Bongard-RWR source matrices**, and each source matrix yields **100 new BP instances**. The resulting benchmark contains **5,400 Bongard problems** and **49 concepts**. The paper additionally notes that **6 source matrices** were discarded because suitable generated images could not be produced reliably [2508.12026].

## 3. Formal structure, subset selection, and benchmark variants

A Bongard-RWR+ instance is defined as
\[
\mathcal{BP} = (L, R, T, C),
\]
where \(L = \{L_1, \ldots, L_P\}\) and \(R = \{R_1, \ldots, R_P\}\) are the left and right context images, \(T = \{T_L, T_R\}\) are two test images, and \(C = (C_L, C_R)\) is the underlying concept description. In the main setting, \(P = 6\), so each side contains six context images plus one test image per side [2508.12026].

The benchmark does not simply sample images arbitrarily from the generated pool. Instead, the paper uses **ViT-L/14 embeddings** to maximize within-side visual diversity. Candidate \(P\)-element subsets are embedded, and the construction algorithm chooses the subset with **minimum maximum pairwise cosine similarity**. An image-removal step can then discard one redundant image after each subset selection to increase variety. This process is repeated until it yields \(M = 10\) diverse subsets per side. Since there are **10 left-side subsets** and **10 right-side subsets**, the Cartesian product produces
\[
|L'| = 10,\quad |R'| = 10,\quad |L'| \times |R'| = 100
\]
Bongard matrices per source BP [2508.12026].

Several benchmark variants are defined. **Bongard-RWR+/GS** is a **grayscale** version in which all images are converted to grayscale. **Bongard-RWR+/LP** disables the image-removal step; the paper associates this with fewer unique images, lower average intra-side embedding similarity, and greater image reuse. This LP variant is instantiated for
\[
P = 2,3,4,5,6
\]
to study the effect of the number of demonstrations. The paper also introduces supervised-learning-oriented splits: **Bongard-RWR+/TVT** with **12,150 matrices** partitioned into **7,290 train**, **1,215 validation**, and **3,645 test**, and **TVT-Large** with **86,400 matrices** partitioned into **51,840 train**, **8,640 validation**, and **25,920 test** [2508.12026].

These design choices matter because the benchmark is intended to test concept recovery rather than memorization of a small hand-curated set. A plausible implication is that diversity-based subset selection makes the benchmark less sensitive to accidental image overlap and more sensitive to actual concept induction.

## 4. Task formulations and evaluation protocol

Bongard-RWR+ defines **six task settings**, ordered by increasing difficulty. **Image-to-Side (I1S)** asks for classification of a **single test image** as LEFT or RIGHT. **Images-to-Sides (I2S)** requires joint classification of **two test images**, one from each class. **Description-to-Side (D1S)** and **Descriptions-to-Sides (D2S)** first convert the visual inputs into captions and then classify one or two test descriptions. **Concept Selection (CS)** is a multiclass formulation in which the correct concept \(C^*\) must be selected from a candidate set
\[
\{C_k\}_{k=1}^{K}, \quad C^* \in \{C_k\},
\]
with \(K \in \{2,4,8,16\}\). **Concept Generation (CG)** requires free-form natural-language description of the concept [2508.12026].

The evaluation covers four open-access VLMs: **InternVL2.5 78B**, **Qwen2-VL 72B-Instruct**, **LLaVA-Next 110B**, and **MiniCPM-o 2.6 8B**. For text-only reasoning in D1S and D2S, the paper additionally evaluates **DeepSeek-R1 70B**. It also includes a **Similarity Classifier (SC)** baseline and supervised models in the TVT setting. All models are run at decoding temperature **0.5**, and structured JSON outputs are enforced via **Outlines** [2508.12026].

The CG task is evaluated with multiple text-generation metrics: **BLEU**—specifically
\[
\text{BLEU}_1, \text{BLEU}_2
\]
—along with **METEOR**, **ROUGE-L**, **CIDEr**, and **BERTScore** precision, recall, and F1. The supervised baselines in the TVT setting are **MLP**, **WReN**, and **SNAIL** [2508.12026].

The benchmark formulations are significant because they separate several subproblems that are often conflated in Bongard evaluation: direct visual-side assignment, caption-mediated classification, constrained concept retrieval, and unconstrained concept verbalization. This suggests that Bongard-RWR+ is structured not only as a dataset but also as an evaluation suite for disentangling different failure modes in multimodal reasoning.

## 5. Empirical results and observed model behavior

The principal empirical result is that current VLMs remain close to chance on the binary Bongard-style classification settings. In the main results table, **I1S** and **I2S** are near random for all evaluated VLMs. The reported scores are:

- **InternVL2.5**: I1S **0.50**, I2S **0.39**, D1S **0.57**, D2S **0.49**
- **Qwen2-VL**: I1S **0.49**, I2S **0.44**, D1S **0.58**, D2S **0.42**
- **LLaVA-Next**: I1S **0.50**, I2S **0.50**, D1S **0.54**, D2S **0.43**
- **MiniCPM-o 2.6**: I1S **0.48**, I2S **0.45**, D1S **0.51**, D2S **0.41**
- **DeepSeek-R1**: D1S **0.57**, D2S **0.56**
- **Similarity Classifier**: I1S **0.52**, I2S **0.54**, D1S **0.49**, D2S **0.50** [2508.12026]

The paper’s interpretation is direct: on raw image-based Bongard classification, **VLMs struggle heavily**, and the simple **Similarity Classifier often matches or exceeds them**. The particularly weak I2S results indicate that jointly reasoning over two test images remains difficult.

The **Concept Selection** task provides a cleaner discriminative view. **InternVL2.5** is the strongest model overall, reaching **91%** at \(K = 2\) but dropping to **57%** at \(K = 16\). **MiniCPM-o** and **LLaVA-Next** are much weaker at large \(K\), around **19%** for \(K=16\). The pattern indicates that models can often reject obviously wrong concepts when the distractor set is small, but fail to identify the correct **fine-grained rule** as the candidate set becomes more competitive [2508.12026].

Concept-group analysis divides the benchmark into **9 semantic categories**: **Size**, **Position**, **Count**, **Branching**, **Similarity**, **Contour**, **Shape**, **Rotation**, and **Angle**. Performance is relatively better on **Size**, **Shape**, **Count**, and some **Branching** concepts, and relatively poor on **Contour**, **Rotation**, and **Angle**. The paper attributes the harder categories to requirements for **subtle geometric comparisons**, **precise spatial relationships**, and **detailed structural sensitivity** [2508.12026].

The paper also reports several controlled comparisons. Increasing model size generally improves performance, but not enough to resolve the hardest concept-selection settings. On **Bongard-RWR+/GS**, performance is similar or sometimes better than with color, especially for lower-performing models, consistent with the benchmark’s origin in black-and-white synthetic concepts. Comparison between **Bongard-RWR** and **Bongard-RWR+/LP** shows similar performance trends, which the paper interprets as evidence that the generated images are **functionally equivalent** to real images for this AVR setting. In the open-ended **Concept Generation** task, scores are generally **low across all models**; **MiniCPM-o 2.6** is often best or near-best in image-based CG, while **Qwen2-VL** tends to lead in several text-based CG metrics, but with modest margins. Finally, the supervised baselines **MLP**, **WReN**, and **SNAIL** achieve roughly **0.48–0.51** test accuracy, indicating that Bongard-RWR+ is difficult not only for VLMs but also for standard supervised AVR models [2508.12026].

## 6. Relation to prior Bongard benchmarks, misconceptions, and limitations

Bongard-RWR+ is best understood as the large-scale successor to **Bongard-RWR**, not as a replacement for the full spectrum of Bongard benchmarks. Synthetic BPs remain the canonical source of abstract concepts. **Bongard-HOI** and **Bongard-OpenWorld** probe performance on real-world imagery but with more **high-level and coarse** semantics. **Bongard-RWR** established the more demanding bridge from synthetic abstraction to real-world imagery by translating synthetic concepts into real-world images; **Bongard-RWR+** preserves that conceptual agenda while scaling the dataset through generation and manual verification [2411.01173].

The benchmark also stands apart from solver-oriented Bongard research. For example, a separate line of work reformulates classical BPs as a **reinforcement learning / contextual bandit** problem over image pairs, with a **skip action** interpreted as a counterfactual intervention and a **Siamese/contrastive** feature extractor for low-level representation learning. That work is methodologically about solving and interpreting BPs, whereas Bongard-RWR+ is primarily a benchmark for evaluating fine-grained abstract visual reasoning under real-world-like image statistics [2206.07196].

A recurrent misconception concerns the acronym **RWR**. In the Bongard literature, **RWR** refers to **Real-World Representations**, as in **Bongard-RWR** and **Bongard-RWR+**. This is unrelated to the **Regression-With-Residuals (RWR)** method used in causal inference and public-policy analysis; the overlap is nominal rather than conceptual [2202.03281].

The limitations reported for Bongard-RWR+ are concrete. The pipeline still requires **manual verification**, so the dataset is not fully automatic. Some concepts remain too difficult for current text-to-image systems to render faithfully and were therefore excluded. Only selected VLMs are evaluated, and the paper explicitly calls for broader community benchmarking. Within those constraints, the central conclusion is stable: present VLMs can often recognize **coarse** or global properties, but they struggle with the **fine-grained abstract visual reasoning** demanded by Bongard-RWR+, especially when the task requires precise relational discrimination or free-form concept description [2508.12026].

Source: https://www.emergentmind.com/topics/bongard-rwr