BIRD: Blocksworld Image Reasoning Dataset

• BIRD is a benchmark dataset featuring real-world block images with spatial and color annotations for inferring minimal rearrangement sequences.
• The dataset supports modular two-stage methods that decouple visual perception from logical reasoning, enabling detailed evaluation of event sequencing.
• BIRD facilitates research in robotics and cognitive science by challenging systems to model complex, inductively generalizable block manipulation tasks.

The Blocksworld Image Reasoning Dataset (BIRD) is a specialized benchmark developed to evaluate and advance computational systems in Image-based Event Sequencing (IES)—the task of inferring a sequence of object rearrangement actions from a pair of images depicting different spatial arrangements. BIRD uniquely combines real-world visual input (photographs of wooden blocks in various configurations) with rich symbolic annotations and exhaustive event sequence labeling, supporting rigorous experimentation in perception, reasoning, and planning from vision.

1. Dataset Structure and Annotation

BIRD consists of 7,267 high-resolution photographs of wooden blocks arranged on a white, uniformly lit background. Each image contains up to five blocks, constrained such that no two blocks share the same color within a single scene. Crucially, arrangements include contact, partial overlap, and stacking, contrasting with synthetic datasets that often enforce stricter separation.

Each image is annotated with:

• A 5×5 color-blind arrangement vector encoding the grid-based spatial positions of blocks, abstracting away color to represent geometry.
• A color vector (5×3 bits) listing the colors of the blocks, encoded in bottom-to-top, left-to-right order. These representations enable precise, structured characterization of both spatial and non-spatial attributes.

The dataset’s key feature is the inclusion, for every ordered image pair $\left(I^S, I^T\right)$, of all possible minimal-length event sequences required to transform $I^S$ into $I^T$. Each event is formalized as

$\text{move}(X, Y, t)$

where $X$ is a block color ($X \in \mathcal{C}$), $Y$ is drawn from $$\mathcal{D} = \mathcal{C} \cup \{\text{table}\} \cup \{\text{out}\}$$, and $t$ indexes the time step. For instance, $\text{move}(\text{R}, \text{G}, 2)$ denotes “move the Red block onto the Green block at $t = 2$.” All permutations adhere to constraints (e.g., $X \ne Y$), encompassing non-trivial transformation paths.

2. Image-based Event Sequencing (IES) Task Definition

The IES task extends traditional recognition by demanding temporal, causal reasoning over visual states. Given source and target images, the system must predict a valid sequence $M = [m_1, m_2, ..., m_L]$ such that

$I^S \xrightarrow{M} I^T$

Here, the search space is combinatorially large; for 8-step sequences, with each step offering 48 possible moves, $|M| = 48^8 \approx 2.8 \times 10^{13}$.

This structure necessitates that systems:

• Perceive not just object identity and position, but also stack hierarchies and contact relationships,
• Infer minimal rearrangement steps between disparate configurations,
• Operate robustly as the sequence length and spatial complexity grow.

3. Modular Two-Stage Learning and Reasoning Approaches

To address the above, BIRD supports and motivates a modular two-step solution:

• Stage-1: Visual Perception. This module encodes the RGB input into structured, interpretable representations suitable for reasoning. An 8-layer convolutional arrangement encoder ($f_A$) localizes blocks within a 5×5 lattice, while a ResNet-50-based color grounding module ($f_C$) extracts ordered color attributes into a vectorized form.
• Stage-2: Event Sequencing. Given structured representations for the source and target, this stage infers the minimal event sequence. Multiple architectures are compared:
  • Fully Connected Neural Networks (FC): Multi-label classifiers trained to output full length event sequences.
  • Q-Learning (QL): Reinforcement learning approaches modeling sequencing as a finite MDP, seeking policies that minimize transition cost.
  • Inductive Logic Programming (ILP): Symbolic systems that learn update rules such as

  $$\text{on}(X, Y, t+1) \! \leftarrow \! \text{move}(X, Y, t)$$

  enabling deterministic rollouts. A logic engine function ($g_\ell$) applies moves to update abstract configurations, supporting sequential reasoning detached from pixel input.

The decoupling of perception and reasoning simplifies the output search space and allows sequencing modules to be reapplied across domains, including to natural images whose objects have been mapped into block-like representations.

4. Performance Benchmarking and Generalizability

BIRD provides two evaluation metrics:

• Full Sequence Accuracy (FSA): Proportion of test samples with a completely correct predicted event sequence.
• Step Level Accuracy (SLA): Fraction of correctly predicted actions within a sequence.

Empirical results indicate that end-to-end convolutional and relational networks (e.g., ResNet-50, PSPNet, Relational Networks) achieve modest FSA (typically 30–35%), due in part to the high-dimensional output space and failure to reliably model causal transitions. In contrast, the modular two-stage pipeline—especially employing ILP for sequencing—attains 100% FSA and SLA in the oracle perception scenario.

A critical property of BIRD is its design for evaluating inductive generalizability: systems must generate correct event sequences longer than those encountered during training. Two-stage approaches with explicit logical reasoning (ILP) maintain robust performance as required sequence length increases, while end-to-end models do not extrapolate well beyond the training distribution.

5. Application and Reuse Beyond Synthetic Data

An advantage of the structured arrangement and color abstractions in BIRD is the ability to transfer learned sequencing modules to other domains. Experiments involving Mask-RCNN object detectors confirm that natural images, when processed into BIRD’s abstract state representations, are amenable to the same sequencing algorithms. Improved accuracy in both FSA and SLA substantiates the utility and flexibility of BIRD-aligned reasoning pipelines across visual complexity gradients.

Moreover, the sequencing challenge in BIRD reflects demands found in robotics (manipulation, rearrangement planning) and cognitive science (event causality modeling), where learning and reasoning must interplay seamlessly.

6. Implications for Future Benchmarks and Extensions

Future directions for BIRD include:

• Relaxing the current constraints to allow larger numbers of blocks, richer block characteristics, and a wider variety of permissible actions, thereby expanding the diversity of event sequences.
• Further exploration of hybrid reasoning pipelines that blend deep learning and formal logic, especially in the context of environments with less regular structure than grid worlds.
• Investigating real-world applications such as robotic planning, complex scene understanding, and interactive AI systems requiring robust action sequence inference.

A plausible implication is that datasets like BIRD, which require modular perception-reasoning decompositions and provide exhaustive event sequence annotations, will influence the design of next-generation neural-symbolic learning and planning systems.

7. Significance in the Context of Symbolic and Neural Reasoning

BIRD establishes a standard for IES at the intersection of computer vision, classical planning, and logical reasoning. Through its emphasis on:

• Realistic visual input and unconstrained spatial arrangements,
• Exhaustive labeling of minimal action sequences,
• Explicit support for causal and temporal abstraction,
• Rigorous benchmarking of modular and end-to-end methods,

BIRD reveals key limitations of monolithic end-to-end systems, underscores the strengths of modular approaches (especially logic-informed learners), and operationalizes the concept of inductive generalizability. Its design principles and challenges are increasingly reflected in more recent planning and reasoning benchmarks, such as multi-step reasoning tasks addressed using Monte Carlo Tree Search–based methods (Gao et al., 2 Oct 2024), indicating its enduring relevance for research in interpretable and effective reasoning from real-world visual data (Gokhale et al., 2019).