Mystery Blocksworld Event Sequencing

• Mystery Blocksworld is a testbed for studying image-based symbolic state abstraction and event sequencing, transforming visual scene changes into planning problems.
• The framework employs a modular two-stage architecture that separates visual perception from symbolic event sequencing using methods like FC, QL, and ILP to enhance performance.
• Empirical evaluations on the BIRD dataset demonstrate that logic-based planners, particularly ILP, achieve high accuracy and robust inductive generalization even with extended action sequences.

Mystery Blocksworld constitutes a testbed for studying the joint problem of image-based symbolic state abstraction and action-sequence generation. The central task, Image-based Event-Sequencing (IES), requires inferring the minimal sequence of block moves needed to rearrange objects depicted in a source image into a configuration present in a target image. The domain operationalizes core challenges in machine perception, temporal reasoning, and inductive generalization by recasting visual scene transformation as a planning problem over discrete, symbolic states induced from raw images (Gokhale et al., 2019).

1. Formal Model of Image-based Event-Sequencing in Blocksworld

In the IES formulation, the state space $S$ comprises Blocksworld configurations parameterized by arrangement and color vectors. The arrangement vector $A \in \{0,1\}^{5 \times 5}$ specifies which grid cell contains which block, and the color vector $C \in \{0,1\}^{5 \times 3}$ encodes the 3-bit color of up to five blocks in a specified ordering. Thus, any state $s \in S$ is represented as $s = (A, C)$.

The action space consists of atomic moves $m_t = \mathrm{move}(X,Y,t)$, where $X \in \mathcal{C}$ denotes the color of the manipulated block, $Y \in \mathcal{D}$ is the destination (another block, the table, or "out"), and $t$ is the discrete timestep. There are $|\mathcal{C}|=6$ colors and $A \in \{0,1\}^{5 \times 5}$0 valid destinations, encoding each action as a 16-bit one-hot vector.

The transition function $A \in \{0,1\}^{5 \times 5}$1 deterministically advances the state: $A \in \{0,1\}^{5 \times 5}$2. This is implemented by a logic engine $A \in \{0,1\}^{5 \times 5}$3 that enforces domain-specific axioms (exogeneity, freedom, inertia, and sequentialism), ensuring that actions are feasible and respect the natural constraints of block manipulation.

The IES instance is defined by a pair of images $A \in \{0,1\}^{5 \times 5}$4. The system extracts the corresponding states $A \in \{0,1\}^{5 \times 5}$5 and $A \in \{0,1\}^{5 \times 5}$6, with the objective of finding a plan $A \in \{0,1\}^{5 \times 5}$7 of minimal length such that sequential updates reach the goal: $A \in \{0,1\}^{5 \times 5}$8. Inductive generalizability is assessed by testing whether systems trained on sequence lengths up to $A \in \{0,1\}^{5 \times 5}$9 can correctly solve test instances with $C \in \{0,1\}^{5 \times 3}$0.

2. Blocksworld Image Reasoning Dataset (BIRD)

BIRD underpins empirical investigation of IES in the Mystery Blocksworld setting. It contains 7,267 real photographs of wooden blocks (six colors, up to five blocks per image, no color repetition) arranged on a white tabletop under controlled illumination. Every possible arrangement up to five blocks and every color permutation are represented, with each image labeled with explicit $C \in \{0,1\}^{5 \times 3}$1 vectors.

All ordered pairs $C \in \{0,1\}^{5 \times 3}$2 are coupled, and for each pair, minimal-length valid move sequences $C \in \{0,1\}^{5 \times 3}$3 (maximum length 8) are generated using the ASP solver Clingo. Approximately one million $C \in \{0,1\}^{5 \times 3}$4 triplets are uniformly sampled across all sequence lengths $C \in \{0,1\}^{5 \times 3}$5.

State transitions adhere to background axioms:

• Exogeneity: blocks exist unless explicitly moved "out."
• Freedom: a block can only be manipulated if uncovered; placements are allowed only on free blocks or the table.
• Inertia: unacted blocks remain stationary.
• Sequentialism: only one move is permitted per timestep.

3. Modular Two-Stage Solution Architecture

A modular pipeline outperforms end-to-end vision-to-sequence models for IES. This architecture separates visual grounding from event-sequencing:

Stage 1: Visual Perception

Encoders $C \in \{0,1\}^{5 \times 3}$6 and $C \in \{0,1\}^{5 \times 3}$7 predict arrangement $C \in \{0,1\}^{5 \times 3}$8 and color $C \in \{0,1\}^{5 \times 3}$9 respectively from images.

• $s \in S$0: an 8-layer CNN producing a $s \in S$1 grid output.
• $s \in S$2: a ResNet-50-based module extracting color bits.

Corresponding loss functions are cross-entropy for arrangement (over 25 grid cells) and binary cross-entropy for color (15 bits).

Stage 2: Event Sequencing

Given symbolic representations $s \in S$3 and $s \in S$4, three approaches are compared:

• Fully Connected Network (FC): 5-layer MLP predicts the 128-bit action sequence, optimizing binary cross-entropy loss.
• Q-Learning (QL): Trains a policy over MDP tuples $s \in S$5, maximizing discounted reward to the goal.
• Inductive Logic Programming (ILP): Learns ASP rules using $s \in S$6 and axioms, e.g., $s \in S$7.

A logic engine $s \in S$8 maintains all state transitions in event-sequencing.

4. Training and Objective Functions

End-to-end baselines (ResNet-50, PSPNet, Relational Network – RN) attempt to map image pairs directly to bitwise-encoded action sequences via multi-label binary cross-entropy.

Stage-wise architectures are trained as follows:

• Stage 1: arrangement and color classification losses.
• Stage 2, FC: BCE over the 128 output bits.
• Stage 2, QL: temporal-difference loss,

$s \in S$9

• ILP: no numeric loss; instead, symbolic entailment is optimized.

5. Empirical Evaluation and Results

Performance is quantified by:

• Full Sequence Accuracy (FSA): proportion of test cases with exactly correct predicted action sequence.
• Step-Level Accuracy (SLA): average per-step match over all cases and timesteps.

Method	FSA (%)	SLA (%)
End-to-end (ResNet50, PSPNet, RN)	30–35	36–57
Perfect Perception + FC	68.9	72.6
Perfect Perception + QL	84.1	87.8
Perfect Perception + ILP	100.0	100.0
Encoded Perception + FC	56.3	60.2
Encoded Perception + QL	69.0	71.2
Encoded Perception + ILP	83.6	88.5

Perfect-perception plus ILP achieves 100% on both metrics. End-to-end models exhibit significant performance degradation, especially for longer sequences.

Inductive generalization is demonstrated in cases where test sequence lengths exceed those in training. End-to-end FSA drops to 0% for $s = (A, C)$0, while two-stage models (notably Perfect Perception + ILP) retain high accuracy across all tested lengths.

6. Extensions to Natural Images

Transfer to natural images is assessed by remapping Mask-RCNN detection outputs onto the block grid and color format. When symbolic mappings are perfect, Stage 2 (trained on BIRD) with ILP achieves up to 100% FSA and SLA. Using Mask-RCNN perception, FSA is 75.6% and SLA is 80.6%. This suggests feasibility for real-world event-sequencing given sufficiently robust perception modules.

7. Limitations and Prospective Research Directions

BIRD and the Mystery Blocksworld framework exhibit several controlled constraints: small numbers of blocks, fixed color palette, white backgrounds, and moves limited to pick-and-place with grid alignment and minimal occlusion. Dynamics such as sliding and rotation are absent. Expansion paths include increasing visual and physical diversity (continuous poses, lighting, textures, open-world objects), expanding action vocabularies (e.g., push, rotate), and supporting multi-agent or concurrent manipulations. Reinforcing deeper inductive biases within neural planners is identified as a critical trajectory for achieving robust and generalizable IES systems (Gokhale et al., 2019).

In summary, the Mystery Blocksworld domain illustrates the importance of separating symbol grounding from symbolic planning. The use of logic-based planners (ILP), in concert with robust perception, achieves both high task performance and superior inductive generalization versus end-to-end neural architectures. This framework provides a rigorous empirical foundation for research at the intersection of visual reasoning, symbolic learning, and cognitive induction.