MaskedSpatialViLT: Object-Centric Spatial Reasoning

• MaskedSpatialViLT is a variant of SpatialViLT that isolates object-specific regions with masks to capture nuanced spatial relationships.
• It integrates depth, 3D coordinates, and edge maps from segmented areas to enhance directional and topological reasoning metrics.
• The model’s object-centric approach refines multimodal embeddings, aiding applications in robotics, augmented reality, and visual question answering.

MaskedSpatialViLT is an architectural variant within the SpatialViLT framework, specifically designed to enhance visual spatial reasoning in multimodal vision-language transformers by fusing spatial features extracted from object-masked regions. Unlike conventional approaches that aggregate spatial information over entire images, MaskedSpatialViLT isolates and concentrates on object-specific areas using masks generated from object segmentation, allowing the model to capture nuanced spatial relationships that are often diluted when background or global image features dominate. This methodology was introduced to address the challenges faced by VLMs in interpreting complex spatial relations in 3D scenes and intricate object configurations, with a particular focus on directional, topological, and proximity relations in the context of the Visual Spatial Reasoning (VSR) benchmark (Islam et al., 3 Oct 2025).

1. Architectural Foundations and Motivation

MaskedSpatialViLT extends the base ViLT architecture, which itself is characterized by direct patch-based visual tokenization without convolutional backbones or region-based supervision (Kim et al., 2021). ViLT’s core processing represents images as non-overlapping patches, projects each patch into a unified embedding space (via $v_i \in \mathbb{R}^{P^2 C}$ mapped to $\bar{v}_i = v_i \cdot V$), and concatenates these with language tokens for holistic transformer-based multimodal reasoning. MaskedSpatialViLT adapts this pipeline by exploiting explicit object masks to restrict spatial feature extraction to semantically relevant regions, thereby retaining and amplifying spatial cues that are crucial for context-dependent reasoning.

The underlying motivation is to overcome the limitations of global spatial pooling, which may obscure object boundaries, internal structure, and relational attributes necessary for spatial queries such as “is object A adjacent to object B” or “is object C to the left of object D.” By focusing on masked regions, the model avoids background confusion and more accurately models spatial interactions.

2. Spatial Feature Extraction within Object Masks

Central to MaskedSpatialViLT is the integration of spatial modalities—depth, 3D coordinates, and edge maps—extracted exclusively from regions defined by object masks. The segmentation masks are generated using models such as CLIPSeg, guided by image-caption pairs to ensure alignment with semantic targets. Once masks are computed, the following spatial extraction procedures are performed within those regions:

• Depth Maps: Computed via MiDaS or equivalent depth estimation networks, yielding per-pixel depth $z$ for masked regions.
• 3D Coordinates: Derived from camera projection equations,

$$x = (u - c_x) \cdot z / f_x,\qquad y = (v - c_y) \cdot z / f_y$$

where $(u, v)$ are pixel indices, $(c_x, c_y)$ camera center, $f_x, f_y$ focal lengths, and $z$ depth.

• Edge Maps: Created using the Canny edge detection algorithm, further refining region boundaries and providing geometric detail.

All spatial features are computed strictly within the confines of the object mask, reinforcing the object-centric inductive bias.

3. Multi-Task Learning Framework for Spatial Reasoning

MaskedSpatialViLT employs a multi-task learning protocol, with a combined loss function: $$\mathcal{L}_{\text{total}} = \mathcal{L}_c + \lambda_d \mathcal{L}_d + \lambda_r \mathcal{L}_r + \lambda_e \mathcal{L}_e$$ where $\mathcal{L}_c$ is the main spatial relation classification loss, and $\mathcal{L}_d$, $\mathcal{L}_r$, $\mathcal{L}_e$ are auxiliary reconstruction losses for depth, 3D coordinates, and edges, respectively. The auxiliary tasks require the model, from its joint multimodal embeddings, to reconstruct the masked spatial features; this enforces spatial priors and regularization in the embedding space.

During training, the masked region features and corresponding textual inputs (e.g., captions describing spatial configurations) are simultaneously processed, with visual tokens derived only from masked spatial maps. This drives the model to internalize object-level spatial attributes, improving its ability to answer fine-grained spatial reasoning questions.

4. Spatial Reasoning Capabilities and Empirical Performance

MaskedSpatialViLT has been empirically evaluated on the Visual Spatial Reasoning (VSR) dataset, which includes over 10,000 image-caption pairs covering a taxonomy of spatial relations: directional (e.g., left/right), topological (e.g., adjacent, inside), proximity, and orientation. Experimental results indicate:

Meta-Category	MaskedSpatialViLT Accuracy
Topological	75.90%
Directional	~68.52%

The model demonstrates superior performance in topological reasoning, attributable to its focused attention on masked object boundaries where adjacency and attachment relations manifest. For directional reasoning, the isolated features (such as object centroid and edges within masks) provide robust geometric cues, supporting precise answers to questions involving left/right or above/below relations. A plausible implication is that this object-centric masking offers significant advantages in reasoning categories where background context or global pooling would otherwise interfere with discrimination of object-to-object relations.

In comparison, the unmasked SpatialViLT variant excels in proximity and orientation due to access to broader contextual information but may underperform when localization or internal object geometry dominates the reasoning requirement. The SpatialEnsemble approach, which combines both MaskedSpatialViLT and SpatialViLT, achieves the highest aggregate accuracy by leveraging complementary strengths in spatial representation.

5. Impact on Multimodal Embedding and Downstream Tasks

By restricting spatial feature computation to masked regions, MaskedSpatialViLT produces embeddings that are highly discriminative, object-aware, and less susceptible to background noise. This has direct implications for VLM robustness in multimodal reasoning:

• Improved localization and spatial quantification: Embeddings encode object position, extent, and contour fidelity, enhancing downstream tasks like visual question answering, image retrieval, and semantic search.
• Enhanced object interaction modeling: Relations such as adjacency or containment are more accurately captured due to preserved object boundaries.
• Transferability to robotics and augmented reality: Precise spatial reasoning is critical for scene understanding in robotics navigation and AR overlays; MaskedSpatialViLT’s embeddings provide the necessary spatial fidelity.

Applications can include robot path planning (where obstacle boundaries must be localized), AR systems (accurate overlay on segmented objects), and intelligent surveillance (focus on relational attributes in complex scenes).

6. Relation to Prior Work and Theoretical Significance

MaskedSpatialViLT synthesizes principles from several prior works:

• ViLT’s convolution-free patch tokenization and transformer cross-modal fusion (Kim et al., 2021).
• Multi-task learning for spatial feature reconstruction, as found in auxiliary self-supervised vision tasks.
• Object-centric reasoning from segmentation-guided models (e.g., CLIPSeg, as used here).

Compared to approaches that globally pool spatial features or apply region supervision at the level of bounding boxes, MaskedSpatialViLT’s mask-guided pipeline achieves finer granularity and more faithful spatial representations. This suggests a paradigm shift in multimodal transformers, from region-agnostic to object-focused spatial reasoning.

7. Limitations and Future Prospects

While MaskedSpatialViLT shows strong performance in object-centric spatial categories, its relative effectiveness may decline in tasks where relationships require global context (e.g., holistic orientation or scene-level proximity). Ensemble strategies or hybrid masking approaches may further improve overall accuracy by balancing local and global spatial features.

Future research directions include:

• Incorporating learned mask generation to adaptively optimize mask fidelity.
• Exploring self-supervised mask discovery for unlabeled or ambiguous scenes.
• Extending the approach to dynamic spatial reasoning in video sequences.

The framework provides a foundation for developing spatially intelligent vision-LLMs capable of nuanced multimodal understanding and reasoning across diverse domains.