ProCap: Projection-Aware Captioning for Spatial Augmented Reality

Abstract: Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard Vision LLMs (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated segmentation; then it uses region-aware retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP dataset are available on the project page: https://ZimoCao.github.io/ProCap/.

• The paper presents ProCap, a dual-stage pipeline that decouples scene and projection semantics via segmentation and region-aware retrieval.
• It introduces the RGBP benchmark with 65 physical scenes and over 180K overlays, providing dual-caption annotations for both physical and projected content.
• Experimental results demonstrate ProCap’s significant performance gains over standard VLMs in both seen and unseen SAR scenarios.

ProCap: Projection-Aware Captioning for Spatial Augmented Reality

Motivation and Problem Statement

Spatial Augmented Reality (SAR) utilizes projection mapping to overlay digital content directly onto physical environments, enabling applications across design, manufacturing, exhibition, and interaction. However, semantic scene understanding in SAR environments remains an unsolved problem for vision-LLMs (VLMs). Standard VLMs fail to disambiguate between the physical scene and virtual projections, frequently conflating the two, especially under geometric and photometric distortions. This ambiguity results in hallucinated or merged image descriptions, undermining critical downstream tasks like spatial reasoning and instruction following.

RGBP: The First Large-Scale SAR Semantic Benchmark

Addressing the lack of appropriate datasets, the authors introduce RGBP, a large-scale SAR benchmark containing 65 distinct physical scenes and more than 180,000 projected overlays, each paired with dense, decoupled annotations. Data acquisition leverages a hemispherical projector-camera setup and variable lighting and geometry configurations to maximize environmental diversity.

Figure 1: Configuration of the RGBP dataset capture environment, illustrating the complex capture setup for varying projections and physical layouts.

The dataset is uniquely structured to facilitate the semantic isolation of projections and physical content. For every instance, RGBP provides (1) a segmentation mask delineating projected regions, and (2) dual ground-truth captions—one for the physical scene, and one for the projected content.

Figure 2: Four representative RGBP training/evaluation scenes, including projection masks for virtual content decoupling (see color-highlighted boxes).

A dual-captioning evaluation protocol is formalized, prompting models to separately describe physical and projected layers, enabling fine-grained assessment of model performance on each layer.

ProCap: Decoupling Scene and Projection Semantics

The core contribution is ProCap, a modular projection-aware captioning pipeline designed to resolve virtual-physical ambiguity. The architecture is fundamentally a two-stage pipeline:

1. Segmentation: An automated module segments projection regions from the physical scene, providing coarse binary masks.
2. Region-Aware Retrieval: Isolated regions are encoded and used to query an external semantic database (e.g., LVIS-derived visual-name memory) to retrieve clean object names which compensate for severe perceptual distortions.

   Figure 3: Pipeline of the ProCap architecture showing segmentation, region isolation and feature retrieval feeding into a dual-captioning decoder.

Both scene and projection features are processed by separate Q-Formers. For projections, semantic retrieval provides strong priors to the LLM, suppressing errors due to ambiguous or degraded input. The final frozen LLM decoder receives fused, task-conditioned visual-semantic embeddings and produces independent captions for each region.

Experimental Validation and Numerical Results

Comprehensive evaluation on the RGBP dataset demonstrates that ProCap outperforms strong off-the-shelf and fine-tuned VLM baselines in both seen and unseen scenes, for both physical and projection descriptions. The dual-captioning protocol reveals large performance gaps in standard VLMs due to unresolved virtual-physical ambiguity.

Key findings include:

• On seen scenes: ProCap with TinyLlama-1.1B backbone achieves CIDEr scores of 70.27 (scene) and 54.37 (projection) for COCO-derived projected content, dramatically higher than FastVLM-7B’s 2.31 and 7.65, respectively. When fine-tuned on RGBP, Qwen3-VL-8B-Instruct exceeds 120 CIDEr on the projection task, confirming dataset necessity.
• On unseen scenes: ProCap’s region-aware retrieval module generalizes robustly: Vicuna-1.5-7B backbone reaches 86.26 CIDEr for projections on COCO, compared to FastVLM-7B’s 9.7. Scene captioning metrics are lower but remain above all non-specialized baselines.
• Ablations confirm that segmentation, feature refinement, and retrieval augmentations all provide statistically significant boosts in both CIDEr and SPICE across domains.

  Figure 4: Qualitative comparisons between ProCap and FastVLM showing improved decoupling and semantic precision in challenging SAR scenarios.

  

  Figure 5: Scene library examples demonstrating the scale and variety of physical/projection content pairings for robust model training and evaluation.

Implications and Open Challenges

This work enables practical SAR agents to perform context-aware, instruction-driven visual reasoning in mixed-reality scenarios, addressing a key bottleneck for multimodal embodied AI. The modular design allows ProCap to function as a specialized expert agent within Mixture-of-Experts (MoE) architectures, being invoked only when SAR-specific context is detected.

The RGBP dataset offers a scalable paradigm for text-to-SAR synthesis and simulation tasks, encouraging research on disentangled conditional generation and physics-consistent scene relighting. The dual-captioning annotation schema will catalyze the development of frameworks that require independent control of scene and projection semantics—critical for editing, interactive applications, and simulation.

Figure 6: Real-world projection mapping results using ProCap, illustrating robust understanding and decoupling under challenging acquisition conditions.

Limitations remain regarding segmentation (boundary ambiguities in demanding lighting or surface geometries) and out-of-distribution projections (narrow knowledge base coverage). Future research will benefit from more granular, volumetric annotations and expansion to novel semantic domains.

Conclusion

ProCap and RGBP collectively establish the foundation for robust, projection-aware semantic understanding in SAR environments. The dual segmentation–retrieval mechanism enables high-fidelity, disentangled captioning. The protocol and dataset significantly raise the bar for SAR scene understanding, pointing toward practical, multimodal assistants capable of intelligent, context-aware operation in physical-virtual blended environments.