VOIC: Visible-Occluded Interactive Completion

• VOIC is a family of architectures that decouples visible and occluded scene reasoning to reconstruct unobservable content.
• It employs dual decoders and interactive modules for iterative 3D semantic completion, layered scene decomposition, and pose estimation.
• Experimental evaluations show significant performance gains on metrics like IoU, mAP, and PSNR compared to prior methods.

The Visible-Occluded Interactive Completion Network (VOIC) denotes a family of architectures designed for joint inference of visible and occluded content in visual scenes, with instantiations in dense 3D scene completion, layered scene decomposition, and visible-occluded body joint estimation. Such systems are characterized by explicit decoupling of visible and occluded region reasoning, iterative or staged completion mechanisms, and cross-modal or cross-branch interaction for enhanced scene understanding in physically realistic or ambiguous conditions.

1. Conceptual Framework and Problem Definition

VOIC networks address core visual inference tasks where significant portions of the scene may be unobservable due to occlusion or sensor limitations. The central objective is to reconstruct or semantically interpret both the visible and occluded components of the scene, frequently using only monocular or partial sensory input. The signature innovation is the architectural separation—often via dedicated decoder branches or iterative modules—between the high-confidence, directly observed ("visible") subset and the low-confidence, inferred ("occluded") subset, coupled with interaction channels or shared context to propagate cues bi-directionally between these domains. In recent work, this paradigm has been formalized for:

• Monocular 3D Semantic Scene Completion (SSC), where the goal is dense voxelwise labeling and occupancy prediction in both visible and occluded spatial regions from a single camera input (Han et al., 22 Dec 2025).
• Completed Layered Scene Decomposition, involving multi-instance segmentation, hierarchical occlusion ordering, and generative completion of occluded object parts in 2D images (Zheng et al., 2021).
• Large-scale pose estimation and tracking, where VOIC-style networks concurrently detect visible joints and hallucinate positions of occluded joints, leveraging multi-stage visual and temporal cues (Fabbri et al., 2018).

2. Representative VOIC Architectures

a. Dual-Decoder Paradigm for Monocular SSC

The VOIC implementation for monocular 3D SSC (Han et al., 22 Dec 2025) features:

• Visible Region Label Extraction (VRLE): An offline process that constructs a binary 3D visibility mask $\mathbf M_{\rm vis}$ by projecting annotated voxel corners into the image and applying Z-buffer visibility tests. This yields $$\mathbf Y_{\rm vis} = \mathbf Y \odot \mathbf M_{\rm vis}$$ as supervision exclusively for visible voxels, while full $\mathbf Y$ is reserved for occlusion decoding.
• View-Enhanced Feature Construction (VEFC): Fuses 2D image features $\mathbf F_{2D}$ with predicted depth-derived occupancy $\mathbf M$ into a base 3D volumetric feature $\mathbf F_{\rm base}$ using Deformable Attention:

$$\mathbf F_{\rm voxel} = \mathrm{DeformAttn}(\mathbf Q_{\rm content} + \mathbf S_{\rm pos},\, \mathcal P(\mathbf S_{\rm pos}),\, \mathbf F_{2D})$$

• Visible Decoder (VD): Consumes $\mathbf F_{\rm base}$ (masked to visible voxels), instance queries, and image features, employing cross-attention and instance-level context to predict $\hat{\mathbf Y}_{\rm vis}$ and guide occlusion reasoning.
• Occlusion Decoder (OD): Ingests VD-predicted priors and updated image/instance features to deliver full-scene prediction $\hat{\mathbf Y}$, with global context propagated back to VD.

b. Iterative Decomposition and Completion Loop

The approach in completed scene decomposition (Zheng et al., 2021) alternates segmentation/occlusion-inference and image-completion modules. At each iteration, fully-visible instances are segmented and masked out, rendering a "missing" region for the completion module (PICNet-based) to inpaint. This iterative process yields layer-by-layer amodal segmentation, explicit depth ordering, and hallucinated completions for fully occluded regions via repeated application until all objects are inventoried.

c. Four-Branch Interactive Completion for Body Joint Estimation

For pose estimation in crowded/occlusive scenes (Fabbri et al., 2018), VOIC corresponds to a four-branch multi-stage CNN:

• Visible heatmaps (per-joint)
• Occluded heatmaps (hallucinated joint positions)
• Part Affinity Fields (spatial linkage)
• Temporal Affinity Fields (motion linkage across frames)

Explicit supervision masks and stage-wise feature concatenation enable branches to update and regularize each other iteratively, yielding robust visible and occluded pose estimates across single images and short video clips.

3. Mathematical Formulation, Supervision, and Losses

Central to VOIC frameworks is the decoupled supervision scheme:

Component	Supervision Target	Loss Structure
Visible Decoder / Branch	Masked visible ground truth ($\mathbf Y_{\rm vis}$, visible joints)	Cross-entropy, mIoU, geometric loss (Han et al., 22 Dec 2025); masked squared error (Fabbri et al., 2018)
Occlusion Decoder / Branch	Full scene ground truth ($\mathbf Y$) including invisible content	Same as above, but on unmasked/full regions

Bidirectional cross-modal attention or stage-wise feature sharing reinforces mutual information. Typical total loss for the dual-decoder (Han et al., 22 Dec 2025):

$$\mathcal L_{\rm total} = \mathcal L_{\rm VD} + \mathcal L_{\rm OD}$$

Each $\mathcal L_X$ (with $X\in\{\rm VD, OD\}$) includes weighted semantic, geometric, and cross-entropy terms, e.g.:

$$\mathcal L_X = \lambda_{\rm scal}\,\mathcal L_{\rm scal}^{\rm geo} + \lambda_{\rm ce}\,\mathcal L_{\rm ce} + \lambda_{\rm miou}\,\mathcal L_{\rm miou}$$

Stage-wise squared-error losses for pose estimation incorporate explicit masking, preventing penalization of the visible branch on jointly occluded locations (Fabbri et al., 2018).

4. Experimental Protocols and Quantitative Evaluation

VOIC systems are evaluated using application-appropriate metrics:

• SSC: Intersection-over-Union (IoU), mean IoU (mIoU) over all classes (Han et al., 22 Dec 2025), with benchmark datasets including SemanticKITTI and SSCBench-KITTI360.
• Layered Scene Decomposition: Amodal segmentation AP on synthetic and real datasets, depth ordering accuracy (Occlusion-AP), and completion image quality (PSNR, SSIM, RMSE) (Zheng et al., 2021).
• Pose Tracking: Keypoint mean Average Precision (joint-mAP), Multi-Object Tracking Accuracy (MOTA), and additional tracklet consistency measures (Fabbri et al., 2018).

Selected results:

Dataset / Task	Metric	VOIC Perf.	Best Prior
SemanticKITTI (SSC)	IoU	45.69 %	CGFormer 44.41 %
SSCBench-KITTI360 (SSC)	mIoU	21.37 %	CGFormer 20.05 %
Synthetic CSD (Amodal Seg.)	Mask AP	50.3 %	HTC 47.9 %
Synthetic CSD (Completion)	PSNR/SSIM	30.45/0.877	SeGAN-V: 16.0/0.60
JTA (Body Joint Detection)	joint-mAP	59.3	w/o occlusion 50.9

Ablation studies consistently demonstrate that explicit decoupling of visible/occluded reasoning with cross-modal or interactive feedback yields substantial gains over end-to-end or single-branch models.

5. Data, Training Regimes, and Implementation

Typical VOIC pipelines leverage either synthetic or real-world datasets with granular occlusion annotations:

• VOIC for SSC (Han et al., 22 Dec 2025): SemanticKITTI and SSCBench-KITTI360, RGB+predicted depth, with VRLE for ground-truth decoupling; ResNet-50/MobileStereo backbone; AdamW optimization; multi-step scheduler; batch size 1; up to 256×256×32 voxel grids.
• Completed Scene Decomposition (Zheng et al., 2021): CSD synthetic dataset derived from SUNCG; paired RGBA + depth; RPN-driven segmentation; dual backbones; SGD/Adam.
• Pose Estimation (Fabbri et al., 2018): JTA dataset synthesized from GTA-V; VGG-19 backbone; staged training; domain gap addressed through photorealism and augmentation.

Data augmentation, domain transfer via pseudo ground-truth, and separate training phases for segmentation/completion are common. Inference is typically efficient (<0.3s/frame for SSC (Han et al., 22 Dec 2025); 25–65ms/frame for pose estimation (Fabbri et al., 2018)).

6. Limitations, Challenges, and Future Directions

Documented limitations across VOIC implementations include:

• Domain gap: Synthetic-to-real adaptation is only partially alleviated through pseudo-labeling or photometric fine-tuning (Zheng et al., 2021, Han et al., 22 Dec 2025).
• Cascaded error accumulation: Early-stage errors in segmentation or VD propagate to occlusion reasoning and completion (Zheng et al., 2021).
• Representation granularity: Challenging scene hierarchies with numerous small occluders or cast shadows remain difficult (Zheng et al., 2021).
• Temporal consistency: Dynamic or complex motion remains a challenge, particularly in crowded tracking.

Proposed research directions:

• Integrate differentiable rendering and explicit shadow models for more physically accurate reconstructions (Zheng et al., 2021).
• Extend architectures to richer 3D representations (e.g., explicit 6-DoF pose, mesh-level completion) (Zheng et al., 2021).
• Employ transformer-based or cross-scale global inpainting modules for enhanced long-range dependency modeling (Zheng et al., 2021).
• Quantify and propagate uncertainty estimates with human-in-the-loop correction (Zheng et al., 2021).

Future VOIC systems are expected to generalize to a broader range of occluded scene reconstruction and reasoning tasks in both 2D and 3D, across diverse domains such as robotics, autonomous navigation, augmented reality, and long-term object/object part tracking.

---

References

• "VOIC: Visible-Occluded Decoupling for Monocular 3D Semantic Scene Completion" (Han et al., 22 Dec 2025)
• "Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition" (Zheng et al., 2021)
• "Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World" (Fabbri et al., 2018)