Self-Occlusion Optimization Strategy

• Self-occlusion optimization strategy is a set of methods that explicitly manage loss of observability caused by an object’s own geometry, enhancing performance in classification, depth, and reconstruction tasks.
• Techniques include decoupled feedback, occlusion masking, and explicit representation to correct supervisory signal errors and recover occluded information in various imaging frameworks.
• Empirical results demonstrate substantial gains, such as improved stereo matching accuracy and reduced error metrics, underscoring the practical benefits of integrating occlusion-aware supervision in vision pipelines.

Self-occlusion optimization strategy denotes a class of methods that explicitly manage the loss of observability created when parts of an object, scene, or robot block other parts from a camera, sensor, or virtual viewpoint. In the cited literature, self-occlusion appears as fixed epipolar information loss in direct self-supervised stereo, corrupted photometric supervision in monocular depth from video, conservative recoverability assumptions in non-line-of-sight imaging, multi-refraction ambiguity in transparent-object reconstruction, missing topology in tree and cloth reconstruction, degraded pose and scene-flow estimates, and reduced perception quality or task success in embodied systems (Yang et al., 2024, Schellevis, 2019, Grau et al., 2022, Li et al., 2023, Huang et al., 2022, He et al., 2022).

1. Problem setting and canonical failure modes

Self-occlusion, in the TEOS formulation, is the loss of observable object surface due to the object’s own geometry when viewed from different perspectives. TEOS was designed to investigate the role of self-occlusion in object classification, and its baseline evaluations show that classification accuracy decreases monotonically as self-occlusion increases; at higher occlusion rates $>85\%$, accuracy drops to near-chance, and certain viewpoint regions yield higher classification scores—up to $68\%$ better in some cases (Solbach et al., 2021).

In direct self-supervised stereo matching, the failure mode is more specific. The crux of the occlusion challenge is that the positions of occluded pixels consistently align with the epipolar search direction defined by the input stereo images, so persistent information loss and erroneous feedback recur at fixed locations during self-supervised training. In self-supervised single-view depth estimation from video, occlusion instead refers to parts of a scene that can be observed in the target frame but not in a frame used for image reconstruction; reconstructed occluded areas therefore corrupt the supervisory signal (Yang et al., 2024, Schellevis, 2019).

Other domains expose analogous but not identical pathologies. In transparent-object reconstruction, previous refraction-tracing methods assume that every ray passing through a solid transparent object refracts exactly twice, but self-occluded rays may intersect the surface more than twice, producing incorrect or conflicting gradients and unstable optimization. In non-line-of-sight reconstruction, the commonly employed Fermat path criterion is described as rather conservative, because it classifies some surfaces as unrecoverable although they contribute to the signal (Li et al., 2023, Grau et al., 2022).

A recurring implication is that self-occlusion is not merely a nuisance variable. In the cited work, it changes the effective optimization landscape by making some supervisory signals systematically wrong, some geometry partially unrecoverable under naive criteria, and some control objectives incompatible unless visibility is explicitly modeled.

2. Decoupled feedback, masking, and occlusion-aware losses

A prominent strategy is to change the supervision pathway rather than only the prediction model. In "Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching" (Yang et al., 2024), the core intervention is to decouple the input stereo images from the feedback stereo images. The original right view is replaced with a pseudo-image $\widetilde{I}^R$ generated from the current network’s disparity predictions, while photometric feedback is always computed against original images. Because the selection of which view acts as input is randomized, the network probabilistically samples information from both sides of an occluding object, with $p=0.5$ for each configuration. The corresponding photometric loss is

$$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$

where $pe(\cdot,\cdot)$ combines SSIM and $L_1$ loss with $\alpha=0.85$, and $\widetilde{O}$ masks out occluded pixels. On KITTI 2015, the reported test result is $D1\text{-all}=4.68\%$, versus $68\%$0 for PASMNet and $68\%$1 for UHP; in ablation, the base method gives $68\%$2, pseudo-stereo only improves this to $68\%$3 but exhibits overfitting, and pseudo-stereo plus occlusion masking plus improved pseudo-image rendering plus fully pseudo-stereo reaches $68\%$4 and $68\%$5 with stable training (Yang et al., 2024).

A second line of work keeps the usual view-synthesis framework but explicitly suppresses gradients from occluded regions. In "Improving Self-Supervised Single View Depth Estimation by Masking Occlusion" (Schellevis, 2019), the occlusion mask is constructed entirely from predicted depth: $68\%$6 with tolerance set to $68\%$7. Two losses are introduced: $68\%$8 and

$68\%$9

On the KITTI validation split, the non-occluded minimum reprojection loss achieves $\widetilde{I}^R$0, $\widetilde{I}^R$1, $\widetilde{I}^R$2, $\widetilde{I}^R$3, and $\widetilde{I}^R$4, slightly improving on per-pixel min reprojection. The same paper also notes that per-pixel minimum reprojection can ignore some reprojection error caused by object motion, not only by occlusion (Schellevis, 2019).

The endoscopic setting introduces a more aggressive augmentation-based variant. "Occlusion-Aware Self-Supervised Monocular Depth Estimation for Weak-Texture Endoscopic Images" (Huang et al., 24 Apr 2025) overlays a random block mask of width and height equal to $\widetilde{I}^R$5 of image size on the target frame, so the mask covers $\widetilde{I}^R$6 of the target frame. The masked target frame and unoccluded source frame are concatenated and fed to DepthNet and PoseNet, and depth supervision is applied only to unoccluded regions: $\widetilde{I}^R$7 In ablation on SCARED, adding the MASK-based data augmentation module improves Abs-Rel from $\widetilde{I}^R$8 to $\widetilde{I}^R$9 (Huang et al., 24 Apr 2025).

Occlusion-aware supervision also appears outside monocular or stereo depth. LoLep defines an occlusion-aware reprojection loss for single-view view synthesis, where a target-view pixel is considered occluded in the source if $p=0.5$0 with $p=0.5$1, and penalizes only non-occluded reprojection error: $p=0.5$2 Against MINE, LoLep reports an LPIPS reduction of $p=0.5$3 and an RV reduction of $p=0.5$4 (Wang et al., 2023). In 3D point-cloud scene flow, 3D-OGFlow constructs an occlusion-weighted cost volume

$p=0.5$5

and combines non-occluded Chamfer loss, smoothness, synthetic flow loss, and synthetic occlusion loss in self-supervised training. On FlyingThings3D, self-supervised EPE improves from $p=0.5$6 for PointPWC-Net to $p=0.5$7, and supervised EPE improves from $p=0.5$8 to $p=0.5$9 (Ouyang et al., 2021).

3. Explicit representations of occluded structure

A second major family of strategies makes self-occlusion a first-class representational object. "Occlusion Fields: An Implicit Representation for Non-Line-of-Sight Surface Reconstruction" (Grau et al., 2022) models the NLoS surface as the decision boundary of a neural network that classifies points in 3D space as either wall-visible or occluded relative to the observed wall. For a point $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$0 and wall sensor positions $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$1, global occlusion is defined as

$$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$2

and is approximated by a neural classifier

$$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$3

The network is trained with binary cross entropy against ray-traced visibility, and surfaces are extracted by thresholding followed by marching cubes. The paper states that a surface patch is recoverable if it is directly visible from some point on the wall and occludes the space behind itself, and reports robustness to significant amounts of self-occlusion as well as recovery of features beyond the Fermat path criterion (Grau et al., 2022).

SO-Pose introduces a two-layer object representation for direct 6D pose estimation. The first layer is the standard 2D-3D correspondence field for visible surface points; the second is a self-occlusion map formed by intersecting each camera ray with the three object coordinate planes, yielding up to three points $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$4. The object is written as

$$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$5

where $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$6 is the set of visible 3D points and $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$7 encodes self-occlusion geometry. These maps are predicted by separate decoders and fused for direct 6DoF regression; on LineMOD, LMO, and YCB-V, the resulting framework is reported to surpass or rival state-of-the-art methods, and ablation shows clear drops when the cross-layer losses are removed (Di et al., 2021).

RiCS provides a 2D representation for volumetric self-occlusion. For each image pixel that hits the object surface, a set of sampled directions in camera space is tested for self-intersection, producing a binary tensor $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$8. In the reported configuration, $$L_p = \frac{1}{N} \sum_{ij} \left[ p \cdot pe(I_{ij}^L, I_{ij}^R) + (1-p) \cdot pe(I_{ij}^R, \widetilde{I}_{ij}^R) \right] \cdot \widetilde{O}_{ij},$$9 directions are sampled at $pe(\cdot,\cdot)$0 increments in azimuth and elevation. This self-occlusion map is concatenated with surface normals and albedo for harmonization, and the full method reports $pe(\cdot,\cdot)$1, $pe(\cdot,\cdot)$2, and $pe(\cdot,\cdot)$3, with stated improvements of $pe(\cdot,\cdot)$4, $pe(\cdot,\cdot)$5, and $pe(\cdot,\cdot)$6 over the best baseline on those metrics (Jang et al., 2022).

The same representational logic appears in prompt-based segmentation. S$pe(\cdot,\cdot)$7POT first generates an occlusion-free reference face, then contrasts encoder tokens from the raw and reference images, and finally forms positive and negative prompts by greedy matching and self-attention screening. The supervision uses three objectives—occlusion prompt recall, face prompt recall, and face prompt penalty—without occlusion ground-truth masks. Reported results are $pe(\cdot,\cdot)$8 IoU in standard training and $pe(\cdot,\cdot)$9 IoU in per-image overfit; ablation further states that using only one loss fails badly with IoU $L_1$0 (Wang et al., 31 Jan 2026).

MoDOT treats occlusion boundaries as a task that can explicitly sharpen depth. It jointly estimates depth and occlusion boundaries, uses CASM to leverage mid-level boundary features for depth prediction, and adds the Occlusion Boundary-Depth Constraint Loss

$L_1$1

$L_1$2

The reported outcome is state-of-the-art on the proposed synthetic datasets and NYUD-v2, with real-world transfer comparable to competitors while preserving sharp occlusion boundaries (Xu et al., 27 May 2025).

4. Completion, reconstruction, and geometry-constrained inference

Self-occlusion optimization often takes the form of hidden-structure completion. "Self-Supervised Scene De-occlusion" (Zhan et al., 2020) decomposes the problem into mask completion and content completion. PCNet-M is trained from overlaid modal masks using two cases: partial completion, where one object erases part of another and the network must recover the original mask, and a regularizing case where the second object is not an occluder and the network must not hallucinate new pixels. The overall mask loss is

$L_1$3

with $L_1$4. Progressive inference then performs ordering recovery, amodal completion, and content completion. Reported pairwise ordering accuracy is $L_1$5 on COCOA and $L_1$6 on KINS; amodal mask mIoU reaches $L_1$7 and $L_1$8, respectively (Zhan et al., 2020).

In tree-canopy skeleton extraction, visible RGB-D branch segments are converted into a 3D likelihood occupancy grid and missing structure is recovered by minimum-cost path search through occluded regions. The observed branch probability around a segment is

$L_1$9

the voxel update is

$\alpha=0.85$0

and graph-edge cost is

$\alpha=0.85$1

The proposed method outperforms MST and FTSEM under heavy occlusion, maintains precision $\alpha=0.85$2, and yields higher OSR as occlusion increases (Kim et al., 2023).

Cloth manipulation replaces partial observability with explicit mesh completion and test-time optimization. The predicted observation-space vertex is

$\alpha=0.85$3

and refinement uses a unidirectional Chamfer loss

$\alpha=0.85$4

together with a mapping consistency loss

$\alpha=0.85$5

The full system reports Normalized Improvement $\alpha=0.85$6, versus $\alpha=0.85$7 without finetuning, $\alpha=0.85$8 for VCD, $\alpha=0.85$9 for GarmentNets, and $\widetilde{O}$0 when ground-truth mesh is used with learned dynamics (Huang et al., 2022).

Transparent-object reconstruction and embodied pose tracking use different geometry but a similar logic: detect when standard assumptions break, then optimize around the failure. NeTO uses an implicit SDF and a self-occlusion-aware refractive ray tracing procedure that excludes rays from the refraction loss if any sampled point on the segment $\widetilde{O}$1 has positive SDF, indicating more than two refractions. The refraction loss is then

$\widetilde{O}$2

where $\widetilde{O}$3 contains only non-self-occluded rays. The paper reports that removing the self-occlusion check significantly increases errors, especially in problematic regions (Li et al., 2023). UNOC, by contrast, simulates egocentric occlusion patterns with ray casting and trains a GRU with position, kinematic, and occlusion losses: $\widetilde{O}$4 On occluded body joints, RMSJPE drops from $\widetilde{O}$5 cm to $\widetilde{O}$6 cm; on occluded fingers, it drops from $\widetilde{O}$7 cm to $\widetilde{O}$8 cm (Parger et al., 2020).

5. Active perception, control, and decision-making

Embodied systems convert self-occlusion from a passive nuisance into an active control variable. The Visibility Maximization Controller formulates line-of-sight preservation as a soft constraint in a quadratic program over joint, head, and base velocities. For each robot link, the rate of approach to the line of sight is constrained by a velocity damper,

$\widetilde{O}$9

while end-effector tracking, camera orientation, joint-limit avoidance, and other terms remain in the same optimization. The reported trade-off is explicit: in real moving-object tracking, occlusion drops from $D1\text{-all}=4.68\%$0 for the baseline to $D1\text{-all}=4.68\%$1 for VMC; in fixed-base multi-object tracking, it drops from about $D1\text{-all}=4.68\%$2 to $D1\text{-all}=4.68\%$3; in mobile manipulation, from $D1\text{-all}=4.68\%$4 to $D1\text{-all}=4.68\%$5. A planning-based baseline achieves lower occlusion rates, but with highly inefficient paths and a significant drop in task success (He et al., 2022).

Pad-AI addresses occlusion-aware autonomous driving through active perception. It encodes visible region polylines, lane lines, and agent history with a hierarchical GNN, acts over semantic motion primitives rather than low-level controls, and filters candidate actions through a prediction module and Responsibility-Sensitive Safety constraints before PPO updates. The framework is described as a self-reinforcing method that learns occlusion-aware decision-making through active perception and demonstrates efficient and general perception-aware exploration in challenging scenarios under both dynamic and static occlusions (Jia et al., 2024).

The TEOS analysis provides a related but more diagnostic result. Humans are described as handling self-occlusion by changing the viewpoint or manipulating the scene, whereas the tested CNNs are passive systems restricted to a single view. This suggests why active vision and next-best-view planning recur as proposed remedies whenever self-occlusion is treated as an information-acquisition problem rather than only as a post hoc inference problem (Solbach et al., 2021).

6. Empirical outcomes, misconceptions, and open tensions

Across the cited work, numerical gains are often substantial when self-occlusion is modeled explicitly rather than treated as noise or ignored. The reported improvements span supervised, self-supervised, inference-time, and control-time settings.

Setting	Strategy	Reported outcome
Direct self-supervised stereo (Yang et al., 2024)	Pseudo-stereo inputs, occlusion masking, improved pseudo-image rendering, fully pseudo-stereo	KITTI 2015 test $D1\text{-all}=4.68\%$6; ablation best $D1\text{-all}=4.68\%$7, $D1\text{-all}=4.68\%$8
Single-view view synthesis (Wang et al., 2023)	Occlusion-aware reprojection loss and BS-SA	LPIPS reduction $D1\text{-all}=4.68\%$9; RV reduction $68\%$00 versus MINE
3-D video coding (Gao et al., 2018)	Allowable depth distortions with dynamic programming	Up to $68\%$01 bitrate reduction
Robotic manipulation control (He et al., 2022)	Line-of-sight visibility as a soft QP constraint	Real moving-object tracking occlusion drops from $68\%$02 to $68\%$03
TEOS classification (Solbach et al., 2021)	Single-view CNN baselines under controlled self-occlusion	MobileNet-V2: $68\%$04 top-1 on $68\%$05, $68\%$06 on $68\%$07; high self-occlusion drives accuracy near chance

Several misconceptions are directly challenged. One is that heavily occluded regions are necessarily unrecoverable. Occlusion Fields argues that the Fermat path criterion is too restrictive and that a surface can be reconstructed if it is visible from at least one wall sensor and occludes the space behind itself (Grau et al., 2022). Pseudo-stereo reaches a similar conclusion in self-supervised stereo by showing that persistent occlusion is partly an artifact of the fixed coupling between input and feedback images rather than an absolute impossibility of learning those regions (Yang et al., 2024).

A second misconception is that occluded pixels can simply be ignored without further consequence. The 3-D video coding work explicitly rejects the common assumption that occlusion-inducing depth pixels are negligible because their associated texture pixels are occluded in the synthesized view. It formulates a joint rate-distortion objective over allowable depth distortions and solves it with dynamic programming while preserving occlusion order, reporting the largest BDBR reduction among compared methods and gains that become more pronounced as the number of occluded pixels increases (Gao et al., 2018).

A third misconception is that lower occlusion alone implies the better system. The VMC experiments show that a planning-based baseline can achieve lower occlusion rates than the reactive controller, but only at the expense of highly inefficient paths and a significant drop in task success (He et al., 2022). The monocular-depth masking study likewise shows that per-pixel min reprojection can perform well partly because it ignores motion-induced error in addition to occlusion-induced error (Schellevis, 2019). These results suggest that self-occlusion optimization is usually entangled with other desiderata—stability, realism, dynamic-scene robustness, path efficiency, or safety—rather than being a single-objective problem.

A final recurring implication is that explicit self-occlusion handling is often necessary but not sufficient. Pseudo-stereo without masking and realism improvements overfits; S$68\%$08POT reports that any single self-supervised objective fails badly; MoDOT improves depth boundaries by combining architectural interaction and explicit boundary-depth constraints; and NeTO requires both implicit SDF optimization and self-occlusion-aware ray filtering. In that sense, the literature portrays self-occlusion optimization less as an isolated module than as a systems-level design principle that reshapes supervision, representation, inference, and control around visibility structure (Yang et al., 2024, Wang et al., 31 Jan 2026, Xu et al., 27 May 2025, Li et al., 2023).