Pose-guided Visibility Predictor (PVP)

• The paper introduces PVP as a computational module that predicts scene visibility from pose data using both analytic and neural methods.
• PVP is applied in generative 3D pose estimation, occluded person re-identification, and VR streaming, where differentiable and statistical formulations yield improved accuracy.
• Empirical results show that PVP enhances re-id accuracy, pose estimation robustness, and VR efficiency by effectively managing occlusion, motion, and camera changes.

A Pose-guided Visibility Predictor (PVP) is a computational module designed to estimate the visibility of scene elements (e.g., body parts, pixels, or 3D locations) given pose information. PVPs address occlusion, motion, and camera changes by leveraging geometric, statistical, or neural modeling of pose-dependent visibility. The PVP concept has been instantiated across diverse domains, including human re-identification under occlusion (Gao et al., 2020), marker-less motion capture and pose estimation (Rhodin et al., 2016), and real-time VR frame prediction and foveated streaming (Chen et al., 2022), each adopting domain-specific formulations and architectures.

1. PVP in Generative and Discriminative Vision Pipelines

The Pose-guided Visibility Predictor was first described in generative 3D pose estimation, where it offers an alternative to hard mesh-based occlusion rendering. Instead, articulated objects are modeled as continuous density fields (typically Gaussian mixtures), enabling closed-form, differentiable visibility computations. In discriminative settings such as occluded person re-identification, PVPs are utilized as neural modules outputting per-part visibility scores, enabling weighted feature aggregation to focus on unoccluded regions only. In VR rendering systems, PVPs predict the fraction of pixels likely to remain visible following user motion, thus facilitating adaptive resource allocation for real-time frame composition.

2. Mathematical Formulations

PVP modeling differs by application but is united by its pose-conditioned nature. The principal mathematical strategies include:

• Gaussian Density and Radiative Transport (Pose-guided Generative PVP) Articulated objects are parameterized by a set of Gaussians $Q$ with centers $\mu_n(p)$ driven by the pose $p$. The aggregate extinction density at spatial location $x$ is $D(x) = \sum_{n\in Q} G_n(x)$, $G_n(x) = c_n \exp(-\|x-\mu_n\|^2/(2\sigma_n^2))$. The visibility at a point $x$ from camera origin $o$ along direction $n$ at distance $s$ is the transmittance:

$$T(o, n, s, p) = \exp\left(-\int_0^s D(o + t n, p)\, dt\right)$$

For a single Gaussian, all integrals are reducible to error-functions. Visibility is thus continuous and differentiable in pose $p$ (Rhodin et al., 2016).

• Neural Part-wise Visibility (Discriminative PVP in ReID) Given pose features $F_{pose}$, the PVP computes per-part visibilities via a feedforward head:

$$\hat v = \sigma(\mathrm{BN}(\mathrm{Conv}_{1\times 1}(\mathrm{GlobalAvgPool}(F_{pose}))))$$

Output $\hat v_i \in (0, 1)$ for each part $i$ describes the probability of being unoccluded (Gao et al., 2020).

• Statistical Prediction of Per-pixel Visibility (VR PVP) The PVP estimates the survival function $ViS(d)$, i.e., expected fraction of depth-$d$ pixels visible after pose perturbation $(\Delta p, \Delta \theta, \Delta \phi)$. Closed-form, moment-based approximations for $ViS_{fov}$ and $ViS_{dst}(d)$ use analytic models of head pose/position trajectories (Chen et al., 2022).

Setting	Pose Parametrization	Visibility Output	Core Computation Domain
Generative 3D pose	Articulated $\mu_n(p)$	$V_n(o, n, p)$	Dense geometry, integrals
Person ReID	2D pose/keypoints, $F_{pose}$	$\hat v_i \in (0,1)$	Neural, per-part
VR streaming	$(p_{ref}, \theta_{ref}, \phi_{ref})$	$ViS(d)$	Analytical statistics

3. Supervision and Training Mechanisms

Supervision of PVP modules reflects ground-truth limitations in visibility labeling:

• Self-mined Pseudo-labels (ReID):

No ground-truth labels for occlusion necessitate constructing pseudo-labels via self-mining: for each positive identity pair, a graph-matching quadratic program is solved on extracted part features to determine visible/occluded parts. These binary codes $v^*$ supervise the PVP by minimizing $$L_v = -\sum_{i=1}^{N_p} v^*_i \log(\hat v_i^p \cdot \hat v_i^g)$$ (Gao et al., 2020).

• Analytical Gradients (Generative 3D):

Because all visibility and transmittance expressions are analytic in the pose parameters, gradients $\partial V_n/\partial p$ are computable exactly, enabling gradient-based optimization of photo-consistency or data terms in pose estimation (Rhodin et al., 2016).

4. Integration into Downstream Tasks

PVP directly modulates downstream loss and inference computations:

• In ReID, PVP outputs serve as weights in the part-wise matching distance computation:

$$d(I_p, I_g) = \frac{\sum_{i=1}^{N_p} \hat v_i^p \hat v_i^g d_i}{\sum_{i=1}^{N_p} \hat v_i^p \hat v_i^g}$$

where $d_i$ are cosine distances between part embeddings (Gao et al., 2020).

• In generative fitting, predicted colors at each pixel are weighted sums of Gaussians’ albedos and visibilities, used in photo-consistency objectives (Rhodin et al., 2016).
• In VR, the PVP's $ViS(d)$ is used within the ALG-ViS algorithm to split each frame into foreground (to be re-rendered) and background (to be reused), enabling adaptive bandwidth and compute utilization (Chen et al., 2022).

5. Empirical and Computational Impact

PVPs demonstrate measurable improvements across applications:

• Occluded Person ReID On Occluded-REID, "PVP only" yields rank-1 accuracy 65.2% vs. 59.3% for the plain PCB baseline; full PVPM rises to 66.8%. Across Partial-REID and P-DukeMTMC, absolute improvements of up to 7.7 percentage points are recorded, demonstrating the importance of visibility estimation (Gao et al., 2020).
• Pose Estimation Robustness In generative pose estimation, PVP regularizes the energy landscape, yielding a larger radius of convergence and reducing spurious minima compared to binary mesh-based visibility. For "Marker" sequences using only two cameras, average joint-error is 3.7 cm for PVP versus 7 cm baseline; in multi-subject settings, joint optimization reduces limb-error by ≈10% (Rhodin et al., 2016).
• VR Rendering Efficiency In VR, PVP + ALG-ViS achieves SSIM $\geq 0.945$ compared to 0.915 for a heuristic baseline, with 16.1–33.4% lower rendering time and 11.3% lower average bandwidth. Per-frame bandwidth variance drops by 88.4% over ML-based predictors, evidencing stability and resource efficiency (Chen et al., 2022).

6. Methodological Limitations and Extensions

• Smooth Visibility vs. Sharp Boundaries:

The Gaussian-based PVP necessarily diffuses sharp occlusion boundaries, trading infinitesimal accuracy at silhouettes for analytic, globally smooth visibility suitable for optimization (Rhodin et al., 2016).

• Data-driven PVPs are contingent on the quality of pose estimation (e.g., OpenPose accuracy) and the expressivity of self-mined pseudo-labels (Gao et al., 2020).
• General Applicability:

The PVP abstraction extends seamlessly between analytic, geometric approaches and purely neural, data-driven frameworks, provided a pose representation amenable to visibility conditioning can be defined.

• A plausible implication is that as richer pose representations and labeling techniques emerge, PVP architectures can be further specialized or generalized, supporting broader visual inference tasks under occlusion.

7. Connections to Related Literature

PVP builds on classical visibility analysis in rendering and pose estimation, extending to differentiable models (Rhodin et al., 2016), and is complementary to attention-based feature selection in deep learning (Gao et al., 2020). In VR and streaming, it parallels analytic resource management using signal prediction (Chen et al., 2022). The principle of integrating pose as a conditioning variable for visibility—rather than as a late-stage filter—distinguishes PVP from heuristic or solely appearance-based occlusion handling.