AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors

Abstract: We present AHOY, a method for reconstructing complete, animatable 3D Gaussian avatars from in-the-wild monocular video despite heavy occlusion. Existing methods assume unoccluded input-a fully visible subject, often in a canonical pose-excluding the vast majority of real-world footage where people are routinely occluded by furniture, objects, or other people. Reconstructing from such footage poses fundamental challenges: large body regions may never be observed, and multi-view supervision per pose is unavailable. We address these challenges with four contributions: (i) a hallucination-as-supervision pipeline that uses identity-finetuned diffusion models to generate dense supervision for previously unobserved body regions; (ii) a two-stage canonical-to-pose-dependent architecture that bootstraps from sparse observations to full pose-dependent Gaussian maps; (iii) a map-pose/LBS-pose decoupling that absorbs multi-view inconsistencies from the generated data; (iv) a head/body split supervision strategy that preserves facial identity. We evaluate on YouTube videos and on multi-view capture data with significant occlusion and demonstrate state-of-the-art reconstruction quality. We also demonstrate that the resulting avatars are robust enough to be animated with novel poses and composited into 3DGS scenes captured using cell-phone video. Our project page is available at https://miraymen.github.io/ahoy/

• The paper introduces an end-to-end pipeline using Gaussian splatting and diffusion priors to robustly reconstruct animatable 3D avatars from occluded videos.
• It employs dense UV fusion, hallucinated supervision, and decoupled pose mapping to inpaint unseen body regions and ensure identity preservation.
• Empirical validation shows significant gains in PSNR, SSIM, and LPIPS, outperforming previous methods in both novel-pose and novel-view settings.

Occlusion-Robust Animatable 3D Gaussian Avatars from Monocular YouTube Video

Motivation and Problem Statement

Robust photorealistic 3D human reconstruction under occlusion, especially from in-the-wild monocular video, is largely unsolved. Prior NeRF/3DGS and implicit/mesh-based approaches require the subject to be fully visible, severely constraining in-the-wild applicability. The authors introduce the AHOY framework to overcome intrinsic limitations resulting from unseen body regions, lack of multi-view per-pose supervision, and identity preservation for high-fidelity animation from highly occluded monocular videos.

Pipeline and Technical Contributions

The end-to-end AHOY pipeline systematically bootstraps complete animatable 3D Gaussian avatars from incomplete in-the-wild sequences. The approach is decomposed into complementary stages, visualized in the following figure:

Figure 1: The AHOY pipeline: dense UV-based canonical map fusion, identity-finetuned video diffusion priors, RF-inversion supervision, pose-dependent mapping, joint map-pose/LBS-pose optimization, and split head/body supervision for robust animatable avatar learning under occlusion.

The pipeline consists of:

• Coarse Gaussian Avatar Construction: The system fuses partial texture observations over a video sequence using DensePose UV correspondences. Missing regions are inpainted in a canonical pose with a diffusion-based inpainting model (FLUX). Multi-view canonical renderings are generated via a multi-view diffusion model for spatial coverage, supporting the initial learning of canonical, pose-independent Gaussian maps.
• Hallucinated Supervision via Identity-finetuned Video Diffusion: To compensate for persistent unobserved regions, an identity LoRA-finetuned version of a powerful video diffusion model (Wan 2.2) is used to generate (via rectified flow inversion) dense supervision videos. The coarse canonical avatar’s structured pose renderings are refined with these priors to “hallucinate” synthetic, identity-preserving, view-consistent textures for previously unseen body surfaces and novel poses.
• Canonical-to-Pose-Dependent Gaussian Map Advancement: With the hallucinated video set, the system moves to a pose-dependent StyleUNet/SMPL representation, decoupling per-frame LBS-pose deformations from the map pose. This map/LBS-pose decoupling soaks up multi-view inconsistencies, ensuring avatar consistency even from inconsistent diffusion priors.
• Head/Body Supervision Split: To mitigate identity loss or inconsistency in faces generated by video diffusion, only body regions are trained with hallucinated supervision, while the head region is reconstructed via a separate multi-view facial diffusion model, leveraging SMPL-to-FLAME correspondence for region partitioning and FLAME for head pose/expressivity.

The system implements joint optimization of StyleUNet parameters, LBS-pose/camera corrections, and head/body split losses for best possible dense, animatable, and photorealistic 3DGS avatar recovery.

Empirical Validation and Results

The authors present comprehensive qualitative and quantitative evaluation:

• YouTube In-the-Wild Animation and Reconstruction: On complex in-the-wild videos with extensive body occlusion, the pipeline outperforms both image- and video-based animatable avatar baselines. Notably, under significant occlusion, AHOY achieves a PSNR of 22.81, SSIM of 0.887, and LPIPS of 0.107, compared to 19.03/0.821/0.181 for LHM and 16.52/0.769/0.231 for IDOL (all scores in the novel-pose setting), a significant advance in the occluded reconstruction regime.
• Canonical-Pose Input Comparison: Even when other baselines are provided with additional canonical-pose view inpainting (thus giving them the best possible setting), AHOY avatars yield higher fidelity results, demonstrating the effectiveness of the pipeline’s pose-dependent learning and hallucinated supervision rather than mere reliance on inpainting.
• Multi-View and Novel-Pose Synthesis on BEHAVE: With multi-view ground truth on the BEHAVE dataset, AHOY achieves PSNR 24.12 (SSIM 0.905, LPIPS 0.09) in the novel-view regime, and 22.81 (0.889, 0.11) in the novel-pose regime, compared to 18.21/0.807/0.203 and 16.93/0.774/0.238 for the best previous baseline, confirming robust generalization and photorealistic rendering.

Figure 3: AHOY’s avatars rendered from held-out BEHAVE side cameras deliver far higher structural and appearance fidelity than previous approaches, even under extreme self-occlusion and novel poses.

• Ablation Study: The system’s gains are parcellated via ablation: removal of hallucinated supervision, RF-inversion, map/LBS pose decoupling, or head/body splitting each degrades novel-view/novel-pose scores—demonstrating that all pipeline stages are critical to effective occlusion-robust animatable avatar learning.

Theoretical and Practical Implications

The combination of identity-finetuned latent video diffusion priors and explicit pose-dependent 3DGS representations, equipped with mechanisms to absorb multi-view inconsistencies, opens up learning from massive unconstrained video corpora. This pipeline removes the dependence on controlled captures, un-occluded single shots, and enables training on open-world YouTube-scale video for both research and downstream production.

Practically, the pipeline supports compositing into photorealistic 3DGS-captured scenes and robust animation in downstream AR/VR, gaming, and film scenarios, even for avatars reconstructed from highly occluded, casual video.

Theoretically, the work demonstrates the viability of generative priors for completion of missing scene geometry, bridging minimal-supervision/sparse-input settings with high-fidelity explicit 3D parameterizations suitable for driving and animation. The method motivates future work in geometric-consistency-constrained diffusion guidance, acceleration for low-latency inference, and further integration of stronger backbone video diffusion models.

Conclusion

AHOY presents a full-stack, occlusion-tolerant pipeline for animatable 3D Gaussian avatars learned from in-the-wild monocular video, incorporating identity-conditioned diffusion priors, canonical-to-pose-dependent Gaussian mapping, multi-view hallucinated supervision with explicit inconsistency absorption, and identity-preserving split region learning. The empirical gains (e.g., state-of-the-art PSNR, SSIM, and LPIPS under heavy occlusion and complex scenes) highlight the efficacy and generalization potential of using diffusion priors for supervision beyond the classical multi-view geometry paradigm. The approach establishes a scalable foundation for mining the vast diversity of human appearance from open-world, unconstrained monocular video archives, with broad implications for 3D vision, graphics, and AR/VR.