SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes (2104.03953v3)

Abstract: Neural implicit surface representations have emerged as a promising paradigm to capture 3D shapes in a continuous and resolution-independent manner. However, adapting them to articulated shapes is non-trivial. Existing approaches learn a backward warp field that maps deformed to canonical points. However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn. To address this, we introduce SNARF, which combines the advantages of linear blend skinning (LBS) for polygonal meshes with those of neural implicit surfaces by learning a forward deformation field without direct supervision. This deformation field is defined in canonical, pose-independent space, allowing for generalization to unseen poses. Learning the deformation field from posed meshes alone is challenging since the correspondences of deformed points are defined implicitly and may not be unique under changes of topology. We propose a forward skinning model that finds all canonical correspondences of any deformed point using iterative root finding. We derive analytical gradients via implicit differentiation, enabling end-to-end training from 3D meshes with bone transformations. Compared to state-of-the-art neural implicit representations, our approach generalizes better to unseen poses while preserving accuracy. We demonstrate our method in challenging scenarios on (clothed) 3D humans in diverse and unseen poses.

• The paper introduces a forward skinning model that learns pose-independent deformations in a canonical space, outperforming state-of-the-art methods in generalization.
• The paper employs an iterative root-finding approach with implicit differentiation to enable end-to-end training without relying on predefined skinning weights.
• The paper demonstrates robust performance by animating complex, non-rigid shapes—including realistic clothed human figures—across varied poses and topological changes.

Overview of SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes

The paper "SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes" presented by Chen et al. introduces a novel method to represent and animate articulated 3D shapes using neural implicit surfaces. This approach is significant as it bridges the gap between traditional linear blend skinning (LBS) techniques used for polygonal meshes and the modern neural implicit representations of 3D shapes. The proposed method, SNARF (Skinned Neural Articulated Representations with Forward Skinning), enhances generalization to unseen poses without the need for ground-truth skinning weights or hand-crafted part correspondences.

Technical Contributions

The primary contribution of SNARF lies in its ability to learn a forward skinning model that represents the deformation of 3D shapes in a pose-independent, canonical space. This advance is achieved through a novel iterative root-finding approach that identifies all canonical correspondences for any deformed point. Key technical innovations include:

1. Forward Skinning Mechanism: By employing a forward deformation field rather than a pose-dependent backward one, SNARF enhances generalization capabilities. The iterative root-finding process facilitates the identification of all potential correspondences, even under topological changes, allowing for robust handling of complex deformations.
2. End-to-End Differentiable Learning: The differentiation of the forward skinning procedure is enabled through analytical gradients derived using implicit differentiation, supporting the training of the deformation field alongside 3D meshes and bone transformations.
3. Pose-Conditioned Neural Implicit Function: SNARF models local, pose-dependent deformations by conditioning an occupancy-based neural implicit function on joint angles. This adaptability is crucial for capturing subtle variations like clothing wrinkles and muscle movements.

Results and Experimental Validation

Through extensive experiments on both synthetic 2D data and realistic 3D human models, the effectiveness of SNARF is consistently demonstrated. Key results include:

• Superior Generalization: SNARF significantly outperforms existing state-of-the-art methods such as NASA in terms of intersection-over-union (IoU) measures of near-surface points, especially when challenged with poses far outside the training distribution.
• Handling of Topological Changes: Unlike traditional backward skinning approaches, SNARF manages topological variations seamlessly, as evidenced in the synthetic 2D stick experiments.
• Practical Application to Clothed Humans: The ability of SNARF to model dynamically clothed human figures with rich detail is illustrated through qualitative results on the CAPE dataset.

Implications and Future Directions

The successful integration of neural implicit surfaces with traditional LBS concepts opens new avenues for creating more flexible and realistic animated 3D models. By reducing the dependency on manually predefined skinning weights and improving generalization across poses, SNARF can potentially transform applications in virtual reality, gaming, and animation.

Looking forward, interesting challenges remain in enhancing scalability across subjects and reducing reliance on specific input sources, such as direct 3D meshes. Incorporating advanced techniques like differentiable rendering could broaden SNARF's applicability further, potentially enabling learning from richly annotated 2D imagery.

In conclusion, "SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes" contributes significantly to the field of 3D shape representation and animation, blending neural implicit representations with forward skinning mechanics to achieve versatility and precision in modeling articulated and non-rigid objects.