Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos

Abstract: We introduce a new method for learning a generative model of articulated 3D animal motions from raw, unlabeled online videos. Unlike existing approaches for 3D motion synthesis, our model requires no pose annotations or parametric shape models for training; it learns purely from a collection of unlabeled web video clips, leveraging semantic correspondences distilled from self-supervised image features. At the core of our method is a video Photo-Geometric Auto-Encoding framework that decomposes each training video clip into a set of explicit geometric and photometric representations, including a rest-pose 3D shape, an articulated pose sequence, and texture, with the objective of re-rendering the input video via a differentiable renderer. This decomposition allows us to learn a generative model over the underlying articulated pose sequences akin to a Variational Auto-Encoding (VAE) formulation, but without requiring any external pose annotations. At inference time, we can generate new motion sequences by sampling from the learned motion VAE, and create plausible 4D animations of an animal automatically within seconds given a single input image.

• The paper introduces a novel model that learns articulated 3D animal motions from raw online videos without relying on pose annotations.
• It extends the MagicPony framework by incorporating a spatio-temporal transformer VAE to generate accurate 3D reconstructions and animations.
• The method leverages unlabeled video data to produce plausible animations from a single image, demonstrating competitive results on standard datasets.

Overview

The paper introduces a novel generative model called 'Ponymation', designed for learning 3D animal motions from unlabeled online videos. Unlike previous motion synthesis methodologies that require pose annotations or parametric shape models for training, Ponymation is capable of learning from raw video collections. Utilizing online videos, the model develops a generative model for diverse 3D animations. At its core, it extends from an existing framework, MagicPony, that learns from single images, augmenting it with a training pipeline that includes temporal regulations for better reconstruction accuracy.

Training Data and Model Capabilities

The method leverages available online video data and learns to generate articulated 3D motions alongside a category-specific 3D reconstruction model. This learning occurs without any dependence on pose annotations or shape templates. Given a single test image, the algorithm reconstructs the articulated 3D mesh of the animal and generates plausible animations by drawing from a motion latent space learned during the training process.

Methodology

The process begins by collecting video clips from the internet of various animal categories. These clips are then used to train a spatio-temporal transformer Variational Auto-Encoder (VAE), different from frameworks that focus on individual static images. The architecture of the transformer VAE takes a sequence of images, encodes it into latent space, and decodes a sequence of articulated 3D poses. Training is conducted without explicit pose annotations by minimizing 2D reconstruction losses on video frames.

Contributions and Results

Their approach brings several key contributions to the table. It presents a new method that does not rely on manual supervision for learning complex motion patterns; it innovatively employs a spatio-temporal transformer VAE architecture effective in extracting motion information from videos; and finally, at inference, the model is capable of generating animations of new animal instances from a single image. Compared to baselines trained on static images, this video training framework exhibits improved reconstruction accuracy. Quantitative evaluations on datasets like PASCAL VOC have shown the method’s competitiveness with other methods that utilize more explicit annotations.

The model still has areas for enhancement, particularly concerning the need for predefined bone topology, which may limit its applicability to a broader range of animal species. Future work could aim to discover the articulation structure automatically while training on videos.