Whole-Body Self-Supervised Learning

• Whole-body self-supervised representation learning is a framework that extracts structured, transferable representations of articulated bodies from unlabeled sensory data using techniques like temporal coherence, masking, and cross-modal correspondence.
• Key methodologies involve part-to-whole tokenization, hierarchical motion segmentation, and disentangled latent spaces that separately encode geometry, appearance, and articulation.
• Applications span human motion analysis, surgical robotics, medical imaging, and embodied navigation, enabling improved prediction, reconstruction, and transfer learning.

Whole-body self-supervised representation learning denotes a family of methods that learn transferable representations of an entire articulated body, a body-scale motion sequence, or a body-scale volumetric scan from unlabeled sensory data. Across human motion, dressed-human geometry, full-body scan analysis, whole-body MRI, articulated objects, and embodied agents, the learned representation may take the form of hierarchical motion tokens, dense depth and surface-normal fields, disentangled geometry/appearance/articulation codes, mesh-invariant latent body parameters, or frozen whole-body embeddings reused for downstream prediction. What unifies these approaches is that supervision is induced from temporal coherence, masking, cross-modal correspondence, differentiable rendering, or active exploration rather than manual semantic labels (Kinoshita et al., 30 Apr 2026, Jafarian et al., 2021, Wei et al., 2022, Hartman et al., 2024, Seletkov et al., 4 Aug 2025, Du et al., 2021).

1. Scope and conceptual foundations

The term “whole-body” is not restricted to human full-body pose in the narrow skeletal sense. In the literature it encompasses several related problem formulations: full-body human movement represented from 3D pose sequences; dressed-human geometry represented as dense visible-surface structure; full 3D body scans and 4D body motions encoded into latent body-model spaces; whole-body medical volumes represented as systemic phenotypes; and, by close analogy, articulated objects or whole-motion surgical activity represented as coherent articulated or temporal units. The survey literature places these methods within the broader SSL taxonomy of pretext-task methods, information-maximization methods, teacher-student methods, contrastive representation learning, and clustering-based methods, typically organized around an encoder $y=f_\theta(x)$ with optional projector and predictor modules (Uelwer et al., 2023).

A recurrent conceptual shift is from frame-level or patch-level encoding toward representations of structured wholes. In surgical robotics, for example, self-supervision is applied not to isolated images but to a short gesture segment formed from synchronized video-derived optical flow and 76-dimensional kinematics, so that the latent code captures the full temporal action unit rather than an instantaneous descriptor (Tamhane et al., 2020). In human motion, the same shift appears as variable-length body-movement segments rather than fixed-length clips, while in whole-body MRI it appears as a body-wide latent phenotype rather than a collection of organ-specific engineered measurements (Kinoshita et al., 30 Apr 2026, Seletkov et al., 4 Aug 2025).

This breadth matters because it corrects a common misconception: self-supervised whole-body representation learning is not synonymous with a single architecture class or a single modality. It includes discriminative and generative models, 2D and 3D inputs, passive datasets and active embodied exploration, and latent spaces that may be flat, hierarchical, disentangled, or explicitly geometric.

2. Part–whole structure, tokenization, and latent organization

A central design principle is that wholes are learned through parts. The analysis of part-aware pretraining argues that contrastive learning behaves as a part-to-whole task: random crops frequently contain only part of an object, the encoder learns a part-level representation, and the projection layer “hallucinates” a whole-object representation from that part representation. Masked image modeling is correspondingly interpreted as a part-to-part task, because masked regions are inferred from visible regions. Empirically, self-supervised encoders outperform supervised models on several part-level tasks, and the combination of contrastive learning and masked image modeling improves performance further (Zhu et al., 2023).

CO-SSL makes this principle explicit by aligning local representations before pooling with a global image representation. In CO-BYOL, local spatial features are projected and predicted so that each local representation agrees with the global target embedding, and the whole is in turn aligned to the set of local targets. The reported consequence is that CO-SSL learns highly redundant local representations and gains robustness to corruption, internal feature masking, and small adversarial attacks; on ImageNet-1K, CO-BYOL with RF99-ResNet50 reaches 71.5% Top-1 accuracy after 100 pre-training epochs (Aubret et al., 6 Jan 2025). For whole-body learning, this supports a part–whole regime in which limbs, joints, or anatomical subregions are not only encoded locally but trained to be predictive of a global body state.

A related strategy appears in large structured visual objects. Slide Pre-trained Transformers represent a whole slide image as a set of patch tokens with coordinates, aggregate them with a 6-layer transformer using a learnable Fourier-feature relative positional embedding, and form two self-supervised views through a sequential split $\rightarrow$ crop $\rightarrow$ mask pipeline. Although developed for gigapixel pathology rather than bodies, this design gives a direct template for whole-body scans or large body-centric scenes: tokenize local regions, retain coordinates, reduce token count through masking and cropping, and train a global transformer representation to be invariant across semantically related views (Hou et al., 2024).

3. Temporal compositionality and whole-motion representations

In human motion, whole-body representation learning increasingly treats movement as hierarchically compositional. A4Mer learns a two-level representation from 3D pose sequences: Action Atoms, which are variable-length atomic body-movement segments, and Action Motifs, which are recurring temporal compositions of those atoms. The model uses a nested latent Transformer with segment-wise self-attention, cross-attention to consolidate each segment into a latent token, and a JEPA-style masked latent prediction objective with local/global decomposition to avoid collapse. Action Atom boundaries are initialized by thresholding nonlinear changes in joint trajectories, Action Atom sequences are discretized with k-means into 512 codes, and frequent subsequences are mined with the Generalized Sequential Pattern algorithm to define Action Motifs. On downstream tasks, A4Mer reports 31.7 / 59.0 top-1 / top-3 k-NN action-recognition accuracy and achieves 38.1 mm MPJPE on HiK in the zero-shot transfer setting for motion prediction (Kinoshita et al., 30 Apr 2026).

The data substrate for this hierarchical approach is equally significant. The Action Motif Dataset contains 50 subjects, 24 cameras, 14.2 hours of footage at 30 fps, and 129 daily actions, with full SMPL annotations. A distinctive annotation strategy mounts tiny cameras on the feet and uses ChArUco markers on ceilings and tables to improve pose recovery under frequent leg and foot occlusions. This reflects a broader tendency in whole-body SSL: the quality of the learned motion semantics is tightly coupled to the quality and coverage of the latent body state used during pretraining (Kinoshita et al., 30 Apr 2026).

Whole-body learning also includes “whole-motion” representations in domains that are not human-pose-centric. In surgical robotics, a self-supervised encoder–decoder maps an optical-flow segment to a latent representation and decodes it into synchronized kinematics. Optical flow is computed with the Farnebäck algorithm, the encoder is a 2D CNN similar in spirit to a two-stream visual backbone, temporal context is a 50-frame window subsampled to 25 frames, and the decoder predicts 25 kinematics vectors with an MSE loss,

$$\min_{\theta, \phi}\frac{1}{n}\sum_{i=1}^{n} \left\| \mathcal{D}\big(r(T(\mathcal{V}_i);\theta);\phi\big) - \mathcal{K}_i \right\|_2^2.$$

Frozen embeddings evaluated with XGBoost yield gesture-recognition accuracies of 76.2% on Knot Tying, 69.6% on Needle Passing, and 77.8% on Suturing; skill-classification accuracies of 76.8%, 80.8%, and 81.2%; and cross-task transfer accuracies ranging from 44.6% to 64.8% (Tamhane et al., 2020). Although the domain is surgical robotics, the methodological point is directly relevant: a synchronized complementary modality can define the self-supervisory target for a segment-level motion representation.

4. Geometry, articulation, and generative whole-body latent spaces

A major branch of the field learns structured 3D latent spaces in which geometry, appearance, and articulation are explicitly separated. “Self-supervised Neural Articulated Shape and Appearance Models” represents an articulated instance in a state as a triplet $(\theta_i,\phi_i,\psi_j)$, where $\theta_i$ is a geometry code, $\phi_i$ an appearance code, and $\psi_j$ an articulation code. Geometry is modeled as a category-level neural signed distance function, appearance as a view-dependent radiance function, and articulation through an optional deformation field,

$x' = x + \mathcal{D}_\Psi(x,\theta_i,\psi_j;\Psi),$

that maps observation-space points to a canonical pose. Training uses masked multi-view RGB images, masks, camera parameters, RGB reconstruction, mask loss, Eikonal regularization, and latent-code regularization, without 3D supervision or articulation labels. On SAPIEN, with 2346 articulated objects across 46 categories, the method supports few-shot reconstruction, latent swapping, articulation interpolation, and novel view-synthesis, while handling revolute joints, prismatic joints, and combinations thereof (Wei et al., 2022).

For human bodies, an analogous but denser representation is learned in “Self-supervised 3D Representation Learning of Dressed Humans from Social Media Videos.” Instead of regressing only coarse parametric body variables, the model predicts a depth map $z=g(\mathbf{x};\mathbf{I})$ and a surface-normal map $\rightarrow$0 over the visible dressed human. Self-supervision comes from local part-wise 3D warping between frames using DensePose UV correspondences, a temporal coherence loss on warped 3D points, a photometric loss after warping, and depth–normal geometric consistency. HDNet uses a stacked hourglass architecture in a Siamese temporal configuration, and training combines supervised losses on RenderPeople with self-supervised losses on a TikTok dataset of over 340 dance video sequences and more than 100K frames. The method is reported to outperform prior human depth estimation and human shape recovery approaches on both real and rendered data (Jafarian et al., 2021).

A different line of work operates directly on full 3D body scans and 4D body motions. VariShaPE estimates latent body-model parameters from arbitrary whole-body meshes without requiring mesh registration, using a varifold-based, remeshing-invariant descriptor called VariGrad. MoGeN then learns the geometry of the latent motion space itself by lifting pose parameters into a higher-dimensional Euclidean space in which motion mini-sequences are approximated by linear interpolation and extrapolation. In experiments based on SMPL and DFAUST, the paper reports a mean vertex distance of 2.1 mm for the VariShaPE model on registered data and shows that MoGeN improves interpolation and extrapolation relative to direct linear interpolation in SMPL space and ARAPReg (Hartman et al., 2024). This suggests that whole-body SSL is not only about learning latent codes, but also about learning the geometry through which those codes evolve.

5. Multimodal grounding, active exploration, and online adaptation

Whole-body representations need not be learned from a fixed passive corpus. Curious Representation Learning treats representation learning as an embodied minimax game between a visual encoder and an exploration policy:

$\rightarrow$1

The agent acts in Habitat with discrete navigation and camera-control actions, observes egocentric RGB images of size $\rightarrow$2, and trains a ResNet-50 encoder with a SimCLR-style contrastive objective while a PPO policy is rewarded for seeking observations that maximize current representation error. Because the full contrastive denominator can be exploited by standing still, the intrinsic reward is defined from the positive-pair similarity term, $\rightarrow$3, with a constant shift of $\rightarrow$4 and reward normalization. With 10 million interactions in 16 environments, CRL improves transfer to ImageNav and ObjectNav under RGB-only settings, achieving ImageNav SPL 0.0324, Soft SPL 0.219, Success 0.058, and ObjectNav SPL 0.0144, Soft SPL 0.119, Success 0.040 (Du et al., 2021).

The same embodied logic appears in object-centric online learning from monocular robot video. Online Object Representations with Contrastive Learning mines its own positives by nearest-neighbor matching between detected object crops across frames and trains an Object-Contrastive Network with an n-pairs loss. The longer the model observes a scene, the lower the identification error; in one reported example the online model reaches about 2.2% error while an offline supervised baseline remains around 52.4% error, and in robotic pointing the learned embedding yields about 72% recognition accuracy over five classes and about 89% accuracy on the binary “is-container” attribute (Pirk et al., 2019). Although object-centric, this provides a template for whole-body learning in situated systems: correspondences can emerge from temporal continuity, and the data distribution can be shaped by the agent’s own motion.

Cross-modal grounding is the complementary pattern to active exploration. In surgical gesture learning, optical flow predicts kinematics; in articulated-object modeling, RGB and masks with camera parameters supervise geometry, appearance, and articulation; and in whole-body MRI risk modeling, masked image reconstruction pretrains an encoder later fused with cardiac MRI-derived features (Tamhane et al., 2020, Wei et al., 2022, Seletkov et al., 4 Aug 2025). A plausible implication is that whole-body SSL is strongest when the latent state must simultaneously explain complementary signals rather than a single sensory stream.

6. Whole-body medical imaging, evaluation regimes, and open problems

A clinically explicit realization of whole-body SSL is “Whole-body Representation Learning For Competing Preclinical Disease Risk Assessment.” The method pretrains a masked autoencoder on approximately 70,000 neck-to-knee T1-weighted dual-echo Dixon whole-body MRIs from UK Biobank, operating on a $\rightarrow$5 tensor with water and fat contrasts, using 3D patches of size $\rightarrow$6, a 4D positional embedding, intensity-based foreground selection, and 70% masking of foreground patches. The pretraining loss is

$\rightarrow$7

The frozen encoder output is then fed into competing-risk survival models such as Deep Survival Machines, Neural Fine–Gray, and DeepHit. Against whole-body radiomics, the learned representation improves DSM time-dependent concordance from 0.614 to 0.628 for CVD, from 0.692 to 0.712 for T2D, from 0.638 to 0.682 for COPD, and from 0.610 to 0.636 for CKD. In CVD subgroups it improves whole-body radiomics from 0.616 to 0.642 for IHD, from 0.632 to 0.650 for HD, and from 0.519 to 0.614 for stroke, while fusion with cardiac features reaches 0.672, 0.665, and 0.617 respectively. The MAE reports a mean PSNR of 32.18 on 100 held-out samples, and t-SNE of the latent space clusters by sex and BMI class without using those labels during pretraining (Seletkov et al., 4 Aug 2025).

The downstream evaluation protocols across the field are correspondingly diverse. Motion-centered papers emphasize action recognition, motion prediction, interpolation, gesture recognition, surgeon skill classification, and transfer across tasks (Kinoshita et al., 30 Apr 2026, Tamhane et al., 2020). Geometry-centered papers emphasize reconstruction fidelity, interpolation, extrapolation, and transfer in latent pose space (Jafarian et al., 2021, Hartman et al., 2024). Embodied papers evaluate navigation, imitation learning, and real-image transfer (Du et al., 2021). Medical imaging emphasizes survival-oriented prediction rather than reconstruction alone (Seletkov et al., 4 Aug 2025). This heterogeneity indicates that “representation quality” is domain-specific: a good whole-body representation may be predictive, generative, transferable, or controllable, and different papers optimize different combinations of these properties.

Several limitations recur. Some methods depend on accurate 3D pose or SMPL annotations, as in AMD, or on camera parameters and object masks, as in articulated implicit rendering (Kinoshita et al., 30 Apr 2026, Wei et al., 2022). Some learn articulation latents without an explicit skeletal graph or known joint parameterization, which increases flexibility but weakens explicit kinematic structure (Wei et al., 2022). Some rely on heuristics for segmentation or frequent-pattern mining rather than end-to-end learned boundary discovery (Kinoshita et al., 30 Apr 2026). Dressed-human reconstruction remains sensitive to uncommon viewpoints, lighting, occlusions, and multiple people (Jafarian et al., 2021). Whole-body MRI results are tied to UK Biobank demographics, scanner context, and administrative event definitions (Seletkov et al., 4 Aug 2025). Embodied approaches remain constrained by simulation, RGB-only sensing, or reward-hacking failure modes (Du et al., 2021).

Taken together, these lines of work indicate that whole-body self-supervised representation learning is converging on several stable principles: the body should be modeled as a structured whole rather than a bag of independent frames; local parts should be informative about global state; temporal or articulated composition should be preserved rather than averaged away; cross-modal or cross-view agreement is often more informative than purely appearance-level invariance; and frozen self-supervised encoders can support a wide range of downstream models. This suggests that future progress will likely come from tighter integration of hierarchical motion structure, explicit geometry, multimodal grounding, and embodied data acquisition within a single representation-learning framework.