Unique Lives, Shared World: Learning from Single-Life Videos (2512.04085v1)

Abstract: We introduce the "single-life" learning paradigm, where we train a distinct vision model exclusively on egocentric videos captured by one individual. We leverage the multiple viewpoints naturally captured within a single life to learn a visual encoder in a self-supervised manner. Our experiments demonstrate three key findings. First, models trained independently on different lives develop a highly aligned geometric understanding. We demonstrate this by training visual encoders on distinct datasets each capturing a different life, both indoors and outdoors, as well as introducing a novel cross-attention-based metric to quantify the functional alignment of the internal representations developed by different models. Second, we show that single-life models learn generalizable geometric representations that effectively transfer to downstream tasks, such as depth estimation, in unseen environments. Third, we demonstrate that training on up to 30 hours from one week of the same person's life leads to comparable performance to training on 30 hours of diverse web data, highlighting the strength of single-life representation learning. Overall, our results establish that the shared structure of the world, both leads to consistency in models trained on individual lives, and provides a powerful signal for visual representation learning.

• The paper demonstrates that single-life models, trained solely on individual egocentric videos, can extract aligned geometric priors for various geometric tasks.
• It employs novel temporal and spatial pairing strategies along with the Correspondence Alignment Score (CAS) to rigorously quantify model alignment.
• The study shows that with sufficient data duration, these models generalize effectively to downstream tasks like depth estimation and segmentation, rivaling diverse-data baselines.

Single-Life Learning: Visual Representation Alignment and Generalization from Egocentric Video

Introduction

The paper "Unique Lives, Shared World: Learning from Single-Life Videos" (2512.04085) presents a rigorous investigation into the single-life learning paradigm, wherein vision models are trained exclusively on egocentric video streams from individual persons. Unlike conventional approaches that aggregate diverse datasets to maximize representation generality, this work isolates learning within the context of a unique person's visual experience, then studies alignment between such independently trained models and their capability to generalize to geometric tasks in unseen domains. The authors focus primarily on the capacity of these single-life models to extract robust and functionally similar geometric priors, introducing the Shared World Hypothesis as an extension of the Platonic Representation Hypothesis.

Figure 1: Single-life learning paradigm, illustrating the independent training of models on egocentric video data from individual lives.

The Single-Life Paradigm and Training Methodology

This paradigm defines a "life" as a set of sequential, egocentric video frames captured by a unique participant. Each model $f_{\theta_i}$ is trained exclusively on $\mathcal{D}_i$, the video data of person $i$, via self-supervised objectives—specifically, cross-view completion as proposed in CroCo [weinzaepfel2022croco]. This model architecture is a weight-sharing ViT-based encoder-decoder transformer, designed to exploit spatiotemporal continuity inherent in egocentric video for the learning of geometric structure.

Critical to the framework are two pairing strategies for constructing effective learning pairs:

• Temporal pairing: Sampling frames close in time to maximize viewpoint overlap.
• Spatial pairing: Leveraging camera poses and 3D point clouds to identify frames with high geometric overlap.

The union of these methods yields the most transferable geometric representations, as validated by downstream probing.

Metrics for Alignment: Correspondence Alignment Score (CAS)

To rigorously quantify the functional alignment between models trained on separate lives, the authors introduce the Correspondence Alignment Score (CAS). This metric exploits decoder cross-attention maps to evaluate patch-wise correspondences between test images processed by distinct models. CAS extends the mutual $k$-nearest neighbor principle, focusing on the intersection of top-k attended patches for each model across shared inputs.

Figure 2: CAS metric calculates intersection-over-top-k correspondences at the patch level using decoder cross-attention maps.

Dataset Specification and Diversity

Twenty single-life models were trained with data sourced from three egocentric video datasets: HD-Epic (single-room kitchen environments with ground truth poses), Walking Tours (outdoor urban traversals), and Anonymous Lives (ALD, week-long daily records spanning broad indoor and outdoor environments). Additionally, control models were trained on "non-life" continuous videos lacking egocentric perspective and a subset of the Kinetics-400 (K400) dataset for comparison.

Figure 3: Statistical distributions of life properties—camera motion, depth, optical flow, brightness, and object frequency—contrasting indoor and outdoor environments.

Experimental Findings

Emergence of Alignment and Scaling with Data Size

CAS measurements indicate the emergence of strong representational alignment after 30–60 minutes of egocentric video exposure. Models trained on non-life control videos fail to manifest alignment; the egocentric perspective is an indispensable ingredient. The scaling trend shows that larger single-life datasets lead to CAS values approaching those attained by diverse web-scale baselines, verifying the Shared World Hypothesis.

Figure 4: CAS with CroCo as a function of life duration for different datasets. Alignment emerges earlier in datasets similar to CroCo's training domain.

Intra- and Inter-Model Alignment

Pairwise CAS matrices reveal pronounced block-diagonal structures—models within similar environments (e.g., all indoor kitchens or outdoor tours) align most strongly. Multidimensional scaling visualizations further confirm dataset-based clustering, suggesting domain-specific geometric priors overlay a shared foundational structure.

Figure 5: CAS score matrix for individual single-life models showing high intra-dataset alignment clusters.

Figure 6: MDS visualization of model alignment, with clusters corresponding to environmental similarity.

Generalization to Downstream Geometric Tasks

Single-life models generalize effectively to depth estimation (NYU-Depth-v2, ScanNet) and zero-shot correspondence (HPatches). Performance scales robustly with data duration, with models trained on 30 hours from ALD matching or surpassing the K400 baseline at equivalent size. Early performance favors diverse datasets, but single-life models converge with larger, temporally diverse experience.

Figure 7: Downstream generalization scores (NYU, ScanNet, HPatches) vs. single-life data duration, single-life attaining competitive performance at larger scales.

Impact of Pairing Strategies

Union of spatial and temporal pairing delivers optimal generalization. Temporal pairing alone is remarkably effective, especially in natural environments, indicating that egocentric physical motion encapsulates salient geometric cues for self-supervised representation learning.

Figure 8: Relative gains in depth estimation with different pairing strategies on HD-Epic, indicating benefit of joint spatial/temporal sampling.

Figure 9: Performance gains with temporal pairing for Walking Tours and ALD, showing robust improvement over baselines.

Architecture Independence

The paradigm is not confined to CroCo; DINOv2 models trained on single lives also yield robust geometric representations, though alignment (via CAS) is less pronounced between models. Task performance differences suggest objective-specific suitability, and stability issues can arise on redundant or non-egocentric data with contrastive regimes.

Qualitative Visual Results

Zero-shot HPatches correspondence predictions show structured matching for ALD lives, indicating higher geometric priors learned. Video segmentation label propagation demonstrates that features from single-life models reliably track novel object identities and positions over time, supporting semantic transfer capabilities.

Figure 10: HPatches qualitative results—pixel correspondences predicted by single-life models on source/target frames.

Figure 11: Zero-shot segmentation tracking on DAVIS-2017 using features from HD-Epic, WT, and ALD lives.

Implications and Future Directions

Empirical results substantiate the Shared World Hypothesis: single-life models converge to highly aligned geometric representations, mirroring physical world regularities despite non-overlapping personal experience. Practical utility extends to depth estimation, correspondence, and zero-shot segmentation, with scaling properties indicating that further increases in single-life data duration can fully close the gap to diverse-data baselines. The paradigm supports the viability of egocentric video as a singular, self-supervised source for learning transferable priors, potentially revolutionizing camera/sensor-specific model deployment or privacy-sensitive applications.

Theoretically, the findings deepen understanding of representation alignment mechanisms. The CAS metric offers a means to probe functional similarity at fine granularity, relevant for comparative studies across architectures and scaling regimes. The robustness of learned representations to environmental redundancy further motivates future theoretical exploration into the sufficiency and necessity of data diversity for foundational perception.

Future research should investigate:

• Extension to semantic and multimodal learning tasks
• Lifelong and adaptive learning across multiple sequential lives
• Model architectures explicitly designed for egocentric sensory streams
• Application to embodied learning, robotics, or human-robot interaction

Conclusion

This work demonstrates that functionally aligned and generalizable geometric models can be learned exclusively from the egocentric video of a single individual. The Shared World Hypothesis is supported both quantitatively and qualitatively. The egocentric perspective, with its structured motion and context, is both necessary and sufficient for synthesizing robust geometric priors, with practical performance matching that of diverse web-mined data at equivalent scale. This paradigm has significant implications for the future of visual foundation models, privacy-aware learning, and representation theory in AI.