P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture

Abstract: The increasing maturity of embodied AI platforms has driven a growing interest in procedural video representation learning to support intelligent assistance systems for complex, multi-step tasks. Leveraging large-scale latent predictive training, video foundation models capture video dynamics, enabling downstream tasks such as activity understanding, spatiotemporal localization, and predictive control. However, procedural videos include actions with long-range dependencies that these models do not support, due to the quadratic complexity of self-attention. Distinct actions, for example, may be visually similar despite appearing at different points in the procedure, such as turning the stove on versus off. Here, we propose a backbone-agnostic approach that learns long-duration video representations by reducing the problem to a dense, frame-aligned action space and predicting pooled masked latent vectors. This approach allows our Procedural Joint Embedding Predictive Architecture (P-JEPA) to ingest videos over 30 minutes long, enabling effective long-form understanding of procedural steps. We evaluate P-JEPA using features extracted with VJEPA2.1, TSM, and I3D over the EgoExo4D, EgoProceL, and Assembly101 datasets, finding that it consistently improves linear separability, streaming inference, and temporal action segmentation performance, achieving state-of-the-art results on EgoExo4D fine-grained action classification while using an order of magnitude fewer parameters than LLM-based methods and running in real time.

• The paper presents P-JEPA, a self-supervised framework that uses masked latent prediction and clip-causal attention to capture procedural structure in long videos.
• It employs dense frame-wise feature extraction with a teacher-student architecture and rotary positional embeddings to ensure effective alignment across segments.
• P-JEPA achieves state-of-the-art accuracy on benchmarks like EgoExo4D and Assembly101 while enabling real-time streaming inference with significantly fewer parameters than existing VideoLLMs.

P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture

Motivation and Context

Procedural video understanding presents a significant challenge in video representation learning, characterized by sequences of actions where contextual dependencies and long-range temporal structure are critical for interpretation. Standard video backbone architectures, optimized for short clips, fail to capture these long-horizon dependencies due to both model capacity constraints (especially the quadratic cost of self-attention) and the difficulties of aligning representations across visually similar but procedurally distinct segments. VideoLLMs mitigate some challenges by leveraging sparse sampling and large-scale context with LLMs but remain limited in their ability to perform authentic sequence-level temporal reasoning, often being outperformed by single-frame baselines on diagnostic tasks.

The limitations of both traditional short-term vision backbones and current VideoLLMs motivate the pursuit of architecture and training protocols explicitly designed for dense, procedure-aware, long-form video representation learning without reliance on language supervision.

Model Architecture and Methodology

P-JEPA (Procedural Joint Embedding Predictive Architecture) introduces a backbone-agnostic self-supervised approach to capture procedural structure in long (up to 30-minute) videos. The procedure consists of:

• Feature Extraction: Dense frame-wise features are extracted using pretrained models such as I3D, TSM, or V-JEPA2.1. Frame-wise tokens are pooled (via mean-pooling or a supervised cross-attention mechanism) to yield one token per frame, ensuring alignment across heterogeneous feature extractors.
• Masked Latent Prediction: A transformer-based encoder is trained to predict the masked latent representations of segments, inspired by joint-embedding predictive architectures (JEPA). The teacher encoder, updated as an exponential moving average of the student, defines targets in latent space. Random masking, typically covering 60–80% of tokens, forces the network to reason about hidden procedure steps from available context.
• Clip-Causal Attention: Critical to procedural modeling, P-JEPA’s encoder applies a segment-causal attention mask, where tokens can attend to tokens within their own segment and to all tokens in preceding segments, but never to future segments. Rotary positional embeddings are applied across both segment and within-segment indices, supporting both global procedural context and precise local order.
• Streaming Inference: At test time, only the procedure-aware encoder is used, providing contextualized representations for downstream probes in an online, frame-by-frame process.

Empirical Results

P-JEPA demonstrates significant and consistent gains across major procedural video benchmarks:

• EgoExo4D Fine-Grained Action Classification: On the EgoExo4D benchmark, P-JEPA combined with a streaming-classifier achieves an accuracy of 56.85%, substantially above TimeSformer (35.1%) and LLM-based Pro VideLLM-11 (44.4%), while using $\sim$1.2B parameters—nearly an order of magnitude less than most VideoLLMs. Inference is real-time (20fps on consumer GPUs).
• Assembly101 Temporal Action Segmentation: Applying P-JEPA features with an LTContext probe yields state-of-the-art segmentation: frame accuracy of 42.9% (TSM), outperforming the strong supervised and TSM baseline. Ablations show the clip-causal attention mask confers the highest linear probe accuracy (40.1%) versus strictly token-causal or bidirectional masking.
• EgoProceL Streaming and Low-data Regimes: In streaming (online) inference on EgoProceL, FACT with P-JEPA features outperforms the original FACT by almost 20 accuracy points in the causal regime, and the gains are especially pronounced under label scarcity—FACT trained on 33% data with P-JEPA features matches or exceeds 50% data performance with raw features.

Qualitative analyses such as t-SNE visualizations and feature-space progress metrics corroborate that P-JEPA organizes features to respect procedural order and contextual differentiation, capturing not only local visual evidence but also placement within the procedural trajectory.

Implications and Theoretical Significance

P-JEPA answers whether long-horizon procedural video understanding can be achieved in a self-supervised regime, without text supervision, complex language modeling, or external knowledge graphs. The results indicate that self-supervised masked latent prediction across densely aligned features, when scaffolded by well-designed temporal attention masks, organizes the representation space for strong linear separability and temporal modeling. Notably, the resulting representations are both highly reusable and efficient, serving as input to existing temporal models and improving their robustness (especially with limited labeled data).

There are significant practical implications: real-time streaming inference becomes feasible for complex multi-step tasks in robotics, AR assistance, and process monitoring without introducing LLM-driven latency/bloat. Theoretically, P-JEPA bridges hierarchical procedural modeling and self-supervised representation learning, and decouples semantic procedural abstraction from direct language grounding.

Limitations and Future Directions

P-JEPA’s reliance on predefined segment boundaries (for clip-causal attention) may limit deployment in fully unconstrained real-world video streams, where preprocessing to infer boundaries is non-trivial. Additionally, action-pooled features require some supervision for training the cross-attention pooler, which introduces a partial dependence on labeled action data during feature extraction. The mean-pool approach, though label-free, underperforms.

Further research opportunities include block-causal attention (removing boundary reliance), integrating dynamic boundary discovery, and compressive video tokenization for even longer sequences. Extending the JEPA objective to incorporate weak or sparse procedural cues (narration, graphs) and cross-modal alignment could yield further improvements.

Conclusion

P-JEPA establishes a highly effective and efficient paradigm for procedural video representation learning, producing state-of-the-art results on fine-grained classification and temporal segmentation in long videos with substantial parameter and inference efficiency. The approach demonstrates that dense, self-supervised, and procedure-aware representations can surpass existing short-clip architectures and VideoLLMs on diagnostics requiring authentic temporal reasoning and procedural context, thus advancing the foundation for embodied AI video understanding (2606.23256).