P-JEPA: Procedural Joint Embedding Predictive Architecture

• P-JEPA is a JEPA-based architecture that learns procedural representations by predicting masked latent tokens from temporally ordered, dense feature streams.
• It employs a clip-causal attention mechanism that aggregates context within current segments while enforcing causality across procedural steps.
• The model supports both long-form video understanding and deterministic MDP settings, improving action recognition and streaming inference efficiency.

Searching arXiv for papers on P-JEPA and closely related JEPA variants. Procedural Joint Embedding Predictive Architecture (P-JEPA) denotes a JEPA-family architecture for long-horizon procedural representation learning in which masked latent prediction is applied to temporally ordered, procedure-sensitive inputs rather than to isolated static regions. In the procedural-video formulation, P-JEPA operates over dense, frame-aligned feature streams extracted from long videos and learns to predict masked teacher latents from visible context while preserving causal procedural structure, thereby targeting representations that encode fine-grained actions, temporal progress, and context-dependent action meaning over takes exceeding 30 minutes (Tristram et al., 22 Jun 2026). The same abbreviation has also been used for a simplified action-conditioned JEPA variant in deterministic MDPs, where an encoder, latent transition model, and auxiliary head are jointly trained to prevent unhealthy representation collapse through dynamics and auxiliary supervision (Yu et al., 12 Sep 2025). This dual usage is consequential: one line treats P-JEPA as a procedural-video foundation architecture, while the other treats it as a procedural or predictive latent world-model abstraction.

1. Terminology, scope, and lineage

The procedural-video meaning of P-JEPA originates in “P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture,” which studies long, ordered, multi-step videos such as cooking, repair, and assembly, emphasizing that procedural understanding requires representations of fine-grained local motion, state changes induced by previous actions, temporal progress through the task, and context-dependent action semantics (Tristram et al., 22 Jun 2026). The motivating difficulty is that procedural videos are often tens of minutes long, whereas most video foundation models operate on short clips, and self-attention scales quadratically in token count. A further difficulty is procedural ambiguity: actions such as turning a stove on versus off may be visually similar yet semantically distinct because of their position in the procedure (Tristram et al., 22 Jun 2026).

A separate usage appears in “Why and How Auxiliary Tasks Improve JEPA Representations,” which calls a simplified model-based-RL JEPA variant “P-JEPA.” There, the architecture consists of an encoder $z_t = E_\phi(o_t)$, a latent transition model $\hat z_{t+1} = T_\psi(z_t, a_t)$, a target embedding path $z_{t+1} = E_\phi(o_{t+1})$, and an auxiliary head $P_\theta(z_t)$, trained jointly with latent dynamics and an auxiliary prediction loss (Yu et al., 12 Sep 2025). In that setting, “P-JEPA” denotes a procedural or predictive JEPA in deterministic MDPs rather than a long-form procedural-video encoder.

This terminological ambiguity matters because the two usages emphasize different aspects of procedure. The procedural-video formulation is backbone-agnostic, clip-causal, and focused on long-context representation learning over dense feature streams (Tristram et al., 22 Jun 2026). The deterministic-MDP formulation is focused on representation collapse, bisimulation-like distinctions, and auxiliary-task-induced state abstraction (Yu et al., 12 Sep 2025). A plausible synthesis is that both instantiate the broader JEPA idea—predict latent targets from context in embedding space rather than reconstructing observations—but they operationalize “procedure” at different levels.

The broader JEPA literature provides several bridges into this procedural interpretation. TS-JEPA shows how the JEPA recipe transfers from static inputs to temporally ordered sequences using visible-versus-hidden temporal patches, an EMA teacher, and latent $L_1$ regression (Ennadir et al., 29 Sep 2025). Seq-JEPA goes further by introducing action-conditioned sequential prediction over observation sequences and an aggregate latent state, which can be read as a short-horizon world-model precursor to procedural JEPA (Ghaemi et al., 6 May 2025). These works do not define procedural video P-JEPA, but they clarify how JEPA changes when the prediction problem is temporal, ordered, and action- or transition-sensitive.

2. Procedural-video P-JEPA architecture

In the procedural-video formulation, the key reduction is from raw RGB to a dense, frame-aligned action space. A long video is converted into one token per frame timestep at 4 fps using a pretrained short-range backbone such as V-JEPA 2.1, TSM, or I3D, and P-JEPA then learns a procedural contextualizer over that feature stream rather than over RGB patches (Tristram et al., 22 Jun 2026). The paper states this explicitly as reducing long-duration video modeling to a dense, frame-aligned action space and predicting pooled masked latent vectors (Tristram et al., 22 Jun 2026).

For a video organized into annotated segments $s=1,\dots,S$, segment $s$ contributes

$$\mathbf{X}_s = (\mathbf{x}_1,\ldots,\mathbf{x}_{m_s}), \qquad \mathbf{x}_t \in \mathbb{R}^{d_{\mathrm{in}}},$$

and the full take is

$$\mathbf{X} = \mathbf{X}_1 \Vert \cdots \Vert \mathbf{X}_S,$$

with valid length

$T = \sum_{s=1}^S m_s.$

Each token index $\hat z_{t+1} = T_\psi(z_t, a_t)$0 carries both a segment index $\hat z_{t+1} = T_\psi(z_t, a_t)$1 and a within-segment index $\hat z_{t+1} = T_\psi(z_t, a_t)$2 (Tristram et al., 22 Jun 2026). This dual indexing is central because the model encodes both global procedural progress and local order inside the current step.

The learnable components are a student encoder $\hat z_{t+1} = T_\psi(z_t, a_t)$3, a predictor $\hat z_{t+1} = T_\psi(z_t, a_t)$4, and a teacher encoder $\hat z_{t+1} = T_\psi(z_t, a_t)$5, where the teacher is an EMA copy of the student with momentum $\hat z_{t+1} = T_\psi(z_t, a_t)$6 (Tristram et al., 22 Jun 2026). During training, the student sees only visible context tokens, the teacher sees the full valid stream, and the predictor fills masked positions with learned mask tokens and predicts teacher latents at masked positions. At inference, the teacher and predictor are discarded and only the student encoder is retained (Tristram et al., 22 Jun 2026).

The masking mechanism samples a valid-token subset $\hat z_{t+1} = T_\psi(z_t, a_t)$7 with masking ratio $\hat z_{t+1} = T_\psi(z_t, a_t)$8, and the visible context is $\hat z_{t+1} = T_\psi(z_t, a_t)$9. Reported masking ratios are 60% or 80%, with supplementary notes specifying $z_{t+1} = E_\phi(o_{t+1})$0 for EgoExo4D and EgoProceL and $z_{t+1} = E_\phi(o_{t+1})$1 for Assembly101 TSM runs (Tristram et al., 22 Jun 2026). The latent targets and predictions are

$z_{t+1} = E_\phi(o_{t+1})$2

and the loss is

$z_{t+1} = E_\phi(o_{t+1})$3

The design is intentionally simple: latent prediction only, EMA teacher, and average $z_{t+1} = E_\phi(o_{t+1})$4 loss on masked positions, with no additional normalization losses, contrastive terms, variance regularizers, or auxiliary objectives reported in the paper (Tristram et al., 22 Jun 2026).

A distinctive architectural contribution is clip-causal attention. The encoder allows a token to attend to all tokens in the same segment and all tokens in previous segments, but never to future segments: $z_{t+1} = E_\phi(o_{t+1})$5 Thus, attention is fully visible within a segment but causal across segments (Tristram et al., 22 Jun 2026). The predictor uses standard token-causal attention over the flattened stream: $z_{t+1} = E_\phi(o_{t+1})$6 This asymmetry is important: the encoder aggregates step-local context while respecting procedural directionality across segments, whereas the predictor unfolds left-to-right (Tristram et al., 22 Jun 2026).

The positional scheme is a 2D rotary positional embedding over segment index $z_{t+1} = E_\phi(o_{t+1})$7 and within-segment position $z_{t+1} = E_\phi(o_{t+1})$8. For each attention head,

$z_{t+1} = E_\phi(o_{t+1})$9

$P_\theta(z_t)$0

The intended semantics are explicit: $P_\theta(z_t)$1 encodes progress through the procedure, and $P_\theta(z_t)$2 encodes local order inside the segment (Tristram et al., 22 Jun 2026). This makes procedural order a first-class part of the latent geometry.

3. Feature pipelines, training regime, and long-context modeling

P-JEPA is backbone-agnostic in the sense that it operates on frame-aligned dense features rather than requiring a specific video encoder (Tristram et al., 22 Jun 2026). The paper evaluates three feature families: V-JEPA 2.1, TSM, and I3D. For I3D and TSM, it uses existing pre-extracted features. For V-JEPA 2.1, which emits patchwise outputs, the paper reduces each clip to one token per frame using either mean pooling or a learnable action-pool module (Tristram et al., 22 Jun 2026).

The action-pool variant first applies a small self-attention stack to frozen V-JEPA outputs $P_\theta(z_t)$3, then uses cross-attention from $P_\theta(z_t)$4 learnable queries: $P_\theta(z_t)$5 After training a small attentive classifier with action labels, the classifier is discarded and the pooled tokens are extracted with non-overlapping sliding windows, with $P_\theta(z_t)$6 set to match the number of input frames so that the result is again one token per frame (Tristram et al., 22 Jun 2026). This supervised action-pooling stage is outside the JEPA loss itself but materially affects the quality of the procedural token stream.

A learned LayerNorm-plus-linear projection maps features from $P_\theta(z_t)$7 to model width $P_\theta(z_t)$8 when necessary (Tristram et al., 22 Jun 2026). The P-JEPA stack itself is shallow: encoder depth 4, predictor depth 2, trained for 1000 epochs with learning rate $P_\theta(z_t)$9, step decay by factor $L_1$0, on one Nvidia H200 GPU for roughly 8–12 hours depending on dataset (Tristram et al., 22 Jun 2026). The longest reported training takes are 34.5 minutes on EgoExo4D, 36.6 minutes on EgoProceL, and 30.8 minutes on Assembly101 (Tristram et al., 22 Jun 2026).

The feasibility of these durations follows from the input reduction rather than from an exotic transformer trick. P-JEPA uses one token per frame at 4 fps after feature extraction and optional pooling, making token count far smaller than in raw spatiotemporal patch transformers (Tristram et al., 22 Jun 2026). This permits full-take modeling while focusing the JEPA objective on procedural regularities rather than low-level appearance. A plausible implication is that P-JEPA is best viewed not as a replacement for short-clip video encoders, but as a procedural contextualization layer on top of them.

The paper explicitly evaluates streaming inference. At test time, the student encoder runs on the valid stream with no masking, producing clip-causal contextualized token features in which each token uses evidence from its own segment and all preceding segments, but not future segments (Tristram et al., 22 Jun 2026). Because the representation is trained with segment-causal structure, downstream online models degrade less than when using raw features.

4. Empirical performance and representational behavior

The empirical evaluation spans three datasets and several downstream protocols. EgoExo4D is used for fine-grained action classification with 278 classes and longest training take 34.5 minutes; EgoProceL is used for temporal action segmentation, streaming inference, sparse-label analysis, and feature analysis; Assembly101 is used for linear probing, temporal action segmentation, attention-mask ablation, and qualitative boundary analysis (Tristram et al., 22 Jun 2026).

The strongest headline result is on EgoExo4D fine-grained action classification. The official benchmark comparison reports top-1 accuracy of 35.13 for TimeSformer, 39.10 for EgoVLPv2, 38.19 for Viewpoint Distillation, 40.34 for View-Invariant Encoding, 42.62 for VideoLLM-MoD, 44.36 for ProVideLLM-8B/11, and 56.85 for P-JEPA + Streaming Probe (Tristram et al., 22 Jun 2026). The same section reports that the total stack has 1.2B parameters, roughly an order of magnitude fewer parameters than typical VideoLLMs, and runs at around 20 fps inference on an RTX 3090, including the V-JEPA 2.1 encoder, pooler, P-JEPA encoder, and streaming classifier (Tristram et al., 22 Jun 2026).

On Assembly101 linear probing, the gains indicate that P-JEPA moves useful procedural information directly into the representation. For V-JEPA mean-pool features, raw performance is 16.1 Acc / 3.3 Edit / 2.7,1.9,1.1 F1, and +P-JEPA gives 29.4 / 18.6 / 16.0,13.5,10.3. For V-JEPA action-pool, raw is 34.0 / 18.5 / 17.8,15.5,10.2 and +P-JEPA gives 40.0 / 27.8 / 26.0,23.0,16.9. For TSM, raw is 23.5 / 3.0 / 2.4,1.1,0.4 and +P-JEPA gives 40.1 / 15.0 / 15.2,13.1,9.8 (Tristram et al., 22 Jun 2026).

On Assembly101 temporal action segmentation with LTContext, P-JEPA also improves every backbone: V-JEPA mean-pool rises from 30.6 / 26.2 / 26.0,21.7,14.3 to 33.8 / 27.6 / 29.3,25.0,18.7; V-JEPA action-pool rises from 41.6 / 32.7 / 34.4,30.1,21.9 to 43.0 / 34.2 / 35.9,31.5,22.3; TSM rises from 40.2 / 32.2 / 34.3,30.5,22.4 to 42.9 / 35.9 / 37.3,34.0,26.4 (Tristram et al., 22 Jun 2026). The paper states that LTContext with TSM + P-JEPA sets a new state of the art on Assembly101 in their single-view setting (Tristram et al., 22 Jun 2026).

Streaming evaluation on EgoProceL is particularly relevant to procedural understanding because the benchmark prohibits future information leakage. With I3D features and FACT, raw streaming performance is 69.2 Acc$L_1$1 / 19.2 Acc / 18.1 Edit / 14.8,13.8,10.9 F1, and +P-JEPA gives 76.8 / 39.1 / 28.8 / 26.9,25.9,24.1. With LT-Causal, raw is 78.6 / 55.3 / 40.2 / 35.0,32.0,25.7 and +P-JEPA gives 80.8 / 58.8 / 48.9 / 42.1,40.3,34.4 (Tristram et al., 22 Jun 2026). The paper interprets this as evidence that P-JEPA substantially reduces degradation under streaming conditions.

The sparse-label analysis on EgoProceL shows the strongest relative benefit when labels are scarce. With FACT trained on 15% of data, I3D gives 76.1 / 50.8 / 41.8 / 41.5,37.8,26.3 and P-JEPA gives 78.9 / 50.1 / 52.1 / 48.6,45.6,36.9. At 33%, I3D gives 80.7 / 61.4 / 53.9 / 52.3,49.5,38.5 and P-JEPA gives 84.5 / 66.1 / 65.2 / 60.1,57.9,52.2. At 50%, I3D gives 83.7 / 67.5 / 64.6 / 62.5,60.0,48.1 and P-JEPA gives 88.0 / 74.4 / 72.7 / 67.3,65.3,59.8. At 100%, the gap narrows (Tristram et al., 22 Jun 2026). The paper notes that FACT + P-JEPA with 33% of data performs on par with or better than FACT on raw I3D with 50% of data (Tristram et al., 22 Jun 2026).

The attention-mask ablation on Assembly101 isolates the value of clip-causal structure. With TSM features and linear probing, token-causal gives 35.6 Acc / 7.6 Edit / 7.0,5.6,1.0, bidirectional gives 37.6 / 12.2 / 12.2,9.8,7.2, and clip-causal gives 40.1 / 15.0 / 15.2,13.1,9.8 (Tristram et al., 22 Jun 2026). The paper’s interpretation is concise: strict token-causal is too restrictive, bidirectional helps but is still worse, and the right temporal inductive bias is free interaction inside the current local step and causality across steps.

The supplementary feature analyses reinforce the claim that P-JEPA reorganizes the representation around procedural progress. On EgoProceL, Time-progress Spearman increases from 0.6078 to 0.9351, progress efficiency from 0.3512 to 0.7685, and adjacent temporal nearest-neighbor fraction from 0.4737 to 0.8000 (Tristram et al., 22 Jun 2026). The paper also reports that raw I3D features fail to separate the similar actions “place the motherboard and fix it” and “remove the motherboard,” whereas P-JEPA features form two distinct clusters and permit a linear probe to recognize the “remove the motherboard” class at all (Tristram et al., 22 Jun 2026). The trustworthy takeaway is that P-JEPA materially improves temporal alignment and context disambiguation, not merely classification scores.

5. Theoretical and conceptual extensions beyond procedural video

The simplified deterministic-MDP P-JEPA formulation contributes a different kind of result. It studies a jointly trained encoder, latent transition model, and auxiliary prediction head with objective

$L_1$2

where

$L_1$3

Under deterministic MDP assumptions and perfect fitting,

$L_1$4

the paper proves a “No Unhealthy Representation Collapse” theorem: $L_1$5 where $L_1$6 is the largest bisimulation over the MDP and auxiliary function $L_1$7 (Yu et al., 12 Sep 2025). The direct procedural-video paper does not use this theorem, but the result is highly relevant conceptually: auxiliary tasks can determine which distinctions a procedural latent space must preserve.

Seq-JEPA offers another useful precursor. It processes short sequences of observations $L_1$8 and relative transformations $L_1$9, forms an aggregate representation

$s=1,\dots,S$0

and predicts the next latent

$s=1,\dots,S$1

using an EMA target encoder and cosine prediction loss (Ghaemi et al., 6 May 2025). The paper argues that encoder outputs become transformation-equivariant while the aggregate representation becomes more invariant (Ghaemi et al., 6 May 2025). This suggests that a procedural JEPA may benefit from architectural role separation between instantaneous observation/action-sensitive latents and higher-level procedure memory.

TS-JEPA shows a simpler temporal translation of JEPA. It tokenizes a time series into 10 non-overlapping patches, masks 70% of them, uses a transformer encoder on visible patches, a transformer predictor for masked latents, and an EMA-encoder with $s=1,\dots,S$2 for target latents, training with average $s=1,\dots,S$3 loss in latent space (Ennadir et al., 29 Sep 2025). The paper notes that prediction is performed strictly in latent space over time rather than by reconstructing raw values, and that this may be more robust to noise and confounding factors (Ennadir et al., 29 Sep 2025). A plausible implication is that procedural JEPA objectives should target latent future descriptions rather than exact future observations whenever procedural traces contain nuisance variation.

Theoretical work on JEPA geometry adds a further conceptual layer. “Intrinsic-Energy Joint Embedding Predictive Architectures Induce Quasimetric Spaces” proves that if a JEPA energy is an intrinsic least-action energy

$s=1,\dots,S$4

then $s=1,\dots,S$5 is a quasimetric (Kobanda et al., 12 Feb 2026). The paper also argues that symmetric finite energies are structurally mismatched with directed reachability (Kobanda et al., 12 Feb 2026). This is not a P-JEPA algorithm, but it suggests that procedural latent spaces oriented toward executability or cost-to-go should often be asymmetric rather than Euclidean.

Finally, “Value-guided action planning with JEPA world models” makes this control alignment explicit by shaping the latent geometry so that the negative goal-conditioned value approximates a distance or quasi-distance between state embeddings. Its value loss is

$s=1,\dots,S$6

with

$s=1,\dots,S$7

and $s=1,\dots,S$8 either Euclidean or quasimetric (Destrade et al., 28 Dec 2025). This work is not the same as procedural-video P-JEPA, but it clarifies a central issue for procedural JEPA more broadly: predictive latent dynamics are not automatically planning-friendly.

6. Limitations, misconceptions, and future directions

A common misconception is that P-JEPA is already a unified, standardized architecture. The literature instead shows at least two distinct usages of the term: a procedural-video JEPA for long-form video understanding (Tristram et al., 22 Jun 2026) and a simplified action-conditioned JEPA with auxiliary supervision in deterministic MDPs (Yu et al., 12 Sep 2025). The shared core is latent prediction from context, but the surrounding architecture, objective, and intended use differ substantially.

Another misconception is that procedural JEPA is inherently equivalent to generic video JEPA. The direct P-JEPA paper explicitly argues that short-window visual evidence is often insufficient for procedural ambiguity and that standard video models do not support long-range procedural dependencies because of quadratic self-attention cost (Tristram et al., 22 Jun 2026). P-JEPA addresses this by reducing the problem to dense frame-aligned action features and imposing clip-causal procedural structure. This is not the same as merely extending a short-clip encoder.

The procedural-video formulation also has explicit limitations. Clip-causal attention requires segment boundaries, which may not be available in live deployment; the authors suggest fixed-window block-causal alternatives as future work (Tristram et al., 22 Jun 2026). P-JEPA depends on frame-aligned action-relevant features, so weak backbones degrade performance; this is visible in the inferior performance of V-JEPA mean-pool relative to action-pool (Tristram et al., 22 Jun 2026). The stronger V-JEPA action-pool features require supervised training of the pooling/classification stage, so the overall best-performing pipeline can include supervised backbone-side adaptation even though P-JEPA itself is trained self-supervised (Tristram et al., 22 Jun 2026). The architecture also does not explicitly model hierarchy, subgoals, or symbolic state transitions (Tristram et al., 22 Jun 2026).

More generally, adjacent JEPA work indicates unresolved issues for any broader P-JEPA agenda. TS-JEPA is observation-only and not action-conditioned (Ennadir et al., 29 Sep 2025). Seq-JEPA uses short sequences and known relative transformations rather than latent procedural abstractions (Ghaemi et al., 6 May 2025). Value-guided JEPA planning highlights locality problems and suggests hierarchical representation spaces as future work (Destrade et al., 28 Dec 2025). The deterministic-MDP P-JEPA theorem is restricted to deterministic dynamics and zero-loss assumptions (Yu et al., 12 Sep 2025). These limitations suggest that current P-JEPA formulations are better understood as strong procedural representation learners than as complete procedural world models.

The most plausible future direction, stated directly in the procedural-video paper, is toward long-context embodied and assistive systems that require online procedural understanding over extended takes (Tristram et al., 22 Jun 2026). A plausible implication from the surrounding literature is that the next stage would combine several ingredients that currently appear separately: long-form clip-causal procedural context (Tristram et al., 22 Jun 2026), action-conditioned latent prediction and auxiliary-task-shaped abstraction (Yu et al., 12 Sep 2025), multi-step action-observation aggregation (Ghaemi et al., 6 May 2025), and geometry aligned with goal-conditioned controllability or quasimetric reachability (Destrade et al., 28 Dec 2025, Kobanda et al., 12 Feb 2026). If such components are integrated, P-JEPA would move from procedural representation learning toward modular causal state representations that are not only descriptive of procedures but operational for planning within them.