LeNEPA: Latent Euclidean Embedding Prediction

• LeNEPA is a latent predictive framework where models forecast next-step embeddings in Euclidean space rather than reconstructing raw inputs.
• It employs diverse loss metrics and regularization strategies, such as Euclidean and cosine losses, to align predicted embeddings with observed ones.
• The approach is versatile, having been applied in mobility, vision, EEG, time-series, and pixel-based world modeling scenarios.

Latent Euclidean Next-Embedding Prediction Architecture (LeNEPA) denotes a family of latent predictive models in which the training target is a future, masked, or next-step embedding in a Euclidean latent space rather than a reconstructed raw observation. In the literature, the term is used both as a conceptual framework and as the name of a concrete no-augmentation time-series self-supervised learning recipe. Across these uses, the recurring elements are a latent space in $\mathbb{R}^d$, a mechanism that maps available context into a predictive state, a metric or energy defined over latent embeddings, and an objective that brings predicted next embeddings closer to observed ones while avoiding collapse or spurious transitivity. The framework has been instantiated in mobility modeling, visual self-supervision, EEG foundation modeling, time-series representation learning, and world modeling from pixels (Chen et al., 2020, Xu et al., 18 Dec 2025, Panchavati et al., 17 Mar 2026, Chemeris et al., 1 Jul 2026, Maes et al., 13 Mar 2026).

1. Conceptual scope and defining pattern

A concise formulation appears in the mobility-pattern interpretation of LeNEPA: it posits “a shared latent Euclidean space for all entities relevant to next-step prediction,” a composition function $f$ that maps context into a context vector $V^c$, a scoring function that is a monotonic transform of negative Euclidean distance, and a training objective that aligns observed next steps closer than negatives. In the time-series formulation, the same idea becomes an augmentation-free self-supervised recipe in which a causal transformer predicts the next latent patch-token, the predictive loss is computed in a lightweight projected space that is discarded at evaluation, and stability is supplied by temporal isotropy regularization rather than an EMA teacher or stop-gradient target. In vision, the term is used for an explicit Euclidean extension of Next-Embedding Predictive Autoregression (NEPA), retaining causal next-embedding prediction in continuous space while making the Euclidean geometry explicit (Chen et al., 2020, Chemeris et al., 1 Jul 2026, Xu et al., 18 Dec 2025).

This usage suggests that LeNEPA is best understood as a design family rather than a single algorithm. Some instantiations are strictly causal and autoregressive; others are masked and become “next-embedding” predictors when the masked block lies to the right of visible context. Some use direct squared Euclidean regression; others use cosine regression on normalized vectors, pairwise Euclidean ranking, or robust Euclidean losses. What remains invariant is that prediction is performed in latent space and that latent geometry is treated as a primary modeling object (Panchavati et al., 17 Mar 2026, He et al., 21 Nov 2025).

Instantiation	Domain	Characteristic form
MPE	Traffic trajectory prediction	Additive context vector and Euclidean next-location scoring
NEPA / LeNEPA extension	Vision self-supervision	Causal Transformer predicts next patch embeddings
Laya	EEG foundation modeling	Masked temporal latent prediction with Euclidean loss and SIGReg
LeNEPA	Time-series SSL	No-augmentation next-latent-token prediction with temporal SIGReg
LeWorldModel	Pixel-based world models	Next-embedding prediction plus Gaussian latent regularization

2. Mathematical formulations

A canonical LeNEPA objective in time-series SSL takes raw signals $x \in \mathbb{R}^{B\times C\times L}$, tokenizes them into latent patches $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$, applies a causal backbone $f_\phi$, and computes prediction loss after a one-step shift in a projected space:

$$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$

The full objective adds temporal isotropy regularization,

$$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$

with the main PTB-XL/Diag setting using $\lambda_{\mathrm{pred}}=1$, $\lambda_T=20$, and $f$0 for SIGReg placement (Chemeris et al., 1 Jul 2026).

In visual NEPA, the autoregressive target is the next patch embedding, with teacher forcing and stop-gradient on targets:

$f$1

and the baseline objective is negative cosine similarity after $f$2 normalization,

$f$3

The Euclidean LeNEPA variant replaces or complements this with

$f$4

Because normalized embeddings satisfy $f$5, the cosine and Euclidean views are equivalent up to scale and shift under unit normalization (Xu et al., 18 Dec 2025).

Earlier Euclidean next-step formulations appear in mobility prediction. MPE defines a context vector

$f$6

and scores candidate next locations by negative squared Euclidean distance:

$f$7

Training replaces full normalization with a pairwise logistic ranking loss using negative sampling, enforcing that the observed next location is closer to $f$8 than sampled negatives. In other LeNEPA-style variants, the loss can instead be a robust Euclidean criterion, as in DSeq-JEPA’s Huber regression over region latents, or Euclidean masked latent prediction, as in Laya’s

$f$9

These formulations differ in optimization details but share latent-target prediction as the central training signal (Chen et al., 2020, He et al., 21 Nov 2025, Panchavati et al., 17 Mar 2026).

3. Latent geometry, isometry, and stabilization

The “Euclidean” component of LeNEPA is not merely a choice of norm; it is often tied to explicit geometric conditioning. In visual NEPA, latent geometry is shaped by unit-length normalization, causal conditioning, and stop-gradient on targets. The causal mask is strictly triangular, Rotary Position Embedding (RoPE) is applied to queries and keys at all layers, and the architecture uses LayerNorm, QK-Norm, LayerScale, and SwiGLU. Empirically, removing stop-gradient collapses training, with loss converging to $V^c$0 under cosine regression because all embeddings become identical; no teacher-student or momentum encoder is used for targets, and the reported EMA with decay $V^c$1 serves only to smooth validation accuracy during ablations (Xu et al., 18 Dec 2025).

The time-series LeNEPA paper replaces the stop-gradient/EMA mechanism used by vanilla NEPA with temporal SIGReg-based isotropy regularization over per-sample temporal token sets. The predictive loss is computed in a lightweight MLP+BatchNorm+ReLU projected space of dimension $V^c$2, and this projector is discarded at evaluation, a design identified as “Guillotine-style” loss-space decoupling. The reported ablations state that computing prediction and SIGReg losses in the projected space and discarding the projector at evaluation improves 22/24 last-step probe comparisons, while temporal per-sample SIGReg is the only single-component placement with sustained frozen-backbone gains across PTB-XL and Diag (Chemeris et al., 1 Jul 2026).

Laya combines Euclidean masked latent prediction with stop-gradient and Sketched Isotropic Gaussian Regularization. Its total objective is

$V^c$3

with $V^c$4 in practice; removing SIGReg or reducing $V^c$5 below approximately $V^c$6 caused collapse. Batch normalization in the projector contributes additional conditioning, and SIGReg operates on projected summary embeddings to enforce isotropic, well-conditioned geometry (Panchavati et al., 17 Mar 2026).

A stronger geometric statement is supplied by the isometric autoencoder literature. The Rate-Distortion Optimization guided autoencoder establishes a constantly-scaled orthonormal system in the decoder Jacobian,

$V^c$7

with $V^c$8, yielding a scaled isometry between data-space and Euclidean latent-space tangent vectors. This supports the broader LeNEPA intuition that next-embedding prediction is easier and more stable when latent coordinates are whitened, isotropic, and metrically well behaved (Kato et al., 2019).

4. Architectural realizations across domains

In mobility modeling, LeNEPA appears as an additive factor model over users, time slots, and locations. MPE embeds objects, time slots, current locations, and next locations into a shared low-dimensional latent space, with role-separated location embeddings $V^c$9 and $x \in \mathbb{R}^{B\times C\times L}$0 introduced specifically to suppress “phantom transitions.” Sequential, personal, and temporal factors are combined linearly in $x \in \mathbb{R}^{B\times C\times L}$1, candidate restriction is imposed through the road-transition graph, and inference ranks next locations over $x \in \mathbb{R}^{B\times C\times L}$2 by Euclidean energy (Chen et al., 2020).

In vision self-supervision, the architecture is a single-stream causal Transformer over Conv2d patch embeddings. NEPA reports ViT-B and ViT-L backbones with patch size $x \in \mathbb{R}^{B\times C\times L}$3, RoPE, LayerScale initialized to $x \in \mathbb{R}^{B\times C\times L}$4, SwiGLU gating, QK-Norm, and strictly causal attention. No decoder, no masking, no contrastive negatives, and no task-specific heads are used in pretraining; the only objective is next-embedding prediction in continuous space (Xu et al., 18 Dec 2025).

In EEG, Laya uses a markedly different front end while keeping the latent-prediction principle. Raw EEG $x \in \mathbb{R}^{B\times C\times L}$5 is patchified by depthwise 1D convolution per channel, then passed through a dynamic channel mixer that uses learned queries with cross-attention over channels and Fourier-encoded electrode coordinates as spatial priors. A Transformer encoder with RoPE produces token embeddings and a mean-pooled summary, a 3-layer MLP projector with batch normalization maps them into a projected space, and a lightweight Transformer predictor infers masked temporal targets under contiguous block masking with ratio approximately $x \in \mathbb{R}^{B\times C\times L}$6 and block sizes 5–10 patches (Panchavati et al., 17 Mar 2026).

DSeq-JEPA extends latent next-embedding prediction to an explicitly ordered region sequence. A target encoder produces a saliency map from cosine similarity between a [CLS] token and patch tokens, Otsu thresholding and connected components define irregular non-overlapping regions, and these regions are ranked by discriminative score. A context encoder sees only the union of the first $x \in \mathbb{R}^{B\times C\times L}$7 regions, and a lightweight sequential predictor estimates the latent of the $x \in \mathbb{R}^{B\times C\times L}$8-th region under a Huber loss with $x \in \mathbb{R}^{B\times C\times L}$9. The resulting chain is “GPT-style” in order but JEPA-style in latent prediction (He et al., 21 Nov 2025).

In world modeling from pixels, LeWorldModel and NE-Dreamer show two different routes. LeWorldModel uses a ViT-Tiny encoder with a one-layer MLP projector and BatchNorm, a 6-layer action-conditioned transformer predictor with Adaptive LayerNorm, and a two-term objective consisting only of next-embedding prediction and Gaussian latent regularization via SIGReg. NE-Dreamer instead combines an RSSM backbone with a lightweight causal temporal transformer over histories of $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$0 and aligns predicted next encoder embeddings to stop-gradient targets using a Barlow Twins cross-correlation objective. The latter is explicitly described as a mapping to LeNEPA, but the paper notes that similarity is operationalized through cross-correlation rather than explicit Euclidean or cosine distance (Maes et al., 13 Mar 2026, Bredis et al., 3 Mar 2026).

5. Empirical behavior and reported results

The reported results show that latent next-embedding prediction is effective across markedly different data modalities. In vision, NEPA pretrained on ImageNet-1K attains $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$1 and $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$2 top-1 accuracy on ImageNet-1K with ViT-B and ViT-L after fine-tuning, and reaches $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$3 and $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$4 mIoU on ADE20K with a UPerNet head. Key ablations report $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$5 accuracy at 50k steps with shift+causal+stop-grad, a drop to $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$6 when causal masking is removed, divergence when the prediction shift is removed, and collapse when stop-gradient is removed (Xu et al., 18 Dec 2025).

In time-series SSL, the fixed-horizon frozen-probe protocol on PTB-XL and Diag is central. On PTB-XL classification, LeNEPA with $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$7, layers $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$8, and fixed L4 readout reports AUROC/AUPRC of $z=\tau_\theta(x)\in\mathbb{R}^{B\times T\times D}$9; on Diag it reports AUROC $f_\phi$0, AUPRC $f_\phi$1, MSE $f_\phi$2, MAE $f_\phi$3, Pearson $f_\phi$4, and $f_\phi$5 $f_\phi$6. Learning-curve analysis states that LeNEPA reaches $f_\phi$7 of its final AUROC/AUPRC gain after 2–5k updates, compared with 5–10k updates for the faster JEPA readout. As a separate frozen-encoder check, LeNEPA-CauKer reaches $f_\phi$8 mean UCR-128 Random-Forest accuracy in a single-seed, best-checkpoint run, within 1.16 points of Mantis and within 0.24 points of MOMENT (Chemeris et al., 1 Jul 2026).

In EEG, Laya-S trained on approximately $f_\phi$9 of the corpus for 10k steps achieves the best mean clinical balanced accuracy, $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$0, compared with $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$1 for LaBraM and $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$2 for LUNA. On seizure detection, Laya-S reaches $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$3 balanced accuracy and Laya-full reaches $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$4, compared with $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$5 for LaBraM. Under combined 10 dB SNR noise on abnormal detection, Laya retains $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$6 of clean accuracy, whereas LaBraM drops to $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$7 (Panchavati et al., 17 Mar 2026).

In mobility prediction, MPE reports top-3 accuracy and average precision@3 of $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$8 and $$\mathcal{L}_{\mathrm{pred}}= \frac{1}{B(T-1)} \sum_{b=1}^{B}\sum_{t=1}^{T-1} \left\| h_\psi(\hat z_{b,t})-h_\psi(z_{b,t+1}) \right\|_2^2.$$9 on VPR, compared with $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$0 and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$1 for Geo-Teaser and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$2 and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$3 for Bayes. On Taxi, MPE reports $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$4 top-3 accuracy and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$5 average precision@3, compared with $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$6 and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$7 for Geo-Teaser and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$8 and $$\mathcal{L}_{\mathrm{total}}= \lambda_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}}+ \lambda_T \mathcal{L}_{\mathrm{SIG}^{\mathrm{time}}},$$9 for Bayes. The ablations show MPE-object $\lambda_{\mathrm{pred}}=1$0 MPE-time $\lambda_{\mathrm{pred}}=1$1 MPE-plain on both datasets, indicating that personal preferences have a stronger effect than temporal signals in these data (Chen et al., 2020).

In image JEPA variants, DSeq-JEPA improves linear probing on ImageNet from $\lambda_{\mathrm{pred}}=1$2 to $\lambda_{\mathrm{pred}}=1$3 for ViT-B/16 and from $\lambda_{\mathrm{pred}}=1$4 to $\lambda_{\mathrm{pred}}=1$5 for ViT-L/16 relative to I-JEPA, while also improving fine-grained categorization, MS-COCO detection and segmentation, ADE20K segmentation, and CLEVR reasoning benchmarks. In pixel-based world modeling, LeWorldModel reports PushT success rate $\lambda_{\mathrm{pred}}=1$6, compared with $\lambda_{\mathrm{pred}}=1$7 for PLDM and $\lambda_{\mathrm{pred}}=1$8 for DINO-WM, and plans in under one second while being reported as up to $\lambda_{\mathrm{pred}}=1$9 faster than foundation-model-based world models (He et al., 21 Nov 2025, Maes et al., 13 Mar 2026).

6. Limitations, misconceptions, and open directions

A recurring misconception is that LeNEPA denotes a single loss or a single stabilization method. The record is more heterogeneous. MPE uses pairwise logistic ranking under Euclidean energy; visual NEPA uses normalized cosine regression with stop-gradient and admits an explicit $\lambda_T=20$0 interpretation; Laya uses Euclidean masked latent prediction with SIGReg and projector batch normalization; the time-series LeNEPA recipe removes stop-gradient and EMA entirely and relies on temporal SIGReg in a disposable projected space; DSeq-JEPA uses a Huber loss over region latents; and NE-Dreamer is only a mapped relative, since its alignment is implemented with a Barlow Twins objective rather than an explicit Euclidean or cosine metric (Chen et al., 2020, Xu et al., 18 Dec 2025, Panchavati et al., 17 Mar 2026, Chemeris et al., 1 Jul 2026, He et al., 21 Nov 2025, Bredis et al., 3 Mar 2026).

The limitations are similarly domain-specific. MPE is first-order and uses only the immediate previous location; static embeddings do not capture drift, seasonality, or events, and road topology is only indirectly encoded. Visual NEPA is sensitive to stop-gradient, causal masking, and proper shifting; removing any of these causes collapse or divergence. Laya shows that scaling to more data and longer training does not automatically improve mean clinical balanced accuracy, and subject-specific motor-imagery decoding remains difficult under linear probing. The time-series LeNEPA recipe still depends on tokenizer choices, SIGReg scale, and layer-placement hyperparameters, while intermediate-layer probing is often preferable to final-layer probing. LeWorldModel is limited by short-horizon rollouts, offline data coverage, and the requirement for action labels, and SCMHSA-based video prediction underperforms on KTH despite strong gains on UCSD, UCF Sports, and Penn Action (Maes et al., 13 Mar 2026, Nguyen et al., 28 Jan 2025, Chemeris et al., 1 Jul 2026, Panchavati et al., 17 Mar 2026, Chen et al., 2020).

The dominant future directions already appear within the literature itself. Proposed extensions include replacing additive composition with RNNs or Transformers over longer histories, augmenting location embeddings with GNN layers over adjacency graphs, adding multi-step prediction horizons, introducing dynamic or time-conditioned embeddings, coupling latent predictors to lightweight image decoders or diffusion heads, and extending latent next-embedding prediction to multimodal settings such as text-image or audio. A plausible implication is that LeNEPA will remain most coherent as a geometric principle—predict in latent space, regularize the latent geometry, and evaluate whether the resulting embeddings retain probe-useful structure across domains—rather than as a rigidly fixed algorithmic template (Chen et al., 2020, Xu et al., 18 Dec 2025, Chemeris et al., 1 Jul 2026, Maes et al., 13 Mar 2026).