SkyJEPA: Latent Dynamics for Quadrotors

• SkyJEPA is a Joint Embedding Predictive Architecture that leverages latent space dynamics to enable high-frequency, long-horizon control of quadrotors in zero-shot sim-to-real scenarios.
• It uses parallel state and action encoders coupled with a physics-inspired prober to mitigate compounding errors typical of autoregressive predictors.
• Empirical results demonstrate robust sim-to-real transfer with improved trajectory accuracy, noise resilience, and reduced prediction errors compared to conventional methods.

SkyJEPA is a Joint Embedding Predictive Architecture (JEPA) specifically developed for high-frequency, long-horizon control of quadrotors in the zero-shot sim-to-real regime. By situating dynamics learning and control entirely in latent space and coupling this with a physics-inspired probing mechanism, SkyJEPA achieves accurate, interpretable, and robust prediction and control of agile aerial vehicles across diverse operating conditions without requiring any real-world fine-tuning. The architecture directly addresses the compounding error limitations of autoregressive state-space predictors and demonstrates empirical superiority in both open- and closed-loop settings, even under significant structural and environmental domain gap challenges (Rao et al., 22 Jun 2026).

1. Model Architecture and Latent Dynamics Formulation

SkyJEPA receives as input a history window of length $H$ containing full-state observations $x_{t-H},\ldots,x_t$ and corresponding actions $a_{t-H},\ldots,a_t$ at time $t$. The architecture consists of two parallel encoder networks:

• $\mathrm{Enc}_\theta$: Maps state history $(x_{t-H},…,x_t)$ to a latent $s_t \in \mathbb{R}^D$.
• $\mathrm{Enc}_\phi$: Encodes action history $(a_{t-H},…,a_t)$ to a latent $z_t \in \mathbb{R}^{D'}$.

Prediction is performed by a latent dynamics predictor $x_{t-H},\ldots,x_t$0, computing a one-step prediction

$x_{t-H},\ldots,x_t$1

and recursively rolling out for $x_{t-H},\ldots,x_t$2 steps using the encoded future actions, producing $x_{t-H},\ldots,x_t$3 for $x_{t-H},\ldots,x_t$4.

The latent model is trained with a multi-step prediction loss

$x_{t-H},\ldots,x_t$5

where $x_{t-H},\ldots,x_t$6 encodes the ground-truth future. An anti-collapse regularizer (SIGReg) with projection-based characteristic function alignment further regularizes the latent space.

This structure explicitly avoids autoregressive reconstruction into metric state at each step, mitigating the error accumulation endemic to state-space models under multi-step rollout (Rao et al., 22 Jun 2026).

2. Physics-Inspired Prober and Metric State Integration

Once the latent dynamics model is trained, a parameter-efficient prober $x_{t-H},\ldots,x_t$7 is learned (with frozen encoder/predictor weights) to project each latent $x_{t-H},\ldots,x_t$8 into interpretable physical residuals:

$x_{t-H},\ldots,x_t$9

where $a_{t-H},\ldots,a_t$0 denotes translational acceleration residual and $a_{t-H},\ldots,a_t$1 the rotational acceleration control matrix. These augment a nominal, parameter-free discrete-time kinematic integrator on $a_{t-H},\ldots,a_t$2: \begin{align*} \dot{v}t &= \frac{1}{m}\sum{i=0}^3 f_{i,t} R_t e_3 - g + \Delta \dot{v}t\ \Delta \tau_t &= K_t a_t\ p{t+1} &= p_t + v_t \Delta t\ v_{t+1} &= v_t + \dot{v}t \Delta t\ R{t+1} &= R_t\,\exp([\omega_t]\times \Delta t)\ \omega{t+1} &= \omega_t + \Delta \tau_t\,\Delta t \end{align*}

The prober’s parameters are optimized for metric accuracy over a horizon:

$a_{t-H},\ldots,a_t$3

where $a_{t-H},\ldots,a_t$4 denotes the integrator-predicted state (Rao et al., 22 Jun 2026).

3. Model Predictive Path Integral Control Integration

SkyJEPA is paired with a sampling-based model predictive path integral (MPPI) controller. The core routine involves:

• Maintaining a nominal action sequence $a_{t-H},\ldots,a_t$5,
• Sampling $a_{t-H},\ldots,a_t$6 candidate sequences via Gaussian perturbations,
• For each, encoding, rolling out latents, probing to obtain metric rollouts,
• Computing a trajectory cost in metric state and action space,

$a_{t-H},\ldots,a_t$7

• Softmax-weighted update of nominal sequence and receding-horizon execution (Rao et al., 22 Jun 2026).

This leverages SkyJEPA’s latent model for anticipatory control without reconstructing high-dimensional observation sequences at every planning step.

4. Automated Dataset Generation and Domain Randomization

To support robust sim-to-real transfer without incurring real-world data collection costs, SkyJEPA employs a structured simulation data pipeline:

• Physical parameters $a_{t-H},\ldots,a_t$8 are randomized within a domain-adapted envelope,
• Reference spatial trajectories are generated via zero-mean Gaussian processes composed with periodic kernels,
• Analytic flatness is used to compute corresponding velocities, accelerations, attitudes, and angular velocities,
• Closed-loop tracking is performed in simulation (via NMPC and MPPI) to generate realistic, dynamically feasible trajectories,
• Resulting datasets comprise 20,000 rollouts of 10 seconds sampled at 20 Hz over 500 randomized environments, split into 80/10/10 train/val/test (Rao et al., 22 Jun 2026).

This approach yields diverse, high-quality training data spanning broad variations in quadrotor dynamics and control tasks, narrowing the sim-to-real gap.

5. Empirical Performance and Comparative Results

SkyJEPA demonstrates significant empirical advantages:

• Long-horizon accuracy: Compounding ratio stays near 1 for much longer than autoregressive baselines, with per-step error growth $a_{t-H},\ldots,a_t$9 0.06 versus 0.23 at $t$0.
• Latent alignment: High cosine alignment (mean $t$1 0.75) of latent trajectories, as opposed to negative alignment for autoregressive predictive baselines.
• Metric accuracy: Physics-inspired prober reduces open-loop position RMSE from $t$28.8 m to $t$31.43 m, and attitude error from $t$453° to $t$54.7°.
• Noise robustness: Under increasing input noise, SkyJEPA maintains median pose RMSE 10–30% lower than predictive baselines.
• Sim-to-real transfer: Zero-shot sim-to-real closed-loop experiments show position RMSE reduction of 26–38%, and attitude RMSE reduction of 33–54% compared to MPPI with predictive world models across multiple reference trajectories and geometric perturbations.
• Generalization: Demonstrated robustness to payload (e.g., +300 g) or hardware modifications (propeller swap) with 25–35% accuracy improvement over non-JEPA baselines, without retraining.
• Data scaling: Increasing dataset volume improves Trajectory Distribution Quality (TDQ) from 0.01 to 0.94 and reduces state RMSE from 5.4 m to 1.4 m (Rao et al., 22 Jun 2026).

6. Architectural Innovation: Comparison and Relation to UWM-JEPA

The design of SkyJEPA builds on insights from broader JEPA literature, particularly the role of latent structure and dynamics for predictive world modeling under uncertainty. UWM-JEPA (Radha et al., 25 May 2026) introduces density-matrix latent geometry and unitary predictor dynamics to preserve belief states under blind rollout in partially observed domains. While SkyJEPA employs a vector-valued latent and deterministic latent dynamics predictor, the UWM-JEPA findings underscore the importance of the predictor’s geometry for belief propagation, counterfactual reasoning, and action sensitivity.

The use of counterfactual training and the separation of the prober stage in SkyJEPA echo recommendations from UWM-JEPA to avoid teacher-forcing artifacts and enhance genuine action-conditioned prediction. A plausible implication is that an extension of SkyJEPA to partially observed settings or high-dimensional sensory inputs (such as vision) could benefit from adopting density-matrix latent representations or approximately unitary predictors, enabling richer belief tracking and principled uncertainty handling (Radha et al., 25 May 2026).

7. Strengths, Limitations, and Potential Extensions

Strengths:

• Accurate long-horizon latent predictions with minimal compounding error.
• Physically interpretable metric rollouts through the prober and integrator.
• Supports real-time, onboard inference on embedded hardware (9,000 parameters, TensorRT acceleration).
• Robust, zero-shot sim-to-real deployment without real-world data adaptation.
• Efficient data pipeline facilitating broad domain randomization.

Limitations:

• Relies on full high-precision state observation; nontrivial extension to raw sensor data (vision).
• No built-in uncertainty quantification; planning under epistemic uncertainty would require additional mechanisms such as Bayesian JEPA or GP posterior integration.
• No explicit obstacle or map awareness; integration of environment context into the latent or prober is not addressed.

Potential extensions:

• Vision-based encoder integration to enable end-to-end visual MPC.
• Incorporating safety-constraint representations or barrier certificates within control rollouts.
• Active learning for domain refinement and real-world adaptation.
• Generalization to tasks with variable mass, multi-agent settings, or complex aerodynamics.

SkyJEPA advances the capability of JEPA-style world models and establishes a scalable, interpretable, and data-efficient framework for agile vehicle control under significant domain variation, with a foundation well-suited for adaptation to partially observable and high-dimensional settings (Rao et al., 22 Jun 2026, Radha et al., 25 May 2026).