AdaJEPA: Adaptive Latent World Models That Learn While They Plan
This presentation introduces AdaJEPA, a framework that fundamentally reimagines how AI systems plan and act in uncertain environments. Traditional world models freeze after training, causing them to fail when reality shifts. AdaJEPA breaks this paradigm by continuously recalibrating its predictions during deployment through lightweight self-supervised adaptation, achieving robust control across severe distribution shifts in manipulation and navigation tasks without requiring any reward signals or expert demonstrations.Script
When a robot plans its next move, it imagines the future using an internal model of the world. But what happens when that model encounters something it has never seen before? Traditional systems freeze and fail. AdaJEPA solves this by learning continuously during deployment, adapting its predictions with every action it takes.
The core innovation is a closed-loop protocol the authors call Plan, Act, Adapt, Replan. After solving a model predictive control problem in latent space and executing the first action, the system uses the observed transition to update its world model through a single gradient step on a self-supervised prediction objective. Then it replans with the recalibrated model and repeats, creating a continuous cycle of refinement.
The results are striking across three kinds of distribution shift. For novel object shapes in contact-rich manipulation, visual corruptions like blur and noise, and altered physics or maze layouts in navigation, even a single adaptation step per action dramatically improves goal-reaching success rates compared to frozen baselines.
AdaJEPA also fundamentally changes data efficiency. The researchers show that with test-time adaptation, a model trained on just 10 trajectories per shape can match or exceed the performance of frozen models trained on 500 trajectories. This reveals that adaptation can compensate for limited offline diversity, though the pretrained features still set an upper bound on what adaptation can recover.
What makes this practical is the minimal computational overhead. Adaptation targets only a small subset of encoder or predictor parameters with a single stochastic gradient descent step, adding negligible latency per control cycle. The efficiency comes from selective recalibration rather than wholesale retraining, and the faster goal attainment often more than compensates for the slight update cost.
AdaJEPA establishes that world models do not need to be frozen artifacts. By embedding lightweight self-supervised adaptation directly into the planning loop, the authors demonstrate a path toward truly resilient AI that recalibrates itself in nonstationary environments without reward signals or expert guidance. To dive deeper into adaptive world models and create your own video explanations, visit EmergentMind.com.
