Towards Human-like Physical Intelligence: LifelongVision-Language-Action Learning for Robotic Manipulation

Abstract: Similar to the natural capabilities of humans to sequentially learn new tasks, robots with Vision-Language-Action (VLA) models should possess lifelong learning ability to learn a new task when deployed in open-world environments. However, most recently proposed lifelong learning models aim to effectively learn the current task (plasticity) or maintain high accuracy on previous tasks (stability), while the plasticity-stability trade-off remains largely unsolved in robotic manipulation models. To address this fundamental challenge, we propose a cache-efficient lifelong Vision-Language-Action learning framework for robotic manipulation (i.e., LifelongVLA), which alleviates the plasticity-stability trade-off with a dual-timescale adaptation mechanism while achieving low-cost robotic deployment with a cache-efficient replay strategy. More concretely, we propose a dual-timescale LoRA gating module to decompose VLA adaptation into two lightweight pathways: a short-term adapter for plasticity and a long-term adapter for stable consolidation. These pathways are integrated via a task-aware gate, enabling explicit control of the plasticity-stability trade-off. In the skill replay phase, a cache-efficient stochastic replay strategy is proposed to preserve more balanced retention signals without full-trajectory storage. Finally, experiments show that LifelongVLA outperforms existing baselines, demonstrating efficient skill expansion, robust retention of learned manipulation behaviors, and reduced reliance on retraining for real-world deployment on an xArm robot.

• The paper introduces LifelongVLA, which combines dual-timescale LoRA gating with cache-efficient latent replay to balance rapid learning of new manipulation skills with retention of earlier ones.
• The method achieves 83.2% average success and 11.4% forgetting across 10 sequential LIBERO tasks, outperforming episodic replay by 13.0 success points and 8.2 forgetting points.
• The framework stores 95.70 MiB per task and reaches over 80% success across five real-robot tasks without task identity at inference, although longer task streams and varied language remain untested.

LifelongVLA addresses the plasticity–stability trade-off in continual learning for Vision-Language-Action (VLA) policies, combining a dual-timescale LoRA adaptation mechanism with a cache-efficient replay strategy. Built on a frozen PaliGemma backbone (Gemma 2B) with a 300M continuous diffusion-style action decoder following the $\pi_0$ design, the framework targets open-world robotic manipulation where new tasks arrive sequentially and task identity is unavailable at inference. On a 10-task LIBERO benchmark stream, it achieves an 83.2% average success rate and an 11.4% average forgetting rate, outperforming episodic replay (ER) by 13.0 points in success rate and 8.2 points in forgetting rate while cutting replay memory to 95.70 MiB per task.

Problem setting and motivation

The paper defines a lifelong VLA setting in which a policy $F_{\Theta_0}$ with frozen backbone parameters is adapted over a sequence of manipulation tasks $\mathcal{T}_1,\ldots,\mathcal{T}_T$. Only the current dataset is available at each step, previous data is limited to a bounded cache, and future distributions are unknown. The authors argue that existing skill-incremental approaches—replay buffers, architectural expansion, parameter isolation—leave the plasticity–stability dilemma unresolved: high plasticity enables rapid acquisition of new skills but overwrites prior behaviors, while strong stability impedes future learning. They also note that full-trajectory replay for VLA backbones is expensive because image-rich episodes must be re-encoded repeatedly, and expansion-based designs add routing cost and inference latency. A related observation in the literature—that pretrained VLA models are surprisingly resistant to forgetting under simple experience replay (Liu et al., 4 Mar 2026)—is acknowledged, but the authors contend such solutions remain costly for large backbones.

Dual-timescale LoRA gating

The core architectural contribution decomposes adaptation into two LoRA pathways per layer: a short-term pathway $$\Delta W_{\mathrm{sh}} = A_{\mathrm{sh}}B_{\mathrm{sh}}$$ trained only on the current-task denoising loss, and a long-term pathway $$\Delta W_{\mathrm{lg}} = A_{\mathrm{lg}}B_{\mathrm{lg}}$$ trained only on replay and distillation signals with a smaller learning rate ($\eta_{\mathrm{lg}} < \eta_{\mathrm{sh}}$). A task-aware gate composes the two pathways in weight space:

$$\Delta \widehat{W}_t^\ell(c) = (1-\alpha_t^\ell(c))\,\Delta W_{\mathrm{sh},t}^\ell + \alpha_t^\ell(c)\,\Delta W_{\mathrm{lg},t}^\ell$$

where the gating coefficient $$\alpha_t^\ell(c) = \sigma(W_g^\ell\phi^\ell(c) + b_g^\ell)$$ is computed from stop-gradient frozen prefix features, deliberately avoiding the circular dependency that would arise if the gate context depended on the gated weights themselves. Because composition occurs on LoRA weights rather than outputs, inference requires only a single forward pass—no dual-pass overhead. The design provides an explicit, per-layer control interface over the plasticity–stability balance, in contrast to single-adapter approaches where both objectives share one update pathway.

Cache-efficient stochastic replay

The replay module stores only stop-gradient prefix tokens $\bar{h}^{\mathrm{pre}}$ from a small reservoir (cache quota $c_t=50$ per task, reservoir size $F_{\Theta_0}$0) together with compact state–action supervision; raw images, language tokens, suffix tokens, diffusion times, and noise are not stored. At replay time, fresh diffusion variables $F_{\Theta_0}$1 and $F_{\Theta_0}$2 are sampled and the suffix token is recomputed with the current model, so each cached prefix yields multiple stochastic denoising instances. The cached prefix thus acts as a stable "memory anchor" while recomputed suffixes avoid the staleness problem of dense feature replay, where cached representations drift out of sync with the evolving adapted model.

Training uses a masked mixed-batch objective $F_{\Theta_0}$3 with separately normalized current and replay terms, preventing batch-size imbalance from letting either source dominate. A detached snapshot $F_{\Theta_0}$4 of the model after the previous task serves as a distillation teacher on replay samples, with $F_{\Theta_0}$5 and $F_{\Theta_0}$6.

Empirical results

On 10 LIBERO tasks trained sequentially (10,000 steps per task, LoRA rank $F_{\Theta_0}$7, scaling $F_{\Theta_0}$8), LifelongVLA reaches an 83.2% average success rate against a 89.6% joint-training upper bound, with the lowest forgetting rate of all compared methods. Sequential fine-tuning and LwF-LoRA collapse to 7.8% average success, and information-theoretic regularization (Info-VLA) and skill-level expert specialization (AtomicVLA) both plateau at 28.6%, indicating that neither regularization nor expert expansion alone resolves the trade-off in this setting. Task-level results include 96% on task 5, 98% on task 9 (with zero forgetting), and 96% on task 10.

The ablation isolates both components. Latent replay with a single LoRA reduces memory from 167.62 MiB/task (raw data replay) to 95.70 MiB/task while lowering forgetting from 18.44% to 21.78% at a cost in success rate (81.8% to 74.6%); adding dual LoRA restores 83.2% success and 11.4% forgetting, at the price of doubling trainable parameters to 99.97M—still far below AtomicVLA's 226.49M parameters and 452.98 MiB/task. This confirms that separating plastic and consolidation pathways, not merely compact replay, drives the retention gains.

Real-robot evaluation on an xArm with a five-task stream (50 training and 50 evaluation episodes per task) yields success rates above 80% on every task after all five are learned, with no task identity at test time, supporting the transferability of the retention mechanism beyond simulation. The authors note, however, that detailed per-task real-world forgetting statistics are deferred to supplementary material.

Limitations and open questions

The paper concedes several constraints on the generality of its results. Evaluation covers a 10-task simulation stream and a 5-task real-robot stream; longer, more heterogeneous sequences, randomized task orders, and variance across multiple streams are not yet assessed, so long-term stability under extended deployment remains unverified. The language conditioning is restricted to clear instructions; robustness to paraphrase, ellipsis, coreference, and context-dependent commands is untested. Additionally, the dual-LoRA design doubles trainable parameters relative to single-adapter baselines, and the fixed hyperparameters ($F_{\Theta_0}$9, $\mathcal{T}_1,\ldots,\mathcal{T}_T$0, $\mathcal{T}_1,\ldots,\mathcal{T}_T$1) are not studied for sensitivity to stream length or task difficulty.

Conclusion

LifelongVLA offers a concrete mechanism for lifelong VLA learning that makes the plasticity–stability trade-off an explicit, controllable quantity via weight-level gating of short- and long-term LoRA pathways, paired with a latent replay scheme that substitutes stochastic suffix recomputation for full-trajectory storage. The reported 83.2% success rate against a 89.6% joint upper bound, 11.4% forgetting, and 95.70 MiB/task memory represent a favorable operating point among current lifelong manipulation methods, with the principal open questions concerning scale of task streams and linguistic robustness.