- The paper introduces LifeLong-RFT, a chunk-level GRPO method with quantized action, continuous trajectory, and format rewards that adapts VLA policies without simulator or real-robot interaction.
- LifeLong-RFT improves standard multi-task performance, reaching 95.6% on LIBERO, 87.5% on four real-world Franka tasks, and a 22% average continual-learning gain over SFT with 20% of the data.
- The method substantially improves forward transfer and learning efficiency while reducing forgetting, but currently depends on reference demonstrations and applies only to discrete-action VLA models.
Motivation and problem setting
Vision-Language-Action (VLA) models pretrained on large-scale datasets are typically adapted to downstream tasks through supervised fine-tuning (SFT). The authors identify two coupled failure modes of this paradigm: SFT demands substantial task-specific demonstration data, and it induces catastrophic forgetting of previously acquired skills. These limitations are mutually reinforcing—improving data efficiency tends to worsen forgetting, and preserving prior knowledge restricts learning from scarce new data—so the plasticity–stability trade-off remains unresolved even for foundation-model-based policies. The paper's central question is whether on-policy reinforcement learning, which recent LLM studies suggest is more robust to forgetting than SFT, can be adapted to VLA post-training to support continual skill acquisition without environment interaction.
Method
LifeLong-RFT performs chunking-level on-policy RL using GRPO over expert demonstrations, eliminating the need for simulator rollouts, world models, or real-robot interaction. For each observation–instruction pair, a group of G=8 action chunks is sampled from the current policy; advantages are computed by group-wise normalization of rewards, and the policy is updated with clipped ratio objectives plus KL regularization against a reference policy.
The key contribution is the Multi-Dimensional Process Reward (MDPR), which scores each sampled action chunk against ground-truth demonstrations along three axes:
- Quantized Action Consistency Reward (QACR): position-wise token matching between predicted and target action-token sequences (produced via the Fast+ tokenizer), normalized by sequence length.
- Continuous Trajectory Alignment Reward (CTAR): after decoding tokens into continuous chunks, an exponentially decaying reward on normalized L1 pose error combined with a binary gripper-state match, averaged over the chunk horizon (α=5, β=0.8).
- Format Compliance Reward (FCR): a binary reward enforcing structural validity (correct action dimensions and chunk size) so outputs remain decodable and executable.
The composite reward weights these terms as ω⋅QACR+(1−ω)⋅CTAR+λ⋅FCR with ω=0.7, λ=0.1. Because rewards are computed per-chunk against reference trajectories rather than from task outcomes, no environmental feedback or learned reward model is required—avoiding both sim-to-real barriers and reward hacking from imperfect reward estimators. The method is applied to NORA-Long with full-parameter optimization on 8 H20 GPUs.
Multi-task results
On SimplerEnv under Visual Matching, LifeLong-RFT improves the NORA-Long SFT baseline by 3.5 points on WidowX (65.5 → 69.0 average success rate) and 4.4 points on Google Robot (74.7 → 79.1). On LIBERO, it reaches a 95.6% average success rate across the four suites—the highest reported among compared continuous and discrete action models, exceeding OpenVLA-OFT (95.4%) and π0​ (94.2%). Real-world experiments on a Franka robot across four tasks show an 8.7-point gain over the SFT baseline (78.8% → 87.5%), including a 15-point improvement on the dexterous "Hang Chinese Knot" task. These gains are consistent but modest in simulation; the more consequential claims concern continual learning.
Continual learning results
Following the LOTUS protocol, each LIBERO suite is trained on six base tasks (50 demonstrations each), then four new tasks sequentially with only 10 demonstrations per new task and 5 retained per prior task for experience replay. LifeLong-RFT outperforms the SFT baseline on all three metrics across all suites:
| Suite |
FWT gain |
NBT reduction |
AUC gain |
| LIBERO-Object |
+11.2 |
−5.3 |
+15.1 |
| LIBERO-Spatial |
+11.2 |
−10.3 |
+19.5 |
| LIBERO-Goal |
+19.6 |
−22.1 |
+35.9 |
| LIBERO-Long |
+13.2 |
−4.5 |
+17.2 |
The abstract's headline claim—a 22% average success-rate gain over SFT in continual learning while adapting with only 20% of the training data—is consistent with these AUC improvements. Real-world sequential learning shows even larger margins: +23.7 FWT, NBT reduced to 6.1, and +31.7 AUC over the SFT baseline. Notably, detailed per-task logs reveal instances of positive backward transfer (e.g., performance improving on previously learned tasks after training Task 8 in LIBERO-Goal), supporting the claim that on-policy RL updates preserve rather than overwrite prior capabilities.
Ablations confirm that CTAR is the dominant component: removing it collapses LIBERO performance from 95.6% to 4.7%, whereas removing QACR or FCR costs only 2.8 and 2.6 points respectively. This indicates the continuous trajectory signal, not discrete token matching, drives most of the optimization. Sensitivity analysis shows robustness to ω except at 0.9, where down-weighting CTAR drops performance to 90.0%.
Data efficiency
Adaptation-efficiency experiments demonstrate strong few-shot behavior: on "Pick Orange Juice," LifeLong-RFT reaches 100% success with 5 demonstrations, while SFT achieves 98% even with 50; on "Put Wine Bottle" it exceeds SFT by 30 points at full data. The practical implication is that RFT can substitute for large demonstration collection when integrating new tasks into a deployed policy.
Limitations and open questions
The paper concedes several constraints. First, the method applies only to discrete-action VLAs built on autoregressive token generation; extending MDPR to continuous-action models (diffusion or flow-matching heads) remains open, and the authors note discrete models currently underperform continuous ones. Second, long-horizon performance degrades under few-shot conditions—certain LIBERO-Long tasks fall to 34–36% success—and the deformable-object real-world task plateaus at 60%. Third, although MDPR avoids environment interaction, rewards are still computed against ground-truth demonstrations, meaning the approach presupposes access to reference trajectories for every new task; how the method behaves when only outcome-level or unlabeled data is available is not addressed. Finally, experience replay retains a small buffer of past demonstrations, so the contribution of pure on-policy RL versus replay to forgetting resistance is not fully disentangled.
Conclusion
LifeLong-RFT demonstrates that chunk-level on-policy RL with verifiable, demonstration-grounded process rewards can serve as a post-training paradigm that improves multi-task performance while substantially mitigating catastrophic forgetting and reducing data requirements. The strongest evidence lies in continual learning on LIBERO (+35.9 AUC on Goal) and real-world sequential adaptation (+31.7 AUC), though the dependence on reference trajectories and restriction to discrete action spaces bound its immediate applicability.
