Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning

Abstract: Recent studies suggest that Reinforcement Fine-Tuning (RFT) is inherently more resilient to catastrophic forgetting than Supervised Fine-Tuning (SFT). However, whether RFT (e.g., GRPO) can effectively overcome forgetting in challenging visual continual learning settings, such as class-incremental learning (CIL) and domain-incremental learning (DIL), remains an open problem. Through a pilot study, we confirm that while RFT consistently outperforms SFT, it still suffers from non-negligible forgetting. We empirically trace this bottleneck to Trajectory-level Drift Agnosticism: among candidate rollouts achieving identical task rewards, the KL divergence from the preceding-task policy varies substantially, which strongly correlates with catastrophic forgetting across sequential tasks. Motivated by this insight, we propose Retention-aware Policy Optimization (RaPO), a simple yet effective RFT method that explicitly mitigates forgetting through trajectory-level reward shaping. Specifically, RaPO comprises two core components: (1) Retention Reward that converts trajectory-level distribution drift into a continuous reward signal, preferentially reinforcing knowledge-preserving rollouts within each group; (2) Cross-Task Advantage Normalization (CTAN), which maintains a persistent exponential moving average of reward statistics across task boundaries to stabilize the optimization progress during continual learning. Leveraging the free-form textual generalization of MLLMs, we comprehensively evaluate RaPO across five visual continual learning settings. Extensive experiments demonstrate that RaPO achieves leading performance, substantially reducing catastrophic forgetting while preserving strong plasticity. To the best of our knowledge, this work represents the first systematic exploration of RFT in visual continual learning, offering insights that we hope will inspire future research.

• The paper introduces RaPO, combining a retention reward with cross-task advantage normalization to preserve prior-task knowledge while maintaining adaptation to new visual tasks.
• RaPO reduces ImageNet-R forgetting from 20.02% with GRPO to 4.69% while raising final accuracy from 74.67% to 85.92%, with gains also reported for detection and video classification.
• The findings show that rollout drift is a practical forgetting signal, but evaluation remains limited by the computational cost of reinforcement fine-tuning, short task streams, explicit task boundaries, and closed-world rewards.

Motivation and problem setting

This paper investigates whether reinforcement fine-tuning (RFT) with verifiable rewards—exemplified by GRPO—can serve as a viable paradigm for rehearsal-free visual continual learning, specifically class-incremental learning (CIL) and domain-incremental learning (DIL). Prior work has established that on-policy RFT is inherently more resistant to catastrophic forgetting than supervised fine-tuning (SFT), because it biases optimization toward low-drift solutions. However, whether this resilience transfers to challenging visual settings with disjoint label spaces, few-shot data budgets, and dense prediction tasks had not been systematically examined. The authors claim this is the first systematic exploration of RFT in visual continual learning.

The pilot study uses Qwen2-VL-2B on a 5-shot, 10-task CIL split of ImageNet-R. GRPO consistently outperforms SFT and classical continual learning baselines (L2 regularization, EWC, LwF), confirming that RFT's forgetting resilience carries over to vision. Yet GRPO still forgets substantially—for example, joint-training GRPO reaches 91.50% accuracy while sequential GRPO retains only 74.67%—indicating that task-reward-driven RFT alone does not resolve the stability–plasticity tension.

Trajectory-level drift agnosticism

The paper's central empirical finding is a phenomenon termed trajectory-level drift agnosticism: among rollout groups achieving identical maximal task reward, the token-level KL divergence from the frozen preceding-task policy $\pi_{t-1}$ varies substantially, and this variance grows as the task sequence progresses. The authors validate causality with two diagnostic GRPO variants applied to all-correct groups in later tasks: one that zeroes rewards for high-KL rollouts mitigates forgetting, while the mirror-image variant that preserves rewards only for high-drift rollouts exacerbates it. This establishes trajectory-level drift as an actionable signal tied to forgetting, though the hard-thresholding intervention is impractical—it collapses fine-grained credit assignment and is unstable under small rollout counts where marginal KL differences trigger winner-take-all updates.

Retention-aware Policy Optimization

RaPO converts the diagnostic insight into a practical method via two components:

Retention Reward: For each rollout $y_i$, the length-normalized log-probability ratio between actor $\pi_t$ and anchor $\pi_{t-1}$ is computed and one-sidedly truncated at zero (preventing reward hacking via deliberately low-confidence outputs). This drift measure is mapped through an exponentially decaying function $$R_{\mathrm{ret}} = \exp(-\alpha \bar{D}_{\mathrm{drift}}) \in (0,1]$$ and added to the task reward before group-relative advantage computation. Because the signal enters inside the composite reward rather than as a loss-level regularizer, it re-ranks rollouts within a group: among equally-rewarded trajectories, knowledge-preserving ones receive larger advantages. The paper carefully distinguishes this from standard KL loss regularization—which only constrains update magnitude and cannot differentiate same-reward rollouts—and from on-policy distillation, whose dense token-level pull toward $\pi_{t-1}$ would suppress plasticity on new-task data. An ablation confirms loss-level KL contributes only modestly to forgetting resistance.

Cross-Task Advantage Normalization (CTAN): Reward statistics shift abruptly at task boundaries (small within-batch spread near the end of a task inflates advantages; higher-variance rewards at the start of a new task compress them). CTAN replaces the per-batch reward standard deviation with a persistent exponential moving average $\hat{\sigma}$ carried across task boundaries, stabilizing credit-assignment scale throughout the stream.

The appendix provides a policy-gradient compatibility analysis: under a detached-reward surrogate view, the retention term introduces no pathwise distillation gradient, rewards and advantages remain bounded ($|A_i| \leq B/\epsilon$), and under standard smoothness/unbiasedness/bounded-variance assumptions the idealized update inherits the usual $\mathcal{O}(K^{-1/2})$ stationary-point convergence rate. The authors explicitly note this is not a formal no-forgetting guarantee, and that clipping and finite-group normalization make the practical estimator an approximation of the ideal identity.

Experimental results

Evaluation spans five settings: class-incremental image classification (ImageNet-R, ImageNet-A, TinyImageNet, CUB-200 at 10/20 tasks), class-incremental object detection (COCO 2017, 5/10 tasks, Qwen2-VL-7B), class-incremental video classification (UCF-101, Kinetics-200), and DIL for both classification (DomainNet, OfficeHome) and detection (Pascal Series). All experiments use 5-shot protocols, three random class orders, and Qwen2-VL backbones.

Selected results illustrate the magnitude of improvement over the strongest baseline (GRPO):

Benchmark	Metric	GRPO	RaPO
ImageNet-R (10 tasks)	Last acc. / Forgetting	74.67 / 20.02	85.92 / 4.69
ImageNet-A (10 tasks)	Last acc. / Forgetting	37.37 / 28.88	44.61 / 20.16
CUB-200 (10 tasks)	Last acc. / Forgetting	28.67 / 45.50	45.15 / 19.10
COCO (5 tasks)	Box AP / Box forgetting	14.64 / 6.67	19.31 / 1.39
Kinetics-200 (5 tasks)	Last acc. / Forgetting	70.33 / 30.37	74.18 / 16.76

Notably, on domain-incremental Pascal detection RaPO's 37.18% mean AP exceeds even joint-training SFT (35.40%), approaching joint-training GRPO (39.13%). Classical continual learning baselines (L2, EWC, LwF) provide marginal or detrimental benefits in most settings, although LwF becomes competitive with GRPO in object detection, suggesting output distillation is particularly useful for preserving structured localization behavior. DIL exhibits intrinsically lower forgetting than CIL across all methods, consistent with its fixed label space avoiding inter-task confusion.

Ablations confirm both components contribute complementarily (retention reward alone yields larger gains than CTAN alone), the preceding-task anchor $\pi_{t-1}$ outperforms EMA, first-task, and pre-trained anchors, hyperparameters are robust (accuracy varies by ~1.7% across tested configurations), and the components transfer to SAPO, indicating they are optimizer-agnostic rather than GRPO-specific. Results also hold on Qwen2.5-VL backbones. One notable secondary finding: removing explicit reasoning degrades GRPO severely on detection (AP drops from 14.64 to 9.82) but affects RaPO far less, which the authors attribute to the retention reward anchoring implicitly learned reasoning patterns—an implication worth noting since it suggests retention shaping confers robustness beyond pure knowledge preservation.

Limitations and open questions

The paper concedes several constraints plainly. RFT training is computationally expensive, limiting evaluation to sequences of at most 20 tasks. Experiments cover only Qwen2-VL at 2B and 7B scale, leaving open whether RaPO extends to larger MLLMs. The study adopts closed-world settings with explicit task boundaries and verifiable rewards; extension to open-world streams without such boundaries remains unaddressed. Evaluation uses closed-set prompts to isolate recognition gains from naming variability, acknowledging but not solving open-set recognition. Finally, the convergence analysis is idealized and provides no formal no-forgetting guarantee.

Conclusion

This work establishes RFT as a credible foundation for visual continual learning and identifies trajectory-level drift agnosticism as the mechanism through which vanilla GRPO still forgets. RaPO addresses this with a minimal modification—reward-level retention shaping plus cross-task advantage normalization—that reduces forgetting dramatically (e.g., from 20.02% to 4.69% on ImageNet-R CIL) while preserving plasticity, generalizes across classification, detection, and video modalities, and transfers to other policy optimizers. The main open questions concern scaling to longer task streams, larger models, and boundary-free open-world settings.