Source of Positive Transfer in RL Fine-Tuning

Determine the origins of the positive transfer observed when fine-tuning reinforcement learning agents with forgetting mitigation by disentangling and quantifying the respective contributions of reused pretrained representations and initialized policy parameters to improved learning efficiency and performance.

Background

The paper studies how forgetting of pre-trained capabilities undermines reinforcement learning fine-tuning and shows that knowledge retention methods (e.g., behavioral cloning, kickstarting, EWC) can restore and improve transfer across NetHack, Montezuma's Revenge, and Meta-World RoboticSequence.

In the robotic manipulation analysis, the authors observe significant positive transfer when forgetting is mitigated but note that the underlying mechanism—whether stemming from the reused representation, the policy initialization, or both—has not been established. They perform a diagnostic experiment by resetting the policy’s last layer to isolate representation reuse, finding reduced yet nontrivial transfer, which motivates isolating and measuring the distinct sources of the transfer effect.

References

Although we see significant positive transfer once the forgetting problem is addressed, it remains an open question where this impact comes from.

— Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem  (2402.02868 - Wołczyk et al., 2024) in Appendix, Section "Analysis of forgetting in robotic manipulation tasks", subsection "Impact of representation vs policy on transfer"