Cross-Domain Lifelong Reinforcement Learning

• CD-L2RL is a lifelong reinforcement learning framework where agents adapt to evolving tasks and domains by reusing skills, goals, and policies.
• Researchers formalize CD-L2RL via finite MDP mixtures, non-stationary POMDPs, and Bayesian models to capture shifts in dynamics and observation spaces.
• Key mechanisms such as policy reuse, similarity-based transfer, and modulating masks help mitigate negative transfer and catastrophic forgetting.

Searching arXiv for recent and foundational papers on cross-domain lifelong reinforcement learning and closely related transfer/continual RL mechanisms. Cross-Domain Lifelong Reinforcement Learning (CD-L2RL) denotes a lifelong reinforcement-learning setting in which an agent faces many tasks over time, possibly in different environments or sensory modalities, and must reuse knowledge such as skills, goals, reward structures, task models, or policies while dynamics, rewards, observations, and sometimes state or action spaces evolve (Edwards et al., 2017, Khetarpal et al., 2018, Firouzjaei et al., 25 Aug 2025). Across the literature, CD-L2RL is motivated by the need for continual adaptation without re-engineering rewards per task, resistance to catastrophic forgetting, structured transfer across repeated and novel tasks, and the ability to exploit latent common structure rather than relearn each domain from scratch (Brunskill et al., 2015, Wang et al., 2022, Steinparz et al., 2022).

1. Conceptual scope and distinguishing features

A lifelong RL agent is described as learning from “a stream of experience spanning many tasks (possibly of different nature) over their lifetime” and as an autonomous agent trained on “a sequence of tasks with no final task” (Khetarpal et al., 2018). In this sense, lifelong RL already presupposes incremental learning, skill accumulation, skill composition, and non-forgetting. CD-L2RL strengthens that requirement by allowing task variation not only in rewards or goals but also in dynamics, observations, sensory modalities, and, in some formulations, the dimensionality or semantics of the decision space (Edwards et al., 2017, Steinparz et al., 2022, Firouzjaei et al., 25 Aug 2025).

The literature does not use a single uniform formalization. Some works study the same physical domain with multiple hidden MDP types sharing common state and action sets, which is explicitly presented as a “natural stepping stone toward CD-L2RL” rather than the full problem (Brunskill et al., 2015). Other works define continual domain shifts in a non-stationary POMDP, where transition, reward, and observation functions depend on time and where explicit task labels or boundaries are absent (Steinparz et al., 2022). Still others reserve the term “domain” for changes in state-action structure itself, such as changes in the number of EH nodes in wireless sensor networks or different robot embodiments in VLA continual learning (Firouzjaei et al., 25 Aug 2025, Liu et al., 11 Feb 2026).

A common misconception is that lifelong RL and cross-domain RL are interchangeable. The surveyed papers separate them: lifelong RL can remain within a common $\mathcal{S},\mathcal{A}$ setting, while cross-domain transfer may additionally require mappings across different observation spaces, action spaces, or modalities (Brunskill et al., 2015, Serrano et al., 2023, Chen et al., 12 Mar 2026). Another misconception is that transfer is always beneficial. Multiple papers emphasize the danger of negative transfer when a new task or domain is misidentified as an old one, or when a source critic, policy, or reward model is inconsistent with the target domain (Brunskill et al., 2015, Joshi et al., 2018, Chen et al., 12 Mar 2026).

2. Formalizations of tasks, domains, and shared structure

One formalization treats lifelong RL as a sequence of tasks $M_t$ drawn from a finite set of discounted MDPs $M=\{M_1,\dots,M_K\}$, all sharing $\mathcal{S}$ and $\mathcal{A}$ but differing in transition and reward functions, with possibly adversarial task order (Brunskill et al., 2015). This yields a cross-task discovery problem: the agent must determine whether the current task is novel or a recurrence of a previously seen MDP type. The Online Coupon-Collector Problem abstracts this as the tradeoff between probing for novelty and skipping to exploit known structure (Brunskill et al., 2015).

A second formalization models lifelong RL as a non-stationary POMDP $(S,A,\mathcal{T},\mathcal{R},\Omega,O,\gamma)$, where $\mathcal{T}_t$, $\mathcal{R}_t$, and $O_t$ vary over time (Steinparz et al., 2022). In that view, a “continual domain shift” is not merely a discrete task change but a time-evolving distribution over observations, rewards, or dynamics. This formulation is especially relevant when task boundaries are absent and the agent must react online to gradual or abrupt drift (Steinparz et al., 2022).

A third formalization, explicit in the wireless-sensor-network literature, distinguishes tasks from domains. A task is a single learning problem under a fixed environmental configuration,

$$\mathcal{F}^j = \langle \mathcal{S},\mathcal{A},\mathcal{P}^j,\mathcal{R}^j,\gamma^j \rangle,$$

while a domain groups structurally related tasks that share the same state/action space structure but differ in parameters such as EH efficiency, channel scales, or arrival rates (Firouzjaei et al., 25 Aug 2025). Cross-domain change then means that the decision space itself evolves, not only the optimal strategy.

Hierarchical Bayesian formulations capture shared structure differently. In model-based lifelong RL with Bayesian exploration, tasks are drawn from a HiP-MDP with hidden parameter $M_t$0, and task dynamics and rewards are expressed as $M_t$1 and $M_t$2 (Fu et al., 2022). The world-model posterior $M_t$3 serves as the prior for new tasks, so forward transfer arises from a better posterior initialization and backward transfer from later refinement of the shared world model (Fu et al., 2022).

These formalizations suggest that CD-L2RL is best understood as a family of problems rather than a single benchmark setting. Some variants emphasize latent task identity under fixed interfaces; others emphasize continual shifts in a single evolving environment; others emphasize changing state-action structure itself (Brunskill et al., 2015, Steinparz et al., 2022, Firouzjaei et al., 25 Aug 2025).

3. Mechanisms for cross-domain knowledge transfer

Several lines of work attack the transfer problem through explicit reuse of policies, critics, task models, or masked subnetworks. The following representative mechanisms recur across the literature.

Mechanism	Core object transferred	Representative papers
Policy reuse with adaptation	Source optimal policy plus adaptive correction	(Joshi et al., 2018)
Similarity-based transfer	Encoders/decoders, task similarity, source policy	(Serrano et al., 2023)
Hybrid critics	Source and target $M_t$4-functions with adaptive weighting	(Chen et al., 12 Mar 2026)
Bayesian nonparametric task models	Mixture components $M_t$5	(Wang et al., 2022)
Parameter isolation	Task-specific modulating masks over a fixed backbone	(Ben-Iwhiwhu et al., 2022, Nath et al., 2023)

In “Cross-Domain Transfer in Reinforcement Learning using Target Apprentice,” the source optimal policy is mapped into the target state space and corrected by an adaptive term derived from the discrepancy between a projected source model and a learned target apprentice model (Joshi et al., 2018). The resulting target control is

$M_t$6

and the paper shows that the augmented policy can be $M_t$7-optimal in the target task when the apprentice model approximates the true target model sufficiently well (Joshi et al., 2018). This mechanism directly addresses a central CD-L2RL requirement: reusing old policies without rerunning full target-task RL from scratch.

“Similarity-based Knowledge Transfer for Cross-Domain Reinforcement Learning” instead learns state and action encoders/decoders into a shared latent space and uses reward-based semi-supervised alignment to compute task similarity and select a source policy (Serrano et al., 2023). The alignment objective

$M_t$8

uses reward proximity as the anchor for cross-domain correspondence, without requiring paired trajectories or expert-aligned data (Serrano et al., 2023). This literature treats source selection itself as a core transfer problem, because incorrect source choice can produce negative transfer.

“Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics” introduces cross-domain Bellman consistency as a transferability criterion and defines a hybrid critic

$M_t$9

with an adaptive, hyperparameter-free weight $M=\{M_1,\dots,M_K\}$0 determined by target TD error and cross-domain Bellman error (Chen et al., 12 Mar 2026). When the mapped source critic is Bellman-consistent, transfer is amplified; when it is unreliable, the method reverts toward the target critic, which is explicitly intended to avoid negative transfer (Chen et al., 12 Mar 2026).

A different approach expands capacity online. “A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning” maintains a mixture of task models $M=\{M_1,\dots,M_K\}$1, treats task identity as latent, and uses a Chinese restaurant process prior to instantiate new components when current data are not well explained by existing ones (Wang et al., 2022). This is notable because it does not require explicit task boundaries or fixed task counts; instead it combines responsibility-weighted updates with domain-randomized robust priors, which the paper argues is suitable for non-stationary and cross-domain lifelong settings (Wang et al., 2022).

Parameter-isolation methods provide yet another route. “Lifelong Reinforcement Learning with Modulating Masks” freezes a backbone network and learns task-specific masks over weights, while linear combinations of previously learned masks are used to accelerate learning on new tasks (Ben-Iwhiwhu et al., 2022). “Sharing Lifelong Reinforcement Learning Knowledge via Modulating Masks” extends that idea to fully distributed and asynchronous settings, where agents issue IDQ, QR, MR, and MTR messages to request and transfer specific masks on demand (Nath et al., 2023). A plausible implication is that mask-based modularity is attractive for CD-L2RL when domains are recurrent but communication, storage, or synchronization are constrained.

4. Reward and goal transfer across domains

Reward specification is a central bottleneck in CD-L2RL because many lifelong settings require goals to be expressed in modalities or environments different from the agent’s own. “Cross-Domain Perceptual Reward Functions” addresses this by defining a learned cross-domain reward

$M=\{M_1,\dots,M_K\}$2

where $M=\{M_1,\dots,M_K\}$3 is the agent-state image, $M=\{M_1,\dots,M_K\}$4 is a cross-domain goal image, and both are embedded by encoder networks into a shared feature space (Edwards et al., 2017). The paper evaluates genuinely cross-domain goals, including handshape images, speech spectrograms, guitar spectrograms, and sheet music, and reports Goal Retrieval Accuracy values of approximately $M=\{M_1,\dots,M_K\}$5 and $M=\{M_1,\dots,M_K\}$6 for maze handshape and speech, and $M=\{M_1,\dots,M_K\}$7 and $M=\{M_1,\dots,M_K\}$8 for music guitar and sheet music, respectively (Edwards et al., 2017). At the same time, it explicitly notes that high GRA does not guarantee optimal RL performance, because false positives and intermediate rewards can mislead the learned policy (Edwards et al., 2017).

Another reward-side mechanism penalizes source-domain experience that does not match target dynamics. “Cross-Domain Policy Adaptation by Capturing Representation Mismatch” learns target-only latent dynamics representations and defines a modified source reward

$M=\{M_1,\dots,M_K\}$9

so that source transitions with large representation deviation are downweighted during adaptation (Lyu et al., 2024). The paper further shows that representation deviation upper bounds performance difference of a given policy in the source and target domains, which is the formal motivation for the reward penalty (Lyu et al., 2024).

In VLA continual learning, reward is again reconstructed rather than read from the environment. “Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning” defines a Multi-Dimensional Process Reward,

$\mathcal{S}$0

combining Quantized Action Consistency Reward, Continuous Trajectory Alignment Reward, and Format Compliance Reward (Liu et al., 11 Feb 2026). This enables chunking-level on-policy reinforcement learning without online environmental feedback or pre-trained reward models, and the paper reports a 22% gain in average success rate over SFT on continual learning for LIBERO while using only 20% of the training data for new tasks (Liu et al., 11 Feb 2026).

A more radical conceptual position argues that reward in lifelong systems should be treated as a formal language rather than only a scalar return. “Some Insights into Lifelong Reinforcement Learning Systems” models reward as $\mathcal{S}$1, decomposes it into reward state and reward value, and organizes local goals as recurring episodic structures within a lifetime (Li, 2020). This suggests that CD-L2RL reward design is not only about shaping a scalar objective, but also about encoding which domain, which local goal, and which form of guidance are currently active.

5. Exploration, adaptation, and non-stationarity

CD-L2RL requires exploration policies that can discover novelty while still exploiting accumulated structure. In the finite-MDP setting, the Online Coupon-Collector Problem formalizes task-level exploration as deciding whether to probe or skip a current task when its underlying type may be novel (Brunskill et al., 2015). The ForcedExp strategy uses a decaying probing probability $\mathcal{S}$2, and with $\mathcal{S}$3 achieves $\mathcal{S}$4 regret, which the paper shows is optimal in the OCCP setting (Brunskill et al., 2015). This is directly relevant when a lifelong agent must decide whether a newly encountered domain is sufficiently different to justify costly exploration.

Reactive online adaptation is treated differently in non-stationary POMDPs. “Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning” augments the reward with prediction-error terms,

$\mathcal{S}$5

where $\mathcal{S}$6 is dynamics surprise and $\mathcal{S}$7 is reward surprise (Steinparz et al., 2022). The paper empirically shows that policy-gradient methods are better suited for lifelong learning under continual domain shifts than Q-learning, because on-policy updates adapt more quickly to current distributions while replay buffers in off-policy methods mix incompatible regimes (Steinparz et al., 2022). This finding is especially pertinent for CD-L2RL systems with continual observation, reward, or dynamics drift.

Model-based Bayesian exploration supplies a third exploration mechanism. In “Model-based Lifelong Reinforcement Learning with Bayesian Exploration,” the agent maintains a hierarchical Bayesian posterior over shared latent task structure and periodically samples models from that posterior for planning (Fu et al., 2022). In finite MDPs the analysis relates sample complexity to the prior-mass radius $\mathcal{S}$8, so posterior initialization quality becomes a formal determinant of sample efficiency (Fu et al., 2022). In continuous-state domains, Variational Bayesian Lifelong Reinforcement Learning combines Bayesian neural-network dynamics with model-based planning and is reported to exhibit backward transfer in challenging domains (Fu et al., 2022).

These lines of work converge on a shared point: in CD-L2RL, exploration is not merely within-task state-action search. It is also cross-task or cross-domain inference about whether previous models, rewards, or critics remain trustworthy, and whether the current regime is sufficiently novel to warrant new structure.

6. Evaluation protocols, empirical patterns, and open issues

The environment literature argues that lifelong RL testbeds should support a multitude of tasks of different difficulty, composition of previously acquired skills, incremental expansion through adding or removing objects, both short-term and long-term planning, and repeated testing on previously seen tasks to assess catastrophic forgetting (Khetarpal et al., 2018). L2Explorer operationalizes this perspective with a Unity-based first-person 3D exploration environment, JSON-driven task reconfiguration, and a curriculum framework that supports learning blocks and evaluation blocks under controlled changes to $\mathcal{S}$9, $\mathcal{A}$0, $\mathcal{A}$1, $\mathcal{A}$2, $\mathcal{A}$3, $\mathcal{A}$4, and $\mathcal{A}$5 (Johnson et al., 2022). It integrates forward transfer, backward transfer, performance maintenance, relative performance, and sample efficiency as lifelong evaluation metrics (Johnson et al., 2022).

Empirically, the field reports gains across very different domains. In the wireless-sensor-network setting, the proposed CD-L2RL algorithm improves adaptation speed by up to 35% over standard reinforcement learning and up to 70% over Lyapunov-based optimization, while also increasing total harvested energy (Firouzjaei et al., 25 Aug 2025). In continual VLA learning, LifeLong-RFT reports a 22% gain in average success rate over SFT on LIBERO and substantial AUC gains on real-world tasks (Liu et al., 11 Feb 2026). In scalable lifelong RL with a Dirichlet process mixture, the proposed method outperforms fine-tuning, replay, consolidation, and progressive networks on navigation, MuJoCo Reacher, and Hopper variants (Wang et al., 2022). In cross-domain policy optimization, QAvatar shows favorable transferability across locomotion and robot-arm manipulation benchmarks while reducing the risk of negative transfer through Bellman-consistency-based weighting (Chen et al., 12 Mar 2026).

The literature is also explicit about open problems. Many theoretically grounded works still assume common state and action spaces, or dynamics-only shifts, which simplifies task identification and transfer (Brunskill et al., 2015, Steinparz et al., 2022, Lyu et al., 2024). Similarity-based transfer methods often assume dense rewards and currently scale pairwise across source-target combinations, which makes large task libraries expensive (Serrano et al., 2023). Mask-sharing systems commonly rely on a task oracle or common backbone architecture for direct interoperability (Nath et al., 2023). Representation-mismatch penalties in PAR depend on manual tuning of $\mathcal{A}$6, and the method is presently formulated for single-source to single-target adaptation rather than lifelong multi-source reuse (Lyu et al., 2024). DPMM-based systems avoid explicit task boundaries, but long-term computational cost still grows with the number of mixture components (Wang et al., 2022).

A final misconception is that a single mechanism suffices for the whole problem. The surveyed work instead suggests a layered architecture: transferable reward or goal representations for cross-domain specification (Edwards et al., 2017), principled source selection or critic weighting to control negative transfer (Serrano et al., 2023, Chen et al., 12 Mar 2026), novelty-sensitive exploration for non-stationary regimes (Brunskill et al., 2015, Steinparz et al., 2022), and evaluation curricula that expose both transfer and forgetting across qualitatively different domains (Khetarpal et al., 2018, Johnson et al., 2022). This suggests that CD-L2RL is less a single algorithmic family than a research program organized around one persistent requirement: an agent must accumulate reusable structure over a lifetime while domains, goals, modalities, and decision interfaces continue to change.