Cross-Task Policy Guidance (CTPG)

• CTPG is a framework in multi-task reinforcement learning that trains a guide policy to select donor control policies, accelerating exploration and credit assignment.
• It employs deterministic gating mechanisms, such as the policy-filter and guide-block gates, to safeguard and optimize donor policy selection.
• CTPG extends to language-conditioned transfer and logic-guided planning, demonstrating increased sample efficiency and faster policy transfer in complex environments.

Cross-Task Policy Guidance (CTPG) is a framework in multi-task reinforcement learning that trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks’ control policies, generating better training trajectories (He et al., 9 Jul 2025). In a language-conditioned transfer setting, the same idea is realized by aligning natural language and policy embeddings in a unified representation space and using nearest-neighbor look-up to form a weighted policy initialization for a new task (Gautam et al., 1 Dec 2025). Related realizations of cross-task policy guidance appear in context-aware policy reuse, task-augmented policy search, logic specifications for soft policy guidance in POMCP, and hierarchical prompt decision transformers (Li et al., 2018, Deisenroth et al., 2013, Mazzi et al., 2023, Wang et al., 2024).

1. Formal definition in multi-task reinforcement learning

In the explicit CTPG framework, there are $N$ tasks, each an MDP

$\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$

The state space $S$, action space $A$, and discount factor $\gamma$ are shared, while the transition kernel $P_i$ and reward $R_i$ are task-specific. Each task $i$ has a control policy $\pi_i(a\mid s)$, a learned Q-function $Q_i(s,a)$, and a value $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$0. CTPG adds a guide policy $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$1, which is a distribution over indices $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$2 and chooses which task’s control policy will serve as the behavior policy for the next $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$3 steps (He et al., 9 Jul 2025).

The central intuition is that if a policy $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$4 of some donor task $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$5 is already good in states similar to those encountered by task $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$6, then using $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$7 to generate experience for task $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$8 can greatly accelerate exploration and credit assignment. At sampling time $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$9, one draws

$S$0

and for the next $S$1 steps actions are sampled from $S$2. The transitions $S$3 are stored in a replay buffer for both control-policy and guide-policy learning (He et al., 9 Jul 2025).

A defining feature of this formulation is that it is an explicit-sharing mechanism rather than a replacement for parameter sharing. The framework is stated to be adaptable to existing parameter sharing approaches, including one-hot conditioning, multi-head, modular routing, parameter-compositional, and gradient surgery methods, while leaving the underlying control-policy parameter sharing unchanged (He et al., 9 Jul 2025).

2. Guide-policy dynamics, gating, and optimization

The guide policy for task $S$4 acts every $S$5 steps and is trained through a fixed-horizon Bellman equation. Its guide Q-function $S$6 satisfies

$S$7

where the guide reward is the cumulative discounted return over the next $S$8 steps under the selected donor policy: $S$9 This makes the guide policy a long-horizon selector over donor behaviors rather than a one-step action chooser (He et al., 9 Jul 2025).

Two deterministic gating mechanisms are used to improve learning efficiency. The first is the policy-filter gate. A comparable guide-Q $A$0 is estimated under the same entropy-regularization used by $A$1, compared against

$A$2

and then masked by

$A$3

Sampling is performed from the renormalized distribution

$A$4

The second is the guide-block gate. Tasks whose log-temperature is above the average log-temperature are blocked from guidance; formally,

$A$5

If $A$6, then $A$7 acts for all steps and no guidance is used (He et al., 9 Jul 2025).

Training combines standard SAC for the control policies with a discrete-action SAC variant for the guide policy. The guide-policy training uses hindsight off-policy correction by reassigning the guide action as

$A$8

The paper emphasizes that no parameters are learned for the two gates; they are deterministic functions of $A$9, $\gamma$0, and $\gamma$1. Empirically, removing the policy-filter gate collapses performance, removing the guide-block gate slows down easy tasks, and removing hindsight off-policy correction makes guide training unstable (He et al., 9 Jul 2025).

3. Language–policy alignment as CTPG in CLIP-RL

“CLIP-RL” extends the philosophy of Contrastive Language-Image Pretraining to reinforcement learning by treating the instruction and corresponding policy of a task as two modalities representing the same concept, the task itself (Gautam et al., 1 Dec 2025). The method begins from $\gamma$2 base tasks, each with instruction $\gamma$3 and a converged base policy $\gamma$4. A fixed sentence embedder $\gamma$5 is mapped into a $\gamma$6-dimensional multi-modal space by a learned projection head $\gamma$7: $\gamma$8 For the policy stream, a vector representation $\gamma$9 is extracted by flattening and optionally sub-sampling the network’s weights, and a second projection head $P_i$0 maps it into the same space: $P_i$1 The full $P_i$2 similarity matrix is

$P_i$3

and the encoders are trained with a symmetric contrastive CLIP-style loss with temperature $P_i$4 (Gautam et al., 1 Dec 2025).

In practice, one may first train all $P_i$5 to convergence with their RL losses, freeze them, then optimize only $P_i$6 via $P_i$7. Once the shared space is learned, transfer to a new target instruction $P_i$8 proceeds by computing $P_i$9, computing cosine similarities to the base instruction embeddings $R_i$0, and forming a weighted average of policy parameters: $R_i$1 The resulting $R_i$2 is reshaped back into network weights to initialize a new policy network $R_i$3 (Gautam et al., 1 Dec 2025).

The transfer pipeline has three stated effects. First, policy initialization starts the new network at $R_i$4 rather than random weights, so the agent already encodes behaviors similar to those needed for $R_i$5. Second, early exploration is biased because $R_i$6 proposes actions that resemble combinations of source policies. Third, standard RL such as PPO or A2C is then run from initialization $R_i$7, minimizing the ordinary policy-gradient loss $R_i$8 on the new MDP (Gautam et al., 1 Dec 2025).

The reported experiments use four grid-worlds of increasing size, $R_i$9, $i$0, $i$1, and $i$2. The base tasks are four instructions—“top left first,” “top left second,” “top right first,” and “top right second”—and the target task is “top left third.” The metric is the number of environment steps or episodes until reaching $i$3 of the optimum reward, averaged over 10 independent trials. The key finding is that the CLIP-RL CTPG method achieves $i$4 fewer training steps on average compared to a pure language-similarity baseline, and that the larger the grid, the larger the speed-up (Gautam et al., 1 Dec 2025).

4. Antecedents: policy reuse and task-conditioned policy search

Context-Aware Policy reuSe (CAPS) realizes cross-task policy guidance through a source-policy library $i$5, a context descriptor $i$6, and temporally abstracted options. Each source policy is stored as an option $i$7 with intra-option policy $i$8, initiation set $i$9, and parameterized termination function $\pi_i(a\mid s)$0. A context-aware selector $\pi_i(a\mid s)$1 uses an $\pi_i(a\mid s)$2-greedy choice on the option-value function $\pi_i(a\mid s)$3, and termination is learned from an advantage-based stopping rule. Under bounded rewards, standard stochastic-approximation conditions, and persistent exploration, the paper states Theorem 1 (Optimal Source Selection) and Theorem 2 (Target-task Optimality). Empirically, CAPS converges $\pi_i(a\mid s)$4 faster than baselines in grid-world and reaches the target object $\pi_i(a\mid s)$5 sooner than all baselines in Pygame Learning Environment (Li et al., 2018).

“Multi-Task Policy Search” realizes cross-task policy guidance by making the controller an explicit function of both the state and a task descriptor $\pi_i(a\mid s)$6. The augmented state is

$\pi_i(a\mid s)$7

with $\pi_i(a\mid s)$8 when task and state share coordinates, or $\pi_i(a\mid s)$9 when the descriptor is external. The policy is written as $Q_i(s,a)$0, and the multi-task objective averages the long-term cost over training tasks: $Q_i(s,a)$1 The method is embedded in PILCO, uses moment-matching predictions and analytic gradients, and permits zero-shot generalization to a new $Q_i(s,a)$2 without further training once $Q_i(s,a)$3 is learned (Deisenroth et al., 2013).

These antecedents differ in mechanism but share a common structural theme: useful behavior for a target task is not learned entirely from scratch. CAPS performs context-dependent selection and learned termination over reusable source options, whereas Multi-Task Policy Search absorbs cross-task structure directly into a single policy that varies with a task descriptor (Li et al., 2018, Deisenroth et al., 2013).

5. Logic-guided planning and prompt-based sequence models

Cross-task policy guidance also appears outside standard actor–critic MTRL. In “Learning Logic Specifications for Soft Policy Guidance in POMCP,” high-return traces from small POMCP instances are converted into an inductive logic programming task under answer set programming semantics. Belief states are grounded into high-level categorical features, actions are mapped to ASP atoms, and ILASP learns a hypothesis $Q_i(s,a)$4 of causal rules and weak constraints. At each new tree node, POMCP grounds the current belief into feature atoms, queries whether actions are derivable under $Q_i(s,a)$5, and injects a small prior into UCT by pretending that a suggested action has already been sampled $Q_i(s,a)$6 times with high value $Q_i(s,a)$7. Because the added counts $Q_i(s,a)$8 are finite, the paper states that standard UCT guarantees still apply asymptotically (Mazzi et al., 2023).

In “Hierarchical Prompt Decision Transformer” (HPDT), cross-task guidance is implemented through two layers of soft prompt tokens. A global prompt $Q_i(s,a)$9 is formed by embedding transition tuples from a demonstration segment and averaging them: $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$00 Adaptive prompts $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$01 are retrieved online by nearest-neighbor search over demonstration keys built from $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$02, embedding the retrieved tuples $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$03, and averaging them. The global token is prepended once, while the adaptive prompt is injected at each timestep by vector summation in the embedding space. The policy is trained with an action-prediction MSE objective over training tasks, rollout segments, and paired demonstrations (Wang et al., 2024).

The main cross-task guidance mechanisms described across these works can be organized as follows:

Paper	Guidance object	Transfer mechanism
(He et al., 9 Jul 2025)	Guide policy over task indices	Select donor behavior policy for next $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$04 steps
(Gautam et al., 1 Dec 2025)	Joint language–policy embedding	Weighted policy initialization from nearest neighbors
(Li et al., 2018)	Source-policy options with learned termination	Context-aware reuse and stopping
(Deisenroth et al., 2013)	Task descriptor $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$05 in policy input	Single policy generalizes to new tasks
(Mazzi et al., 2023)	Learned ASP rules	Soft priors in UCT for larger instances
(Wang et al., 2024)	Global and adaptive prompt tokens	Retrieval-augmented in-context policy generation

Empirically, the logic-guided POMCP system reports a $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$06 increase in discounted return in battery, and up to nearly $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$07 improvement in rocksample as horizon grows (Mazzi et al., 2023). HPDT reports that it outperforms all baselines on $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$08 tasks, with ablations showing that removing $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$09 collapses performance on tasks with discrete identity shifts and removing $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$10 hurts tasks with fine-grained local guidance needs (Wang et al., 2024).

6. Empirical profile, misconceptions, and limitations

Across the reported experiments, CTPG is repeatedly associated with gains in sample efficiency, faster transfer, or stronger few-shot generalization, but the mode of guidance differs materially by formulation. In explicit multi-task SAC, incorporating CTPG with five classical parameter-sharing methods significantly enhances performance in manipulation and locomotion benchmarks, often by $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$11 relative on MetaWorld-MT10/50 and by $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$12k return on HalfCheetah-MT5/8 (He et al., 9 Jul 2025). In CLIP-RL, the aligned language–policy space reduces the number of environment steps or episodes required to reach $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$13 of the optimum reward, with larger speed-ups on larger grids (Gautam et al., 1 Dec 2025). In planning and sequence-modeling settings, soft priors and retrieval-conditioned prompts serve analogous guiding roles (Mazzi et al., 2023, Wang et al., 2024).

A common misconception is that cross-task guidance is equivalent to ordinary parameter sharing. The explicit CTPG framework argues the opposite: existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures, whereas CTPG exploits cross-task similarities through explicit guidance from other tasks’ control policies and leaves the control-policy parameter-sharing mechanism unchanged (He et al., 9 Jul 2025). Another misconception is that guidance eliminates the need for task-specific optimization. CLIP-RL still fine-tunes the initialized policy with standard RL on the new MDP, and CAPS still learns a target-task optimal policy through intra-option Q-learning with learned termination (Gautam et al., 1 Dec 2025, Li et al., 2018).

The literature also identifies clear failure modes. Harmful donor selection is a central issue in explicit CTPG, which is why the policy-filter gate and guide-block gate are deterministic safeguards; removing them degrades performance (He et al., 9 Jul 2025). Logic-guided POMCP can fail when examples are insufficient or low quality, when the branching factor is small, or when the user-defined features are poor (Mazzi et al., 2023). CAPS requires that source and target share a common state/action space, and tabular CAPS does not scale; function approximation adds complexity and potential instability (Li et al., 2018). Multi-Task Policy Search notes that task representation $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$14 is hand-designed and that no online re-training is performed on a new test $\M_i = \bigl(S, A, P_i, R_i, \gamma\bigr),\quad i\in\{1,\dots,N\}.$15 unless the PILCO loop is rerun (Deisenroth et al., 2013).

Taken together, these results suggest that “cross-task policy guidance” is best understood not as a single algorithmic primitive but as a design pattern for transferring behavioral structure across tasks. The guiding signal may be a donor control policy, a weighted initialization in a shared embedding space, a reusable option with learned termination, a task descriptor inside a unified controller, a logic rule used as a soft prior, or a retrieved prompt injected into a sequence model. What remains constant is the operational role of guidance: to bias exploration, initialization, or action selection toward behaviors already shown to be useful in related tasks (He et al., 9 Jul 2025, Gautam et al., 1 Dec 2025, Li et al., 2018, Deisenroth et al., 2013, Mazzi et al., 2023, Wang et al., 2024).