Dynamic Multiplexing Policy in RL

• Dynamic Multiplexing Policy is a reinforcement learning strategy that combines a trained task policy with a user-intent model derived from trajectory-level feedback.
• It employs RUDDER-based reward redistribution, Boltzmann fusion, and dynamic temperature modulation to adapt the influence of human feedback on action selection.
• The approach enables zero-shot personalization, ensuring task performance while conservatively adapting to user preferences without additional environment interactions.

Dynamic Multiplexing Policy, in the reinforcement-learning sense formalized in "Dynamic Policy Fusion for User Alignment Without Re-Interaction," denotes a personalized control policy that combines a trained task policy with a user-intent policy inferred from trajectory-level human feedback, while requiring no additional interaction with the environment. In this usage, multiplexing means dynamically blending a task channel and a user channel so that the agent continues to achieve the intended task while adhering to preference, avoidance, or mixed user criteria. The defining features are trajectory-level feedback on previously collected rollouts, RUDDER-based reward redistribution, product-style policy fusion with an invariability constraint, and state/history-dependent temperature modulation of the user policy (Palattuparambil et al., 2024).

1. Problem formulation and objective

The underlying decision process is a standard Markov decision process

$M = (S, A, P, r_T, \gamma),$

with states $S$, actions $A$, transition kernel $P$, task reward $r_T$, and discount $\gamma$. The already trained task-specific policy is denoted $\pi_\phi(a \mid s)$ and is learned by a DQN from $Q(s,a)$. This policy maximizes task return, but it may not align with personal preferences such as avoiding certain states or preferring certain lanes or routes (Palattuparambil et al., 2024).

The personalization problem is posed without access to a user-specific reward function and without retraining from scratch. Instead, the method uses trajectory-level feedback collected on the same trajectories used to train $\pi_\phi$. In the reported experiments, human feedback is simulated as numeric trajectory scores $l$, defined as counts of how often a trajectory meets user criteria. The objective is to infer a user-intent policy $S$0 from these labels and to fuse it with $S$1 into a personalized policy that balances task performance and user preferences.

This formulation is distinct from state-action labeling and from pairwise preference comparisons. The trajectory-level rating mechanism is described as more feasible than state-action feedback and as avoiding pairwise labeling. A plausible implication is that the policy is designed for settings where feedback bandwidth is limited but retrospective evaluation of whole trajectories is available.

2. Intent inference from trajectory-level feedback

The user-intent model is learned with RUDDER. An LSTM-based predictor is trained to output $S$2, interpreted as an estimate of expected cumulative human-induced preference value along a trajectory (Palattuparambil et al., 2024). The training loss is composite: $S$3

$S$4

$S$5

and

$S$6

Here $S$7 is the trajectory score, $S$8 is the LSTM output at time $S$9, $A$0 is the horizon, and $A$1 is a forward prediction head.

RUDDER is then used for reward redistribution: $A$2 These redistributed human-induced rewards are mean-shifted,

$A$3

to produce a signed signal irrespective of whether the intent is preference or avoidance. The LSTM-derived $A$4 and the redistributed rewards $A$5 are the two quantities that connect trajectory-level labels to control-time decisions: $A$6 induces a user policy, and $A$7 drives the dynamic gate.

The implementation details reported for this stage are specific. The LSTM is single layer with 64 units; actions are FiLM-conditioned on states; optimization uses Adam with learning rate $A$8 and gradient clipping $A$9. If observations are pixels, a VAE is pre-trained to reduce dimensionality. These details matter because the quality of the redistributed signal is explicitly identified as central to stable gating.

3. Policy fusion and the dynamic gating rule

The task policy and the user-intent policy are constructed as Boltzmann policies: $P$0

$P$1

Boltzmannization is motivated by the need to avoid zero-support issues and to make fusion well-defined over all actions (Palattuparambil et al., 2024).

The static fused policy is

$P$2

with $P$3 a normalizer. This square-root product form is claimed to satisfy two structural requirements. First, it acts on the intersection of supports. Second, it satisfies an invariability constraint: if $P$4, then $P$5. The paper also states that naive product fusion $P$6 fails invariability unless $P$7 is random.

The policy becomes dynamic through temperature modulation of the user channel. Accumulated shifted human-induced reward is defined as

$P$8

The user-policy temperature is then updated by

$P$9

When $r_T$0 exceeds $r_T$1, $r_T$2 increases toward $r_T$3, flattening $r_T$4 and reducing its dominance. When $r_T$5 is below $r_T$6, $r_T$7 decreases toward $r_T$8, sharpening $r_T$9 and increasing its influence. The multiplexing is therefore state/history-dependent through $\gamma$0 and $\gamma$1, rather than being a fixed convex or product combination.

In the reported experiments, action selection is greedy with respect to the fused policy: $\gamma$2 This corresponds to low-temperature exploitation of $\gamma$3.

4. Offline workflow and deployment procedure

The end-to-end procedure is explicitly given as a seven-step pipeline (Palattuparambil et al., 2024).

1. Collect trajectories $\gamma$4 from $\gamma$5, used for DQN training.
2. Obtain human trajectory-level feedback $\gamma$6 on $\gamma$7.
3. Train an LSTM to predict $\gamma$8 from state-action sequences using the RUDDER-style loss $\gamma$9.
4. Compute human-induced rewards through reward redistribution and mean-shift them to obtain $\pi_\phi(a \mid s)$0.
5. Construct Boltzmann policies $\pi_\phi(a \mid s)$1 and $\pi_\phi(a \mid s)$2, fuse them via $\pi_\phi(a \mid s)$3, and dynamically modulate $\pi_\phi(a \mid s)$4 using $\pi_\phi(a \mid s)$5.
6. Perform offline validation on held-out trajectories or via replay, without new interactions.
7. Deploy by acting greedily with on-the-fly $\pi_\phi(a \mid s)$6 modulation.

The zero-shot property follows directly from this organization: the human intent model is trained on the same trajectories used to learn the task policy, and deployment occurs without collecting additional preference data. The paper states that training $\pi_\phi(a \mid s)$7 and computing $\pi_\phi(a \mid s)$8 do not require importance sampling or off-policy evaluation.

Hyperparameters are environment-specific. For 2D Navigation, the reported values are $\pi_\phi(a \mid s)$9, $Q(s,a)$0, $Q(s,a)$1, and $Q(s,a)$2. For Highway, the reported values are $Q(s,a)$3, $Q(s,a)$4, $Q(s,a)$5, and $Q(s,a)$6. The slope parameter $Q(s,a)$7 controls responsiveness, while $Q(s,a)$8, $Q(s,a)$9, and $\pi_\phi$0 calibrate how quickly the system suppresses over-adherence or reinforces under-adherence.

5. Zero-shot adaptation, bounded deviation, and safety properties

The theoretical part is framed around conservative deviation from the task policy (Palattuparambil et al., 2024). The invariability lemma states that if $\pi_\phi$1, then $\pi_\phi$2 and $\pi_\phi$3 are invariant policies with zero KL divergence. This property is used to justify the square-root product fusion as a stable personalization operator.

The main bounded-divergence theorem assumes bounded deviations in $\pi_\phi$4-values and temperatures: $\pi_\phi$5 Under these assumptions,

$\pi_\phi$6

where $\pi_\phi$7 is the normalizer of $\pi_\phi$8, $\pi_\phi$9, and

$l$0

This bound quantifies how much the personalized policy can deviate from the task policy as a function of $l$1- and $l$2-deviations.

The safety interpretation is operational rather than formal in the control-theoretic sense. The paper attributes conservative zero-shot adaptation to the KL-type bound together with temperature limits $l$3 and $l$4. It also notes practical calibration effects: larger $l$5 delays increases in $l$6, thereby keeping $l$7 sharper longer, while larger $l$8 caps how uniform $l$9 can become when it is over-dominant.

6. Empirical behavior and ablation results

The empirical study covers 2D Navigation, Highway, and Pong, with preference, avoidance, and mixed modes where applicable. Baselines are DQN, RUDDER, static fusion, and MORL with scalarized reward $S$00 (Palattuparambil et al., 2024).

A central result is that static fusion can over-adhere to the user policy. In 2D Navigation, static fusion is reported to repeatedly revisit desired regions, severely degrading task score in preference and mixed modes, and tuning $S$01 is described as brittle. Dynamic fusion is presented as the mechanism that prevents this looping failure mode.

In 2D Navigation, dynamic fusion achieves perfect task scores of approximately $S$02 while increasing desired-region visits without looping. In Preference mode, the reported values are desired visits $S$03 and score $S$04. In Mixed mode, undesired visits are approximately $S$05 and score is $S$06.

In Highway, dynamic fusion is reported to strongly adhere to lane preferences or avoidance while retaining near-DQN task performance. In Avoidance mode, the reported values are undesired lane $S$07, hits $S$08, and score $S$09. In Preference mode, the reported values are desired lane $S$10, hits $S$11, and score $S$12.

In Pong, dynamic fusion spends $S$13 in the preferred region while scoring $S$14, compared with DQN at $S$15. The reported adherence is substantially better than MORL at $S$16, while retaining a nonzero score.

The ablations isolate the gating parameters. Higher $S$17 keeps $S$18 sharper longer and increases preference adherence, but may reduce task score slightly in Highway. Larger $S$19 weakens $S$20 more strongly when it becomes over-dominant, reducing desired-lane visits and increasing undesired-lane visits. These results suggest that the dynamic gate is not merely a regularizer but the mechanism that determines whether personalization remains compatible with task completion.

7. Limitations, extensions, and broader uses of the term

The reported limitations are concrete. The current implementation assumes discrete actions. The policy is sensitive to temperature hyperparameters and sigmoid slope. Noisy or mis-specified preferences can misguide $S$21 and $S$22. Distribution shift between training trajectories and deployment can degrade alignment. Overfitting to limited feedback is also identified, and the stability of gating is said to depend on RUDDER signal quality (Palattuparambil et al., 2024).

The extensions listed in the source are multi-user multiplexing through multiple $S$23, time-varying preferences via meta-RL or adaptive LSTM fine-tuning, integration with MORL frontiers so that $S$24 acts as a personalized point on a Pareto surface, and hierarchical gating that combines per-episode and session-level feedback. These directions preserve the same basic abstraction: a task channel, one or more user channels, and a gating rule that modulates their relative effect.

Outside zero-shot RL personalization, the phrase "dynamic multiplexing policy" appears in several unrelated technical literatures. In rollout–training disaggregation for RL post-training, "RollMux" uses phase-level interleaving of rollout and training jobs through co-execution groups, conservative stochastic admission, and round-robin scheduling (Wu et al., 12 Dec 2025). In IEEE 802.11be multi-link WLANs, MCAB periodically reallocates active flows across enabled interfaces according to available airtime $S$25 and a timer $S$26 (López-Raventós et al., 2022). In optical metasurfaces, a dynamic multiplexing policy denotes the design and decoding rules that map per-channel amplitudes to orientations of single-sized Sb$S$27S$S$28 meta-atoms and polarization keys (Liu et al., 2024). In multiplexed quantum repeaters, it denotes the rule that, after a link is established with one end node, assigns all remaining chips to the opposite end node (Grimbergen et al., 8 Jul 2026). In microarchitecture, it denotes runtime selection among policy combinations for prefetching and cache replacement across 20M-instruction execution phases (Zhang et al., 6 May 2026).

These usages suggest that the phrase is field-dependent rather than canonical. In the reinforcement-learning sense, however, its most specific meaning is the dynamic fusion of a task policy and a user-intent policy under a zero-shot, no-reinteraction constraint, with reward-redistributed human feedback supplying the control signal for online modulation.