LEXPOL: Language-Conditioned Multi-Task RL

• LEXPOL is a language-conditioned multi-task RL framework that decomposes control into expert sub-policies guided by natural-language descriptions.
• It employs a pretrained language model (BERT) and a learned gating network to softly select or combine reusable skills across tasks.
• Empirical results on MetaWorld demonstrate that LEXPOL achieves improved sample efficiency and competitive performance compared to traditional multi-task methods.

Searching arXiv for the primary LEXPOL paper and closely related multi-task RL baselines to ground the article with current paper metadata and arXiv ids. Lexical Policy Networks (LEXPOL) denote a language-conditioned mixture-of-policies architecture for multi-task reinforcement learning (MTRL) in which task metadata, typically brief natural-language descriptions, are encoded and used by a learned gating module to softly select or combine multiple sub-policies into a final action. In this formulation, multi-task behavior is not represented by one monolithic shared controller, but by a reusable collection of experts that all observe the same environment state while language determines which skills should dominate control. The architecture was introduced in “Multi-Task Reinforcement Learning with Language-Encoded Gated Policy Networks” (Arora, 7 Oct 2025), where it is evaluated primarily on MetaWorld and interpreted as a mechanism for indexing and recombining reusable skills through natural-language task descriptions.

1. Definition and problem setting

LEXPOL is defined as a language-conditioned mixture-of-policies architecture for multi-task reinforcement learning (Arora, 7 Oct 2025). Its central claim is that multi-task control can be improved by separating behavior into a set of sub-policies or experts and using language-conditioned gating over their outputs, rather than relying on a single shared policy to absorb all task variation.

The paper studies multi-task RL settings in which each task comes with task metadata, especially brief natural-language descriptions. The motivating benchmark is MetaWorld, where tasks share the same state and action dimensionality, but the semantic meaning of state dimensions differs across tasks. This setting is viewed through a Contextual MDP / Block-Contextual MDP lens, where a context $c$ specifies task-dependent reward, transitions, and possibly relevant subspace or semantics: $$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$

The difficulty, as formulated in the paper, is that in shared-state multi-task environments the same state coordinates can have different meanings under different tasks. A standard shared policy must therefore implicitly infer both task identity and the relevant behavior primitives. LEXPOL addresses this by making task decomposition explicit: it performs task decomposition into reusable skills together with language-conditioned gating over policy outputs (Arora, 7 Oct 2025).

A key conceptual distinction drawn in the paper is with CARE. In CARE, language gates among state encoders and a universal policy acts on the resulting representation. In LEXPOL, language gates among policies or skills, while all policies receive the same state. This means language functions not primarily as a representation conditioner, but as a selector or composer of reusable control modules. This suggests that the method is aimed at semantic ambiguity in policy reuse rather than only task-conditioned representation learning.

2. Architectural organization

The architecture is divided into three parts: a context encoder, a mixture of policies, and a gating network (Arora, 7 Oct 2025). The high-level computation is: input state $s$ and task text $x$, encode the text into a task embedding, run all $k$ policies on the same state, compute gating weights from the text embedding, and combine the expert outputs into a final action.

The task metadata are encoded with a pretrained LLM, specifically BERT, optionally followed by a small MLP. The output is written as

$z_{\text{context}} \in \mathbb{R}^n,$

or equivalently

$z = f_{\text{text}}(x).$

The appendix reports a two-layer feedforward task encoder with hidden and output dimension 50. The pseudocode also includes

$$\bar z_{\text{context}}^m = \mathrm{stopgrad}(z_{\text{context}}^m),$$

before passing the embedding to the gate. The paper does not explain this design choice in detail.

The expert set consists of $k$ policies

$P_i,\quad i=1,\dots,k,$

each receiving the same state $$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$0 and producing an action

$$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$1

The number of experts $$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$2 is a hyperparameter. The paper states that any policy optimization algorithm could in principle be used, but the experiments use Soft Actor-Critic (SAC). The actor and critic are described only at a coarse level as feedforward networks with three fully connected layers of 400 units and ReLU nonlinearity.

The gating module is an MLP

$$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$3

that maps the context embedding to $$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$4 logits,

$$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$5

followed by a softmax,

$$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$6

The resulting routing is explicitly a soft mixture, not hard selection. The paper refers to these coefficients as soft attention, gating weights, and softmax gating embedding. A plausible implication is that the architecture is closer to continuous policy composition than to discrete expert arbitration.

The final action is the weighted sum of expert actions,

$$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$7

or

$$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$8

The paper’s pseudocode is typographically unclear at this line, but the surrounding description makes the intended computation the dot product between expert actions and attention weights (Arora, 7 Oct 2025).

3. Learning dynamics and optimization regime

LEXPOL is trained with Soft Actor-Critic, and the default regime is end-to-end training (Arora, 7 Oct 2025). The training workflow is stated as: provide task metadata, encode it with BERT plus MLP, compute gate weights, run all experts on the current state, form the weighted action mixture, execute the action in the environment, and update expert policies and gate jointly via SAC. The paper states that “The entire algorithm is learned end-to-end.” It also states that “The policy loss is used to update all the parameters as all networks directly result in the generation of the final action.”

The paper gives the recoverable core equations but does not provide a full LEXPOL-specific SAC derivation. What can be stated directly is that LEXPOL uses the SAC RL loss, the final mixed action interacts with the environment, and gradients update the expert policies and gating module jointly in the end-to-end setting (Arora, 7 Oct 2025). The exact objective as instantiated in LEXPOL is not written in formula form, and the paper does not specify whether critics are shared across experts, assigned per expert, or defined only on the mixed action.

The appendix reports the following optimization details: batch size $$\mathcal{M}'(c) = \{\mathcal{R}^c, \mathcal{T}^c, \mathcal{S}^c\}.$$9 number of tasks, policy learning rate $s$0, Q-function learning rate $s$1, Adam with betas $s$2, discount 0.99, episode horizon 150, reward scale 1.0, uniform exploration policy for 1500 steps, and 1 env step / 1 train step. The modular component count is given as 6 for MT10 and 10 for MT50, though the appendix phrases this as “number of encoders” and notes “each for LEXPOL + CARE,” making the wording slightly ambiguous.

Task descriptions are used at both training and inference time. The paper gives examples such as “go to the blue goal,” “go to the red goal,” and “go to the red goal, then the blue goal,” but does not enumerate all MetaWorld language templates. It states only that natural-language metadata are used and that the evaluation follows CARE’s protocol (Arora, 7 Oct 2025).

4. Frozen experts and compositional skill reuse

A distinctive variant in the paper is the frozen-experts setting, in which policies are first trained independently, then frozen, and only the gating module and embedding MLPs are learned for a new composite task (Arora, 7 Oct 2025). The paper explicitly states that “it is possible to freeze the policy parameters and just learn the gating.” This is important because it frames LEXPOL not merely as a jointly optimized multi-task controller, but as a compositional layer over a pre-existing skill library.

The clearest mechanistic analysis is carried out in a continuous T-shaped environment with two base tasks, “go to the blue goal” and “go to the red goal.” One policy is trained per task to convergence, both policies are frozen, and then the LEXPOL gate is trained on the new composite instruction “go to the red goal, then the blue goal.” The reward gives credit for reaching red first and blue second, with penalties for the wrong order or taking too long (Arora, 7 Oct 2025).

The visualization reported in the paper shows, for each state, which policy dominates the gate: red shading for the red-goal policy, blue shading for the blue-goal policy, with intensity reflecting confidence or dominance. The reported result is that the gate switches or blends the two experts in a way appropriate to the sequential instruction. This is presented as the strongest evidence that the language gate can recombine fixed skills for a novel task description (Arora, 7 Oct 2025).

The paper further states that when trained end-to-end rather than with pretrained experts, the learned experts resemble the same primitive skills observed in the frozen-expert analysis. This suggests that joint optimization can induce reusable modular behavior even without explicit pretraining of the component skills. A plausible implication is that language is functioning both as an index into a skill library and as a compositional selector over that library.

5. Empirical evaluation on MetaWorld

The primary benchmark is MetaWorld, comprising 50 robotic manipulation tasks with shared state and action spaces but task-dependent semantics (Arora, 7 Oct 2025). The paper evaluates on MT10 and MT50. Baselines include Multi-task SAC, Soft Modularization, SAC + FiLM, SAC + CARE, SAC + MoE / AMESAC, and one SAC agent per task as an upper bound.

The evaluation protocol follows CARE’s setup: evaluation at regular intervals, 5 trials per test environment, mean success rate across trials, averaging across environments, 10 random seeds, averaging the time series across seeds, and reporting the best mean success rate observed during training. MetaWorld returns binary success at episode end, so success rate is the primary metric (Arora, 7 Oct 2025).

The main quantitative results are as follows.

Setting	LEXPOL	Best comparison shown
MT10, 100k steps	0.39 ± 0.052	CARE: 0.35 ± 0.038
MT10, 0.5M steps	0.69 ± 0.093	CARE: 0.66 ± 0.034
MT10, 2M steps	0.86 ± 0.063	CARE: 0.82 ± 0.054
MT50, 100k steps	0.42 ± 0.012	CARE: 0.38 ± 0.082
MT50, 0.5M steps	0.52 ± 0.057	CARE: 0.49 ± 0.056
MT50, 2M steps	0.64 ± 0.057	CARE: 0.56 ± 0.032

At 2M steps, the full MT10 table gives: Multi-task SAC $s$3, Soft Modularization $s$4, SAC + FiLM $s$5, SAC + CARE $s$6, SAC + MoE $s$7, LEXPOL $s$8, and one SAC per task $s$9. At 2M steps on MT50: Multi-task SAC $x$0, Soft Modularization $x$1, SAC + FiLM $x$2, SAC + CARE $x$3, SAC + MoE $x$4, LEXPOL $x$5, and one SAC per task $x$6 (Arora, 7 Oct 2025).

The paper’s direct interpretation of these tables is that LEXPOL consistently matches or exceeds strong multi-task baselines, performs especially well in sample efficiency, and substantially outperforms plain multi-task SAC. It also notes an important significance nuance: LEXPOL is significantly better than weaker baselines such as Multi-task SAC, Soft Modularization, and SAC + FiLM, but differences relative to stronger baselines such as CARE and MoE are not statistically significant under the paper’s test (Arora, 7 Oct 2025). Accordingly, the strongest precise reading is not unconditional dominance, but consistent top-tier performance with strong early-learning behavior.

6. Mechanistic interpretation, scope, and limitations

The paper’s analysis suggests that the language-conditioned gate learns at least two roles. First, language functions as an index over skills: for ordinary tasks, task text indicates which expert or experts should dominate. Second, language functions as a compositional selector: in composite tasks, it appears to organize the recombination of multiple skills into a novel behavior (Arora, 7 Oct 2025).

A technically important limitation is that the gating weights are computed from task context only, not directly from the environment state: $x$7 Thus the gate is task-conditioned rather than explicitly state-conditioned. All experts see the current state, but the mixture coefficients are primarily fixed by the task description. The paper occasionally discusses that humans vary skill usage across states within the same task, but the described architecture does not implement state-dependent gating (Arora, 7 Oct 2025). This suggests that intra-episode phase switching can only arise indirectly through the experts’ different state responses under a fixed or task-level mixture.

Several implementation details are left underspecified. The paper does not fully specify the exact SAC objective as instantiated with mixed actions, whether critics are shared or per-expert, exact language preprocessing, or the full MetaWorld task-description templates. It also does not provide a systematic ablation over number of experts $x$8, sensitivity to language encoder choice, gate sparsity diagnostics, or detailed gate-distribution analyses across MetaWorld (Arora, 7 Oct 2025). Re-implementation is therefore possible at a high level, but lower-level design choices would need to be inferred.

The paper also reports that a LEXPOL + CARE hybrid improves long-run performance but underperforms at 100k steps, which it attributes to higher network complexity. This suggests that richer modular decompositions may improve asymptotic behavior while worsening low-data optimization when the architecture becomes larger (Arora, 7 Oct 2025).

7. Relation to adjacent “lexical policy” uses and broader interpretation

The term “LEXPOL” is specific in (Arora, 7 Oct 2025): it refers to text-conditioned soft routing over expert policies in multi-task RL. However, adjacent work uses related lexical-to-policy ideas in other senses. “Natural Language Access Control (NLAC): From Help Desk Requests to Structured Policies” (Wessner et al., 4 Jun 2026) studies translation from help-desk-style natural language into structured access-control intents, and is therefore relevant to lexical policy translation, but not to mixture-of-policies RL. “NeLLCom-Lex: A Neural-agent Framework to Study the Interplay between Lexical Systems and Language Use” (Zhang et al., 26 Sep 2025) learns a speaker policy

$x$9

over lexical choices in a referential game, which is relevant to lexical decision policies, but not to language-conditioned control over continuous-action experts. These neighboring usages indicate that “lexical policy” can refer either to language-conditioned action selection, lexical-to-structured-policy mapping, or lexical choice as an action space. In the strict technical sense established by (Arora, 7 Oct 2025), however, LEXPOL is a modular multi-task RL architecture.

Its principal contribution is therefore architectural rather than merely representational: language is used to gate over policies themselves, not simply to modulate a shared representation or concatenate an instruction embedding to a monolithic policy. The paper’s summary claim is that natural language can index and recombine reusable control skills within a single multi-task policy framework (Arora, 7 Oct 2025). A plausible broader implication is that the method is most natural in domains where tasks share state and action spaces, task identity is available as metadata, and the task family admits reusable skill structure.