---
title: 'Composition-RL: Modular Reuse in Reinforcement Learning'
url: https://www.emergentmind.com/topics/composition-rl
type: topic
---

# Composition-RL: Modular Reuse in Reinforcement Learning

Composition-RL denotes a family of reinforcement-learning methods in which previously solved components are recombined to solve new problems, satisfy structured specifications, or improve generalization to unseen combinations. In its canonical early form, the problem is posed as follows: a lifelong-learning agent has already solved a library of “basis” tasks and wants to tackle a new task immediately—without any further learning—by re-using its existing value functions [1807.04439]. The same label is now used across several related settings, including exact and approximate value-function composition, assume–guarantee synthesis for stochastic control systems, modular policy composition, language-grounded task composition, and the automatic composition of verifiable prompts or environments for RL with verifiable rewards [2212.01174, 2312.01456, 2602.12036].

## 1. Conceptual scope

A recurring feature of Composition-RL is that the constituent tasks share a common substrate while differing in structured ways. In value-function composition, tasks share the same state and action spaces and differ only in reward functions, often only on absorbing goal states [1807.04439]. In stochastic-control formulations, continuous-space subsystems are abstracted into finite MDPs or stochastic games and then synthesized compositionally via assume–guarantee reasoning or graph-structured reach-avoid decompositions [2208.03485, 2312.01456]. In lifelong and language-grounded settings, the common substrate is a reusable library of neural modules, world value functions, or parsed symbolic operators [2207.00429, 2501.12539]. In recent RLVR work, the common substrate is a pool of verifiable prompts or deterministic environments that can be fused into harder training instances [2602.12036, 2606.12373].

| Setting | Composition object | Representative papers |
|---|---|---|
| Absorbing-task transfer | Soft or standard \(Q\)-functions | [1807.04439], [2212.01174], [2303.02557] |
| Stochastic control systems | Abstract MDPs, edge policies, subsystem specifications | [2208.03485], [2312.01456], [2309.06420] |
| Lifelong and embodied RL | Neural modules, diagnostic subtask structure, Boolean WVFs | [2207.00429], [2001.04418], [2501.12539] |
| Compositional generalization | Outcome-level policy optimization over outputs | [2605.04920], [2505.19406], [2509.25123] |
| RL with verifiable rewards | Composed prompts or recursive environment chains | [2602.12036], [2606.12373] |

This distribution of uses suggests that Composition-RL is not a single algorithmic template but a research program organized around the same operational goal: exploit compositional structure so that solving a new task requires less exploration, less retraining, or stronger guarantees than monolithic RL.

## 2. Value-function composition in entropy-regularized and standard RL

The foundational formulation of Composition-RL is given in entropy-regularized RL. Let \(\mathcal S\) be the state space, \(\mathcal A\) the finite action space, \(\rho(s'|s,a)\) deterministic, and \(\mathcal G\subset \mathcal S\) an absorbing goal set. A task MDP is specified by a reward function \(r(s,a)\) that differs across tasks only on the absorbing states \(\mathcal G\). With temperature \(\tau>0\), the optimal policy is the Boltzmann distribution
\[
\pi^*_s(a)\propto \bar\pi_s(a)\exp(Q^*(s,a)/\tau),
\]
and the soft composition rule for a library of optimal entropy-regularized \(Q\)-functions \(\{Q^{*,k}_\tau\}_{k=1}^n\) is
\[
Q_{\mathrm{comp}}(s,a)=\tau\log\Bigl(\sum_{k=1}^n w_k \exp(Q^{*,k}_\tau(s,a)/\tau)\Bigr),
\]
for non-negative weights \(w_k\) summing to \(1\) [1807.04439]. Under deterministic \(\rho\) and an MDP family where \(r\) differs only on \(\mathcal G\), if the new task’s terminal reward satisfies
\[
r(g)=\tau\log\sum_k w_k\exp(r_k(g)/\tau)\qquad \forall g\in\mathcal G,
\]
then this operator yields the true optimal \(Q^{*}_{\tau,\mathrm{new}}\). In desirability variables \(Z_k(s,a)=\exp(Q^{*,k}_\tau(s,a)/\tau)\), composition is linear:
\[
Z_{\mathrm{comp}}(s,a)=\sum_k w_k Z_k(s,a),\qquad Q_{\mathrm{comp}}=\tau\log Z_{\mathrm{comp}}.
\]

As \(\tau\to0^+\), the entropy penalty vanishes and the composition rule reduces to
\[
Q^{0}_{\mathrm{comp}}(s,a)=\max_{k=1,\dots,n}Q^{*,k}_0(s,a),
\]
independent of \(w\) [1807.04439]. In this limit, the rule coincides with the “generalised policy improvement” or “max-over-heads” principle and is exactly optimal for new tasks whose terminal rewards are the pointwise max of the basis rewards. The same work reports exact OR-style composition, approximate AND-style composition by averaging \(Q_{\mathrm{Blue}}\) and \(Q_{\mathrm{Square}}\), and temporal composition that creates an “always-collect-any-remaining-item” policy in an 11×11 pixel-based grid world with six collectible items.

Later soft-RL work broadens this picture from exact closed-form composition to correction-based composition. For two entropy-regularized tasks \(T^{(1)}=(p,r)\) and \(T^{(2)}=(p,\tilde r)\), with reward difference \(k(s,a,s')=\tilde r(s,a,s')-r(s,a,s')\), the optimal soft value functions satisfy
\[
Q^*_{(2)}(s,a)=Q^*_{(1)}(s,a)+K^*(s,a),
\]
where \(K^*\) is the optimal soft \(Q\)-function of the auxiliary task with reward \(k\) [2212.01174]. The same framework shows that potential-based reward shaping remains policy-invariant in the entropy-regularized case:
\[
r'(s,a,s')=r(s,a,s')+\gamma \Phi(s')-\Phi(s),
\]
with
\[
Q'^*(s,a)=Q^*(s,a)-\Phi(s),\qquad V'^*(s)=V^*(s)-\Phi(s).
\]
For a composite reward \(r_{\rm comp}=f(r^{(1)},\dots,r^{(M)})\), the exact optimal soft \(Q\)-function has the form
\[
Q^*_{\rm comp}(s,a)=Q_{\rm naive}(s,a)+K^*(s,a),
\]
so a naïve composition can be corrected by learning a residual soft value function [2212.01174].

A more general analysis replaces exact equalities by double-sided bounds. For a broad class of convex-sublinear or concave-superlinear composition functions, the optimal composite value \(\widetilde Q^*\) is bounded above and below by transformations of primitive \(Q\)-functions plus auxiliary slack terms; this yields regret bounds for zero-shot policies and motivates hard clipping, soft clipping, and test-time clipping during fine-tuning [2303.02557]. This suggests a three-tier taxonomy inside value-function Composition-RL: exact zero-shot composition under restrictive structural assumptions, exact composition with learned corrections in soft RL, and bounded zero-shot transfer with auxiliary uncertainty control.

## 3. Compositional synthesis with guarantees in stochastic control and safety

In discrete-time stochastic control systems, Composition-RL is formulated over a network of continuous-space stochastic subsystems \(\Sigma_i=(X_i,U_i,W_i,\zeta_i,f_i,Y_i,h_i)\), with unknown dynamics \(f_i\) that are only assumed Lipschitz in \(x_i\) and \(w_i\) with known constants \(H_{x_i},H_{w_i}\) [2208.03485]. Each subsystem is implicitly abstracted by a finite MDP \(\mathcal M_i=(S_i,U_i,W_i,T_i)\) obtained from a uniform quantizer \(\alpha_i:X_i\to S_i\). The abstraction error in finite-horizon satisfaction is bounded by
\[
\epsilon_i=T_i\cdot L_i\cdot(\delta_i H_{x_i}+\mu_i H_{w_i}),
\]
where \(T_i\) is the horizon, \(L_i\) the Lebesgue measure of \(X_i\), \(\delta_i\) the state-grid spacing, and \(\mu_i\) the input-grid spacing. Each abstract MDP is viewed as a two-player stochastic game, synthesized locally by minimax-Q learning under worst-case internal input, with no knowledge of neighbors required. The network-level guarantee is a compositional lower bound:
\[
P_{\mathrm{network}}\ge \prod_{i=1}^N P_i-\delta,\qquad
\delta=\tfrac12[(1+\epsilon)^N-(1-\epsilon)^N],\quad \epsilon=\max_i \epsilon_i.
\]
The same framework compiles finite-horizon co-safe LTL formulas into automata-based reward functions and uses potential-based reward shaping to densify sparse signals [2208.03485].

A closely related line uses logical specifications provided in SpectRL. Any SpectRL formula \(\phi\) is compiled into a directed acyclic abstract graph \(G=(V,E,\beta,s,t)\), where vertices carry “vertex regions” and edges carry “safety regions” [2312.01456]. Edge policies are learned jointly with reach-avoid supermartingales (RASMs), and a multiplicative RASM yields a tighter lower bound on reach-avoid probability:
\[
P[\mathrm{ReachAvoid}(X^T,X^U)]\ge 1-\frac1\lambda \gamma^N,
\qquad
N=\Bigl\lfloor\frac{\lambda-1}{L_V\Delta}\Bigr\rfloor.
\]
The algorithm “Claps” then propagates lower bounds over the DAG via
\[
\mathrm{Prob}[v_i]\leftarrow \max\{\mathrm{Prob}[v_i],\,p_{v\to v_i}\cdot \mathrm{Prob}[v]\},
\]
and, if \(\mathrm{Prob}[t]\ge p_{\mathrm{global}}\), back-tracks the maximizing path and stitches the edge policies in sequence [2312.01456].

Verifiable compositional RL systems based on a high-level parametric MDP (pMDP) make the same decomposition explicit at the subsystem level. A collection of RL subsystems, each with entry conditions \(I_c\), exit conditions \(F_c\), and horizon \(T_c\), is abstracted into a pMDP whose parameters \(p_c\) represent subtask success probabilities [2309.06420]. If each subsystem policy \(\pi_c\) satisfies
\[
\Pr^s_M(\Diamond_{\le T_c}F_c\mid \pi_c)\ge p_c\qquad \forall s\in I_c,
\]
then any high-level pMDP policy that reaches the abstract goal with probability at least \(1-\delta\) induces a composed policy in the original environment that satisfies the overall task with at least the same probability. When empirical subsystem performance falls below specification, the framework re-solves a bilinear program to update the subtask thresholds and re-route the high-level policy [2309.06420].

Safety-aware task composition adds a distinct Boolean layer. In a deterministic labeled MDP, conjunction and disjunction of extended optimal \(Q\)-functions are implemented by pointwise \(\min\) and \(\max\), while analytical negation for minimum-violation semantics is
\[
\neg\bar Q^*(s,g,a)=\bigl[\bar Q^*_{\max}(s,g,a)+\bar Q^*_{\min}(s,g,a)\bigr]-\bar Q^*(s,g,a).
\]
The framework distinguishes minimum-violation paths from prioritized-safety paths and extends Boolean composition from discrete action spaces to continuous action spaces via TD3, with actor selection determined by the composed critics [2306.17033]. This makes explicit a trade-off already implicit in earlier composition rules: exact recombination is often easiest for reachability or OR-type objectives, while safety and avoidance introduce extra semantics, extra assumptions, or approximate reasoning.

## 4. Neural modules, language grounding, and embodied policy composition

A modular lifelong formulation assumes a fixed set of shared subproblem solution spaces \(\mathcal F=\{F_1,\dots,F_k\}\) and a corresponding set of neural modules \(M=\{m_1,\dots,m_k\}\), such that the optimal policy for task \(t\) can be written as a composition
\[
\pi^{(t)*}(s)=m_{i_D}(\dots m_{i_2}(m_{i_1}(\phi_{i_1}(s)))\dots).
\]
Modules are organized into ordered depths, one module is selected per depth, online exploration is performed with PPO on a module copy, and offline consolidation uses Batch-Constrained Q-learning over replay buffers from all tasks seen so far [2207.00429]. The reported settings include 64 tasks in a discrete 2-D grid domain and 48 tasks in robotic manipulation. The method exhibits zero-shot generalization when structure is given, forward transfer during sequential learning, and retention via offline replay. A plausible implication is that Composition-RL in lifelong settings is as much about routing and reuse as about closed-form value algebra.

Language-conditioned Composition-RL makes the compositional structure explicit in either latent state or symbolic parses. In BabyAI-style instruction following, a diagnostic classifier is attached to the agent’s LSTM hidden state \(h_t\) to predict which subtask is currently active, with combined objective
\[
L_{\mathrm{total}}(\theta)=L^{\mathrm{PPO}}(\theta)+\beta L_{\mathrm{class}}(\theta).
\]
The classifier’s gradient shapes the hidden states so that they encode “current objective,” yielding more interpretable clustering and improved performance on repeated-visit and zero-shot transfer settings [2001.04418]. The effect is not uniform: the paper reports negligible effect on the simplest levels, but a reduction in episode length on “Before-repeat” and improved zero-shot success on novel attributes.

CERLLA pushes this symbolic direction much further. It pretrains \(N=9\) world value functions, one per atomic attribute, and composes them through Boolean operators
\[
Q_{i\lor j}(s,a)=\max\{Q_i(s,a),Q_j(s,a)\},\qquad
Q_{i\land j}(s,a)=\min\{Q_i(s,a),Q_j(s,a)\},
\]
with negation
\[
Q_{\neg i}(s,a)=(Q_{\mathrm{MAX}}(s,a)+Q_{\mathrm{MIN}}(s,a))-Q_i(s,a).
\]
A language instruction is mapped by a semantic parser to a Boolean expression over the symbols, and the resulting composed \(Q\)-function is used greedily [2501.12539]. The parser itself is improved by RL-style feedback: BM25 retrieves up to \(K=10\) previous examples, an LLM proposes a beam of \(B=10\) candidate expressions, and a candidate parse is accepted when the composed policy reaches a success-rate equal to the oracle upper-bound performance of \(92\%\). On 162 BabyAI tasks, the method reaches a success rate equal to the oracle policy’s upper-bound performance of \(92\%\), while the non-compositional baseline reaches \(80\%\) with the same number of environment steps [2501.12539].

Concept learning provides another embodied route to compositionality. In a 3D Unity environment where instructions specify color-shape targets, A2C agents trained directly on color-and-shape combinations require about \(67.4\)K episodes on train combinations and about \(94.8\)K episodes on held-out combinations. When agents are first trained on color-only or shape-only instructions and then fine-tuned on color-and-shape combinations, train learning drops to \(0.6\)K episodes and held-out learning to \(5.5\)K episodes; only these concept-then-compose agents solve a more complex zero-shot \(C\wedge S\wedge S\) environment [2309.04504]. This suggests that Composition-RL can emerge either from explicit operators over learned primitives or from representational factorization that makes later recombination much easier.

## 5. Outcome-level RL and compositional generalization in generative models

A distinct contemporary meaning of Composition-RL concerns models that must interpret or generate unseen combinations of known primitives. One formulation defines a compositional type \(c=(p,r)\), where \(p=(p_1,\dots,p_k)\in P^k\) is an ordered tuple of primitives and \(r\in R\) is a composition rule. Instead of token-level cross-entropy, the policy \(\pi_\theta(y\mid x)\) is optimized at the outcome level with Group Relative Policy Optimization (GRPO). Two rewards are studied: a binary exact-match reward and a composite reward built from primitive coverage and a compositional-skeleton term [2605.04920]. On SCAN, COGS, GeoQuery, and CFQ, GRPO improves compositional generalization over supervised fine-tuning. The exact-match averages over three runs reported for SFT, GRPO-Bin, and GRPO-Comp are \(77.4\), \(81.5\), and \(83.1\), with especially large gains on SCAN-length and CFQ-MCD3 [2605.04920]. The paper further reports that supervised models exhibit higher mean training-data trigram frequency among incorrect predictions, whereas GRPO reduces this copying bias and sharpens the output distribution.

Vision-language reasoning exposes a harder version of the same problem. ComPABench trains on isolated skills such as Shape Area and Grid Position, then evaluates cross-modal, cross-task, and out-of-distribution compositions [2505.19406]. For the 7B model, pure-text SFT attains near-perfect performance on the component tasks but drops to \(2.2\%\) on pure-text compositional evaluation; pure-text RL reaches \(81.2\%\). In multimodal composition, SFT attains \(7.2\%\) and RL \(31.2\%\). The paper argues that current VLMs trained with RL or other post-training strategies still struggle compositionally under cross-modal and cross-task scenario, and proposes RL-Ground, which combines “caption-before-thinking” with progressive vision-to-text grounding rewards. On multimodal composition, the 7B RL-Ground variant reaches \(52.8\%\), compared with \(31.2\%\) for baseline RL [2505.19406].

A synthetic string-transformation study addresses a central controversy directly: whether RL teaches genuinely new skills or merely activates existing ones. Atomic skills are deterministic functions \(f:\Sigma^*\to\Sigma^*\), \(g:\Sigma^*\to\Sigma^*\), and the compositional target is \(h(x)=g(f(x))\). Stage 1 trains atomic skills; Stage 2 performs RL on composed tasks with a terminal binary reward [2509.25123]. RL on atomic data only succeeds on Level 1 but fails to generalize compositionally, while RL on Level 2 compositions generalizes to deeper unseen levels and transfers to a different target task, Countdown, when the model already has the target’s atomic skills. The same paper reports that next-token training on the same compositional data does not produce these effects. This provides explicit evidence for a compositional-skill acquisition account of RL in at least one controlled setting [2509.25123].

## 6. Verifiable prompt and environment composition for RL with verifiable rewards

In RL with verifiable rewards, Composition-RL is used to create new training tasks automatically. One proposal targets the pass-rate-1 problem in RLVR: policy-gradient updates vanish when a prompt’s sampled rollouts are all correct or all incorrect, and easy prompts become increasingly prevalent as training proceeds [2602.12036]. Sequential Prompt Composition (SPC) defines a recursive operator that combines multiple original prompts into a new verifiable question of compositional depth \(K\). Training uses GRPO over the surrogate compositional dataset \(\hat{\mathcal D}_{C_K}\), optionally with a curriculum \(K(t)=1\), then \(2\), then \(3\). On Qwen3-4B, the overall average rises from \(28.1\) for baseline RL on original prompts to \(31.4\) for depth-2 Composition-RL; the curriculum variant reaches \(36.4\). In cross-domain training, Physics-Math-Composition reaches \(32.6\), compared with \(26.8\) for mix training and \(29.5\) for sequential Math-then-Physics [2602.12036].

A related but more general construction treats verifiable environments themselves as composable objects. In RACES, each environment is a four-tuple \(e=(G_e,f_e,D_e,V_e)\), and two environments are composable when the codomain of one matches the domain of the next [2606.12373]. Recursive composition yields chains \(F_{\pi_t}=f_{e_t}\circ\cdots\circ f_{e_1}\), and four operators instantiate RL tasks from these chains: \(\textsc{SEQUENTIAL}\), \(\textsc{PARALLEL}\), \(\textsc{SORT}\), and \(\textsc{SELECT}\). Path discovery uses a frontier-based BFS with quality filters on runtime errors, timeouts \(>2\) s, excessive steps \(>400\), and degenerate outputs. Across six unseen benchmarks, RACES improves DeepSeek-R1-Distill-Qwen-14B from \(48.2\) to \(51.3\) and Qwen3-14B from \(58.8\) to \(61.1\). On Qwen3-4B-Instruct-2507, RL on 50 base environments composed by RACES reaches \(50.8\), exceeding \(50.4\) from RL on 300 individual environments [2606.12373].

These RLVR formulations invert the direction of classical Composition-RL. Instead of composing policies or value functions to solve a new downstream task, they compose tasks themselves to generate a richer on-policy curriculum. The common mechanism remains structural reuse: existing solved or verifiable units are not discarded after training but reassembled into more difficult instances whose reward remains exact.

Across these strands, several limitations recur. Exact value-function composition typically requires tasks that differ only in terminal-state rewards and, in the 2018 recipe, deterministic dynamics; AND-type composition is only approximate there [1807.04439]. Soft correction and bound-based methods often assume shared state and action spaces and remain primarily tabular in their formal analyses [2212.01174, 2303.02557]. Formal-control approaches depend on abstraction error bounds, verification machinery, or graph decompositions [2208.03485, 2312.01456]. RLVR composition depends on reliable verifiers and incurs longer prompts or deeper environment chains as compositional depth increases [2602.12036, 2606.12373]. Even with these constraints, the literature consistently treats composition not as a peripheral convenience but as the central mechanism for zero-shot transfer, safety-preserving synthesis, modular reuse, and compositional generalization.

Source: https://www.emergentmind.com/topics/composition-rl