---
title: General Policy Composition Techniques
url: https://www.emergentmind.com/topics/general-policy-composition-gpc
type: topic
---

# General Policy Composition Techniques

Searching arXiv for recent papers on General Policy Composition and closely related policy composition work.
General Policy Composition (GPC) denotes a family of methods for constructing a single actionable policy by composing multiple pre-existing policies or policy-like objects, rather than training one monolithic model over pooled heterogeneous data. Across the literature, GPC appears in several formal guises: distribution-level composition of diffusion or flow-based robot policies, product-of-experts and score-space composition of heterogeneous diffusion policies, prioritized composition of soft \(Q\)-functions under task ordering, compositional evaluation of switching policies through geometric horizon models, modular calling of reusable policy sketches, and free-energy-based gating of policy primitives. A recurring theme is that composition is performed in a representation aligned with the underlying decision model—scores, log-densities, \(Q\)-functions, visitation models, or module semantics—so that independently trained components can be reused while preserving structure needed for generalization, controllability, or safety [2503.12466][2510.01068][2402.02511].

## 1. Distribution-level composition in generative robot policies

In recent robotics work, GPC is instantiated as *test-time distribution-level composition* of pre-trained generative policies. The central formulation is given by “Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition” [2510.01068], which defines GPC as a training-free framework that “forms a stronger, more adaptable policy by convexly combining the distributional scores of multiple pre-trained diffusion-based or flow-based policies, and then sampling actions from the composed score.” A closely related precursor is “Modality-Composable Diffusion Policy (MCDP) via Inference-Time Distribution-level Composition” [2503.12466], which composes pre-trained single-modality diffusion policies during inference without retraining.

For diffusion-based policies, the basic sampler is written as
\[
\tau_{t-1} = \alpha_t \tau_t + \beta_t s_\theta(\tau_t, t) + \gamma_t \eta,\qquad \eta \sim N(0,\sigma_t^2 I),
\]
and GPC composes the parent policies at the score level:
\[
\hat{s}_{\mathrm{comp}}(\tau_t,t)=\sum_{i=1}^{n} w_i\, s_\theta(\tau_t,t,c_i),\qquad w_i\ge 0,\quad \sum_{i=1}^{n} w_i=1.
\]
The resulting action trajectory is sampled from the composed score rather than from any parent individually [2510.01068]. In MCDP, the corresponding rule is expressed in the common \(\epsilon\)-parameterization:
\[
\hat{\epsilon}_{\mathcal{M}^*}(\tau_t,t)=\sum_{i=1}^{n} w_i\,\epsilon_\theta(\tau_t,t,\mathcal{M}_i),\qquad \sum_i w_i=1,
\]
with the reverse update
\[
\tau^{t-1}=\alpha^t\big(\tau^t-\gamma^t\hat{\epsilon}_{\mathcal{M}^*}(\tau_t,t)+\xi\big),\qquad \xi\sim \mathcal{N}(0,\sigma_t^2 I).
\]
The paper emphasizes that this is *CFG-free* and avoids additional training or architectural modification of the parent experts [2503.12466].

These methods treat policy composition as an operation on conditional trajectory distributions rather than as feature concatenation or action averaging. In MCDP, each expert models \(p_\theta(\tau\mid o^{(m)})\) for a single modality such as RGB or point cloud, and GPC combines these experts “in a principled, distribution-level manner” to improve “cross-modality, cross-domain, and cross-embodiment” generalization [2503.12466]. In the broader GPC formulation, heterogeneous parents may include vision-action and vision-language-action policies, as well as diffusion-based and flow-matching policies, provided their outputs can be mapped into a common score representation [2510.01068].

## 2. Mathematical principles and theoretical guarantees

A major line of GPC work grounds composition in product-of-experts or additive score algebra. MCDP states the product-of-experts view explicitly:
\[
p_{\mathrm{comp}}(\tau)\propto \prod_i p_i(\tau\mid o^{(m_i)})^{w_i},
\]
so that
\[
\log p_{\mathrm{comp}}=\sum_i w_i \log p_i + C,\qquad
\nabla_\tau \log p_{\mathrm{comp}}=\sum_i w_i \nabla_\tau \log p_i.
\]
Under score-based diffusion, adding scores corresponds, up to scale, to adding noise predictions, which yields the practical inference rule used by MCDP [2503.12466]. PoCo adopts an analogous score-space formulation for heterogeneous diffusion policies and analytic costs:
\[
s^*(x_t,o)=\sum_{i=1}^{N} w_i\, s_i(x_t,o)-\lambda \nabla_{x_t}R(x_t,o),
\]
with the corresponding policy-level product-of-experts form
\[
\pi^*(a\mid s,o)\propto \prod_{i=1}^{N}\pi_i(a\mid s,o)^{w_i},
\qquad
\log \pi^*(a\mid s,o)=\sum_i w_i \log \pi_i(a\mid s,o)-\lambda R(a,s,o)
\]
[2402.02511].

Theoretical guarantees are developed most explicitly in the 2025 GPC paper [2510.01068]. Its first proposition establishes *single-step improvement via convex combination*: if two score estimators have different bias and noise components, the convex combination
\[
\epsilon(w)=w\epsilon_1+(1-w)\epsilon_2
\]
has mean-squared error
\[
Q(w)=\mathbb{E}\|\epsilon(w)-s^*\|^2
\]
with a minimizer \(w^\star\) satisfying
\[
Q(w^\star)\le \min\{Q(0),Q(1)\},
\]
with strict inequality unless the errors are perfectly aligned. This formalizes the claim that convex averaging can cancel misaligned error components [2510.01068].

The same paper then proves a *score-to-sample stability* result using a Grönwall-type bound. For dynamics
\[
\dot{x}(t)=F(t,x(t),s(t,x(t))),
\]
if the estimated score \(\hat{s}\) has uniformly bounded error \(\kappa(t)\) and the dynamics satisfy Lipschitz conditions, then terminal sampling error is bounded by
\[
\mathbb{E}\|x_{\hat{s}}(T)-x^*(T)\|
\le
\left[\int_0^T \exp\!\Big(2\int_t^T \tilde{L}(\tau)\,d\tau\Big)L_s(t)^2\,dt\right]^{1/2}
\left[\int_0^T \kappa(t)^2\,dt\right]^{1/2}.
\]
A corollary states that if convex score composition reduces integrated score MSE, then it reduces terminal trajectory error as well [2510.01068]. This is the strongest formal result among the cited generative-policy GPC papers.

Other GPC formulations rely on different mathematical objects. “Generalised Policy Improvement with Geometric Policy Composition” composes *geometric horizon models* (GHMs) to evaluate non-Markov geometric switching policies without additional learning [2206.08736]. Given a switching sequence \(\nu=\pi_1 \overset{\alpha}{\rightarrow}\pi_2\overset{\alpha}{\rightarrow}\cdots\overset{\alpha}{\rightarrow}\pi_n\), it shows that careful composition of base-policy GHMs yields an unbiased estimator of \(Q^\nu_\gamma(x,a)\), after which generalized policy improvement (GGPI) produces a new Markov policy satisfying
\[
Q_\gamma^{\pi'}(x,a)\ge \max_{\nu\in \Pi} Q_\gamma^\nu(x,a)
\]
for suffix-closed collections \(\Pi\) of switching policies [2206.08736]. By contrast, “Towards Task-Prioritized Policy Composition” composes maximum-entropy soft \(Q\)-functions with state-action dependent indifference weights,
\[
Q_{1\succeq 2}(s_t,a_t)=Q_1^*(s_t,a_t)+w_{1\succeq 2}(s_t,a_t)\,Q_2^*(s_t,a_t),
\]
where the weight is constructed from the Jacobian and Hessian of the higher-priority \(Q_1^*\), with the goal of preserving high-priority constraints locally [2209.09536].

## 3. Heterogeneity, modularity, and representation choices

A defining feature of GPC is that the constituent experts need not share identical inputs or training data. PoCo addresses “heterogeneous robot learning” by training separate diffusion policies on simulation, human video, and real-robot data, across RGB, depth/point-cloud, tactile, and proprioceptive modalities, provided all policies share the same action space: “6-DoF end-effector velocity: xyz + roll, pitch, yaw” [2402.02511]. Composition is then performed in score-space over trajectories rather than over observations. This allows domain-level fusion of separate policies trained on different modalities and domains [2402.02511].

MCDP makes the same design choice in a more specific setting. It leverages standard image-based DP and point-cloud-based DP3 policies from RoboTwin and composes them at inference time without architectural changes [2503.12466]. Practical composition requires *scheduler alignment*: in the RoboTwin experiments, DP3’s DDIM with 10 steps was replaced by DDPM with 100 steps so that the image and point-cloud experts shared compatible \(\sigma_t\) and update magnitudes [2503.12466]. The broader GPC paper generalizes this normalization principle by converting heterogeneous outputs—\(\epsilon\)-prediction, \(x_0\)-prediction, \(v\)-prediction, or flow-style representations—into a common score representation before composition [2510.01068].

GPC also appears in explicitly symbolic or programmatic forms. “On Policy Reuse: An Expressive Language for Representing and Executing General Policies that Call Other Policies” extends a sketch language with internal finite memory states, indexical features over dynamically loaded object registers, and reusable modules that call other modules with parameter passing [2403.16824]. Here composition is not probabilistic but semantic: a module
\[
\mathrm{mod}=\langle X,Z,M,R,\Phi,m_0,\mathcal{R}\rangle
\]
encapsulates arguments, memory, registers, features, and rules; call rules transfer control to a callee module, and do rules select ground actions directly. Composition is thus realized as nested calls with stack-based scoping, proper returns, and width-bounded search-based execution [2403.16824].

A different representational axis is policy gating. “Neural Policy Composition from Free Energy Minimization” models the composed policy as a convex mixture of primitives,
\[
p(u_k\mid x_{k-1})=\sum_{\alpha=1}^{n} w_k^\alpha\,\pi_\alpha(u_k\mid x_{k-1}),
\]
with the weights \(w_k\) optimized by minimizing a free-energy objective subject to the simplex constraint [2512.04745]. The equilibrium satisfies
\[
w_k^\star=\operatorname{softmax}\!\big(-\epsilon^{-1}\nabla F(w_k^\star)\big),
\]
so task structure, encoded through the generative model and cost, determines the gate rather than the gate being fixed heuristically [2512.04745]. This suggests a general distinction within GPC between *composing distributions directly* and *learning or computing a gating distribution over reusable primitives*.

## 4. Empirical evidence across robotics and reinforcement learning

The empirical record for GPC is strongest in robot policy composition. On RoboTwin manipulation tasks, MCDP composes image-based DP and point-cloud-based DP3 and reports several cases where the composed policy exceeds both parents. For example, on *Empty Cup Place*, the unimodal experts achieve \(0.42\) and \(0.62\) success, while MCDP reaches \(0.86\) at \(w_{\mathrm{img}}=0.4\); on *Shoe Place*, it improves from \(0.37\) and \(0.36\) to \(0.60\) at \(w_{\mathrm{img}}=0.5\) [2503.12466]. The same study also documents failure regimes: when one expert is much weaker, the composed policy may not exceed the stronger parent, as in *Pick Apple Messy* and *Block Hammer Beat* [2503.12466].

The later GPC paper broadens the evidence base. On Robomimic and PushT, convex score composition yields average improvements such as DP+MP from \(39.19\) and \(29.64\) to \(41.41\), Florence-D+DP to \(66.76\), Florence-F+FP to \(86.39\), and \(\pi0\)+FP to \(88.94\) [2510.01068]. On RoboTwin 2.0, the base policies have average success rates \(0.46\) for DPimg, \(0.65\) for DPpcd, and \(0.40\) for RDT; convex composition raises these to \(0.70\), \(0.53\), and \(0.72\) for the evaluated pairs [2510.01068]. Real-world robotic evaluations on 20 trials per task show GPC improving over both DPimg and DPpcd on *Place Bottles*, *Hang Mug*, *Clean Table*, and *Punch Holes* [2510.01068].

PoCo provides empirical support for heterogeneous-domain and behavior-level composition. In simulation, adding smoothness or workspace costs reduces their respective metrics while slightly reducing success: normal success \(0.70\), +smoothness success \(0.67\) with smoothness reduced from \(0.027\) to \(0.016\), and +workspace success \(0.67\) with workspace reduced from \(0.030\) to \(0.022\) [2402.02511]. In real multi-task composition with classifier-free guidance \(\alpha=2\), PoCo improves average success to \(24/40\) relative to \(20/40\) for multi-task unconditioned and \(21/40\) for multi-task conditioned, though single-task specialists remain strongest at \(26/40\) [2402.02511]. In hammering, domain composition of simulation and real data achieves \(20/20\) across four generalization axes, exceeding the human-only, simulation-only, and real-only policies [2402.02511].

Outside generative robotics, the evidence is more varied but still supportive. In multi-objective RL, “Policy composition in reinforcement learning via multi-objective policy optimization” uses teacher-policy adherence objectives inside MO-MPO, enabling sequential and parallel composition in humanoid and point-mass domains [2308.15470]. The paper reports that teacher policies can markedly speed learning, especially in sparse-reward settings, but also notes that strong adherence can cap final performance if teachers conflict with the optimal task policy [2308.15470]. In geometric policy composition, depth-2 GGPI on sparse-reward MuJoCo Ant outperforms standard GPI, with approximately \(100\%\) success at 100 samples and favorable pairwise comparisons across seeds and episodes [2206.08736]. In object-centric policy refactorization for compositional generalizability, the object-centric GNN student with self-supervised proposals outperforms CNN and RelationNet baselines on Multi-MNIST, BigFish, and FallingDigit under changes in object count and background, though Pacman shows that object-centric GNN is not always the strongest student architecture [2011.00971].

## 5. Constraints, priorities, and security-oriented composition

Not all GPC work is aimed at performance improvement alone. Several formulations focus on safety, ordering, or hard constraints. In prioritized reinforcement learning, the objective is explicitly asymmetric: task \(r_1\) has higher priority than \(r_2\), and the compound reward is
\[
r_{1\succeq 2}(s_t,a_t)=r_1(s_t,a_t)+w_{1\succeq 2}(s_t,a_t)\,r_2(s_t,a_t),
\]
with \(w_{1\succeq 2}(s_t,a_t)\in[0,1]\) suppressing the lower-priority objective whenever it would considerably worsen the higher-priority return [2209.09536]. The paper interprets this as an RL analogue of null-space control and introduces the “indifference-space” of a policy to identify actions that do not materially degrade the high-priority task [2209.09536]. This is a distinctly non-symmetric conception of GPC: not consensus among peers, but filtered insertion of lower-priority objectives into the admissible slack of a dominant task.

PoCo adds another constrained composition mechanism by incorporating analytic energy terms such as smoothness and workspace safety directly into the composed score [2402.02511]. This permits *behavior-level composition* in addition to domain- and task-level composition. Because \(-\lambda \nabla R\) is added in score-space, the method can shape the policy at inference time without retraining. This suggests a general GPC pattern in which learned experts and analytic constraints coexist as additive terms in a shared representation.

A more recent non-robotic extension appears in “Securing Multi-Tool AI Agent Chains With Dynamic, Real-Time Compositional Policies” [2607.03423]. Here GPC is defined over per-tool security policies for tool-using agents rather than over action distributions. The Dynamic Security Control Compositor (DSCC) computes an effective chain-level policy by combining control bindings, classification levels, prohibitions, zones, and time-to-live via monotone operators such as max, intersection, and min. The paper proves a monotonicity invariant: if a tool chain \(T\) is extended to \(T'\), then the composed policy can only become more restrictive, never weaker [2607.03423]. Runtime taint tracking then further tightens the effective policy as more sensitive data are touched. This is a different domain from robot control, but it retains the core GPC idea of composing local policies into a chain-level policy with explicit semantics and guarantees.

## 6. Limitations, failure modes, and open directions

Across the literature, GPC methods inherit limitations from the representations they compose. Distribution-level score composition requires calibration and alignment. MCDP identifies *expert miscalibration*—different \(\sigma_t\) schedules or score scales—as a source of dominance or cancellation, and notes that conflicting signals and sensitivity to manually tuned weights can degrade performance [2503.12466]. The broader GPC paper similarly notes that gains saturate when parent errors are nearly identical, while poor parents can harm performance if assigned large weights [2510.01068]. Superposition operators such as logical AND and logical OR can produce larger gains in some settings, but they require per-step reweighting and are less directly applicable to flow-based models [2510.01068].

PoCo emphasizes that heterogeneous pooling of all data into one shared model can perform poorly despite larger model size; in its ablation, the data-pooling baseline is reported at “\(-75\%\) success vs composed policies” [2402.02511]. The same paper records sensitivity to domain-composition weights and failure modes under conflicting distributions, such as bad point-cloud segmentation against RGB priors [2402.02511]. In multi-objective teacher composition, teachers can act as beneficial priors early in learning yet impose *performance ceilings* if their behavior is suboptimal for the final task, so annealing or state-dependent teacher weighting becomes necessary [2308.15470].

Prioritized and geometric formulations have their own restrictions. Task-prioritized composition depends on differentiability of the higher-priority \(Q\)-function with respect to actions and can become expensive in high-dimensional action spaces because Jacobians and Hessians are required [2209.09536]. Geometric policy composition requires fixed switching probability \(\alpha\), suffix-closed candidate sets for the main guarantee, and sufficiently accurate GHMs; deeper composition expands the candidate set combinatorially and increases computation [2206.08736]. GateMod assumes convexity of the free-energy objective in the gating weights, entropic regularization \(\epsilon>0\), and time-scale separation in its neural implementation [2512.04745].

Several future directions recur across papers. The generative robotics literature repeatedly points to automatic or online weight selection, uncertainty-aware weighting, calibration estimation, and stronger theory for stability under conflicting experts [2503.12466][2510.01068]. PoCo suggests distilling composed policies into faster architectures to address diffusion-sampling cost, and extending composition to more temporally mismatched domains [2402.02511]. GateMod proposes broader regularizers and gating rules beyond softmax and integration with learning of transition kernels or primitives [2512.04745]. DSCC identifies non-monotonic constraints and safe exception mechanisms as open problems for compositional security [2607.03423]. Taken together, these directions suggest that GPC remains an active research area whose central open problem is not merely how to combine policies, but how to do so with calibration, guarantees, and manageable compute across increasingly heterogeneous experts and deployment constraints.

Source: https://www.emergentmind.com/topics/general-policy-composition-gpc