---
title: Factorized Diffusion Policies in Robotics
url: https://www.emergentmind.com/topics/factorized-diffusion-policies-fdp
type: topic
---

# Factorized Diffusion Policies in Robotics

Factorized Diffusion Policies (FDP) comprise a class of techniques that decompose diffusion-based policies into modular components for improved sample efficiency, robustness, multitask generalization, and adaptability in robot skill learning. FDP frameworks enable either (a) observation modality prioritization—where the influence of distinct sensing modalities can be explicitly controlled—or (b) factorization of the action distribution into a product or mixture of expert diffusion models, each capturing different behavioral sub-modes or subtasks. This modularization can be achieved by architectural partitioning, score aggregation, or via parameter space decomposition. FDP models have demonstrated significant empirical benefits in both low-data regimes and settings characterized by distributional shift or catastrophic forgetting.

## 1. Fundamentals of Diffusion Policies in Robot Learning

Diffusion models are generative frameworks that define a fixed Gaussian noising process ("forward" process) and learn to reverse it via a parameterized denoising kernel ("reverse" process), typically predicting the added noise at each step. In robotic skill imitation, these methodologies have been leveraged to map multi-modal sensory observations (e.g., proprioception, vision, tactile) to target action trajectories. The canonical approach formulates the reverse kernel as
$$
p(a_{t-1}\mid a_t, o^{1},\dots,o^{M}) \approx \mathcal N(a_{t} - w_t\,\varepsilon_\theta(a_t, o^{1},\dots,o^{M}, t), \sigma^2_t\mathbf{I}),
$$
where $\varepsilon_\theta$ is a neural network predicting the conditional noise. In standard diffusion policy implementations, all modalities are concatenated or otherwise fused prior to conditioning (“joint” conditioning). However, this approach yields suboptimal data efficiency when modalities are unequally informative and exposes the policy to robustness failures from spurious correlations or sensor-specific noise [2509.16830].

## 2. Mathematical Formulations and Model Factorizations

FDP methodologies operationalize modularity by separating the generative policy or its conditioning space in a principled manner:

### 2.1 Observation Modality Prioritization

Rather than conditioning the denoising kernel jointly on all modalities, FDP selects $k < M$ prioritized modalities $o^{1:k}$ and treats the remaining $o^{k+1:M}$ as secondary. The policy score $\nabla_{a_t}\log p(a_t|o^{1:M})$ is factorized as
$$
\nabla_{a_t}\log p(a_t|o^{1:M})
= \nabla_{a_t}\log p(a_t|o^{1:k}) + \nabla_{a_t}\log p(o^{k+1:M} | a_t, o^{1:k}),
$$
corresponding to a base score model conditioned on the priority modalities and a residual model capturing the information gain from the secondary ones. The two models are trained sequentially using mean-squared error (MSE) losses on the conditional noise prediction, with the residual model correcting the frozen base [2509.16830].

### 2.2 Modular Task and Behavioral Factorization

For multitask or highly multimodal action distributions, FDP can be instantiated as a composition of $N$ specialized diffusion experts. Each expert $p_i(a|o)$ models a distinct behavioral sub-mode. The overall action distribution is represented as a product-of-experts:
$$
p(a|o) \propto \prod_{i=1}^N p_i(a|o)^{w_i(o)},
$$
with a learned router predicting the convex weights $w_i(o)$. At each diffusion step, the aggregate score is
$$
\nabla_{a^k}\log p(a^k|o) \approx \sum_{i=1}^N w_i(o)\,\varepsilon_{\theta_i}(a^k, o, k),
$$
enabling the policy to exploit regime-specific expert predictions [2512.21898].

### 2.3 Parameter Space Decomposition

An alternative form of factorization involves decomposing model parameters, e.g., via truncated Singular Value Decomposition (SVD) of network weights. In rank-$r$ factorized diffusion policies, each layer weight is split into a low-rank (trainable) and orthogonal (frozen) component, modulating network expressivity and computational cost as training progresses [2502.03822].

## 3. FDP Architectures and Training Procedures

The architectural instantiations of FDP depend on the targeted factorization:

- **Observation-prioritized FDP**: The base network (e.g., UNet or DiT) is conditioned on prioritized modalities only; the residual network uses all modalities and injects corrections via FiLM-style or zero-initialized adapter connections. Training proceeds with the base model first, after which it is frozen and the residual is trained [2509.16830].
- **Expert/Modular FDP**: Each expert is an independent denoising network (typically sharing lower-level encoders) and receives a weight from a separate router MLP per observation. All modules are trained end-to-end via the aggregated noise-prediction loss [2512.21898].
- **Parameter-factorized FDP**: Each layer is decomposed via SVD; the number of trainable singular vectors is adjusted over epochs via a scheduling strategy. This implementation does not alter the computation graph for the forward pass, but reduces backpropagation cost [2502.03822].

### Example FDP Training Loop (Observation Prioritization)

```python
# Pseudocode (from [2509.16830])
for epoch in base_training_epochs:
    sample (a_0, o^{1:k}), t, ε
    compute a_t = √ᾱ_t a_0 + √(1−ᾱ_t) ε
    L_base = ||ε - ε_base(a_t, o^{1:k}, t)||^2
    update ε_base

freeze(ε_base)

for epoch in residual_training_epochs:
    sample (a_0, o^{1:M}), t, ε
    compute a_t = √ᾱ_t a_0 + √(1−ᾱ_t) ε
    L_res = ||ε - ε_base(a_t, o^{1:k}, t) - ε_res(a_t, o^{1:M}, t)||^2
    update ε_res
```

## 4. Empirical Results and Evaluation

FDP methods have been evaluated on diverse robotic manipulation benchmarks, including RLBench (vision + proprioception), Adroit (hand, state + prop), Robomimic (env-state + prop), M3L insertion (vision + tactile), and real-world tasks (Close Drawer, Put Block in Bowl, etc.).

### Summary of Key Empirical Findings

| Setting                             | Metric                                   | FDP Result                                        | Baseline              | Gain                    |
|--------------------------------------|------------------------------------------|---------------------------------------------------|-----------------------|-------------------------|
| RLBench, 10 demos                   | Success rate                             | 44% (prop>vision)                                 | 29% (joint DiT)       | +15 pt                  |
| M3L insertion, 100 demos             | Success rate                             | 48–50% (vision>tactile)                           | 22%                   | +26 pt                  |
| RLBench distractor shift             | Success rate under distribution shift    | ≈70% (prop>vision)                                | ≈30% (joint DiT)      | +40 pt                  |
| Real-robot (occlusion, distractor)   | Success rate (Fold Towel, Put In Bowl…)  | ~60% (prop>vision)                                | ~5–15% (joint DiT)    | +40 pt                  |
| MetaWorld multitask [2512.21898]     | Avg. success over 6 tasks                | 74.8% (FDP)                                       | 70.8% (DP), 69.8% SDP | +4–5 pt                 |
| Adaptation/fine-tuning (few demos)   | Retention after adaptation               | >90% with 27% params                              | Full-finetune baseline| Comparable              |
| Parameter factorization [2502.03822] | Training time, simulated tasks           | 7–10% reduction, no SR drop (CT: 4.03h vs. 4.35h) | -                     | 7–10% faster            |
| Parameter factorization [2502.03822] | Training time, real tasks                | Up to 18% speedup online                          | -                     | -                       |

Additional findings:

- **Robustness:** Factorized observation policies drastically outperform joint models under vision corruptions (distractors, occlusion).
- **Low-data efficiency:** Prioritizing informative modalities yields absolute gains up to 20 pt over baselines in sparse demonstration settings.
- **Multitask transfer and forgetting:** Modular experts yield efficient adaptation to new tasks with no catastrophic loss on old skills, especially when combined with a small replay buffer.
- **Parameter factorization:** Rank scheduling (e.g., sigmoid, $\tau=0.5$) enables faster batch times (up to $\sim$20%) with minimal or no loss in performance on a range of simulated and real setups [2502.03822].

## 5. Analysis, Implications, and Limitations

FDPs introduce a task/setting-dependent flexibility:

- **Sample efficiency** is improved by enforcing a strong prior over primary modalities, reducing overfitting to redundant or noisy sensory features [2509.16830].
- **Robustness** arises since the residual corrector cannot fully override the base predictions, limiting the failure modes induced by sensor noise or novel distractors.
- **Efficient multitask learning**: Modular FDPs (expert composition) facilitate specialization, allowing individual modules to adapt or be replaced without disrupting baseline competencies, thereby addressing catastrophic forgetting [2512.21898].
- **Parameter management**: Dynamic low-rank scheduling decreases computational demands, promoting practical training in resource-constrained or online-interactive (e.g., DAgger) settings [2502.03822].

Limitations cited across FDP literature include:

- **Hyperparameter burden**: Modality ranking, number/ordering of experts, and rank schedule require tuning; automated strategies are an open area.
- **Static prioritization**: Current observation-priority schemes are trajectory-invariant; dynamic or state-dependent prioritization may provide further gains.
- **Scope**: Some approaches (notably rank-based) have been explored primarily in imitation learning, not fully on-policy RL.
- **Component specialization**: While component-wise specialization is demonstrated empirically, systematic analysis of functional roles remains an open area.

## 6. Connections to Related Modularization Strategies

FDPs relate fundamentally to modular and compositional learning, mixture-of-experts (MoE) models, and conditional or product-of-experts score aggregation in generative modeling. FDP’s soft aggregation of per-expert scores mitigates MoE routing instabilities and encourages skill specialization [2512.21898]. The residual correction paradigm inherits from classifier guidance/classifier-free guidance in diffusion models (see Dhariwal & Nichol 2021). Parameter factorization is conceptually aligned with adaptive pruning, low-rank neural adaptation, and efficient subspace optimization in large-scale models.

A plausible implication is that FDP-style decompositions can serve as a substrate for future advances in data-efficient, robust, and scalable policy learning in diverse real-world robotic domains [2509.16830; 2512.21898; 2502.03822].

## 7. Outlook and Future Directions

Prominent open directions for FDP include:

- **Automated prioritization and dynamic routing** over observation modalities and experts, enabling context-sensitive adaptation.
- **Integration with Vision-Language-Action (VLA) models** for safe finetuning and generalization to previously unseen input modalities [2509.16830].
- **Heterogeneous architectures** for expert modules (e.g., combining UNet and transformer backbones).
- **Systematic removal or ablation of expert components** to elucidate distributed skill encoding [2512.21898].
- **Application to lifelong and continual learning** scenarios with ongoing task acquisition, seeking robust knowledge retention beyond replay buffer approaches.

Factorized Diffusion Policies thus offer a unified perspective on compositional generative policy design, delivering quantifiable gains in efficiency, robustness, and flexibility across robot learning challenges.

Source: https://www.emergentmind.com/topics/factorized-diffusion-policies-fdp