---
title: Omni-Vision Diffusion Policy (OmniD)
url: https://www.emergentmind.com/topics/omni-vision-diffusion-policy-omnid
type: topic
---

# Omni-Vision Diffusion Policy (OmniD)

Searching arXiv for the cited OmniD papers and closely related diffusion-policy work.
{"query":"arXiv:2508.11898 OmniD Generalizable Robot Manipulation Policy via Image-Based BEV Representation","max_results":5}
{"query":"arXiv:2503.12466 Modality-Composable Diffusion Policy via Inference-Time Distribution-level Composition","max_results":5}
{"query":"2508.11898","max_results":10}
{"query":"Diffusion Policy robotics action diffusion arXiv", "max_results": 5}
Omni-Vision Diffusion Policy (OmniD) denotes a 2025 line of diffusion-policy research aimed at improving visuomotor generalization by integrating richer visual evidence than conventional single-modality policies. In the literature represented by "Modality-Composable Diffusion Policy via Inference-Time Distribution-level Composition" [2503.12466] and "OmniD: Generalizable Robot Manipulation Policy via Image-Based BEV Representation" [2508.11898], the term has two closely related but technically distinct meanings. One usage treats OmniD as an extension of inference-time composition over multiple modality-specific diffusion policies; the other names a concrete multi-view RGB framework that synthesizes image observations into a unified bird’s-eye view (BEV) representation before conditional action diffusion. Both formulations target the same failure modes—overfitting to narrow observation regimes and weak generalization under distribution shift—but they intervene at different levels of the policy stack.

## 1. Terminological scope and problem setting

The modality-composition paper introduces a Modality-Composable Diffusion Policy (MCDP) and presents OmniD as the natural extension of that paradigm to “any number of visual streams—RGB, depth, infrared, optical flow, even learned latent features—alongside proprioceptive or tactile DPs,” composed at inference time by aggregating score estimates with weights [2503.12466]. The BEV paper, by contrast, uses OmniD as the name of a specific multi-view fusion framework in which multi-view RGB images and proprioceptive state are mapped into a unified BEV feature and then fed to a diffusion policy [2508.11898].

| Usage in the literature | Core input structure | Fusion mechanism |
|---|---|---|
| OmniD as an extension of MCDP | Multiple pre-trained unimodal DPs conditioned on different modalities | Weighted score or noise composition at inference time |
| OmniD as BEV-based policy architecture | Multi-view RGB images and robot proprioceptive state | Deformable-attention OFG into BEV, followed by conditional diffusion |

This dual usage matters because the two formulations answer different questions. The MCDP line asks how existing unimodal diffusion policies can be reused without additional training. The BEV-based OmniD asks how a single policy can be trained to suppress view-specific noise and improve multi-view 3D scene understanding. A common source of confusion is to treat “OmniD” as a single canonical architecture; the 2025 literature instead supports reading it as a family resemblance centered on omni-visual integration rather than a unique implementation.

## 2. Diffusion-policy foundations

The compositional formulation inherits the score-based interpretation of Diffusion Policy (DP). Action trajectories $\tau$ are treated as samples from a learned diffusion model; with noisy action variable $x_t \in \mathbb{R}^d$ at diffusion timestep $t$, the policy score is defined as
$$
s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).
$$
Inference then uses a discretized Langevin-style sampler, with the noise-prediction network $\epsilon_\theta(x_t,t)$ related to the score by
$$
s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),
$$
and training is performed by denoising score matching to recover the true score $\nabla_x \log p(x_t)$ [2503.12466].

The BEV-based OmniD instantiates the same general family of action-diffusion policies, but formulates it explicitly as a conditional Gaussian diffusion process over an action sequence $a_0 \in \mathbb{R}^m$. Its forward process is
$$
a_t = \sqrt{\bar\alpha_t}\,a_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),
$$
and a U-Net or MLP $g_\theta(a_t,t,f_{\mathrm{bev}},e_s)$ is trained to predict $\epsilon$ using the objective
$$
\mathcal L
=
\mathbb E_{a_0,\epsilon,t}
\Bigl\|
\epsilon -
g_\theta\bigl(a_t,t \mid f_{\mathrm{bev}},e_s\bigr)
\Bigr\|^2.
$$
Sampling starts from $a_T\sim\mathcal N(0,I)$ and iteratively denoises to $\hat a_0$, which is mapped directly to robot commands such as joint-space targets [2508.11898].

Taken together, these formulations show that OmniD is not a departure from diffusion-policy machinery. Its novelty lies in how conditioning information is constructed or combined: either by score-level composition across modality-specific policies, or by building a stronger fused visual representation before denoising.

## 3. Inference-time distribution-level composition

The central mechanism of MCDP is product-of-experts-style trajectory composition. Given $M$ pre-trained unimodal diffusion policies with conditional densities $p_{\theta_m}(\tau \mid M_m)$, the composed policy is defined up to proportionality by
$$
p_{\mathrm{composed}}(\tau) \propto \prod_{m=1}^M p_{\theta_m}(\tau\mid M_m).
$$
Taking logarithms and gradients yields additive score composition. At diffusion time $t$,
$$
s_{\mathrm{composed}}(x_t,t)
=
\sum_{m=1}^M w_m\, s_{\theta_m}\bigl(x_t^{(m)},t\bigr),
\qquad
\sum_{m=1}^M w_m = 1,\; w_m \ge 0,
$$
and the equivalent noise-prediction form is
$$
\epsilon_{\mathrm{composed}}(x_t,t)
=
\sum_{m=1}^M w_m\,\epsilon_{\theta_m}(x_t^{(m)},t).
$$
Sampling proceeds by initializing $x_T \sim \mathcal N(0,I)$, evaluating each modality-specific policy at each timestep, averaging their noise estimates using the weight vector $w$, and updating the trajectory through the diffusion dynamics [2503.12466].

The operational significance is that the policy is assembled entirely at inference time. The paper explicitly frames this as avoiding the “substantial computational and data-related costs” of training a generalized multimodal DP, while reusing multiple pre-trained DPs based on individual visual modalities [2503.12466]. This suggests a modular route to multimodal control: new sensing streams can be incorporated by adding a compatible diffusion-policy expert rather than retraining a unified model from scratch.

The same paper then proposes OmniD as the direct generalization of this mechanism. In that extension,
$$
s_{\mathrm{OmniD}}(x_t,t) = \sum_{m=1}^M w_m\, s_{\theta_m}(x_t^{(m)},t),
$$
with the intended scope including cross-domain composition between simulation and real-world policies, cross-embodiment composition through a shared latent action space, and dynamic weighting via a small gating network that selects modality importance online [2503.12466]. These are presented as potential extensions rather than completed empirical results.

## 4. Image-based BEV OmniD architecture

The BEV-based OmniD addresses a different bottleneck: existing methods “have difficulty fusing multi-view information to generate an effective 3D representation,” and visuomotor policies can overfit to fixed camera positions and backgrounds [2508.11898]. Its input consists of multi-view RGB images $\{I_j\}_{j=1}^M$—in practice $M=5$ cameras—and robot proprioceptive state $s\in\mathbb{R}^p$. Each image is encoded by a shared ResNet-18 into feature maps $F_j\in\mathbb{R}^{C\times H'\times W'}$, while the state is mapped through an MLP to an embedding $e_s\in\mathbb{R}^d$ [2508.11898].

The workspace is discretized as a 3D voxel grid. The reported bounds are
$$
(X_{\min},X_{\max})=(0,1.152),\quad
(Y_{\min},Y_{\max})=(-0.64,0.64),\quad
(Z_{\min},Z_{\max})=(0,0.768),
$$
with voxel sizes $\Delta X=0.018$, $\Delta Y=0.08$, and $\Delta Z=0.012$. This yields $N=N_xN_yN_z$ BEV queries $\{q_i\}_{i=1}^N$, each associated with a 3D reference point $q'_i \in \mathbb{R}^3$. Each query point is projected into each camera view using camera intrinsics and extrinsics $(K_j,[R_j\mid t_j])$, with normalization by predicted or nominal depth $z_{ij}$ [2508.11898].

Fusion is handled by the deformable attention-based Omni-Feature Generator (OFG). For each query $q_i$, an offset predictor outputs $K$ image-space offsets per view; with $K=4$, the sampling locations are defined around the projected point. Attention weights are computed from query and sampled features, softmaxed over view-offset pairs, and used to form the fused feature
$$
f_i = \sum_{j=1}^{M}\sum_{k=1}^{K} w_{ijk}\,\mathrm{Bilinear}(F_j,s_{ijk}) \in \mathbb{R}^d.
$$
The architecture uses input query dimension $d=256$, an offset MLP with one linear layer $(256 \rightarrow 2K)$ per view, a weight MLP with one linear layer $(512 \rightarrow K)$, and three stacked OFG blocks with residual connections, producing an $N\times d$ 3D feature tensor [2508.11898].

After channel pooling and flattening to obtain $f_{\mathrm{bev}}$, the model concatenates this feature with the proprioceptive embedding $e_s$ and feeds the result to the diffusion policy. The stated role of OFG is to “selectively abstract task-relevant features while suppressing view-specific noise and background distractions” [2508.11898]. In effect, visual fusion occurs before denoising rather than through policy-level composition.

## 5. Training protocols and empirical results

The BEV-based OmniD is trained on both simulation and real-world data. The simulation dataset contains 3,000 expert episodes over six tasks in VLABench. The real-world dataset contains 20,481 UR5e tele-operation trajectories across 24 tasks using a 5-camera rig. Reported hyperparameters include an observation horizon of 16 steps, batch size 16, total training steps 100k, image resolution $480\times480$, AdamW with learning rate $1\times10^{-4}$, cosine decay, 500-step warmup, and random seeds $\{1,2,3,4\}$. Inference uses $T=50$ diffusion steps under standard DDPM, with an approximately 20–40 ms per-step runtime, or fewer steps via DDIM, where 10-step DDIM reduces total time to approximately 200 ms [2508.11898].

Its evaluation reports strong gains in both simulation and real-world settings. In simulation, Table 1 gives in-distribution success of DP: 82.0 versus OmniD: 91.0. Under out-of-distribution object position shifts in a 10–20 cm annulus, DP achieves 0% and OmniD 18%. Under background OOD with four textures, OmniD reaches 90% versus 18% for the best baseline. In few-shot fine-tuning with 10 trajectories from a new camera A, OmniD reaches 88% intra-task and 80% cross-task, versus baseline performance of at most 2%. In the real-world “pickup pumpkin” setting, BCDE in-distribution performance is DP: 24% versus OmniD: 84%; OOD backgrounds give DP: 0% versus OmniD: 24% and 8%; few-shot adaptation with 10 A-view demonstrations gives DP: 0% versus OmniD: 60% [2508.11898]. The paper summarizes these as average improvements of approximately 11% in-distribution, 17% out-of-distribution, and 84% in few-shot experiments.

The MCDP paper evaluates two unimodal policies, DP_img (RGB) and DP_pcd (point cloud), on the RoboTwin dual-arm benchmark of Mu et al. 2024. It sweeps the image weight $w_1 \in \{0.1,\ldots,0.9\}$ with $w_2=1-w_1$. Reported success rates include: Empty Cup Place, DP_img 0.42, DP_pcd 0.62, and MCDP 0.86 at $w_1=0.4$; Dual Bottles Pick (Hard), 0.49, 0.64, and 0.71 at $w_1=0.4$; Shoe Place, 0.37, 0.36, and 0.60 at $w_1=0.4$; Dual Shoes Place, 0.08, 0.23, and 0.20 at $w_1=0.4$; Pick Apple Messy, 0.05, 0.26, and 0.15 at $w_1=0.4$; Dual Bottles Pick (Easy), 0.77, 0.36, and 0.85 at $w_1=0.4$; and Block Hammer Beat, 0.00, 0.76, and 0.61 at $w_1=0.1$ [2503.12466].

The reported findings are explicit. When both unimodal DPs are reasonably accurate, defined in the paper as greater than 30%, MCDP can exceed both by choosing an intermediate weight. If one DP is weak, MCDP cannot surpass the stronger unimodal policy. Maximum gains occur by assigning larger weight to the better-performing modality. Qualitative visualizations show MCDP correcting directional drift from the image DP using geometric cues from the point-cloud DP, leading to robust grasps [2503.12466].

## 6. Interpretation, misconceptions, and open directions

The two OmniD formulations imply two non-identical routes to generalizable visuomotor diffusion. One route is compositional: preserve separate modality-specific experts and combine their distributions during sampling. The other is representational: learn a unified BEV latent from multi-view RGB observations and condition a single diffusion policy on that representation. This suggests that “omni-vision” can describe either breadth of sensing modalities or breadth of geometric view integration, depending on the paper.

Several misconceptions are directly contradicted by the reported results. First, more modalities do not automatically dominate the best constituent policy: the MCDP experiments explicitly state that if one DP is weak, composition does not surpass the stronger unimodal model [2503.12466]. Second, OmniD in the BEV sense is not a generic all-modality framework; its implemented inputs are multi-view RGB images and proprioceptive state, even though the broader OmniD concept in the compositional paper includes depth, infrared, optical flow, learned latent features, and tactile DPs as possible future components [2508.11898]. Third, the two papers solve different generalization problems. The BEV system targets fixed-camera and background overfitting through view-invariant 3D fusion, whereas the compositional system targets multimodal reuse without additional training.

Open directions are already articulated in the source material. The compositional line proposes cross-domain composition between simulation and real-world policies, cross-embodiment composition via a shared latent action representation, and dynamic weighting through an online gating network [2503.12466]. The BEV line provides ablation evidence that OFG and multi-view fusion are central to its gains: on Select Apple with BCDE cameras, removing OFG reduces performance from 96.0% to 84.3%, using single-view A only reduces it from 96.0% to 84.0%, and replacing ResNet-18 with ResNet101 increases it by 1.3 points to 97.3% [2508.11898]. A plausible implication is that future OmniD variants may combine both strategies—stronger multi-view 3D feature construction within each expert and score-level composition across heterogeneous experts—but that synthesis remains prospective rather than demonstrated in the reported experiments.

Source: https://www.emergentmind.com/topics/omni-vision-diffusion-policy-omnid