Omni-Vision Diffusion Policy (OmniD)

• Omni-Vision Diffusion Policy (OmniD) is a diffusion-based framework that integrates diverse visual inputs to enhance robot manipulation tasks.
• One formulation employs inference-time modality composition, combining pre-trained unimodal policies via weighted score aggregation to improve generalization.
• The BEV-based variant fuses multi-view RGB images into a unified 3D bird’s-eye view, achieving significant gains in both simulation and real-world experiments.

Searching arXiv for the cited OmniD papers and closely related diffusion-policy work. {"query":"(Mao et al., 16 Aug 2025) OmniD Generalizable Robot Manipulation Policy via Image-Based BEV Representation","max_results":5} {"query":"(Cao et al., 16 Mar 2025) Modality-Composable Diffusion Policy via Inference-Time Distribution-level Composition","max_results":5} {"query":"(Mao et al., 16 Aug 2025)","max_results":10} {"query":"Diffusion Policy robotics action diffusion arXiv", "max_results": 5} Omni-Vision Diffusion Policy (OmniD) denotes a 2025 line of diffusion-policy research aimed at improving visuomotor generalization by integrating richer visual evidence than conventional single-modality policies. In the literature represented by "Modality-Composable Diffusion Policy via Inference-Time Distribution-level Composition" (Cao et al., 16 Mar 2025) and "OmniD: Generalizable Robot Manipulation Policy via Image-Based BEV Representation" (Mao et al., 16 Aug 2025), the term has two closely related but technically distinct meanings. One usage treats OmniD as an extension of inference-time composition over multiple modality-specific diffusion policies; the other names a concrete multi-view RGB framework that synthesizes image observations into a unified bird’s-eye view (BEV) representation before conditional action diffusion. Both formulations target the same failure modes—overfitting to narrow observation regimes and weak generalization under distribution shift—but they intervene at different levels of the policy stack.

1. Terminological scope and problem setting

The modality-composition paper introduces a Modality-Composable Diffusion Policy (MCDP) and presents OmniD as the natural extension of that paradigm to “any number of visual streams—RGB, depth, infrared, optical flow, even learned latent features—alongside proprioceptive or tactile DPs,” composed at inference time by aggregating score estimates with weights (Cao et al., 16 Mar 2025). The BEV paper, by contrast, uses OmniD as the name of a specific multi-view fusion framework in which multi-view RGB images and proprioceptive state are mapped into a unified BEV feature and then fed to a diffusion policy (Mao et al., 16 Aug 2025).

Usage in the literature	Core input structure	Fusion mechanism
OmniD as an extension of MCDP	Multiple pre-trained unimodal DPs conditioned on different modalities	Weighted score or noise composition at inference time
OmniD as BEV-based policy architecture	Multi-view RGB images and robot proprioceptive state	Deformable-attention OFG into BEV, followed by conditional diffusion

This dual usage matters because the two formulations answer different questions. The MCDP line asks how existing unimodal diffusion policies can be reused without additional training. The BEV-based OmniD asks how a single policy can be trained to suppress view-specific noise and improve multi-view 3D scene understanding. A common source of confusion is to treat “OmniD” as a single canonical architecture; the 2025 literature instead supports reading it as a family resemblance centered on omni-visual integration rather than a unique implementation.

2. Diffusion-policy foundations

The compositional formulation inherits the score-based interpretation of Diffusion Policy (DP). Action trajectories $\tau$ are treated as samples from a learned diffusion model; with noisy action variable $x_t \in \mathbb{R}^d$ at diffusion timestep $t$, the policy score is defined as

$$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$

Inference then uses a discretized Langevin-style sampler, with the noise-prediction network $\epsilon_\theta(x_t,t)$ related to the score by

$$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$

and training is performed by denoising score matching to recover the true score $\nabla_x \log p(x_t)$ (Cao et al., 16 Mar 2025).

The BEV-based OmniD instantiates the same general family of action-diffusion policies, but formulates it explicitly as a conditional Gaussian diffusion process over an action sequence $a_0 \in \mathbb{R}^m$. Its forward process is

$$a_t = \sqrt{\bar\alpha_t}\,a_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$

and a U-Net or MLP $g_\theta(a_t,t,f_{\mathrm{bev}},e_s)$ is trained to predict $x_t \in \mathbb{R}^d$0 using the objective

$x_t \in \mathbb{R}^d$1

Sampling starts from $x_t \in \mathbb{R}^d$2 and iteratively denoises to $x_t \in \mathbb{R}^d$3, which is mapped directly to robot commands such as joint-space targets (Mao et al., 16 Aug 2025).

Taken together, these formulations show that OmniD is not a departure from diffusion-policy machinery. Its novelty lies in how conditioning information is constructed or combined: either by score-level composition across modality-specific policies, or by building a stronger fused visual representation before denoising.

3. Inference-time distribution-level composition

The central mechanism of MCDP is product-of-experts-style trajectory composition. Given $x_t \in \mathbb{R}^d$4 pre-trained unimodal diffusion policies with conditional densities $x_t \in \mathbb{R}^d$5, the composed policy is defined up to proportionality by

$x_t \in \mathbb{R}^d$6

Taking logarithms and gradients yields additive score composition. At diffusion time $x_t \in \mathbb{R}^d$7,

$x_t \in \mathbb{R}^d$8

and the equivalent noise-prediction form is

$x_t \in \mathbb{R}^d$9

Sampling proceeds by initializing $t$0, evaluating each modality-specific policy at each timestep, averaging their noise estimates using the weight vector $t$1, and updating the trajectory through the diffusion dynamics (Cao et al., 16 Mar 2025).

The operational significance is that the policy is assembled entirely at inference time. The paper explicitly frames this as avoiding the “substantial computational and data-related costs” of training a generalized multimodal DP, while reusing multiple pre-trained DPs based on individual visual modalities (Cao et al., 16 Mar 2025). This suggests a modular route to multimodal control: new sensing streams can be incorporated by adding a compatible diffusion-policy expert rather than retraining a unified model from scratch.

The same paper then proposes OmniD as the direct generalization of this mechanism. In that extension,

$t$2

with the intended scope including cross-domain composition between simulation and real-world policies, cross-embodiment composition through a shared latent action space, and dynamic weighting via a small gating network that selects modality importance online (Cao et al., 16 Mar 2025). These are presented as potential extensions rather than completed empirical results.

4. Image-based BEV OmniD architecture

The BEV-based OmniD addresses a different bottleneck: existing methods “have difficulty fusing multi-view information to generate an effective 3D representation,” and visuomotor policies can overfit to fixed camera positions and backgrounds (Mao et al., 16 Aug 2025). Its input consists of multi-view RGB images $t$3—in practice $t$4 cameras—and robot proprioceptive state $t$5. Each image is encoded by a shared ResNet-18 into feature maps $t$6, while the state is mapped through an MLP to an embedding $t$7 (Mao et al., 16 Aug 2025).

The workspace is discretized as a 3D voxel grid. The reported bounds are

$t$8

with voxel sizes $t$9, $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$0, and $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$1. This yields $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$2 BEV queries $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$3, each associated with a 3D reference point $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$4. Each query point is projected into each camera view using camera intrinsics and extrinsics $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$5, with normalization by predicted or nominal depth $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$6 (Mao et al., 16 Aug 2025).

Fusion is handled by the deformable attention-based Omni-Feature Generator (OFG). For each query $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$7, an offset predictor outputs $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$8 image-space offsets per view; with $$s_\theta(x_t,t) \coloneqq \nabla_{x_t}\log p_\theta(x_t).$$9, the sampling locations are defined around the projected point. Attention weights are computed from query and sampled features, softmaxed over view-offset pairs, and used to form the fused feature

$\epsilon_\theta(x_t,t)$0

The architecture uses input query dimension $\epsilon_\theta(x_t,t)$1, an offset MLP with one linear layer $\epsilon_\theta(x_t,t)$2 per view, a weight MLP with one linear layer $\epsilon_\theta(x_t,t)$3, and three stacked OFG blocks with residual connections, producing an $\epsilon_\theta(x_t,t)$4 3D feature tensor (Mao et al., 16 Aug 2025).

After channel pooling and flattening to obtain $\epsilon_\theta(x_t,t)$5, the model concatenates this feature with the proprioceptive embedding $\epsilon_\theta(x_t,t)$6 and feeds the result to the diffusion policy. The stated role of OFG is to “selectively abstract task-relevant features while suppressing view-specific noise and background distractions” (Mao et al., 16 Aug 2025). In effect, visual fusion occurs before denoising rather than through policy-level composition.

5. Training protocols and empirical results

The BEV-based OmniD is trained on both simulation and real-world data. The simulation dataset contains 3,000 expert episodes over six tasks in VLABench. The real-world dataset contains 20,481 UR5e tele-operation trajectories across 24 tasks using a 5-camera rig. Reported hyperparameters include an observation horizon of 16 steps, batch size 16, total training steps 100k, image resolution $\epsilon_\theta(x_t,t)$7, AdamW with learning rate $\epsilon_\theta(x_t,t)$8, cosine decay, 500-step warmup, and random seeds $\epsilon_\theta(x_t,t)$9. Inference uses $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$0 diffusion steps under standard DDPM, with an approximately 20–40 ms per-step runtime, or fewer steps via DDIM, where 10-step DDIM reduces total time to approximately 200 ms (Mao et al., 16 Aug 2025).

Its evaluation reports strong gains in both simulation and real-world settings. In simulation, Table 1 gives in-distribution success of DP: 82.0 versus OmniD: 91.0. Under out-of-distribution object position shifts in a 10–20 cm annulus, DP achieves 0% and OmniD 18%. Under background OOD with four textures, OmniD reaches 90% versus 18% for the best baseline. In few-shot fine-tuning with 10 trajectories from a new camera A, OmniD reaches 88% intra-task and 80% cross-task, versus baseline performance of at most 2%. In the real-world “pickup pumpkin” setting, BCDE in-distribution performance is DP: 24% versus OmniD: 84%; OOD backgrounds give DP: 0% versus OmniD: 24% and 8%; few-shot adaptation with 10 A-view demonstrations gives DP: 0% versus OmniD: 60% (Mao et al., 16 Aug 2025). The paper summarizes these as average improvements of approximately 11% in-distribution, 17% out-of-distribution, and 84% in few-shot experiments.

The MCDP paper evaluates two unimodal policies, DP_img (RGB) and DP_pcd (point cloud), on the RoboTwin dual-arm benchmark of Mu et al. 2024. It sweeps the image weight $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$1 with $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$2. Reported success rates include: Empty Cup Place, DP_img 0.42, DP_pcd 0.62, and MCDP 0.86 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$3; Dual Bottles Pick (Hard), 0.49, 0.64, and 0.71 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$4; Shoe Place, 0.37, 0.36, and 0.60 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$5; Dual Shoes Place, 0.08, 0.23, and 0.20 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$6; Pick Apple Messy, 0.05, 0.26, and 0.15 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$7; Dual Bottles Pick (Easy), 0.77, 0.36, and 0.85 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$8; and Block Hammer Beat, 0.00, 0.76, and 0.61 at $$s_\theta(x_t,t) = -\frac{1}{\sigma_t}\epsilon_\theta(x_t,t),$$9 (Cao et al., 16 Mar 2025).

The reported findings are explicit. When both unimodal DPs are reasonably accurate, defined in the paper as greater than 30%, MCDP can exceed both by choosing an intermediate weight. If one DP is weak, MCDP cannot surpass the stronger unimodal policy. Maximum gains occur by assigning larger weight to the better-performing modality. Qualitative visualizations show MCDP correcting directional drift from the image DP using geometric cues from the point-cloud DP, leading to robust grasps (Cao et al., 16 Mar 2025).

6. Interpretation, misconceptions, and open directions

The two OmniD formulations imply two non-identical routes to generalizable visuomotor diffusion. One route is compositional: preserve separate modality-specific experts and combine their distributions during sampling. The other is representational: learn a unified BEV latent from multi-view RGB observations and condition a single diffusion policy on that representation. This suggests that “omni-vision” can describe either breadth of sensing modalities or breadth of geometric view integration, depending on the paper.

Several misconceptions are directly contradicted by the reported results. First, more modalities do not automatically dominate the best constituent policy: the MCDP experiments explicitly state that if one DP is weak, composition does not surpass the stronger unimodal model (Cao et al., 16 Mar 2025). Second, OmniD in the BEV sense is not a generic all-modality framework; its implemented inputs are multi-view RGB images and proprioceptive state, even though the broader OmniD concept in the compositional paper includes depth, infrared, optical flow, learned latent features, and tactile DPs as possible future components (Mao et al., 16 Aug 2025). Third, the two papers solve different generalization problems. The BEV system targets fixed-camera and background overfitting through view-invariant 3D fusion, whereas the compositional system targets multimodal reuse without additional training.

Open directions are already articulated in the source material. The compositional line proposes cross-domain composition between simulation and real-world policies, cross-embodiment composition via a shared latent action representation, and dynamic weighting through an online gating network (Cao et al., 16 Mar 2025). The BEV line provides ablation evidence that OFG and multi-view fusion are central to its gains: on Select Apple with BCDE cameras, removing OFG reduces performance from 96.0% to 84.3%, using single-view A only reduces it from 96.0% to 84.0%, and replacing ResNet-18 with ResNet101 increases it by 1.3 points to 97.3% (Mao et al., 16 Aug 2025). A plausible implication is that future OmniD variants may combine both strategies—stronger multi-view 3D feature construction within each expert and score-level composition across heterogeneous experts—but that synthesis remains prospective rather than demonstrated in the reported experiments.