---
title: Mixture of Frames Policy (MoF)
url: https://www.emergentmind.com/topics/mixture-of-frames-policy-mof
type: topic
---

# Mixture of Frames Policy (MoF)

to=arxiv_search  人人中彩票  天天中彩票官方 񎔊ppjson_stringify code: {"query":"id:2607.11884 OR ti:\"Mixture of Frames Policy: Multi-Frame Action Denoising for Bimanual Mobile Manipulation\"","max_results":5,"sort_by":"relevance"}
to=arxiv_search  天天中彩票官方 ￣亚洲json_stringify code: {"query":"id:2605.12167 OR ti:\"From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation\"","max_results":5,"sort_by":"relevance"}
Mixture of Frames Policy (MoF) is a diffusion policy for visuomotor imitation learning in bimanual mobile manipulation that performs synchronized action denoising across multiple coordinate frames rather than committing to a single predefined action frame. In its canonical formulation, MoF maintains one shared noisy action state, re-expresses that state in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in a canonical frame before taking a standard diffusion update step [2607.11884]. The method is motivated by the claim that robotic manipulation is inherently multi-frame: local motions may be simpler in an end-effector frame, whereas transport, upright-object handling, and whole-body coordination are often simpler in a base-aligned frame. MoF therefore treats action frame choice as part of the denoising problem itself rather than as a fixed preprocessing decision.

## 1. Problem setting and conceptual motivation

The MoF formulation in "Mixture of Frames Policy: Multi-Frame Action Denoising for Bimanual Mobile Manipulation" studies visuomotor imitation learning for bimanual mobile manipulation using diffusion policies [2607.11884]. At each control step, the policy receives an observation
\[
o=(\mathcal I, q),
\]
where \(\mathcal I\) are RGB images and \(q\) is proprioception, and predicts an action chunk
\[
a \in \mathbb R^{\tau \times d}
\]
over horizon \(\tau\). Each action timestep contains target poses for the left and right end-effectors plus gripper or hand commands, and a whole-body controller tracks those targets.

The motivating premise is that manipulation statistics depend strongly on coordinate representation. The paper states that approach and grasp can be simpler in an end-effector frame, because similar relative gripper-object configurations can induce similar motions regardless of global placement. By contrast, transporting an upright object, such as a filled cup, is easier in a base or world-aligned frame, because the relevant invariances are tied to gravity or body orientation. In bimanual mobile manipulation, the useful frame may also change during an episode: local reaching and grasping are often arm-centric, whereas coordinated insertion, transport, and navigation-coupled behavior are often base-centric [2607.11884].

This diagnosis is used to explain the failure mode of single-frame diffusion policies. A standard policy chooses one fixed action frame \(F^\star\) and requires a single denoiser to model the entire action distribution in that frame. The paper argues that this can make the distribution unnecessarily broad, multimodal, or phase-dependent. MoF addresses that representational burden by allowing several frame-specialized denoisers to operate on the same underlying noisy action state.

## 2. Core mechanism and mathematical formulation

MoF is defined as a diffusion policy that performs synchronized denoising across multiple action coordinate frames [2607.11884]. The synchronization point is essential: there is one canonical diffusion state
\[
x_c^k
\]
stored in a canonical frame \(F_c\), and all experts denoise re-expressions of that same state rather than running independent diffusion trajectories.

The key fusion equation is
\[
\hat{\epsilon}_c = \sum_{F_m\in\mathcal{F}} w_m\, {}^{c}\mathbf{T}_{m} \!\left( \epsilon^{\theta}_m \left( {}^{m}\mathcal{T}_{c}(x^k_c), o; k \right) \right). \tag{1}
\]
Here, \(\mathcal F=\{F_m\}\) is the set of task-relevant frames, \({}^{m}\mathcal T_c\) transforms the noisy action state from the canonical frame into expert frame \(m\), \(\epsilon_m^\theta\) is the denoiser specialized to frame \(m\), \({}^{c}\mathbf T_m\) transforms the predicted noise back into the canonical frame, and \(w_m\) are mixture weights satisfying
\[
\sum_m w_m = 1.
\]

The training objective is the fused denoising loss
\[
\mathcal{L}_{\mathrm{mix}}(\theta) = \mathbb{E}_{k,\epsilon_c,o} \left[ \left\| \epsilon_c - \sum_{F_m\in\mathcal{F}} w_m\, {}^{c}\mathbf{T}_{m} \!\left( \epsilon^{\theta}_m \left( {}^{m}\mathcal{T}_{c}(x^k_c),o;k \right) \right) \right\|^2 \right]. \tag{2}
\]
The paper states that \(x_c^k\) is constructed in the same way as standard Diffusion Policy, with \(x_c^0=a\), and that inference uses a DDIM scheduler with the fused canonical prediction \(\hat\epsilon_c\) [2607.11884].

Operationally, one reverse-diffusion step consists of re-expressing \(x_c^k\) in every expert frame, predicting noise in each native frame, mapping all noise predictions back into the canonical frame, fusing them by learned or uniform weights, and then taking one standard DDIM step. This design is contrasted with independent per-frame diffusion plus late fusion, which the paper rejects because different experts may converge to different action modes under multimodal action distributions.

## 3. Action parameterization and exact frame transformations

A central technical requirement is that frame changes must remain exact and differentiable even for intermediate noisy diffusion states, which need not correspond to valid rotations on \(SO(3)\). To satisfy this, the paper introduces a column-based 6D rotation representation within an \(SE(3)\) action parameterization [2607.11884].

For one arm at one action timestep, the canonical action channels are written as
\[
x_c = [p_c, c_c^1, c_c^2, g],
\]
where \(p_c \in \mathbb R^3\) is position, \(c_c^1,c_c^2 \in \mathbb R^3\) are the first two orientation columns, and \(g\) is the gripper command. If \({}^{m}T_c = ({}^{m}R_c, {}^{m}t_c)\) is the rigid transform from canonical frame to expert frame, the action-state operator is
\[
{}^{m}\mathcal T_c(x_c) = {}^{m}A_c x_c + {}^{m}b_c,
\]
with channelwise action
\[
p_c \mapsto {}^{m}R_c p_c + {}^{m}t_c,\qquad
c_c^i \mapsto {}^{m}R_c c_c^i,\qquad
g \mapsto g.
\]
For denoiser outputs
\[
\epsilon_c = [\Delta p_c, \Delta c_c^1, \Delta c_c^2, \Delta g],
\]
the noise transform is linear:
\[
{}^{m}\mathbf T_c(\epsilon_c) = {}^{m}A_c \epsilon_c.
\]

The paper’s claim is that left-multiplication by the frame-change rotation remains valid even if \(c_c^1,c_c^2\) are arbitrary noisy 3-vectors rather than columns of a valid rotation matrix. This avoids projecting noisy states back onto \(SO(3)\) during denoising. A practical implication, explicitly supported by ablation, is that orthogonalizing before transformation is harmful: MoF-MoE drops by \(7.0\), and MoF-Ensemble collapses to \(0.0\), when orthogonalization or projection is inserted before the transform [2607.11884].

In simulation, each arm contributes \(9\) pose channels plus either \(1\) gripper command or \(6\) finger-joint commands, yielding \(20\)-dimensional actions for parallel-jaw settings and \(30\)-dimensional actions for dexterous-hand settings. The frame transforms act on positions and orientation columns and leave gripper or finger channels unchanged.

## 4. Expert frames, architecture, and training pipeline

In the simulation setting, MoF uses four frame experts: base-relative, left, right, and relative-trajectory [2607.11884]. The left and right experts use the corresponding end-effector frames at the most recent observation step. The base-relative expert uses the mobile base frame for rotations and translation directions while re-centering each arm’s translation at its observation-step end-effector position. The relative-trajectory expert represents each arm’s future motion in that arm’s own current end-effector frame, removing both global translation and global orientation. The canonical frame in simulation is base-relative.

The simulation backbone is the standard Diffusion Policy conditional 1D U-Net with channel dimensions \([256,512,1024]\), diffusion-step embedding dimension \(128\), kernel size \(5\), and GroupNorm groups \(8\) [2607.11884]. Visual encoding uses a ResNet-18 backbone pretrained on ImageNet, fine-tuned end-to-end, with spatial-softmax feature aggregation and no sharing across the three cameras. The denoiser is conditioned through FiLM. Images are rendered at \(84\times84\), preprocessed to \(76\times76\), randomly cropped in training, and center cropped at test time.

A notable implementation detail is that all experts share the same visual features, because images are not tied to a specific action frame, whereas proprioceptive inputs are transformed into each expert’s parameterization before being provided to that expert. The router in MoF-MoE is an MLP with hidden dimension \(512\), taking the diffuser-conditioning embedding together with the diffusion-step embedding and outputting a softmax over experts. MoF-Ensemble instead sets
\[
w_m = \frac{1}{|\mathcal F|}.
\]

Training includes an auxiliary per-expert loss,
\[
\mathcal{L}_{\mathrm{aux}}(\theta) = \mathbb{E}_{k,\epsilon_c,o} \left[ \sum_{F_m\in\mathcal{F}} \left\| {}^{m}\mathbf{T}_c(\epsilon_c) - \epsilon^\theta_m({}^{m}\mathcal{T}_c(x_c^k),o;k) \right\|^2 \right],
\]
and the total objective is
\[
\mathcal L = \mathcal L_{\mathrm{mix}} + \lambda_{\mathrm{aux}} \mathcal L_{\mathrm{aux}},
\]
with \(\lambda_{\mathrm{aux}}=1\) in simulation [2607.11884]. The paper reports that removing the auxiliary loss causes an average drop of \(8.0\) on selected ablation tasks.

Simulation uses action horizon \(16\), executes the first \(8\) actions before re-planning, trains with \(50\) DDIM training timesteps and \(16\) inference timesteps, and optimizes with AdamW at learning rate \(1\times10^{-4}\), weight decay \(1\times10^{-6}\), \((\beta_1,\beta_2)=(0.95,0.999)\), cosine schedule, \(500\) warmup steps, batch size \(128\), and \(500\) epochs [2607.11884]. The real-world instantiation is integrated into HoMMI, uses a DiT-based diffusion policy, and employs left, right, and relative-trajectory experts, with the left frame as canonical because the base expert is disabled when base frame data are not recorded.

## 5. Empirical performance and ablation evidence

The simulation evaluation spans nine bimanual manipulation tasks across BiGym and DexMimicGen: Flip Cup, Move 2 Plates, Kitchenware, Flip Sandwich, Dishwasher, Threading, 3 Piece Assembly, Box Cleanup, and Drawer Cleanup [2607.11884]. One of the paper’s core empirical claims is that no single frame is universally best. Average success rates of single-frame policies are reported as Left \(55.6\), Right \(55.0\), Base \(61.0\), Rel Traj \(55.8\), Oracle \(63.8\), and Worst \(48.8\). The oracle-best minus worst gap is therefore \(15\) percentage points on average.

The main comparison averages are MoF-MoE \(66.8\), MoF-Ensemble \(65.9\), Oracle Frame \(63.8\), Single-Frame Ensemble \(55.9\), MoE-DP \(55.1\), and DP \(50.3\) [2607.11884]. Thus MoF-MoE exceeds standard DP by \(16.5\) points, MoE-DP by \(11.7\) points, and Oracle Frame by \(3.0\) points. The paper interprets this as evidence that MoF is not merely automating frame selection but exploiting synchronized multi-frame denoising directly.

The strongest task-level gain highlighted in the paper occurs on Threading, where Oracle Frame reaches \(63.1\) and MoF-MoE reaches \(70.8\), a gain of \(+7.7\). Router analysis on that task shows that the relative-trajectory expert dominates early reaching, while the base expert dominates coordinated insertion later. Across the nine tasks, the amplitude of within-episode router changes correlates positively with MoF-MoE’s advantage over Oracle Frame, with Spearman \(\rho = +0.75\), \(p=0.020\), and \(n=9\). This suggests that MoF helps most when the preferred frame changes over task phases.

Real-world results are reported on two bimanual mobile-manipulation tasks. In the pouring task, trained from \(257\) demonstrations and evaluated over \(20\) rollouts with \(5\) color combinations and \(4\) cup placements, MoF-MoE achieves \(85\%\) success, compared with Left \(15\%\), Right \(70\%\), and Rel Traj \(5\%\) [2607.11884]. In the serving task, trained from \(177\) demonstrations and evaluated over \(20\) rollouts with \(3\) plate colors and varying table distances, MoF-MoE achieves \(70\%\), compared with Left \(55\%\), Right \(0\%\), and Rel Traj \(60\%\).

The ablations further support the architectural choices. Changing the canonical frame has modest effect: using rel-traj as canonical yields \(-2.5\), left yields \(-0.7\), and right yields \(-0.1\). Removing the best frame from the expert set causes an average drop of \(-6.2\). Removing the auxiliary loss causes \(-8.0\). The orthogonalization ablation is especially strong, as noted above, and is presented as direct support for the column-based representation [2607.11884].

## 6. Related formulations, terminology, and scope

Within robotics, MoF occupies a specific niche: it is a mixture over action coordinate frames inside the denoising process, not a generic mixture-of-experts over conditioning features. The paper makes this distinction explicitly by comparing against MoE-DP, which introduces a mixture structure in the conditioning pathway while still denoising in one fixed frame, and showing that MoF substantially outperforms it on average [2607.11884].

A separate but conceptually adjacent line appears in "From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation" [2605.12167]. There, a hypothetical MoF baseline is described as a policy that generates future frames, encodes those frames visually, aggregates them over time or views, and feeds the resulting future visual representation directly to the policy. MoLA is presented as an alternative to that design: it transforms imagined future visual transitions into a mixture of latent actions through modality-aware inverse dynamics models rather than conditioning policy execution directly on predicted frames. This contrast clarifies that the term “Mixture of Frames Policy” has also been used in a broader conceptual sense to denote direct frame-conditioned control, whereas the 2026 MoF formulation concerns multi-frame action denoising in coordinate space rather than future-frame aggregation [2605.12167].

The acronym “MOF” is also ambiguous outside robotics. In cross-modal video retrieval, "Efficient Cross-Modal Video Retrieval with Meta-Optimized Frames" uses MOF to denote Meta-Optimized Frames, a bilevel optimization method that learns compressed frame tensors for retrieval rather than a policy over action frames [2210.08452]. A plausible implication is that “MoF” should be interpreted by domain context: in robotics after [2607.11884], it denotes synchronized multi-frame denoising for continuous action generation; in multimedia retrieval, it can denote learned compressed frame representations.

More broadly, MoF can be situated against generic mixture-policy research. Probabilistic mixture-of-experts for continuous-control RL represent actions as state-dependent mixtures over experts and require special gradient estimators to handle discrete routing [2104.09122]; later work on entropy-regularized actor-critic argues that mixture policies can improve solution quality and entropy robustness when trained with marginalized reparameterization [2605.09157]. This suggests that MoF belongs to a wider family of mixture-based policy parameterizations, but its distinctive contribution is to place the mixture over coordinate-frame denoisers that remain synchronized through a single canonical diffusion trajectory.

The paper identifies several limitations: experiments focus on relatively small diffusion policies rather than large VLA backbones, router supervision comes only through the denoising loss, the candidate frame set is designer-specified, accurate transforms are assumed, and frame uncertainty is not studied [2607.11884]. The stated future directions are scaling to larger VLA models, behavior-aware routing objectives, automatic learning or discovery of useful frames, and robustness to uncertain frame transforms. Within that scope, MoF’s main significance is the claim that action frame is not merely a representational convenience but a first-class component of the denoising problem in robot manipulation.

Source: https://www.emergentmind.com/topics/mixture-of-frames-policy-mof