Whole-Body VisuoMotor Attention Policy

• WB-VIMA is a learned control architecture that integrates distributed camera views and joint angles to enable robust whole-body manipulation.
• It employs a diffusion-based transformer model to predict joint action sequences, enhancing accuracy in complex, cluttered environments.
• Robustness is achieved through blink training, which simulates sensor dropout and ensures reliable performance under visual failures.

Whole-Body VisuoMotor Attention (WB-VIMA) policy is a learned control architecture enabling robots to achieve whole-body dexterity by fusing comprehensive visual and proprioceptive input for closed-loop manipulation. Developed in the context of the RoboPanoptes system, WB-VIMA aggregates information from a distributed camera array covering the robot’s body surface alongside proprioceptive joint angles to predict complex whole-body manipulation actions via diffusion-based imitation learning. Architectural integration of multi-view, multi-perspective vision, geometric encoding, and robust sensor dropout handling yields resilience, adaptability, and manipulative proficiency in diverse, cluttered, and constrained environments (Xu et al., 9 Jan 2025).

1. Input Modalities and Preprocessing

The WB-VIMA policy is designed to integrate distributed multimodal sensory data:

• RGB Imagery: 21 synchronized body-mounted USB cameras (640×480, resized to 224×224, with random color-jitter augmentation).
• Proprioceptive Input: 9-dimensional joint angle vector $j\in\mathbb{R}^{9}$.
• Camera Geometrics: 3D position $r_{t,i}\in\mathbb{R}^{3}$ and orientation $o_{t,i}\in\mathbb{R}^6$ (the first two columns of the rotation matrix for the i-th camera at time t).

Each camera image $I_{t,i}$ is embedded via a frozen CLIP ViT-B/16 backbone to obtain a 768-dimensional class token $f_{t,i}$, which is then projected to 384 dimensions: $e_{t,i} = W_v f_{t,i} \in \mathbb{R}^{384}$. The geometric pose is linearly embedded as $ppos_{t,i} = W_{pos} r_{t,i} \in \mathbb{R}^{192}$ and $pori_{t,i} = W_{ori} o_{t,i} \in \mathbb{R}^{192}$, concatenated as $$p_{t,i} = [ppos_{t,i}; pori_{t,i}] \in \mathbb{R}^{384}$$. Each camera’s “whole-body vision token” is $c_{t,i} = [e_{t,i}; p_{t,i}] \in \mathbb{R}^{768}$. Proprioceptive scalars are linearly projected individually as $j_{t,k}\to W_j j_{t,k} \in \mathbb{R}^{768}$.

For each time frame $t$, the observation $O_t$ is a set of 30 tokens (21 camera, 9 joint) of dimension 768. Two consecutive frames ($T_o=2$) are stacked to yield 60 tokens (2×30).

2. Action Parameterization and Diffusion Backbone

The policy formulation predicts a sequence of future robot joint actions using a diffusion process. For a predictive horizon of $T_p=16$ timesteps, joint actions $A_t=[a_{t+1},...,a_{t+T_p}]$ with $a_i\in\mathbb{R}^{9}$ are denoised from diffusion noise.

At each diffusion step $k$, the noisy action $A^k$ is embedded by linearly projecting each slice and adding a learned timestep embedding $PE(k)$:

$$h_{t,s} = W_a a^k_{t,s} + PE(k) \in \mathbb{R}^{768}, \quad s=1..T_p$$

The transformer decoder backbone, following the Diffusion Policy paradigm [Chi et al. 2023], takes these as input tokens $X \in \mathbb{R}^{T_p \times 768}$ and conditions on stacked observation tokens $Y \in \mathbb{R}^{(2 \times 30) \times 768}$.

3. Transformer Attention Mechanism and Multi-Modal Fusion

The core attention mechanism is a sequence of $L=12$ transformer layers with $H=8$ heads and $d_\text{model}=768$ dimensions per head. Each layer uses:

• Masked multi-head self-attention on action tokens $X$
• Multi-head cross-attention (action tokens as queries, observation tokens as keys/values)
• Two-layer feedforward networks (hidden size $3072$) with GeLU activation

Multi-head attention adopts the standard formulation:

$$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $Q = X W^Q$, $K = Y W^K$, $V = Y W^V$, $W^Q, W^K, W^V \in \mathbb{R}^{d \times d_k}$, $W^O\in\mathbb{R}^{Hd_k\times d}$, and $d_k=d/H$.

The policy fuses the class tokens from all camera views (each pre-encoded with 3D pose) via cross-attention to the action token sequence. No explicit spatial alignment or pixelwise correspondence (like homography) is performed; instead, the transformer is trained to learn the correspondence between visual observations and control actions implicitly.

4. Learning Paradigm, Data Collection, and Objective

Training is conducted as diffusion-based imitation learning:

• Data: ~400 leader-follower teleoperation episodes sampled at 10 Hz, comprising full RGB streams and proprioception.
• Objective: Predict the diffusion noise $\epsilon$ added to action samples in a forward noising process, with the mean-squared error

$$L = \mathbb{E}_{A \sim \pi(A),\,k,\,\epsilon} \left\| \epsilon - \epsilon_\theta (A^k, k, O) \right\|_2^2$$

where $$A^k = \sqrt{\alpha_k}A + \sqrt{1-\alpha_k}\,\epsilon$$ and $\alpha_k$ is the forward noise schedule.

Training uses AdamW with learning rate $10^{-4}$, weight decay $10^{-4}$, and batch size $\approx 64$, over approximately 200,000 gradient steps. Color-jitter is applied on images for augmentation; optional dropout in MLPs is noted but not detailed. At inference, $K$ denoising steps yield the first $T_a = 8$ predicted actions.

5. Multi-View Robustness: “Blink Training” and Sensor Failure

The WB-VIMA policy incorporates explicit resilience to visual sensor failures through “blink training”:

• During training, each camera input token $c_{t,i}$ is zeroed independently with probability $p_\text{mask}=0.05$, simulating random dropouts.
• The probability that at least one camera stream is masked at any frame is approximately $65.9\%$.
• At test time, missing camera streams are replaced by zero (or a learned [MASK] token). The transformer’s learned cross-attention enables robust operation under sensor dropout.

6. Implementation Specifics and System Integration

Vision embedding uses CLIP ViT-B/16, with all per-camera encodings shared and frozen for efficient parallelization. Data from distributed cameras is streamed via PCIe-USB extenders; each token (vision or proprioceptive) is mapped to the shared latent space for transformer processing. All geometric encodings use the first two columns of camera rotation matrices to avoid discontinuities in 3D orientation representation [Zhou et al. 2019, as cited in (Xu et al., 9 Jan 2025)].

The table summarizes key architectural parameters:

Component	Dimensionality / Value	Notes
Cameras	21 × 224×224 RGB	Synchronized USB feeds, color-jitter augmented
Proprioception	9 scalars	Joint angles, each mapped to 768D token
Transformer	12 layers, 8 heads, d=768	Masked/cross-attention, 4×768 hidden FFN
Embeddings	Vision: 768→384, Pose: 384	Concat to 768D vision token
Observation hist.	$T_o=2$ frames	Stack to 60 tokens per observation
Action horizon	$T_p=16$, $T_a=8$	Predict 16, execute first 8 joint vectors

System-level integration in RoboPanoptes enables complex manipulation including unboxing in tight spaces, multi-step stowing, and object sweeping—demonstrating improvements in adaptability and efficiency over baseline policies without WB-VIMA’s comprehensive fusion and robustness (Xu et al., 9 Jan 2025).