PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulation

Abstract: Bimanual manipulation in cluttered scenes requires policies that remain stable under occlusions, viewpoint and scene variations. Existing vision-language-action models often fail to generalize because (i) multi-view features are fused via view-agnostic token concatenation, yielding weak 3D-consistent spatial understanding, and (ii) language is injected as global conditioning, resulting in coarse instruction grounding. In this paper, we introduce PEAfowl, a perception-enhanced multi-view VLA policy for bimanual manipulation. For spatial reasoning, PEAfowl predicts per-token depth distributions, performs differentiable 3D lifting, and aggregates local cross-view neighbors to form geometrically grounded, cross-view consistent representations. For instruction grounding, we propose to replace global conditioning with a Perceiver-style text-aware readout over frozen CLIP visual features, enabling iterative evidence accumulation. To overcome noisy and incomplete commodity depth without adding inference overhead, we apply training-only depth distillation from a pretrained depth teacher to supervise the depth-distribution head, providing perception front-end with geometry-aware priors. On RoboTwin 2.0 under domain-randomized setting, PEAfowl improves the strongest baseline by 23.0 pp in success rate, and real-robot experiments further demonstrate reliable sim-to-real transfer and consistent improvements from depth distillation. Project website: https://peafowlvla.github.io/.

• The paper introduces PEAfowl, a novel method achieving a +23.0 pp improvement over the strongest baseline (SEM) under domain randomization through geometry-guided multi-view fusion and language-guided readout.
• Explicit geometric alignment across views, leading to a +300% increase in avg. task performance under domain randomization.
• Iterative text-as-query visual readout stabilizes depth-distribution learning even with noisy depth inputs,increasing bimanual coordination success by 50%

Motivation and problem setting

PEAfowl addresses two structural weaknesses in existing vision-language-action (VLA) policies for bimanual manipulation. First, multi-view inputs are typically fused in a view-agnostic manner: per-camera images are encoded independently and their tokens concatenated before a shared policy head, with no explicit modeling of cross-view geometric correspondence or 3D consistency. Second, language is injected as global conditioning, so attention remains largely vision-centric and instruction grounding degrades in cluttered, multi-object scenes. Both weaknesses are amplified in bimanual settings, where self-occlusions, inter-object occlusions, tightly coupled high-dimensional action spaces, and fine-grained referential instructions are common (2601.17885).

The paper's central claim is that explicit geometric alignment across views and iterative text-as-query visual readout can substantially improve multi-task generalization under domain randomization, without increasing inference cost.

Architecture

PEAfowl has three components: geometry-guided multi-view fusion, language-guided readout, and a SEM-style joint-centric diffusion policy backbone (2601.17885).

Geometry-guided multi-view fusion (GGMVF). For each of $V$ cameras, modality-specific encoders (RGB initialized from Grounding-DINO; depth via a lightweight ResNet) produce multi-scale token pyramids. Co-located RGB–D token pairs predict a discrete depth distribution over $B$ bins; the expected 3D anchor $\bar{\mathbf{x}}^{(v)}_n$ is computed by distribution-weighted backprojection into the robot base frame, together with a depth-aware point embedding. Tokens from other views whose anchors fall within the top-$K$ nearest neighbors are aggregated with distance-based softmax weights and injected through a learnable gated residual. A deliberate design choice is that RGB–D fusion is restricted to co-located pairs within the depth-distribution branch only, leaving the main RGB stream untouched to avoid corrupting pretrained semantics under noisy commodity depth.

Language-guided readout. Frozen CLIP ViT-L/14 features (attn-last patch tokens following ClearCLIP) are queried by text-token latents through $M{=}3$ Perceiver-style latent blocks with ReZero-gated residuals, producing compact instruction-conditioned context tokens that condition the action decoder alongside instruction-summary tokens.

Policy backbone and training. The backbone reuses SEM's joint-centric state encoder (joint tokens with position and quaternion from forward kinematics, encoded via joint-graph attention) and diffusion transformer decoder predicting an $H{=}64$ action chunk. Training combines a weighted diffusion imitation loss, an optional FK-consistency term, and a depth-distillation loss: a pretrained Camera Depth Model (CDM) provides refined depth targets offline, supervising the depth-distribution head with a validity-weighted soft-label BCE. Crucially, raw depth remains the policy input at both training and test time, so distillation adds no inference overhead — a practical point for real deployment.

Simulation results

On RoboTwin 2.0 (Aloha-AgileX, 4-camera RGB-D, 9 tasks, 50 demonstrations/task, 100 trials/task), PEAfowl uses only 300M trainable parameters yet outperforms all baselines:

Setting	DP	DP3	ACT	π₀	RDT	SEM	PEAfowl
Clean avg.	26.1	40.0	34.2	22.0	10.6	51.0	69.6
DR avg.	11.3	14.1	7.8	22.1	6.7	24.1	47.1

The headline result is a +23.0 pp improvement over the strongest baseline (SEM) under domain randomization. Gains concentrate on long-horizon tasks requiring sustained spatial awareness under repeated occlusion (e.g., Stack Blocks Three rises from 1% for SEM to 72% clean, 34% DR), while short-horizon gains are smaller because baselines are already strong. On two held-out tasks, PEAfowl retains 51% success on Stack Blocks Two under DR where SEM collapses from 49% to 14%, indicating better transfer of spatial and semantic structure to unseen but related tasks. t-SNE visualization shows post-aggregation tokens form more view-consistent clusters corresponding to the same physical regions, supporting the claim that GGMVF produces 3D-aligned representations rather than merely improving end metrics.

Real-world results

Experiments use a dual-arm AgileX Piper with four RealSense D435 cameras; notably, a calibration-preserving remap to $320\times256$ narrows the effective field of view and reduces cross-view overlap, deliberately hardening the real-world evaluation. Across 6 tasks (10 trials each), PEAfowl averages 68.3% versus 36.7% without depth distillation and 11.7% for SEM. Depth distillation contributes its largest gains on depth-sensitive tasks such as Put Bottles Dustbin (30% → 80%) and Hanging Mug (10% → 30%). The paper also reports that SEM suffers depth-loss divergence during real-robot fine-tuning — reproducibly across seeds — whereas PEAfowl's local pairwise fusion and cross-view aggregation stabilize depth-distribution learning even when added to a simple visuomotor baseline (Baseline+GGMVF improves the DR average from 31.3% to 35.8%). This suggests the geometry-guided pathway is itself a stabilizer for multi-task depth-aware learning, independent of the language branch.

Ablations

Three ablations isolate component contributions. Replacing the Perceiver-style iterative readout with OTTER's one-step similarity readout costs little in-distribution (69.6% → 63.9% clean) but drops sharply under DR (47.1% → 36.1%), supporting the claim that iterative evidence accumulation matters specifically under clutter and scene shift. Removing vision-grounded text tokens (conditioning only on instruction summaries) yields 62.3% / 38.9%, with larger degradation on attribute- and reference-sensitive tasks. Multi-view ablation under DR shows removing the front view costs 12.1 pp (47.1% → 35.0%), and keeping only the head camera drops performance to 26.8%, with wrist views critical for fine bimanual coordination (Hanging Mug falls to 2%).

Limitations and open questions

Several caveats bear directly on the results. The evaluation suite is limited to 9 simulation tasks and 6 real tasks with 10 trials per task on the real robot, so real-world percentages carry wide binomial uncertainty (e.g., 30% vs. 10% on Hanging Mug differs by only two trials). The framework assumes known calibrated intrinsics/extrinsics for backprojection and cross-view matching; sensitivity to calibration error is not quantified. Depth distillation depends on the availability and quality of the pretrained CDM teacher, and the paper does not examine how distillation benefits scale with teacher quality or with sensor noise beyond the D435 setup. Absolute performance on the hardest long-horizon task under DR (Blocks Ranking Size, 14%) remains low, indicating that perception improvements alone do not close the gap on tasks requiring precise size-based reasoning. Finally, whether the Perceiver-style readout's advantage persists at much larger data scales, or whether the fixed $R{=}64$ readout budget becomes a bottleneck, is left open.

Conclusion

PEAfowl demonstrates that replacing view-agnostic token concatenation with distribution-based 3D lifting and cross-view neighbor aggregation, and replacing global text conditioning with iterative text-as-query readout over frozen CLIP features, yields consistent gains for multi-task bimanual manipulation — most prominently +23.0 pp over the strongest baseline under heavy domain randomization, and reliable sim-to-real transfer on a dual-arm platform with deliberately reduced view overlap. The training-only depth-distillation scheme injects geometric priors at zero inference cost, and the observation that the geometry module prevents depth-loss divergence may be of independent interest for stable multi-task visuomotor training.