Dual-Arm Visuotactile Framework

• Dual-Arm Visuotactile Framework is a robotic architecture that integrates co-registered visual and tactile sensing to enable contact-rich and coordinated manipulation tasks.
• It employs shared latent representations, coupled control strategies, and event-triggered corrections to overcome the limitations of single-arm and vision-only systems.
• The framework has demonstrated high precision in applications like grasping, delicate assembly, and garment manipulation using diverse learning paradigms.

A dual-arm visuotactile framework is a robotic manipulation architecture in which two arms are coordinated through joint visual and tactile sensing, shared or coupled control policies, and task-level coordination mechanisms to perform contact-rich actions that are difficult for single-arm or vision-only systems. In the recent literature, the term spans several concrete instantiations: dual-arm visual grasping for large flat objects (Wang et al., 4 Apr 2025), bimanual tactile estimation from egocentric multi-view video (Zhou et al., 13 May 2026), coordinated delicate assembly with event-triggered compliance (Kumar et al., 22 Nov 2025), bimanual multifingered imitation learning from visuotactile demonstrations (Lin et al., 2024), dual-arm visuotactile regrasp-and-place pipelines (Bauza et al., 2023), and dual-arm garment manipulation with confidence-aware dense correspondence and tactile-supervised grasp affordance (Sunil et al., 4 Sep 2025). Across these systems, the common structure is the coupling of two manipulators through shared perception, shared or coordinated action representations, and contact-aware feedback loops.

1. Conceptual scope and historical technical context

A dual-arm visuotactile framework addresses manipulation regimes in which geometry, contact state, and inter-arm coordination must be inferred jointly rather than sequentially. The motivating failure cases are consistent across the literature: single-arm systems struggle with large flat objects whose thickness is smaller than a gripper’s stroke (Wang et al., 4 Apr 2025); visual-only systems degrade under ambiguous visual cues or severe occlusions in contact-rich tasks (Jiang et al., 12 May 2025); proprioceptive or vision-only systems are brittle in delicate snap-fit assembly when engagement events must be detected in tens of milliseconds (Kumar et al., 22 Nov 2025); and deformable-object pipelines fail when graspability and semantic correspondences are both uncertain (Sunil et al., 4 Sep 2025).

The topic is broader than a single control paradigm. Some works formulate dual-arm coordination as a dense visual action-selection problem with a shared feature map and shared reward (Wang et al., 4 Apr 2025). Others formulate it as bimanual tactile-state estimation from multi-view video and hand pose, yielding bilateral pressure maps in a canonical hand grid (Zhou et al., 13 May 2026). A separate line emphasizes full-stack robot systems: low-cost teleoperation, synchronized visuotactile data acquisition, and diffusion-policy imitation learning with two multifingered hands (Lin et al., 2024). Precision manipulation work such as simPLE combines visuotactile pose estimation, task-aware grasping, and dual-arm regrasp graphs for millimeter-scale placement into tight cavities (Bauza et al., 2023).

The sensing substrate is equally heterogeneous. Optical tactile sensors such as STS provide collocated visual and tactile sensing by switching internal illumination of a semitransparent skin (Hogan et al., 2020). MuxGel instead uses spatial multiplexing to recover simultaneous tactile and external visual fields from a single camera (Hu et al., 10 Mar 2026). MoiréTac uses overlapping micro-gratings to produce dense moiré fields while preserving optical clarity and enabling simultaneous 6-axis force/torque sensing and visual perception (Sou et al., 16 Sep 2025). SEED turns visuotactile sensing into a 6D series-elastic end-effector abstraction, where the relative pose between the gripper frame and the tool frame parameterizes spatial compliance and force control (Suh et al., 2021).

Taken together, these systems suggest that a dual-arm visuotactile framework is best understood as an architectural family. A plausible implication is that the defining feature is not merely “two arms plus touch,” but the use of contact-sensitive latent variables or contact-aligned feedback loops as first-class elements in perception, planning, and control.

2. Sensing substrates and multimodal state representations

The visual side of recent dual-arm systems ranges from overhead RGB-D heightmaps to egocentric multi-view video to fingertip-local collocated vision. In dual-arm grasping of large flat objects, the RL observation is an RGB heightmap image,

$$\mathbf{s}_t = \text{RGB} \in \mathbb{R}^{3 \times 224 \times 224},$$

with depth and masks used for heightmap generation and action execution but not included in the RL state (Wang et al., 4 Apr 2025). In EgoTouch and TouchAnything, the observation comprises head-mounted egocentric RGB, left- and right-wrist RGB, and a bimanual 3D hand-pose sequence

$\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$

used to predict bilateral tactile maps

$$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$

for both hands (Zhou et al., 13 May 2026). In HATO, the learning system consumes three RGB-D cameras—two wrist-mounted and one third-person view—together with proprioception and 60 fingertip pressure sensors across two hands (Lin et al., 2024).

On the tactile side, the representation varies according to sensor physics. MoiréTac uses four primary moiré observables—brightness $I$, phase gradient $\nabla \phi$, orientation $\theta$, and period $\Lambda$—to encode all six wrench axes. Normal force $F_z$ is encoded by global brightness and period changes; shear forces $F_x,F_y$ are encoded by spatial averages of phase gradients; twist torque $T_z$ is approximately proportional to fringe orientation rotation; and tilt moments $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$0 are encoded by brightness centroid shift (Sou et al., 16 Sep 2025). STS uses a semitransparent skin and programmable illumination to alternate between visual mode and tactile mode while preserving a common optical path and common camera viewpoint (Hogan et al., 2020). MuxGel interleaves tactile-sensitive coated regions with transparent windows using a checkerboard pattern and reconstructs a pure visual image and a pure tactile image from a multiplexed input using a shared ResNet-34 encoder and two U-Net-style decoders (Hu et al., 10 Mar 2026).

Several frameworks are explicitly structured to support representation sharing across sensors. AnyTouch defines a unified input format for static images and dynamic tactile clips, treats an image as a “static video,” and learns shared tactile embeddings across GelSight Mini, DIGIT, DuraGel, and Tac3D via masked modeling, touch–vision–language alignment, cross-sensor matching, and universal sensor tokens (Feng et al., 15 Feb 2025). TouchAnything uses a shared DINOv2 ViT-B/14 visual backbone across ego and wrist views, cross-view attention, gated fusion, temporal modeling, and pose–vision cross-attention to predict bilateral pressure fields (Zhou et al., 13 May 2026). OmniVTA compresses tactile 3D displacement fields with a causal spatio-temporal VAE and uses these latents as the tactile state in a predictive world model and high-frequency reflex controller (Zheng et al., 19 Mar 2026).

The most important commonality is co-registration. MoiréTac emphasizes that the same camera captures both the external scene and the internal moiré fringe field, so the contact region seen in the image is exactly co-registered with the area where forces are sensed (Sou et al., 16 Sep 2025). STS and MuxGel make the same structural point through different optics: a single optical path removes cross-device registration between tactile and visual modalities (Hogan et al., 2020, Hu et al., 10 Mar 2026). This suggests that in dual-arm settings, local co-registration at each end-effector and global registration across the two arms are both central design variables.

3. Coordination mechanisms, action abstractions, and contact-aware control

Dual-arm coordination is represented in the literature through several distinct abstractions. In the large-flat-object grasping framework, the policy outputs a dense 2D feature map

$\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$1

which is interpreted as a grasp affordance map (Wang et al., 4 Apr 2025). A grasp decoder then extracts two symmetric grasp points for the two arms by finding the feature-map maximum, searching along candidate axes, refining candidate points on each side of the object, back-projecting them into 3D, and applying inverse kinematics to obtain left and right arm commands. The coordination mechanism is implicit but structurally strong: one shared policy, one shared visual representation, one shared action map, and one binary success reward for the joint lift (Wang et al., 4 Apr 2025).

In TouchAnything, the action is not a robot command but a bilateral tactile estimate. Coordination arises because the same fused visual stream informs both hands, the pose features from both hands are fused in one joint-token set of size 42, and temporal modeling treats the entire dual-hand interaction as one sequence (Zhou et al., 13 May 2026). A plausible implication is that this kind of joint latent state can be used downstream as a common control representation in robot bimanual policies even though the work itself is a perception model.

Task-specific dual-arm coordination is more explicit in delicate assembly. The snap-fit framework defines three phases: coupled pick-up, coupled transport and pre-alignment, and decoupled insertion, with arm 1 holding the mating component and arm 2 performing insertion (Kumar et al., 22 Nov 2025). Each arm has a phase variable $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$2, with coupled or decoupled dynamics: $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$3 where $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$4 in synchronized phases and $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$5 in insertion (Kumar et al., 22 Nov 2025). Interaction control is a Cartesian impedance law,

$\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$6

with event-triggered stiffness decay after snap engagement (Kumar et al., 22 Nov 2025). Engagement is detected by SnapNet, a CNN-GRU with joint attention over joint-velocity windows. On the FR3 platform, online detection recall across all parts is 96.7% with latency under 50 ms at 100 Hz, and on the lens–frame task the event-triggered controller achieved 15/15 success with peak force reduced to $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$7 N, compared to 11/15 and $\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$8 N for fixed impedance (Kumar et al., 22 Nov 2025).

SEED provides a different but complementary control abstraction. The relative pose

$\mathbf{P} \in \mathbb{R}^{T \times 42 \times 3},$9

between a gripper-attached frame $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$0 and a tool frame $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$1 is mapped to a spatial wrench

$$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$2

using a 6D bushing model with translational stiffness $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$3 and rotational stiffness $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$4 (Suh et al., 2021). Hybrid force–position control is then formulated by inverting the stiffness map rather than by directly commanding contact wrench. This is not a dual-arm framework in itself, but it defines a modality-agnostic control interface that can be instantiated on both arms in cooperative tool use or assembly.

For deformable objects, coordination is often implemented as a reactive state machine rather than a continuous coupled controller. In the garment framework, one arm holds the garment while the other searches and grasps a new point using a dense correspondence probability map, a grasp affordance heatmap, and tactile validation. Once two good grasps are obtained, the arms move apart to tension the cloth, and then execute folding or hanging trajectories (Sunil et al., 4 Sep 2025). Crucially, the state machine defers action under low confidence: if confidence is low everywhere, the garment is rotated; if a full 360° search still fails, the system falls back to grasping the lowest point with highest affordance to promote unfurling (Sunil et al., 4 Sep 2025).

These examples show that “dual-arm coordination” is not a single algorithmic primitive. It can be expressed as shared dense action maps, coupled phase dynamics, latent tactile co-prediction, reactive state transitions, or explicit relative-pose/relative-wrench regulation.

4. Learning paradigms: imitation learning, reinforcement learning, representation learning, and world models

Recent dual-arm visuotactile frameworks use four main learning paradigms, often in combination.

The first is model-free reinforcement learning over visual or visuotactile latent spaces. In dual-arm grasping of large flat objects, a frozen large-scale visual backbone based on AVN from GraspNet-1Billion extracts a $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$5 feature map from the $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$6 RGB heightmap, and a CNN-based PPO actor–critic refines this representation for the task (Wang et al., 4 Apr 2025). Shared actor–critic layers outperform independent heads, and freezing the backbone yields better performance and stability than adaptive fine-tuning (Wang et al., 4 Apr 2025). The sparse terminal reward is binary: $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$7 The same policy transfers directly from Isaac Gym to real robots without fine-tuning, achieving 87.3% average success on 10 real objects versus 79.3% for the push-to-edge baseline (Wang et al., 4 Apr 2025).

The second is imitation learning, especially diffusion-policy behavior cloning. HATO uses diffusion policies conditioned on multimodal observations—vision, touch, and proprioception—to output 24-dimensional joint targets for two UR5e arms and two multifingered hands (Lin et al., 2024). In the reported tasks, main success rates were 10/10 on Slippery Handover, 10/10 on Tower Block Stacking, 9/10 on Wine Pouring, and 5/10 on Steak Serving (Lin et al., 2024). Modality ablations are especially revealing: in Steak Serving, the full visuotactile policy achieved 5/10 success, while the policy without touch achieved 0/10 despite 10/10 pickup success, showing that tactile information changed task completion rather than only early-stage reachability (Lin et al., 2024). GelFusion uses the same diffusion-policy family but adds a vision-dominated cross-attention fusion mechanism and a dual-channel tactile representation consisting of static texture–geometric features and dynamic interaction features (Jiang et al., 12 May 2025). The design principle is that vision remains the primary global estimator while tactile disambiguates local contact state under occlusion or ambiguity (Jiang et al., 12 May 2025).

The third is large-scale representation learning. AnyTouch learns unified static–dynamic representations across multiple visuotactile sensors through masked autoencoding, next-frame prediction, touch–vision–language alignment, and cross-sensor matching (Feng et al., 15 Feb 2025). TouchAnything learns a pose-aware, multi-view mapping from egocentric video to bilateral tactile maps with contact-aware regression losses and view dropout (Zhou et al., 13 May 2026). The latter reports that adding wrist-mounted views to egocentric video improves tactile prediction up to 5.0% relative in Contact IoU and 6.1% relative in Volumetric IoU on seen objects (Zhou et al., 13 May 2026). A plausible implication is that this kind of bilateral tactile estimation from vision could serve as a tactile prior or pseudo-sensor in dual-arm systems where only one arm is instrumented or where tactile hardware coverage is incomplete.

The fourth is predictive world modeling. OmniVTA combines a self-supervised tactile encoder, a two-stream visuo-tactile world model, an adaptive visuo-tactile fusion policy, and a 60 Hz reflexive controller (Zheng et al., 19 Mar 2026). The world model predicts short-horizon visual and tactile latent evolution with diffusion transformers and uses dynamic- and amplitude-weighted losses to emphasize contact changes: $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$8 The policy forms a Latent Tactile Differential feature by concatenating current tactile features, predicted future tactile features, and their difference, then gates tactile and visual features based on predicted contact probability (Zheng et al., 19 Mar 2026). A 60 Hz reflexive controller corrects deviations between predicted and observed tactile latents in closed loop. Across all six interaction categories, OmniVTA outperforms vision-only diffusion policies, tactile-concatenation baselines, and Reactive Diffusion Policy, especially under perturbations and unseen geometry changes (Zheng et al., 19 Mar 2026).

Across these paradigms, one architectural pattern recurs: robust priors are learned slowly, often at scale or in simulation, while task-specific adaptation is pushed into a thinner control head, a small policy network, or a high-frequency reactive layer.

5. Task families and benchmarked capabilities

The task range covered by dual-arm visuotactile frameworks is now broad enough to support a categorical view.

Representative task families

Task family	Representative systems	Characteristic feedback
Large-object grasping and lifting	(Wang et al., 4 Apr 2025)	Shared visual map, symmetric dual-arm grasp points
Bimanual skill imitation	(Lin et al., 2024)	Wrist/head vision, fingertip pressure, proprioception
Delicate assembly and insertion	(Kumar et al., 22 Nov 2025, Zheng et al., 19 Mar 2026)	Event detection, impedance modulation, tactile reflexes
Precision pick–regrasp–place	(Bauza et al., 2023)	Visuotactile pose posterior, dual-arm regrasp graph
Garment manipulation	(Sunil et al., 4 Sep 2025)	Dense correspondence confidence, grasp affordance, tactile validation
Sensor-rich local manipulation	(Sou et al., 16 Sep 2025, Hu et al., 10 Mar 2026, Suh et al., 2021)	Collocated vision–touch, 6-axis force/torque, spatial compliance

The large-flat-object system demonstrates strong generalization across unseen objects and direct sim-to-real transfer. On simulation test sets it reports success of at least 90% for all beveled large flat objects with average 94.3%, average 95.3% on irregular shapes, and 84.7% on household objects, outperforming push-to-edge at 78.3% (Wang et al., 4 Apr 2025). On real robots, it generalizes to unseen real objects with sizes from 180–440 mm width and 25–95 mm height (Wang et al., 4 Apr 2025).

HATO focuses on long-horizon bimanual tasks that require simultaneous dexterity and tactile feedback rather than low-level force regulation. It uses 75–300 demonstrations per task and finds that ActionMSE may fail to reflect tactile usefulness: in Steak Serving, policies with and without touch have similar offline error, but real-world success diverges sharply (Lin et al., 2024). This is a recurring theme in visuotactile evaluation: contact-aware metrics or task-level completion under perturbation are often more informative than imitation error alone.

simPLE occupies the high-precision end of the spectrum. It performs pick–localize–regrasp–place from CAD models only, on a dual-arm ABB YuMi with GelSlim 3.0 tactile fingers (Bauza et al., 2023). It achieves successful placements into structured arrangements with 1 mm clearance over 90% of the time for 6 objects, and over 80% of the time for 11 objects (Bauza et al., 2023). The pipeline is notable because visuotactile sensing enters as a discrete posterior over table grasps and because dual-arm hand-to-hand regrasps are modeled explicitly as graph edges (Bauza et al., 2023).

Garment manipulation extends the concept to high-dimensional deformable state spaces. The dense descriptor network produces a per-pixel descriptor map $$\hat{\mathbf{M}} \in \mathbb{R}^{T \times 2 \times 21 \times 21}$$9 with $I$0, and correspondence probability maps are learned with a distributional loss that explicitly handles garment symmetries and yields confidence estimates (Sunil et al., 4 Sep 2025). The same tactile classifier that labels real grasps during affordance fine-tuning is used online to detect successful grasp, too little fabric, or too many layers (Sunil et al., 4 Sep 2025). With the full framework, folding succeeds in 6/10 trials and hanging in 7/10 trials; without affordance fine-tuning, folding drops to 3/10, mainly from grasping too many layers (Sunil et al., 4 Sep 2025).

Local manipulation and sensor papers are not always dual-arm, but they define primitives directly reusable in dual-arm systems. MoiréTac achieves $I$1 across tested force/torque axes and 100% test accuracy on six fruits/vegetables in visual classification through the transparent moiré architecture (Sou et al., 16 Sep 2025). MuxGel demonstrates 100% success on 9 unseen-object grasping tests using local vision for approach and tactile depth to stop gripper closure at an appropriate deformation threshold (Hu et al., 10 Mar 2026). SEED shows that visuotactile relative pose can support force-regulated tool use such as writing and squeegeeing without direct wrist force control (Suh et al., 2021).

6. Limitations, misconceptions, and open directions

Several limitations recur across the literature. First, many dual-arm systems still operate with limited contact modalities. The large-flat-object framework explicitly has no tactile or force feedback and relies on pure position control, which the authors note can damage rigid or fragile objects and suffers from asynchronous dual-arm motion (Wang et al., 4 Apr 2025). The snap-fit framework uses proprioceptive event detection rather than tactile sensing, and its recall degrades on the higher-friction Heal Cobot compared with FR3, indicating hardware dependence of proprioceptive transients (Kumar et al., 22 Nov 2025). HATO provides tactile sensing to the policy but not haptic feedback to the human teleoperator (Lin et al., 2024).

Second, many “dual-arm” systems are only partially dual at the sensing or decision level. TouchAnything predicts bilateral tactile maps, but it is a perception model rather than a full bimanual control architecture (Zhou et al., 13 May 2026). Some sensor papers such as MoiréTac, SEED, GelFusion, and MuxGel are single-arm demonstrations whose relevance to dual-arm systems lies in transferable sensor and control abstractions rather than direct bimanual evaluation (Sou et al., 16 Sep 2025, Suh et al., 2021, Jiang et al., 12 May 2025, Hu et al., 10 Mar 2026). This matters because a common misconception is that adding two copies of a single-arm visuotactile stack automatically yields a dual-arm framework. The surveyed work suggests otherwise: dual-arm systems also need explicit coordination representations, conflict resolution, cross-arm timing, and often role specialization.

Third, sim-to-real remains unevenly solved. The visual RL framework for flat objects transfers without fine-tuning, but its domain is geometrically simple and highly structured (Wang et al., 4 Apr 2025). simPLE succeeds with CAD-only training but still executes open-loop after the initial visuotactile estimate and identifies near-success insertion failures as a major remaining issue (Bauza et al., 2023). MuxGel relies on a carefully engineered simulation pipeline with relighting, tactile image synthesis, and checkerboard-boundary perturbations to bridge the multiplexed sensor gap (Hu et al., 10 Mar 2026). AnyTouch and OmniVTA point toward representation-level robustness across sensors and interaction patterns, but neither yet addresses fully general dual-arm cross-sensor transfer in contact-rich real-world tasks (Feng et al., 15 Feb 2025, Zheng et al., 19 Mar 2026).

Fourth, benchmark coverage is still fragmented. RoboTwin provides a dual-arm benchmark with generative digital twins, simulation-generated experts, and real-world alignment, but it is vision-centric and does not include explicit tactile sensing (Mu et al., 17 Apr 2025). OmniViTac provides large-scale contact-rich data, but only for a single-arm setting (Zheng et al., 19 Mar 2026). This suggests that a comprehensive dual-arm visuotactile benchmark would need to combine RoboTwin’s task diversity and digital-twin generation with OmniViTac-style tactile instrumentation and interaction-pattern taxonomy. That is an inference, but it follows naturally from the complementary strengths of the two efforts.

Open directions are correspondingly clear in the source material. The flat-object framework explicitly identifies tactile and force sensing, pre-grasp manipulation, and richer multi-step tasks as next steps (Wang et al., 4 Apr 2025). MoiréTac highlights high-frequency phase analysis for slip prediction, flexible or miniaturized gratings, and cross-device calibration transfer as future work relevant to multi-sensor setups (Sou et al., 16 Sep 2025). AnyTouch motivates unified multi-sensor tactile backbones for heterogeneous hardware deployments (Feng et al., 15 Feb 2025). OmniVTA suggests that predictive contact modeling plus high-frequency tactile feedback is a better primitive than passive tactile concatenation for contact-rich control (Zheng et al., 19 Mar 2026). The garment framework shows that uncertainty-aware state machines remain useful even when dense correspondences and tactile grasp validation are available (Sunil et al., 4 Sep 2025).

A dual-arm visuotactile framework, in the most mature sense suggested by these works, would therefore combine five elements: co-registered local visuotactile sensing at each hand, a shared or coupled multimodal latent state, an explicit dual-arm coordination mechanism, predictive contact modeling or event detection, and high-frequency contact-aware correction. The literature demonstrates each element separately and, in a few cases, in combination, but not yet as a fully unified standard architecture.