Dream-Tac: Unified Visuotactile Control

• Dream-Tac is a unified visuotactile manipulation framework that integrates predicted tactile futures with visual observations to improve control in contact-rich environments.
• It employs both joint latent-sequence modeling and a two-pass policy refinement to incorporate tactile feedback, addressing limitations of vision-only systems.
• Empirical evaluations show up to 83.3% success rates and a 31.7% action accuracy boost over vision-only methods in challenging manipulation tasks.

Searching arXiv for the specified Dream-Tac papers and closely related context. Dream-Tac denotes a line of visuotactile robot-manipulation systems that use predicted tactile futures to improve control in contact-rich settings. In "Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation," the term refers to a unified Tactile-World Action Model that jointly models actions, future visual observations, and tactile dynamics (Lou et al., 7 Jun 2026). In "Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation," the authors also use Dream-Tac or DreamTac as shorthand for DreamTacVLA, a tactile-grounded Vision-Language-Action framework built around Hierarchical Spatial Alignment and a tactile world model (Ye et al., 29 Dec 2025). In both usages, the motivating observation is the same: vision-only policies and world action models often fail when the decisive state variables are contact onset, slip, local geometry, or force modulation rather than scene appearance.

1. Nomenclature and problem setting

The name "Dream-Tac" is used in two closely related but architecturally distinct works. The 2026 system is formulated as a world action model (WAM): it extends vision-only WAMs by jointly modeling future visual observations, future tactile observations, and action chunks conditioned on current visual observation, current tactile observation, and language instruction (Lou et al., 7 Jun 2026). The 2025 system, DreamTacVLA, is a VLA framework with a two-pass "Think–Dream–Act" policy and a tactile world model that predicts future tactile latents from current tactile state and a draft action (Ye et al., 29 Dec 2025).

Both systems target contact-rich manipulation. The 2026 paper explicitly states that RGB often cannot disambiguate contact onset, slip, and forceful interaction, because these cues are sparse, transient, and localized (Lou et al., 7 Jun 2026). The 2025 paper makes the same point from a VLA perspective, arguing that conventional VLAs are vision-centric and effectively blind to what happens at the contact interface, particularly in tight insertions, gear assembly, slip detection, and tool stabilization (Ye et al., 29 Dec 2025).

A common source of confusion is therefore terminological rather than conceptual. DreamTacVLA, Dream-Tac, and DreamTac all denote the tactile-grounded VLA system in the 2025 paper, whereas the 2026 Dream-Tac paper presents a distinct unified visuotactile WAM (Ye et al., 29 Dec 2025). The shared theme is not a single codebase or model family, but the use of anticipated tactile consequences as part of action generation.

2. Formal formulations

In the 2026 paper, the point of departure is the standard WAM factorization

$$p(a_{1:H}, v_{1:T} \mid o, l) = p(v_{1:T} \mid o, l)\, p(a_{1:H} \mid o, l, v_{1:T}),$$

where a vision-only WAM predicts future visual observations and then conditions action generation on those predicted futures. Dream-Tac extends this to

$p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$

where $o$ is current visual observation, $x$ current tactile observation, $l$ instruction, $v_{1:T}$ future visual observations, and $x_{1:T}$ future tactile observations (Lou et al., 7 Jun 2026). The model jointly denoises the concatenated target latents for future visual frames, future tactile frames, and action tokens, so that action tokens can attend to anticipated visuotactile trajectories.

DreamTacVLA is organized differently. Its policy first predicts a draft action from current aligned observations, then predicts a future tactile latent, and finally refines the action using both current observations and the dreamed tactile future:

$$a_{\text{draft}}^{(t)} = \pi_\theta(H_{\text{align}}^{(t)}, H_{\text{null}}),$$

$$H_{\text{dream}}^{(t+N)} = F_\eta(z_\tau^{(t)}, a_{\text{draft}}^{(t)}),$$

$$a_{\text{final}}^{(t)} = \pi_\theta(H_{\text{align}}^{(t)}, H_{\text{dream}}^{(t+N)}).$$

Here $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$0 is the tactile latent produced by a frozen tactile encoder (Ye et al., 29 Dec 2025).

The two formulations therefore place tactile prediction at different levels. Dream-Tac makes future tactile dynamics part of a single joint denoising problem over actions and observations, while DreamTacVLA inserts tactile forecasting into a two-pass policy. This suggests that the former is designed around unified latent-sequence modeling, whereas the latter is designed around policy refinement by imagined tactile consequences.

3. Sensory stack and architectural organization

The two systems use different sensing stacks, pretrained backbones, and tokenization strategies.

Aspect	Dream-Tac (2026)	DreamTacVLA (2025)
Visual sensors	Two synchronized Intel RealSense D435i RGB cameras	Two Intel RealSense D405 cameras
Tactile sensors	Two Xense Photon fingertip tactile RGB sensors	Two high-resolution GelSight sensors
Visual/tactile latent backbone	Same pretrained video VAE in Wan VAE latent space	CLIP ViT for RGB and language; V-JEPA2 for tactile
Core policy/dynamics model	Pretrained video Diffusion Transformer, Cosmos-Predict2-2B Video2World initialization	CLIP-based multimodal encoder plus Action Expert transformer
Action output	Action chunks injected as latent-frame tokens; chunk length $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$1	7-DOF action over a 45-step horizon

Dream-Tac uses two synchronized Intel RealSense D435i RGB cameras, one third-person fixed and one wrist-mounted egocentric, both at $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$2 and 30 Hz, together with two Xense Photon fingertip tactile RGB sensors synchronized at 30 Hz (Lou et al., 7 Jun 2026). Visual and tactile frames are encoded by the same pretrained video VAE, yielding a unified latent representation across modalities. Text instructions are encoded with a pretrained T5 encoder from a web-video pretrained backbone, and robot state and action chunks are padded as frame-like latent tokens and co-denoised with future latents.

DreamTacVLA uses a three-scale perceptual decomposition: GelSight tactile images as micro-vision, a wrist camera as local vision, and a third-person camera as macro vision (Ye et al., 29 Dec 2025). CLIP ViT encodes third-person images, wrist images, and language prompts into token sequences; a ViT-based tactile backbone, V-JEPA2, encodes tactile images into patch tokens; and an MLP encodes robot state. The paper specifies that the policy backbone processes modality-specific tokens by concatenation in a shared transformer, rather than by introducing an additional cross-attention module beyond unified sequence processing.

This architectural divergence reflects different priors. Dream-Tac seeks a single latent dynamical substrate for vision, touch, and action. DreamTacVLA instead preserves modality-specific encoders and adds explicit spatial grounding between tactile and visual tokens before using a world model to refine control.

4. Contact modeling and tactile prediction

A defining component of Dream-Tac is contact-gated visuotactile fusion. The tactile gate is computed from per-view normalized mean absolute differences between consecutive tactile RGB frames:

$p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$3

$p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$4

followed by a robust gate mapping

$p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$5

The implementation uses $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$6, clips $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$7 to $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$8, and sets $p(a_{1:H}, v_{1:T}, x_{1:T} \mid o, x, l),$9 (Lou et al., 7 Jun 2026).

The gate modulates a contact-aware attention bias (CASA). With token indicators $o$0 for tactile tokens, the gated asymmetric bias activates when a non-tactile query attends to a tactile key:

$o$1

with attention weights

$o$2

The paper’s interpretation is explicit: when contact is salient, non-tactile queries from vision, action, or proprioception increase relative attention to tactile keys, while tactile-to-tactile attention is unchanged (Lou et al., 7 Jun 2026).

DreamTacVLA emphasizes a different mechanism: Hierarchical Spatial Alignment (HSA) and latent tactile forecasting. Using robot forward kinematics and calibrated cameras, it projects the tactile sensor pose into wrist and third-person views to obtain bounding boxes and mean-pools the visual tokens that fall inside those boxes. The HSA loss is presented as an InfoNCE-style contrastive objective that pulls the mean-pooled tactile embedding toward the pooled wrist and third-person embeddings and pushes it away from negative samples (Ye et al., 29 Dec 2025). The world model then predicts a future tactile latent

$o$3

where $o$4 is a lightweight Forecasting MLP and $o$5 is a frozen V-JEPA2 tactile encoder.

These mechanisms instantiate two different hypotheses about where the key inductive bias should sit. Dream-Tac encodes salience by modulating cross-modal attention with a contact gate derived from tactile change. DreamTacVLA first enforces geometric correspondence between touch and vision and then predicts tactile futures for policy refinement. A plausible implication is that the former prioritizes dynamic event gating, while the latter prioritizes cross-scale spatial grounding.

5. Optimization, acceleration, and control

Dream-Tac is trained with a denoising objective on the joint latent target

$o$6

$o$7

with modality-wise decomposition

$o$8

Training adopts the rectified-flow / hybrid-EDM denoising objective inherited from the parent checkpoint; $o$9 is drawn from a hybrid mixture consisting of a log-normal component with $x$0 on $x$1 plus a uniform component on $x$2 (Lou et al., 7 Jun 2026). Optimization uses mixed bfloat16 precision and Fused Adam with $x$3, $x$4, $x$5, weight decay $x$6, 2k-step warmup, decay from 1.0 to 0.3 over the next 20k steps, and then fixed at 0.06.

A central engineering contribution of Dream-Tac is dual-level acceleration. For training, the CASA bias is rewritten as a rank-one term

$x$7

and folded into augmented queries and keys,

$x$8

so that

$x$9

This FlashBias reformulation preserves the fused scaled dot-product kernel and avoids dense $l$0 bias masks (Lou et al., 7 Jun 2026). For inference, the denoiser is decomposed as $l$1, and expensive features from $l$2 are cached at a few timesteps and reused for nearby steps. The practical policy is to perform full forward passes at the first denoising step and at step 3 and reuse cached features for the remaining steps.

DreamTacVLA uses a two-stage curriculum. In Stage 1, the world model is disabled and the policy is trained with action loss and HSA. In Stage 2, the Forecasting MLP is introduced so that the policy can refine its action using the dreamed tactile future (Ye et al., 29 Dec 2025). The tactile encoder $l$3 is frozen; the tactile adapter is a 3-layer bottleneck MLP with GELU and dropout $l$4, a learnable residual scale initialized at 0.1, and attention pooling with one learnable query over 196 adapted patches via 8-head multi-head attention. The reported parameter overhead is approximately 5.5M, or approximately 1.8% of the 300M ViT-L tactile encoder (Ye et al., 29 Dec 2025).

The data strategies also differ. Dream-Tac reports 100 demonstrations per task via 3D SpaceMouse teleoperation, synchronized first-person and wrist RGB, tactile RGB, and proprioceptive state at 30 Hz, trained on 8 NVIDIA H100 GPUs (Lou et al., 7 Jun 2026). DreamTacVLA constructs a hybrid large-scale dataset of 2M tactile frames across 4 manipulation tasks and 9 objects, approximately 80% simulated and 20% real-world, with 1000 demonstrations per task in simulation and 100 expert demonstrations per task in the real world (Ye et al., 29 Dec 2025).

6. Empirical performance, robustness, and limitations

Dream-Tac is evaluated on six real Franka Emika Panda tabletop tasks: Pick Baguette, Insert USB, Clean Whiteboard, Peel Cucumber, Play Mahjong, and Cut Banana, with 20 real-world trials per method per task and success rate as the metric (Lou et al., 7 Jun 2026). The average success rate across the six tasks is reported as 83.3% for Dream-Tac, 51.7% for Cosmos-Policy, 50.8% for ForceVLA, 45.0% for $l$5, and 30.8% for $l$6. Dream-Tac achieves the highest success rate on five of the six tasks: 100% on Pick Baguette, 35% on Insert USB, 90% on Clean Whiteboard, 100% on Play Mahjong, 90% on Cut Banana, and 85% on Peel Cucumber, where it is competitive but slightly below ForceVLA. The paper reports an average 31.7% action accuracy improvement over vision-only WAM baselines in the abstract and a 31.6% improvement over Cosmos Policy in the main text (Lou et al., 7 Jun 2026).

Its ablations isolate the contribution of tactile fusion and CASA. A visual WAM without tactile achieves 51.7%, a visuotactile WAM without bias achieves 74.2%, and a visuotactile WAM with CASA achieves 83.3% (Lou et al., 7 Jun 2026). The reported interpretation is that tactile fusion provides the main lift and CASA adds a further gain by emphasizing salient tactile events. The model is also evaluated under out-of-distribution variations in table height, spatial arrangement, object appearance, and background, where it consistently outperforms Cosmos-Policy on three of four settings and matches it on spatial arrangement.

DreamTacVLA is evaluated on Peg-in-Hole, USB Insertion, Gear Assembly, and Tool (Pen) Stabilization, using success rate over 100 real-world trials per task, reported as mean $l$7 standard deviation over 3 runs (Ye et al., 29 Dec 2025). The full model, labeled "Ours (HSA + Dream)," reports 95.0 $l$8 0.2 on Peg, 85.7 $l$9 0.6 on USB, 81.1 $v_{1:T}$0 0.4 on Gear, and 74.6 $v_{1:T}$1 0.5 on Pen. Baseline results are also given for ACT, Diffusion Policy, $v_{1:T}$2, "Ours (HSA-Only, No Dream)," and "Ours (No HSA, Dream-Only)," with the paper highlighting that the full model outperforms strong VLA baselines and that HSA and dreaming are complementary (Ye et al., 29 Dec 2025).

The runtime profile of Dream-Tac is unusually explicit. Training with FlashBias CASA is up to 2.9$v_{1:T}$3 faster; the largest reported end-to-end training reduction on Peel Cucumber is from 80.82 s to 27.48 s for the tactile-plus-bias setting (Lou et al., 7 Jun 2026). Inference for 10-step diffusion runs at approximately 5 Hz on an A800 GPU, and caching reduces heavy steps to approximately 2, achieving up to 1.8$v_{1:T}$4 speed-up without reported success-rate loss; 85% is maintained with both 10-step full diffusion and cached 2-step heavy passes.

The limitations are correspondingly specific. For Dream-Tac, low-contact tasks or phases where tactile is uninformative keep $v_{1:T}$5 low so that the policy reverts to visual dynamics; if tactile sensors are noisy or miscalibrated, the gate mapping parameters $v_{1:T}$6 and $v_{1:T}$7 control sensitivity, and improper thresholds can under- or over-emphasize touch (Lou et al., 7 Jun 2026). For DreamTacVLA, the two-pass Think–Dream–Act policy introduces inference overhead; the authors also note persistent domain gaps and sensor wear despite hybrid sim-to-real data and a frozen tactile encoder, and they do not disclose the exact numeric value of the forecast horizon $v_{1:T}$8 (Ye et al., 29 Dec 2025).

A plausible implication is that the reported success rates across the two papers should not be read as directly comparable benchmarks, because the task suites, trial counts, and evaluation protocols differ. What is comparable is the direction of the result: both papers report that explicitly modeling future tactile state substantially improves performance in manipulation regimes where contact information is decisive.