Papers
Topics
Authors
Recent
Search
2000 character limit reached

Flex-ππ: A Multi-Stream World-Action Model with Compute Flexibility

Published 11 Aug 2026 in cs.RO and cs.CV | (2608.10860v2)

Abstract: World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-ππ, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7×\times on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than π0.5π_{0.5}. Our project website: https://flex-pi.github.io/

Summary

  • The paper introduces a 6-billion-parameter world-action model that jointly predicts 32-step actions and future RGB, pointmap, and DINOv3 feature streams using shared multimodal representations.
  • The paper’s cross-modality forcing objective improves learning by requiring missing visual modalities to remain mutually predictable, with ablations showing gains of 6.8% from DINO features and a further 20% from pointmaps on RoboTwin.
  • The paper enables compute-flexible deployment, ranging from roughly 60 ms action-only inference to 193 ms full visual generation, while achieving 78.8% RoboTwin success with 50 demonstrations and substantially stronger real-world manipulation performance than baselines.

Flex-π\pi: Multi-Stream World-Action Modeling with Compute-Flexible Inference

Research Problem and Central Thesis

“Flex-π\pi: A Multi-Stream World-Action Model with Compute Flexibility” (2608.10860) addresses a limitation in current world-action models (WAMs): although WAMs jointly predict actions and future visual observations, their visual prediction objectives are typically restricted to RGB latents. RGB reconstruction supplies strong appearance and temporal supervision, but it does not explicitly impose the geometric or object-centric constraints required for precise manipulation. The paper’s central claim is that a WAM can be trained on RGB, 3D geometry, and semantic visual features without requiring new sensors, separately pretrained modality-specific generative priors, or an unavoidable increase in deployment latency.

Flex-π\pi is a 6-billion-parameter WAM that jointly denoises action chunks and future latent streams for RGB observations, pointmaps, and DINOv3 features. Pointmaps are generated from RGB using Depth Anything 3, while semantic features are extracted using a frozen DINOv3 encoder. The model uses the frozen Wan-2.2 VAE to encode both RGB images and pointmaps. This produces the paper’s most unusual architectural observation: a VAE trained exclusively for RGB reconstruction can encode and decode image-shaped pointmaps with high fidelity, preserving scene structure despite the absence of pointmap-specific VAE training.

The resulting model is trained as a multimodal predictive policy but can be deployed under multiple inference regimes. It may generate actions alone, actions together with selected visual streams, or all future visual streams jointly. Thus, Flex-π\pi separates the training objective from the deployment computation budget. Figure 1

Figure 1: Flex-π\pi jointly processes RGB, pointmap, and DINO streams while allowing deployment-time selection of the generated streams.

Architecture and Latent Representation

Flex-π\pi represents each visual modality as a separate token stream. RGB observations and pointmaps are encoded by the same frozen Wan-2.2 VAE, yielding latent streams zoz^o and zpz^p. DINOv3 provides patch-level semantic tokens dd. The model additionally receives language instructions and proprioception as global conditioning. The proprioceptive state is embedded into the language-conditioning pathway, allowing both the visual streams and the action expert to attend to task specifications and robot state through a shared conditioning mechanism.

The backbone is a Mixture-of-Transformers (MoT). A 5-billion-parameter visual transformer, initialized from Wan-2.2, processes the visual streams with shared transformer blocks and modality-specific feedforward projections. A smaller approximately 1-billion-parameter action expert produces 32-step action chunks and cross-attends to the visual representation. Cross-stream attention is restricted to the middle 16 of 30 transformer blocks: early layers remain stream-specific, while the central trunk performs multimodal fusion and later layers decode each stream independently. Figure 2

Figure 2: Flex-π\pi architecture, with shared visual processing, modality-specific adapters, and an action expert that reads current and generated future visual tokens.

The model uses flow matching for all streams. RGB, pointmap, and action heads predict flow velocities, whereas the folded DINO stream uses clean-feature, or π\pi0-, prediction because its token dimensionality is unusually high. DINO patch tokens are folded by a factor of two in each spatial dimension, reducing the token count by four while preserving the feature content through an invertible space-to-channel rearrangement.

The action representation is also carefully normalized across embodiments. The canonical state and action layout contains 32 dimensions, accommodating the two-arm end-effector poses, gripper states, and joint positions. On the YAM platform, actions are represented as body-frame-relative targets anchored to the first state in each chunk. This avoids cumulative integration error and permits the same action representation to transfer across pretraining and downstream platforms.

Stream Dropout and Cross-Modality Forcing

The key mechanism enabling compute flexibility is independent dropout over input and output visual streams. During training, each RGB, DINO, and pointmap input stream is independently retained with probability π\pi1, subject to retaining at least one visual stream. A second mask determines which future streams are visible to the action tokens and to other future visual streams.

Importantly, the output mask is not a loss mask. Every future stream is still denoised and supervised, even when it is not observed at the current timestep or is not read by the action tokens. Consequently, the model must infer missing future modalities from the available modalities. For example, it may generate future pointmaps from RGB and DINO inputs, or generate semantic features from RGB and geometry.

The authors call this mechanism cross-modality forcing. It is more than sensor-dropout robustness. By requiring geometry, semantics, and appearance to remain mutually predictable, the training objective encourages a shared latent representation in which the modalities constrain one another. The paper reports that removing cross-modality forcing reduces RoboTwin success by 21 percentage points in the corresponding ablation, providing evidence that the mechanism improves action prediction rather than merely enabling missing-input inference.

This design also supports a direct theoretical interpretation. Future visual prediction acts as an auxiliary representation-learning objective, but the auxiliary targets are not redundant RGB reconstructions. Pointmaps impose spatial consistency, DINO features impose object-level semantic structure, and RGB preserves appearance and temporal detail. Their joint prediction creates a form of multimodal predictive coding aligned with manipulation-relevant factors.

Training Data and Experimental Protocol

Flex-π\pi2 is pretrained on approximately 500 hours from 100 tasks in AGIBOT World-Beta (AgiBot-World-Contributors et al., 9 Mar 2025). The dataset contains bimanual manipulation demonstrations recorded at 30 Hz from an overhead camera and two wrist cameras. Pointmaps are generated offline from RGB using Depth Anything 3 rather than relying directly on the available depth recordings. This gives all three views a consistent geometry-generation pipeline and avoids propagating incomplete or noisy sensed depth.

Fine-tuning and evaluation cover RoboTwin, LIBERO, LIBERO-Plus, and a real bimanual YAM robot. The real-world evaluation uses five tasks spanning sustained contact, bimanual coordination, long-horizon execution, sub-millimeter assembly, and deformable-object manipulation. In particular, Self-Repair Gripper requires sequentially replacing and fastening the robot’s own gripper, while Soft-Bag Zipping requires opening, loading, and closing a deformable pencil case. Figure 3

Figure 3: Representative real-world tasks, including sequential gripper repair with tight insertion clearances and deformable soft-bag zipping.

The real-world evaluation reports both normalized task completion under partial-credit rubrics and binary full-task success. This distinction is important for sequential tasks: a policy may complete many individual stages yet rarely execute the entire sequence without failure. The paper evaluates between 10 and 20 rollouts per task, with interleaved methods and randomized object configurations.

Real-World Manipulation Results

Flex-π\pi3 leads all evaluated baselines on every real-world task. The comparison includes π\pi4, ManiFlow, and Fast-WAM, with ManiFlow receiving RGB and pointmap inputs. Full joint generation improves task completion over the strongest baseline by 5.0 points on Kitchen Organization, 42.7 points on Self-Repair Gripper, and 27.2 points on Soft-Bag Zipping. Averaged across the five tasks, the model achieves a 2.3-fold higher success rate than the strongest baseline. Figure 4

Figure 4: Flex-π\pi5 achieves the highest task completion and binary success across the five bimanual real-world tasks.

The result on Self-Repair Gripper is particularly informative. The task requires eight ordered stages, including insertion and fastening operations with clearances as small as π\pi6 mm. Full joint Flex-π\pi7 completes the entire sequence in more than half of the rollouts, whereas ManiFlow completes one of 20 and π\pi8 completes none. Relative to action-only inference, joint generation improves normalized partial-credit performance by 9.1% and complete-task success by 10 percentage points.

The Soft-Bag Zipping result probes a different failure mode. The object has no stable rest geometry, and the zipper pull is small, slack, and visually similar to the surrounding fabric. Flex-π\pi9 achieves more than 1.5 times the partial-credit score of the strongest baseline and completes the task twice as often as π\pi0 and eight times as often as ManiFlow. The advantage suggests that multimodal future prediction is beneficial not only for rigid geometric precision but also for maintaining task-relevant state estimates under deformation.

Compute Flexibility and the Speed–Performance Frontier

The deployment regimes expose a clear tradeoff between computation and task performance. With RGB-only input, action-only inference takes approximately 60 ms per call on an RTX 5090. Full joint generation takes approximately 193 ms per call. On RoboTwin, action-only generation reaches 40.2% success at roughly 60 ms, RGB future generation increases success to 60.4%, and generating RGB, DINO, and pointmap futures reaches 63.8% at approximately 193 ms. Figure 5

Figure 5: Action-only inference minimizes latency, while joint visual generation increases latency in exchange for higher accuracy.

The paper emphasizes a stronger claim than a conventional Pareto tradeoff: action-only Flex-π\pi1 is simultaneously faster and more accurate than the evaluated baselines in the real-world experiments. At four Euler denoising steps, action-only inference is faster than π\pi2 while achieving an 18.5% gain in average task completion over the strongest baseline. Joint generation adds roughly three times the latency and provides a further 6.5% gain.

The model also exposes the number of flow-matching steps as a deployment parameter. Action-only success peaks at four Euler steps, attaining 94.5% clean and 94.6% randomized RoboTwin success in the reported sweep. Two or more steps remain within one percentage point of the peak, whereas one step causes success to fall to approximately 51–53%. This behavior indicates that latent action prediction can tolerate aggressive sampling reduction, but not arbitrary single-step approximation.

The practical consequence is that one checkpoint can serve systems with substantially different latency constraints. A high-throughput controller can use action-only inference; a manipulation system with greater computational allowance can generate future visual streams; and the input sensor configuration can be changed without retraining.

Generalization and Demonstration Efficiency

Flex-π\pi3 exhibits its largest gains in low-data and distribution-shifted settings. On RoboTwin with 50 demonstrations per task, the model reaches 78.8% success, compared with 31.4% for π\pi4, 41.9% for Fast-WAM, and 17.2% for LingBot-VA. At 100 demonstrations, Flex-π\pi5 reaches 87.0%, while the corresponding baselines reach 44.7%, 68.1%, and 32.2%. At 500 demonstrations, it reaches 94.8%. Figure 6

Figure 6: Flex-π\pi6 maintains the highest RoboTwin success across all demonstration budgets, with the largest advantage in the low-data regime.

The paper reports a 1.9–4.5-fold advantage over baselines in the lower demonstration regimes. Full joint generation outperforms action-only inference at every data scale, supporting the claim that future visual prediction improves data efficiency rather than merely increasing test-time compute.

On the real robot, Flex-π\pi7 loses only 4.7 task-completion points on average under the evaluated distribution shifts in full joint mode and 4.1 points in action-only mode. The strongest baseline loses 26.7 points, despite having access to depth, while π\pi8 loses 25.6 points on the difficult unseen soft-bag condition. Figure 7

Figure 7: Flex-π\pi9 retains higher task completion under unseen objects, distractors, and reduced fine-tuning data.

Training with only half of the real-world demonstrations still leaves full-joint Flex-π\pi0 ahead of all baselines trained on the full data. Action-only Flex-π\pi1 trained on half the data matches π\pi2 trained on the complete dataset. These results support the paper’s interpretation that the world-action objective supplies representation-level supervision that would otherwise need to be obtained through additional demonstrations.

Simulation Results and Benchmark Interpretation

On RoboTwin, Flex-π\pi3 achieves 94.6% success in action-only mode and 94.6% in full joint mode under the reported clean and randomized evaluation aggregation. This exceeds the strongest VLA baseline, Qwen-RobotManip, at 93.9%, despite Qwen-RobotManip using a pretraining corpus approximately 76 times larger than the approximately 500 hours used for Flex-π\pi4 pretraining. Full joint Flex-π\pi5 also exceeds the reported WAM baselines, including LingBot-VA 2.0.

The near-equivalence of action-only and full-joint performance in the high-data RoboTwin setting is an important qualification. It suggests that the benchmark may approach saturation for the evaluated policy class, limiting the measurable benefit of test-time visual imagination. The data-scaling experiments reveal the benefit more clearly than the full-data endpoint.

On LIBERO, Flex-π\pi6 reaches 98.4% in action-only mode and 98.5% in full joint mode with flexible stream dropout. A fixed-mode fine-tuning variant reaches 99.2% with full joint generation, matching the strongest reported baseline. On LIBERO-Plus, full joint Flex-π\pi7 reaches 80.9%, outperforming Fast-WAM and most VLA baselines but remaining below π\pi8 and Qwen-RobotManip. This outcome is consistent with the authors’ limitation analysis: stronger semantic reasoning backbones and substantially larger robot-data corpora can still provide advantages under broad visual, spatial, language, and embodiment perturbations.

Ablation Evidence for the Multimodal Objective

The ablations isolate the contribution of each stream. Relative to video-only training, adding DINO features increases RoboTwin success by 6.8%. Adding pointmaps on top of RGB and DINO produces a further 20% increase. These gains occur even when the corresponding streams are not necessarily generated at deployment, indicating that the streams function as training-time supervisory signals as well as test-time inputs or predictive targets. Figure 8

Figure 8

Figure 8: The input-stream ablation isolates the incremental contributions of RGB, DINO semantics, and pointmap geometry.

The pointmap input is optional at deployment but not optional in the training objective. On Put Plate on Rack, full joint completion declines from 95.0% with depth-derived pointmaps to 91.7% without them. This modest test-time penalty contrasts with the 20-point RoboTwin degradation caused by removing pointmaps from training, demonstrating a central distinction in Flex-π\pi9: a modality can be essential as supervision while remaining dispensable as an input sensor. Figure 9

Figure 9: Cross-modality forcing allows Flex-π\pi0 to generate plausible geometry even when pointmaps are withheld from the input.

The model can generate pointmap futures from RGB and DINO alone, and the generated geometry remains qualitatively consistent with episodes in which pointmaps are observed. This supports the claim that cross-modality forcing induces internal geometric representations rather than simply exploiting an always-present depth stream.

Theoretical and Practical Implications

The paper’s main theoretical implication is that WAM supervision need not be tied to a single perceptual reconstruction space. A pretrained video VAE can serve as a shared coordinate system for heterogeneous image-shaped signals, while frozen discriminative features can be incorporated as an additional semantic stream. This broadens the role of generative world models from appearance prediction toward structured predictive modeling of manipulation-relevant variables.

The empirical results also challenge a common assumption that test-time future imagination is either unnecessary or uniformly too expensive for policy deployment. Flex-π\pi1 shows that visual imagination can be selectively enabled, and that its value depends on the data regime and task. In saturated simulation benchmarks, action-only and joint generation can converge. In low-data or contact-rich real-world tasks, joint generation provides substantial gains in both partial completion and full-sequence success.

Practically, the architecture supports gradual deployment. A robot can initially operate with RGB-only action inference, avoiding a dependence on depth sensors and minimizing latency. If the task requires additional robustness, pointmap input can be enabled. If computational resources permit, future visual streams can be generated to improve action selection. This is a meaningful systems-level property because it decouples model training from a single fixed hardware configuration.

Several limitations remain. Multistream training and cross-modality forcing increase optimization difficulty and require at least 10 fine-tuning epochs for the reported real-world tasks. Full joint generation remains slower than parameter-comparable VLAs, and the optimized inference stack requires substantial GPU memory. The real-world evaluations are also conducted on one bimanual platform and use RGB-derived geometry during much of pretraining, leaving open questions about transfer across camera calibration, embodiments, object categories, and sensor noise distributions.

Future developments may combine Flex-π\pi2-style multimodal predictive objectives with stronger VLM or VLA semantic backbones, larger heterogeneous robot datasets, learned adaptive stream selection, and uncertainty-aware allocation of inference compute. A policy could decide online whether geometry generation is needed, allocate more denoising steps near contact events, or use predicted visual streams for failure detection and recovery. Another direction is to replace offline monocular depth annotation with jointly calibrated metric geometry and force or tactile streams, extending cross-modality forcing beyond vision.

Conclusion

“Flex-π\pi3: A Multi-Stream World-Action Model with Compute Flexibility” (2608.10860) presents a WAM that jointly predicts actions, RGB futures, 3D pointmaps, and object-centric semantic features in a shared latent architecture. Its principal contribution is not merely the addition of geometric and semantic inputs, but the integration of those streams into a single checkpoint whose input and output computation can be selected at deployment.

The reported results show strong gains in demonstration efficiency, out-of-distribution robustness, and difficult real-world manipulation. Flex-π\pi4 reaches 78.8% RoboTwin success with only 50 demonstrations per task, achieves up to 99.2% on LIBERO under fixed-mode fine-tuning, and improves real-world success rates by factors of 2–7 over selected baselines on demanding bimanual tasks. The ablations indicate that the gains arise from multimodal predictive supervision and cross-modality forcing, not solely from supplying additional sensors at inference. The work therefore provides a technically coherent framework for treating geometry and semantics as optional deployment modalities but essential components of world-action representation learning.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1. What is this paper about?

This paper introduces Flex-</strong>,anewartificialintelligencesystemforrobots.</p><p>Thesystemhelpsrobotsunderstandwhatishappeningaroundthemandchoosebetteractions.Insteadoflookingonlyatnormalcameraimages,Flex−</strong>, a new artificial intelligence system for robots.</p> <p>The system helps robots understand what is happening around them and choose better actions. Instead of looking only at normal camera images, Flex- also learns about:

  • 3D shape and distance: where objects are in space
  • Object meaning: which parts of a scene belong to which objects
  • Future changes: what the scene might look like after the robot acts

The main idea is that a robot can perform better if it learns not only what things look like, but also where they are and what they are.

2. What questions did the researchers ask?

The researchers wanted to find out:

  1. Can Flex-controlarobotwellduringdifficulttasksthatrequirecarefulmovements?</li><li>Canitworkwithfewertrainingexamplesthanotherrobotsystems?</li><li>Canitcontinueworkingwhentheobjects,backgrounds,orinstructionschange?</li><li>Doesadding3Dinformationandobjectinformationactuallyimprovetherobot?</li><li>Canuserschoosebetweenafaster,simplermodeandaslower,morecapablemode?</li></ol><p>Ausefulwaytothinkaboutthelastquestionisacarwithdifferentdrivingsettings.Onesettingmightsaveenergyandrespondquickly,whileanothermightusemorecomputingpowertoprovidebetterperformance.</p><h2class=′paper−heading′id=′how−did−the−researchers−build−and−test−it′>3.Howdidtheresearchersbuildandtestit?</h2><h3class=′paper−heading′id=′what−information−does−flex−use′>WhatinformationdoesFlex− control a robot well during difficult tasks that require careful movements?</li> <li>Can it work with fewer training examples than other robot systems?</li> <li>Can it continue working when the objects, backgrounds, or instructions change?</li> <li>Does adding 3D information and object information actually improve the robot?</li> <li>Can users choose between a faster, simpler mode and a slower, more capable mode?</li> </ol> <p>A useful way to think about the last question is a car with different driving settings. One setting might save energy and respond quickly, while another might use more computing power to provide better performance.</p> <h2 class='paper-heading' id='how-did-the-researchers-build-and-test-it'>3. How did the researchers build and test it?</h2><h3 class='paper-heading' id='what-information-does-flex-use'>What information does Flex- use?

    Flex-usesthreemainkindsofvisualinformation:</p><ul><li><strong>RGBimages</strong>:ordinarycolorpicturesfromcameras</li><li><strong>Pointmaps</strong>:picturesinwhicheverypixelalsocontains3Dlocationinformation,suchashowfarawaysomethingis</li><li><strong>DINOfeatures</strong>:informationthathelpsidentifyandseparateobjectsandtheirparts</li></ul><p>Apointmapissimilartogivingtherobotapictureplusameasuringtapeforeverypointinthepicture.</p><p>DINOisacomputervisionsystemthathelpsrecognizemeaningfulpartsofimages.Forexample,itmayhelpdistinguishacup,ahandle,andatable.</p><h3class=′paper−heading′id=′a−surprising−discovery′>Asurprisingdiscovery</h3><p>Theresearchersusedapre−trainedvideomodelcalleda<strong>VAE</strong>.AVAEisatoolthatchangescomplicatedpicturesintoasmallerinternalrepresentation,calleda<strong>latentspace</strong>.Thisisliketurningalongdescriptionintoashortsetofnoteswhiletryingtokeeptheimportantinformation.</p><p>TheVAEhadoriginallybeentrainedonlyonnormalRGBimages.However,theresearchersdiscoveredthatitcouldalsostoreandreconstruct3Dpointmapsveryaccurately,eventhoughithadneverspecificallybeentrainedforthem.</p><p>Thismeanttheycouldadd3Dinformationwithout:</p><ul><li>installingnewtypesofsensors,</li><li>trainingacompletelynewvisualsystem,or</li><li>makingtherobotslowerduringdeployment.</li></ul><h3class=′paper−heading′id=′how−does−the−model−learn′>Howdoesthemodellearn?</h3><p>Flex− uses three main kinds of visual information:</p> <ul> <li><strong>RGB images</strong>: ordinary color pictures from cameras</li> <li><strong>Pointmaps</strong>: pictures in which every pixel also contains 3D location information, such as how far away something is</li> <li><strong>DINO features</strong>: information that helps identify and separate objects and their parts</li> </ul> <p>A pointmap is similar to giving the robot a picture plus a measuring tape for every point in the picture.</p> <p>DINO is a computer vision system that helps recognize meaningful parts of images. For example, it may help distinguish a cup, a handle, and a table.</p> <h3 class='paper-heading' id='a-surprising-discovery'>A surprising discovery</h3> <p>The researchers used a pre-trained video model called a <strong>VAE</strong>. A VAE is a tool that changes complicated pictures into a smaller internal representation, called a <strong>latent space</strong>. This is like turning a long description into a short set of notes while trying to keep the important information.</p> <p>The VAE had originally been trained only on normal RGB images. However, the researchers discovered that it could also store and reconstruct 3D pointmaps very accurately, even though it had never specifically been trained for them.</p> <p>This meant they could add 3D information without:</p> <ul> <li>installing new types of sensors,</li> <li>training a completely new visual system, or</li> <li>making the robot slower during deployment.</li> </ul> <h3 class='paper-heading' id='how-does-the-model-learn'>How does the model learn?</h3> <p>Flex- is a large model with about 6 billion parameters. During training, it watches examples of people or robots performing tasks. It learns to predict both:

    • the robot’s next actions, and
    • what the visual scene will look like next.

    This is called a world-action model. It is similar to learning a game by predicting both the next move and what the game screen will look like afterward.

    The model uses a Transformer, a type of neural network that learns which pieces of information should be connected. A Mixture-of-Transformers allows different information streams—images, 3D data, object information, and actions—to be processed together while still keeping some parts separate.

    The model is trained using a process called flow matching. In simple terms, it starts with random noise and learns how to gradually turn that noise into a likely future image, future 3D map, or action. This is similar to learning the steps needed to turn a blurry, scrambled drawing into a clear picture.

    Training with missing information

    During training, the researchers sometimes hide one or more information streams. For example, the model might receive an RGB image but not the pointmap. It must still predict the future pointmap.

    This method is called cross-modality forcing. It encourages the model to understand how different kinds of information are related. It is like asking a student to guess the shape of an object from its shadow, or to identify an object from its position and color.

    Because of this training, one Flex-modelcanworkinseveralways:</p><ul><li>usingonlycameraimages,</li><li>usingimagesand3Dinformation,</li><li>predictingonlyactionsforspeed,or</li><li>predictingactionsandseveralpossiblefuturesforbetterperformance.</li></ul><h3class=′paper−heading′id=′where−was−it−tested′>Wherewasittested?</h3><p>TheresearcherstestedFlex− model can work in several ways:</p> <ul> <li>using only camera images,</li> <li>using images and 3D information,</li> <li>predicting only actions for speed, or</li> <li>predicting actions and several possible futures for better performance.</li> </ul> <h3 class='paper-heading' id='where-was-it-tested'>Where was it tested?</h3> <p>The researchers tested Flex- in:

    • RoboTwin, a simulated two-armed robot environment
    • LIBERO, a collection of simulated robot tasks
    • LIBERO-Plus, which changes the scenes and instructions to test generalization
    • A real two-armed YAM robot

    The real robot performed tasks such as organizing kitchen objects, putting plates on racks, repairing its own gripper, and closing a soft pencil case.

    4. What were the main results?

    Better performance on difficult real-world tasks

    Flex-performedbetterthanthecomparisonsystemsonallfivereal−robottasks.</p><p>Itsadvantagewasespeciallylargeontasksrequiring:</p><ul><li>veryprecisemovements,</li><li>manystepsinthecorrectorder,</li><li>handlingsoftorflexibleobjects,and</li><li>continuouscontactwithotherobjects.</li></ul><p>Onaverage,Flex− performed better than the comparison systems on all five real-robot tasks.</p> <p>Its advantage was especially large on tasks requiring:</p> <ul> <li>very precise movements,</li> <li>many steps in the correct order,</li> <li>handling soft or flexible objects, and</li> <li>continuous contact with other objects.</li> </ul> <p>On average, Flex- had about 2.3 times the success rate of the strongest comparison system.

    For example, it was much better at:

    • inserting parts during the robot’s self-repair task, where the allowed error was only about half a millimeter
    • handling and zipping a soft bag whose shape changed from one attempt to the next

    More successful with fewer demonstrations

    A demonstration is an example showing the robot how to perform a task.

    Flex-learnedeffectivelyfromfewerdemonstrationsthantheothersystems.Insomesimulationexperiments,itperformed<strong>1.9to4.5timesbetter</strong>thancompetingsystemswhentrainingdatawaslimited.</p><p>Inthereal−worldtests,Flex− learned effectively from fewer demonstrations than the other systems. In some simulation experiments, it performed <strong>1.9 to 4.5 times better</strong> than competing systems when training data was limited.</p> <p>In the real-world tests, Flex- trained on only half the available examples still performed better than other systems trained on all the examples.

    This is important because collecting robot training data is expensive and time-consuming.

    Stronger generalization

    Flex-alsohandledchangesthatwerenotpresentduringtraining,suchas:</p><ul><li>differentobjects,</li><li>moreclutter,</li><li>distractingobjects,and</li><li>changedvisualconditions.</li></ul><p>Itsperformancedroppedmuchlessthanthatofseveralothersystems.Thissuggeststhatitlearnedmoregeneralideasaboutobjectsandspaceinsteadofmemorizingexactlywhatthetrainingsceneslookedlike.</p><h3class=′paper−heading′id=′high−results−in−simulation′>Highresultsinsimulation</h3><p>OnRoboTwin,Flex− also handled changes that were not present during training, such as:</p> <ul> <li>different objects,</li> <li>more clutter,</li> <li>distracting objects, and</li> <li>changed visual conditions.</li> </ul> <p>Its performance dropped much less than that of several other systems. This suggests that it learned more general ideas about objects and space instead of memorizing exactly what the training scenes looked like.</p> <h3 class='paper-heading' id='high-results-in-simulation'>High results in simulation</h3> <p>On RoboTwin, Flex- reached about 94.6% success in its action-only mode. This was higher than the main VLA comparison system and other world-action models.

    On LIBERO, the strongest Flex-setupreachedabout<strong>99.2<h3class=′paper−heading′id=′extra−visual−information−helped′>Extravisualinformationhelped</h3><p>TheresearcherstestedwhathappenedwhentheyremovedpartsofFlex− setup reached about <strong>99.2% success</strong>, matching the best reported comparison system and outperforming most other methods.</p> <h3 class='paper-heading' id='extra-visual-information-helped'>Extra visual information helped</h3> <p>The researchers tested what happened when they removed parts of Flex-.

    Adding DINO object information to video improved success by about 6.8 percentage points. Adding pointmaps as well produced another improvement of about 20 percentage points in the ablation study.

    This shows that the extra information was not merely decorative. Understanding object identity and 3D structure helped the robot choose better actions.

    Flexible speed and accuracy

    Flex-$ can choose how much computation to use.

    In the fastest mode, it predicts only actions. This mode took about 60 milliseconds per call and was faster than the comparison systems while still performing better.

    If it also predicts future images, DINO features, and pointmaps, it becomes slower but usually more accurate. In one test:

    Mode Approximate latency Success
    Action only 60 ms 40.2%
    Action plus video Not specified 60.4%
    Action plus all visual streams 193 ms 63.8%

    The exact numbers depend on the experiment, but the main point is that one trained model can operate at different points on the speed–performance trade-off.

    Cross-modality forcing was important

    When the researchers removed cross-modality forcing, performance became about 21% worse in one ablation.

    This suggests that making the model predict missing types of information helped it build a stronger understanding of the world, even when those extra types of information were not used during testing.

    5. Why is this research important?

    Robots often struggle with tasks that humans find easy because they must understand both appearance and physical structure. A normal image may show that an object is present, but it does not directly explain its exact distance, shape, or position in 3D space.

    Flex-offersawaytogiverobotsthisricherunderstandingwithoutrequiringexpensivenewsensorsoraseparatelargetrainingprocess.Italsoallowsuserstodecidewhethertheywant:</p><ul><li><strong>quickactions</strong>,usefulwhenspeedisimportant,or</li><li><strong>moredetailedfuturepredictions</strong>,usefulwhenthetaskisdifficultandaccuracymattersmore.</li></ul><p>Theresearchsuggeststhattrainingarobottoimaginefutureimages,objectmeanings,and3Dlayoutscanmakeit:</p><ul><li>moreaccurate,</li><li>moreusefulwithsmallamountsofdata,</li><li>betteratunfamiliarsituations,and</li><li>flexibleenoughfordifferenthardwareandcomputinglimits.</li></ul><h2class=′paper−heading′id=′simple−conclusion′>Simpleconclusion</h2><p>Flex− offers a way to give robots this richer understanding without requiring expensive new sensors or a separate large training process. It also allows users to decide whether they want:</p> <ul> <li><strong>quick actions</strong>, useful when speed is important, or</li> <li><strong>more detailed future predictions</strong>, useful when the task is difficult and accuracy matters more.</li> </ul> <p>The research suggests that training a robot to imagine future images, object meanings, and 3D layouts can make it:</p> <ul> <li>more accurate,</li> <li>more useful with small amounts of data,</li> <li>better at unfamiliar situations, and</li> <li>flexible enough for different hardware and computing limits.</li> </ul> <h2 class='paper-heading' id='simple-conclusion'>Simple conclusion</h2> <p>Flex- teaches robots to think about actions and the future at the same time. It combines ordinary camera images with information about 3D space and objects. The researchers found that this helped robots perform difficult tasks more reliably, especially when training data was limited or the environment changed.

    The most important idea is that a robot may learn to act better when it is also trained to understand what the world will look like, where things are, and what those things mean. This could lead to robots that are more capable in homes, factories, laboratories, and other places where careful physical work is needed.

    View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

  • The evidence for “near-lossless” pointmap encoding is incomplete. The paper shows qualitative reconstruction results but does not report quantitative depth/geometry metrics, error distributions, scale accuracy, or comparisons with pointmap-specific encoders.
  • The generality of the shared RGB–pointmap VAE latent space is unresolved. It is unclear whether the Wan VAE preserves geometry equally well across different cameras, resolutions, viewpoints, lighting conditions, object materials, scene scales, and environments outside the training distribution.
  • The effect of monocular depth-estimation errors is not isolated. Pointmaps are generated by Depth Anything 3 rather than measured with depth sensors, but the paper does not evaluate how depth noise, temporal inconsistency, scale ambiguity, occlusions, or estimator failures affect policy performance.
  • The contribution of estimated geometry versus geometry-specific supervision remains unclear. The experiments do not compare Flex-$ against models trained with ground-truth depth, calibrated RGB-D data, alternative depth estimators, or pointmap losses in a way that separates the value of the representation from the value of the additional training signal.
  • The DINO semantic stream is not sufficiently analyzed. The paper does not determine which object-level information is useful, whether DINO features improve object identity, affordance recognition, pose estimation, or language grounding, or how performance changes with alternative semantic encoders.
  • The causal mechanisms behind the reported gains are unresolved. Improvements may result from additional modalities, auxiliary prediction losses, increased computation during training, architectural changes, or regularization from stream dropout; the current ablations do not fully disentangle these factors.
  • The relative importance of input conditioning and future-stream prediction is underexplored. The experiments show cumulative modality and output ablations, but do not evaluate all input/output modality combinations under matched compute, optimization, and training conditions.
  • The role of cross-modality forcing is not fully characterized. The reported ablation indicates a large performance drop when forcing is removed, but it remains unclear whether the benefit comes from representation learning, denoising regularization, increased task difficulty, or implicit data augmentation.
  • The choice of equal loss weights is insufficiently justified. All stream losses use weight 1, despite differences in dimensionality, scale, noise characteristics, and task relevance; sensitivity to loss weighting and dynamic balancing is not reported.
  • The model’s calibration and reliability under missing or corrupted modalities are unknown. Stream dropout tests deliberate absence, but the paper does not measure performance when inputs are partially degraded, misaligned, stale, noisy, or semantically inconsistent.
  • The temporal consistency of generated RGB, pointmap, and DINO futures is not evaluated. Joint generation may produce individually plausible streams that disagree geometrically or semantically across time; no cross-stream consistency metrics or failure analysis are provided.
  • The usefulness of generated visual futures for action generation is not mechanistically established. The paper reports improved performance when future streams are generated, but does not determine whether actions benefit from visual planning, intermediate latent computation, regularization, or simply additional inference-time compute.
  • The speed–performance comparison is not normalized across systems. Reported latency depends on hardware, compilation, batching, number of flow steps, action horizon, precision, and implementation details; equivalent end-to-end comparisons under standardized conditions are needed.
  • The deployment cost of preprocessing is not fully accounted for. The paper emphasizes no added inference latency from output streams, but does not clearly include the runtime and hardware cost of computing Depth Anything 3 and DINO features when these are used as inputs.
  • The reduction in flow-matching steps may affect control quality and stability. The paper mentions varying inference steps but provides limited analysis of action smoothness, temporal consistency, failure rates, and long-horizon degradation as computation is reduced.
  • Closed-loop robustness over extended operation is insufficiently tested. Real-world experiments use only 10–20 trials per task and do not establish reliability over many-hour deployments, repeated resets, hardware drift, wear, or compounding errors across long sequences.
  • The real-world evaluation is narrow in platform and task diversity. All real experiments use a single bimanual YAM robot and five tasks, limiting conclusions about transfer to different robot embodiments, grippers, camera configurations, workspace geometries, and manipulation domains.
  • The real-world comparisons may be affected by task and data-selection bias. Tasks and demonstration counts were selected by iteratively training the smallest baseline until it reached reasonable completion, which may favor the proposed model or produce unequal difficulty across methods.
  • The statistical strength of the real-world results is limited. The paper reports standard deviations but does not provide confidence intervals, significance tests, per-seed variation, or independent repetitions of data collection and training.
  • The demonstrations and fine-tuning protocol may not be comparable across baselines. It is unclear whether all methods receive identical preprocessing, augmentation, action chunking, camera inputs, optimization budgets, and hyperparameter tuning effort.
  • The impact of AGIBOT World pre-training is not isolated. The paper does not compare training from scratch, RGB-only pre-training, modality-specific pre-training, or pre-training on different datasets to determine how much performance derives from the proposed objective versus the shared pre-training corpus.
  • The claim of strong demonstration efficiency lacks broad low-data validation. Data-scaling experiments use selected budgets and limited task subsets; the behavior at one, few, or zero demonstrations, across more tasks and different domains, remains unknown.
  • Out-of-distribution generalization is evaluated mainly under visual shifts. The paper does not systematically test novel task compositions, unseen language instructions, changed robot dynamics, altered object affordances, camera relocation, calibration errors, or physical interaction changes.
  • LIBERO results primarily measure in-distribution fitting. Because standard LIBERO lacks a conventional train/test split, the high success rates do not establish broad generalization or real-world transfer.
  • The benchmark may be close to saturation. RoboTwin results show little difference between action-only and full-joint variants at high performance, making it difficult to assess whether Flex-$ scales to substantially harder planning, contact, or perception regimes.
  • Long-horizon planning capabilities remain uncertain. Although some tasks are multi-stage, the paper does not analyze error accumulation by stage, recovery behavior, subgoal ordering, or whether the model plans beyond the action chunk horizon.
  • Failure modes are not systematically documented. The paper lacks categorized analyses of failures caused by depth errors, semantic confusion, occlusion, contact dynamics, deformable objects, language ambiguity, action instability, and inconsistent future generation.
  • The model’s language grounding is underexamined. The paper uses language instructions but does not evaluate paraphrases, compositional instructions, ambiguous descriptions, multi-object references, instruction changes during execution, or instruction-following failures.
  • The scalability of the 6B-parameter architecture is unknown. No experiments examine smaller models, larger backbones, parameter-efficient adaptation, or whether comparable gains can be achieved with substantially lower memory and training costs.
  • Training-time compute and energy costs are not reported. The claim that additional modalities are available “for free” concerns sensors, pre-training, and inference latency, but does not quantify the added training compute, memory, storage, or environmental cost of predicting four streams.
  • The assumption that all visual modalities are image-shaped limits applicability. The method’s shared VAE encoding strategy is not shown to extend to sparse point clouds, tactile observations, event cameras, force/torque signals, segmentation masks, or other non-image-shaped modalities.
  • The model’s dependence on frozen upstream encoders is unresolved. The paper does not study whether fine-tuning the VAE or DINO encoder improves performance, causes catastrophic loss of pretrained priors, or enables adaptation to domains where the frozen encoders fail.
  • Pointmap coordinate conventions and calibration requirements are insufficiently specified. It remains unclear how camera intrinsics, extrinsics, coordinate frames, temporal synchronization, and multi-view fusion are handled, and how calibration errors affect control.
  • The reported “no additional sensors” advantage may not hold in all settings. Since pointmaps and DINO features require substantial learned preprocessing from RGB, the method still depends on upstream models and compute whose availability, licensing, robustness, and real-time performance are not evaluated.
  • Reproducibility is potentially constrained by missing implementation details. The paper does not provide enough information in the supplied text about exact preprocessing, masking distributions, flow schedules, action normalization, data filtering, hardware, seeds, and baseline training procedures to fully reproduce the results.
  • The relationship between semantic feature prediction and semantic invariance is unclear. Predicting future DINO features may encourage matching the frozen encoder’s representation rather than learning task-relevant semantics; whether this transfers to novel objects, categories, and affordances remains open.
  • Safety and physical risk are not evaluated. The paper does not assess collision rates, force limits, recovery policies, uncertainty estimates, or safeguards for deployment in environments involving humans or fragile objects.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The paper’s findings support near-term deployment primarily in robotic manipulation, especially where precise geometry, object identity, bimanual coordination, and limited demonstrations are important. The following applications are feasible using existing RGB cameras, pretrained encoders, and a single Flex-checkpoint,subjecttotask−specificfine−tuning.</p><ul><li><strong>Industrialroboticassemblyandkitting—Robotics/manufacturing</strong><ul><li>DeployFlex− checkpoint, subject to task-specific fine-tuning.</p> <ul> <li><strong>Industrial robotic assembly and kitting — Robotics/manufacturing</strong> <ul> <li>Deploy Flex- on bimanual robots for tasks such as inserting parts, placing objects into racks, sorting utensils or components, and performing sustained-contact assembly.

  • The pointmap stream can improve spatial alignment and insertion accuracy, while DINO features can help distinguish objects and parts.
  • A practical workflow would be: collect a relatively small number of demonstrations, generate pointmaps from existing RGB cameras with Depth Anything 3, fine-tune Flex-fortheworkstation,andrunthepolicyinaction−onlyorfull−jointmode.</li><li><strong>Category:</strong>ImmediateApplication.</li><li><strong>Dependencies:</strong>Robotkinematicsandcameracalibrationmustbereliable;thetargettaskmustresemblethetrainingdistributionsufficientlyforsafedeployment;collisioncheckingandlow−levelsafetycontrolremainnecessary.</li></ul></li><li><strong>Precisioninsertion,fastening,andself−maintenance—Manufacturing/fieldrobotics</strong><ul><li>Usethemodelforoperationsinvolvingtighttolerances,suchasinsertingconnectors,screwingcomponents,repairinggrippers,oraligningpartswithsub−millimeterclearance.</li><li>Thepaper’sresultsonthe“Self−RepairGripper”tasksuggestvalueincontact−rich,long−horizonprocedureswheregeometry−awarerepresentationsaremoreusefulthanRGBappearancealone.</li><li><strong>Category:</strong>ImmediateApplication.</li><li><strong>Dependencies:</strong>Thereportedresultsdonotestablishreliabilityforsafety−criticalmaintenance.Forcesensing,compliantcontrol,recoverybehaviors,andhumansupervisionmayberequired.</li></ul></li><li><strong>Handlingdeformableobjects—Logistics,apparel,packaging,andhouseholdrobotics</strong><ul><li>ApplyFlex− for the workstation, and run the policy in action-only or full-joint mode.</li> <li><strong>Category:</strong> Immediate Application.</li> <li><strong>Dependencies:</strong> Robot kinematics and camera calibration must be reliable; the target task must resemble the training distribution sufficiently for safe deployment; collision checking and low-level safety control remain necessary.</li> </ul></li> <li><strong>Precision insertion, fastening, and self-maintenance — Manufacturing/field robotics</strong> <ul> <li>Use the model for operations involving tight tolerances, such as inserting connectors, screwing components, repairing grippers, or aligning parts with sub-millimeter clearance.</li> <li>The paper’s results on the “Self-Repair Gripper” task suggest value in contact-rich, long-horizon procedures where geometry-aware representations are more useful than RGB appearance alone.</li> <li><strong>Category:</strong> Immediate Application.</li> <li><strong>Dependencies:</strong> The reported results do not establish reliability for safety-critical maintenance. Force sensing, compliant control, recovery behaviors, and human supervision may be required.</li> </ul></li> <li><strong>Handling deformable objects — Logistics, apparel, packaging, and household robotics</strong> <ul> <li>Apply Flex- to tasks such as zipping bags, opening packaging, manipulating soft materials, folding fabric, or placing flexible items into containers.
  • Joint prediction of future RGB, geometry, semantics, and actions may help the robot anticipate changing object configurations.
  • Category: Immediate Application.
  • Dependencies: Pointmaps from monocular RGB may be noisy for occluded or highly deformable objects. Additional tactile or force sensing could be needed for robust commercial deployment, even though the paper’s method does not require such sensors.
  • Flexible robot inference modes — Edge robotics and embedded systems
    • Use the same trained policy in different operating modes:
    • action-only for low latency and limited compute;
    • RGB-conditioned action generation when visual prediction is unnecessary;
    • full joint generation when maximum performance is preferred.
    • This enables dynamic adjustment to workload, battery level, thermal constraints, network availability, or task difficulty without retraining the model.
    • Category: Immediate Application.
    • Potential product: A deployment controller that selects an inference regime based on latency and task-risk requirements.
    • Dependencies: The latency-performance tradeoff must be recalibrated for the target hardware, camera resolution, flow-matching step count, and robot control frequency.
  • Low-demonstration customization of robot workcells — Small-scale manufacturing and service robotics
    • Fine-tune a pretrained Flex-$ model for new tasks using fewer demonstrations than conventional action-only policies.</li> <li>This could lower the cost of adapting robots to changing product variants, custom orders, or short production runs.</li> <li><strong>Category:</strong> Immediate Application.</li> <li><strong>Dependencies:</strong> The paper demonstrates improved demonstration efficiency, but the amount and diversity of required data will vary by robot morphology, object set, language interface, and environment.</li> </ul></li> <li><strong>Robotic sorting and object organization — Warehousing, retail, and domestic assistance</strong> <ul> <li>Use object-centric DINO features to support sorting, stacking, placement, and organization tasks involving visually similar objects or cluttered scenes.</li> <li>RGB-only input can be used at deployment while the model retains benefits from semantic and geometric supervision during training.</li> <li><strong>Category:</strong> Immediate Application.</li> <li><strong>Dependencies:</strong> Object categories and task instructions must be represented adequately by the pretrained semantic encoder; performance may degrade for novel objects, transparent objects, or severe occlusion.</li> </ul></li> <li><strong>Simulation-based policy development and benchmarking — Robotics academia and R&amp;D</strong> <ul> <li>Use Flex-$ as a baseline or research platform for testing multimodal policy learning, world-action modeling, demonstration efficiency, and speed-performance tradeoffs.
    • RoboTwin, LIBERO, and LIBERO-Plus-style evaluations can be incorporated into workflows for measuring clean performance and robustness to visual, spatial, and language shifts.
    • Category: Immediate Application.
    • Dependencies: Simulation results may not transfer directly to physical robots. Accurate rendering, camera models, robot dynamics, and domain randomization remain important.
  • Training pipelines for RGB-only robotic datasets — Data engineering and robotics research
    • Automatically augment existing RGB demonstrations with:
    • monocular pointmaps from a depth-estimation model;
    • object-centric DINO features;
    • future latent visual targets.
    • This creates additional training supervision without collecting new sensor data.
    • Category: Immediate Application.
    • Potential tool: An offline dataset-conversion pipeline that adds pointmap and semantic-token streams to standard robot trajectories.
    • Dependencies: The approach assumes that the frozen VAE can encode pointmaps accurately for the new camera geometry and scene types. This should be validated quantitatively before training at scale.
  • Adaptive deployment policies for robot operators — Industrial operations
    • Allow operators or task planners to select the inference regime according to operational requirements: fast action-only control for routine actions and full visual generation for difficult or uncertain actions.
    • Category: Immediate Application.
    • Dependencies: A reliable uncertainty estimator or task-difficulty classifier is needed to decide when additional computation is worthwhile. The paper demonstrates configurability but does not provide a complete risk-aware switching mechanism.
  • Educational and research training environments — Academia
    • Use Flex-toteachmultimodalrepresentationlearning,flowmatching,robotimitationlearning,andembodiedAI.</li><li>StudentscancompareRGB−only,RGB−plus−DINO,RGB−plus−pointmap,andfull−joint−generationvariantsusingthesamecheckpointanddeploymentinterface.</li><li><strong>Category:</strong>ImmediateApplication.</li><li><strong>Dependencies:</strong>Themodel’sapproximately6Bparametersmaybetoolargeforordinaryteachinghardwarewithoutmodelcompression,cloudaccess,orsmallerdistilledvariants.</li></ul></li></ul><h2class=′paper−heading′id=′long−term−applications′>Long−TermApplications</h2><p>Thefollowingusesareplausibleextensionsofthepaper’smethodbutrequireadditionalvalidation,larger−scaledata,safetyengineering,orarchitecturaldevelopment.</p><ul><li><strong>General−purposehouseholdrobots—Dailylifeandconsumerrobotics</strong><ul><li>AhomerobotcoulduseFlex− to teach multimodal representation learning, flow matching, robot imitation learning, and embodied AI.</li> <li>Students can compare RGB-only, RGB-plus-DINO, RGB-plus-pointmap, and full-joint-generation variants using the same checkpoint and deployment interface.</li> <li><strong>Category:</strong> Immediate Application.</li> <li><strong>Dependencies:</strong> The model’s approximately 6B parameters may be too large for ordinary teaching hardware without model compression, cloud access, or smaller distilled variants.</li> </ul></li> </ul> <h2 class='paper-heading' id='long-term-applications'>Long-Term Applications</h2> <p>The following uses are plausible extensions of the paper’s method but require additional validation, larger-scale data, safety engineering, or architectural development.</p> <ul> <li><strong>General-purpose household robots — Daily life and consumer robotics</strong> <ul> <li>A home robot could use Flex--like policies for kitchen organization, laundry handling, packing, opening containers, and manipulating unfamiliar household objects.
    • The combination of language conditioning, object semantics, and geometry could support more general task instructions than fixed-purpose controllers.
    • Category: Long-Term Application.
    • Dependencies: Home environments contain large visual and physical variability, including transparent objects, reflective surfaces, pets, children, and unexpected obstacles. Robust uncertainty estimation, continual learning, safe recovery, and extensive real-world testing are required.
  • Assistive robots for elderly or disabled users — Healthcare and social care
    • Potential tasks include retrieving and organizing objects, preparing simple items, opening packaging, and assisting with household routines.
    • Demonstration efficiency could be valuable because collecting many demonstrations from each user or care setting is impractical.
    • Category: Long-Term Application.
    • Dependencies: Safety, privacy, personalization, and fail-safe human interaction are central requirements. The reported manipulation gains do not by themselves establish suitability for physical assistance around vulnerable users.
  • Autonomous laboratory automation — Life sciences and chemistry
    • Flex-$-style models could control robots that transfer samples, manipulate labware, assemble experimental setups, or handle flexible tubing and packaging.
    • Geometry-aware prediction may improve alignment and contact-rich operations.
    • Category: Long-Term Application.
    • Dependencies: Laboratory workflows require extremely high reliability, contamination control, traceability, and integration with scheduling and inventory systems. Vision-only depth estimation may be insufficient for some liquids, transparent vessels, or fine instruments.
  • Multi-robot and mobile-manipulation systems — Warehousing and service robotics
    • Extend the model from fixed-camera bimanual manipulation to mobile robots that must combine navigation, scene change prediction, and manipulation.
    • The shared latent space could potentially integrate camera observations, pointmaps, semantic features, proprioception, language, and action chunks over longer horizons.
    • Category: Long-Term Application.
    • Dependencies: The paper focuses on manipulation and does not demonstrate navigation, dynamic obstacle avoidance, multi-robot coordination, or long-horizon closed-loop planning. These capabilities would require additional datasets and temporal memory.
  • Closed-loop visual predictive control — Robotics and autonomous systems
    • Generated future RGB, pointmap, and semantic streams could serve not only as auxiliary training targets but also as short-horizon predictive simulations for action selection, replanning, and failure detection.
    • A controller could compare predicted and observed futures to detect slips, incorrect insertions, or object displacement.
    • Category: Long-Term Application.
    • Dependencies: Future predictions must be calibrated and sufficiently accurate under distribution shift. The paper shows improved task success, but it does not establish that generated visual futures are reliable enough for explicit model-predictive control or safety monitoring.
  • Automatic task and motion planning from language — Software and robotics
    • Combine Flex-withahigh−levelplannerthatdecomposesinstructionssuchas“organizethekitchen”intosubtasks,theninvokesthepolicyforeachmanipulationprimitive.</li><li>DINOandpointmapstreamscouldprovideinterfacesforobject−levelandspatialstateestimation.</li><li><strong>Category:</strong>Long−TermApplication.</li><li><strong>Potentialproduct:</strong>Ahierarchicalrobotsoftwarestackwithalanguageplanner,semanticscenerepresentation,Flex− with a high-level planner that decomposes instructions such as “organize the kitchen” into subtasks, then invokes the policy for each manipulation primitive.</li> <li>DINO and pointmap streams could provide interfaces for object-level and spatial state estimation.</li> <li><strong>Category:</strong> Long-Term Application.</li> <li><strong>Potential product:</strong> A hierarchical robot software stack with a language planner, semantic scene representation, Flex- manipulation controller, and recovery manager.
    • Dependencies: Reliable grounding of language to objects and actions, persistent object identity, task-state tracking, and safe subtask transitions are not fully addressed by the paper.
  • Sensor-adaptive robotics under degraded conditions — Field robotics and disaster response
    • The modality-dropout design could support graceful degradation when RGB, depth, or semantic processing becomes unavailable.
    • For example, a robot could switch to action-only generation during compute or bandwidth shortages and use additional streams when resources return.
    • Category: Long-Term Application.
    • Dependencies: The paper’s flexibility concerns model inputs and outputs, not guaranteed robustness to sensor corruption, severe occlusion, darkness, weather, or camera failure. Explicit fault detection and hardware redundancy would be needed.
  • Model compression and edge deployment — Robotics hardware
    • Distill or quantize Flex-$ into smaller models that preserve its multimodal training benefits while running on embedded GPUs or robot accelerators.
    • The action-only mode provides a natural target for low-latency deployment, while full generation could run intermittently for difficult actions.
    • Category: Long-Term Application.
    • Dependencies: Compression must preserve geometric and semantic grounding, not merely action imitation. Performance and latency should be evaluated on the actual robot hardware rather than inferred from the reported setup.
  • Synthetic data generation for rare manipulation events — Industry and academia
    • Use the world-action model to generate plausible future visual states and action-conditioned trajectories for rare events such as failed insertions, object slips, deformable-object changes, or recovery behaviors.
    • These trajectories could supplement demonstrations for data-hungry safety and robustness training.
    • Category: Long-Term Application.
    • Dependencies: Synthetic trajectories may contain physically implausible states or reinforce model bias. They should be filtered with physics simulation, human review, or real-robot validation before being used for safety-critical learning.
  • Policy and standards for data-efficient embodied AI — Public policy and governance
    • The paper supports policy discussions around benchmark standards for:
    • reporting success under limited demonstrations;
    • measuring out-of-distribution generalization;
    • publishing latency-performance curves;
    • documenting which sensors and pretrained components are required.
    • Such standards could improve comparability between robot policies and clarify whether improvements come from data scale, model architecture, sensing, or inference compute.
    • Category: Long-Term Application.
    • Dependencies: Benchmarks must include realistic physical variation, safety failures, energy consumption, calibration requirements, and reproducible hardware specifications. Simulation-only success rates are insufficient for deployment regulation.
  • Privacy-preserving robot learning from existing camera data — Policy, industry, and daily life
    • Because pointmaps and semantic features are derived from RGB rather than requiring additional depth sensors, organizations could retrofit existing camera-based robot systems instead of installing new sensing hardware.
    • In some settings, storing compact latent or geometric representations rather than raw video could reduce data-retention requirements.
    • Category: Long-Term Application.
    • Dependencies: Latent representations may still encode sensitive information and should not automatically be treated as anonymous. Privacy guarantees, access controls, retention policies, and audits would be necessary.
  • Transfer to other embodied domains — Agriculture, energy, and infrastructure
    • A Flex-$-like architecture could potentially control systems that manipulate tools, components, crops, cables, or inspection equipment while reasoning about geometry and object identity.
    • Examples include greenhouse harvesting, solar-panel maintenance, cable routing, and infrastructure repair.
    • Category: Long-Term Application.
    • Dependencies: These domains differ substantially from the bimanual tabletop tasks studied in the paper. New pretraining data, environmental sensing, force interaction, weather robustness, and domain-specific safety validation would be required.
    • View Paper Prompt  View All Prompts 

    Glossary

    • Action chunk: A sequence of actions predicted and executed as a unit rather than one action at a time. “Note that ata_t corresponds to an action chunk at:t+Ha_{t:t+H}”
    • Action expert: A specialized, narrower network component that predicts robot actions using representations from the main visual backbone. “A smaller Action Expert cross-attends to visual streams to produce an action chunk at:t+Ha_{t:t+H}.”
    • Attention masking: Restricting which tokens or modalities may exchange information during transformer processing. “We drop both visual inputs and outputs via attention masking”
    • Binary success: The percentage of trials in which the entire task is completed, without partial credit. “binary success, the percentage of rollouts which solve the entire task”
    • Bimanual manipulation: Robotic manipulation performed using two coordinated robotic arms or hands. “a bimanual YAM robot evaluated on five dexterous tasks”
    • Causal joint generation: Generating multiple outputs together while enforcing a directed information flow between them. “Causal Joint Generation.”
    • Cross-attention: An attention mechanism in which tokens from one stream use information from tokens in another stream. “A smaller Action Expert cross-attends to visual streams to produce an action chunk at:t+Ha_{t:t+H}.”
    • Cross-modality forcing: Training a model to generate a modality even when that modality is absent from its input. “We call this cross-modality forcing”
    • Cross-modal fusion: Combining representations from different input modalities into a shared computational representation. “cross-modal fusion happens in the trunk.”
    • Demonstration efficiency: The ability to achieve high performance using relatively few training demonstrations. “Flex-$ shows strong demonstration efficiency”</li> <li><strong>Denoising</strong>: Predicting a clean data representation from a corrupted or noise-perturbed version. “every future stream is always denoised and incurs its flow-matching loss”</li> <li><strong>Depth Anything 3</strong>: A pretrained model used to estimate depth and construct 3D pointmaps from RGB images. “Pointmaps and object semantics are both derived from the same RGB image---via Depth Anything 3”</li> <li><strong>Diffusion transformer (DiT)</strong>: A transformer architecture trained to model diffusion or flow-based generative processes. “with a flow matching~\citep{lipman2023flow,liu2023flow} diffusion transormer~\citep{peebles2022DiT} model”</li> <li><strong>DINO features</strong>: Visual representations produced by a DINO self-supervised vision encoder, used here to represent object-level semantics. “DINO features $d_t$ from a DINO-v3~\citep{dinov3} encoder”</li> <li><strong>Domain randomization</strong>: Varying environmental properties during training to improve robustness to visual or physical changes. “Success rate (\%) over 50 tasks. Training: $2{,}500clean clean + 25{,}000randomizeddemos.”</li><li><strong>Domainshift</strong>:Achangebetweenthedatadistributionusedfortrainingandthatencounteredduringevaluationordeployment.“Flex− randomized demos.”</li> <li><strong>Domain shift</strong>: A change between the data distribution used for training and that encountered during evaluation or deployment. “Flex- is robust to visual distribution shift.”
    • Dexterous manipulation: Fine-grained manipulation requiring precise control of multiple joints, fingers, or contact interactions. “dexterous, precise, real-world bimanual manipulation tasks”
    • Euler steps: Discrete numerical integration steps used to approximate the solution of a differential equation. “At inference, zt+1z_{t+1} is generated by integrating $v_$ from τ=0\tau{=}0 to τ=1\tau{=}1 with KK Euler steps”
    • Flow matching: A generative-model training method that learns a velocity field transporting noise distributions toward data distributions. “Latent prediction of the ground truth next latent is performed with a flow matching”
    • Flow-matching loss: The objective that trains a model to predict the velocity along a prescribed noise-to-data trajectory. “every future stream is always denoised and incurs its flow-matching loss”
    • Flow ODE: An ordinary differential equation whose integration transforms an initial noise sample into a generated sample. “sampling from π\pi corresponds to integrating $v_$ along the flow ODE over active output streams.”
    • Frozen encoder: An encoder whose parameters remain fixed rather than being updated during training. “We use the frozen encoder and decoder from the VAE of the Wan-2.2-5B”
    • Generalization: The ability of a trained model to perform well on tasks, inputs, or conditions not seen during training. “Flex-$ shows strong demonstration efficiency, generalization, and deployment flexibility”
    • Image-shaped tensor: A multidimensional numerical array arranged with the spatial dimensions of an image, even if its values represent another modality. “the latent space for any image-shaped tensor”
    • Inference latency: The time required to produce a model prediction during deployment. “no inference latency is added.”
    • Latent space: A learned lower-dimensional representation in which data are encoded for modeling or generation. “the VAE directly reconstructs pointmaps despite being trained only on RGB pixels”
    • Mixture-of-Transformers (MoT): An architecture that shares transformer processing across modalities while using modality-specific components. “we employ a mixture-of-transformers (MoT)”
    • Multimodal policy learning: Learning robot control policies from multiple types of input or output information. “Multimodal Policy Learning.”
    • Object-centric representation: A representation organized around individual objects and their properties rather than only global image appearance. “DINO adds object-centric structure”
    • Out-of-distribution (OOD): Describing inputs or conditions that differ from the distribution used for training. “Real World OOD Generalization and Demo Efficiency”
    • Pareto frontier: The set of trade-off points where improving one objective requires worsening another. “used for the Pareto and data-scaling analyses”
    • Per-stream adapter: A modality-specific mapping that converts representations into or out of a shared model space. “the per-stream adapters that map each modality into and out of the shared trunk.”
    • Pointmap: An image-aligned array in which each pixel stores a 3D point, representing scene geometry. “3D information in the form of pointmaps pt∈RH,W,3p_t \in \mathbb{R}^{H, W, 3}”
    • Proprioception: Internal sensing of a robot’s joint positions, velocities, or other body states. “Proprioception & s_t & \triangleright \, \text{Robot joint states”
    • Representation learning: Learning useful data features or embeddings as part of model training rather than specifying them manually. “thereby learning better representations than action generation alone.”
    • Spatiotemporal prior: A learned expectation about spatial structure and how it changes over time. “they inherit strong spatiotemporal priors from video-generation backbones”
    • Stream dropout: Randomly removing modality-specific input or output streams during training to support flexible inference. “We achieve this with visual stream dropout and cross-modality forcing”
    • Token stream: An ordered sequence of vector representations corresponding to one modality. “Flex-embedseveryvisualmodalityintoasingle,sharedlatentspaceoftokenstreams”</li><li><strong>Trunk</strong>:Thesharedcentralportionofaneuralnetworkwhererepresentationsfromdifferentstreamsarejointlyprocessed.“cross−modalfusionhappensinthetrunk.”</li><li><strong>Variationalauto−encoder(VAE)</strong>:Agenerativeneuralnetworkthatencodesdataintoaprobabilisticlatentrepresentationanddecodesitbackintoreconstructeddata.“trainedaspartofavariationalauto−encoder(VAE)”</li><li><strong>Vision−language−actionmodel(VLA)</strong>:Arobotpolicythatusesvisualobservationsandlanguageinstructionstogenerateactions.“haverecentlyemergedasastrongalternativetovision−language−actionmodels(VLAs).”</li><li><strong>World−actionmodel(WAM)</strong>:Arobotpolicythatjointlypredictsfutureobservationsandactions.“Generalistrobotpoliciesthatjointlypredictfutureobservationsandactions−−−world−actionmodels(WAMs)”</li><li><strong>x−prediction</strong>:Adiffusionorflow−modelparameterizationthatpredictsthecleantargetrepresentationdirectlyinsteadofitsnoiseorvelocity.“ForDINOweuse‘‘ embeds every visual modality into a single, shared latent space of token streams”</li> <li><strong>Trunk</strong>: The shared central portion of a neural network where representations from different streams are jointly processed. “cross-modal fusion happens in the trunk.”</li> <li><strong>Variational auto-encoder (VAE)</strong>: A generative neural network that encodes data into a probabilistic latent representation and decodes it back into reconstructed data. “trained as part of a variational auto-encoder (VAE)”</li> <li><strong>Vision-language-action model (VLA)</strong>: A robot policy that uses visual observations and language instructions to generate actions. “have recently emerged as a strong alternative to vision-language-action models (VLAs).”</li> <li><strong>World-action model (WAM)</strong>: A robot policy that jointly predicts future observations and actions. “Generalist robot policies that jointly predict future observations and actions---world-action models (WAMs)”</li> <li><strong>x-prediction</strong>: A diffusion or flow-model parameterization that predicts the clean target representation directly instead of its noise or velocity. “For DINO we use ``x$-prediction''”
    View Paper Prompt  View All Prompts 

    Open Problems

    We haven't generated a list of open problems mentioned in this paper yet.

    Generate Now

    Tweets

    Sign up for free to view the 5 tweets with 108 likes about this paper.

    Sign Up for Free