- The paper introduces LILAC, which combines MLLM-generated visual motion prompts, semantic alignment, and object-centric optical flow to produce full 6-DoF manipulation trajectories in one forward pass.
- LILAC achieves the best flow results on Fractal and BridgeData V2, reducing average displacement error to 26.98 and 29.44 while improving physical task success by 14 percentage points over the strongest baseline.
- The approach uses only 20 robot demonstrations per task for fine-tuning and runs about 1.4× faster than π₀, but remains limited by static-scene assumptions, 2D flow, and errors in target visual prompts.
Motivation and problem setting
LILAC addresses language-conditioned robotic manipulation by recasting planning as a flow-based prediction problem. Given a single pre-manipulation RGB image and a natural language instruction, the model predicts an object-centric 2D optical-flow field and converts it into a 6-DoF manipulator trajectory. This formulation is attractive because pixel-space motion provides dense, embodiment-independent supervision, enabling training on human and web videos and reducing reliance on costly embodiment-specific demonstrations—a notable contrast to VLAs such as π0, which required roughly 10,000 hours of demonstrations for pretraining, and to OpenVLA, which the paper cites as having near-0% zero-shot success.
The authors identify two limitations of prior multimodal flow-based methods such as Track2Act, FLIP, and Im2Flow2Act. First, models trained on closed-domain datasets often generate visually plausible motions that do not align with the instruction. Second, closed-loop schemes re-encode the instruction and current image at every step, inflating inference cost, accumulating errors, and requiring expert data covering many intermediate states. LILAC instead generates the full trajectory offline in a single forward pass.
Architecture
LILAC comprises a flow generation module and an Action De-Tokenizer. In the flow generation module, a multimodal LLM (GPT-4o, chosen over Qwen-2.5-VL after qualitative comparison) produces a displacement vector and target bounding box from the image and instruction. The displacement vector is rendered as an arrow overlaid on the RGB frame, forming a visual prompt—an intermediate, coarse motion sketch that bridges the granularity gap between high-level language and dense flow.
A Prompt-Conditioned Multi-Modal Adapter (a transformer encoder) fuses the RGB image, instruction, and rendered visual prompt into a joint latent representation. A transformer-based Semantic Alignment Decoder then autoregressively generates the 2D flow conditioned on this latent, with the final sequence token acting as a CLS token aligned to the CLIP instruction embedding.
The Semantic Alignment Loss (Lsem) is an L1 loss aligning this CLS token with the instruction embedding, explicitly countering the failure mode in which models over-rely on visual context and ignore the language. Flow coordinates are trained with a cross-entropy loss over discretized W and H classes rather than regression, which the authors argue shrinks the solution space and improves convergence with fewer samples. The Action De-Tokenizer, adapted from Track2Act, first estimates a coarse 6-DoF trajectory via rigid transformation from the flow, depth, and camera parameters, then refines it with an autoregressive transformer decoder conditioned on text and image—replacing Track2Act's diffusion head for faster convergence. Notably, depth is excluded from flow generation to preserve compatibility with RGB-only web video datasets.
Flow generation results
The authors construct the Robot Flow benchmark from Fractal (26,516 episodes) and BridgeData V2 (10,672 episodes), using Qwen-2.5-VL for bounding-box generation and CoTracker-3 for point tracking to derive ground-truth flows, with horizon H=8. LILAC outperformed Im2Flow2Act and FLIP across all metrics on both subsets:
| Method |
Fractal ADE↓ |
Fractal AUC↑ |
Bridge ADE↓ |
Bridge AUC↑ |
| Im2Flow2Act |
56.46 |
0.190 |
63.19 |
0.177 |
| FLIP |
44.41 |
0.165 |
41.95 |
0.200 |
| LILAC (full) |
26.98 |
0.434 |
29.44 |
0.396 |
Against the strongest baseline, LILAC reduced ADE by 17.43 points and improved AUC by 0.244 on Fractal, and reduced ADE by 12.51 points on BridgeData V2. Ablations show both components contribute: removing the Semantic Alignment Loss raises ADE to 28.94/31.14, while removing the visual prompt degrades it to 33.65/37.09—drops of 6.67 and 7.65 points—making the visual prompt the largest single factor.
Physical experiments
Physical evaluation used a Toyota Human Support Robot (HSR) on five mobile manipulation tasks (Mobile Replacing, Moving, Pouring, Tissue Picking, Drawer Opening), with 20 trials per task over held-out objects, free-form instructions, and unseen scene layouts. Training data consisted of 100 human-hand episodes per task plus only 20 robot episodes per task for embodiment fine-tuning. For fairness, all flow-based baselines shared LILAC's Action De-Tokenizer, and the pre-grasp trajectory was heuristic for all methods.
LILAC achieved the highest success rate on every task, exceeding FLIP—the best baseline—by 14 points on average. π0 performed poorly, which the authors attribute to its table-top, fixed-viewpoint pretraining being mismatched with whole-body mobile manipulation involving large viewpoint changes. On inference speed, LILAC ran at 376.6 ms/action, roughly 1.4× faster than π0 (541.0 ms), though slower than FLIP (21.6 ms). A trajectory deviation analysis on a Mobile Picking episode showed π0 drifting farther from the human reference than LILAC, supporting the claim that open-loop generation is more robust to compounding error in this setting. The authors note the caveat that the pre-grasp segment was heuristic and the trajectory represents post-grasp motion only.
Error analysis of visual prompts
A human evaluation of 100 sampled outputs identified 37 visual-prompt failures, categorized into six types. Endpoint-related errors dominated: incorrect end points (12), overly short prompts (6), incorrect start points (5), invalid prompts (5), reversed direction (5), and no prompt generated (4). This indicates that specifying target locations remains the primary weakness of the MLLM-based prompting stage, and that such errors can propagate directly into the generated flow.
Limitations and open questions
The authors concede four limitations. First, the static scene assumption: open-loop generation cannot react to unexpected scene changes, and dynamic environments would require closed-loop replanning or residual control. Second, limited expressiveness of 2D flow: tasks demanding depth reasoning or out-of-plane tool motion (e.g., stirring, obstacle-aware placing) are difficult to encode in the image plane; the failure analysis attributes one physical-execution failure partly to the de-tokenizer's difficulty inferring depth from 2D flow. Third, sensitivity to visual-prompt errors, which propagate to downstream flow generation. Fourth, failures in visually and spatially complex scenes with multiple objects. The proposed remedy—injecting pseudo-depth from monocular estimators such as Depth Anything—is sketched but not evaluated, leaving open whether depth-aware flow prediction preserves the RGB-only scalability advantage.
Conclusion
LILAC demonstrates that visual prompts as intermediate motion sketches, combined with an explicit semantic alignment objective, substantially improve instruction-flow correspondence in open-loop manipulation trajectory generation. It achieves the best reported flow quality on the Robot Flow benchmark and a 14-point average success-rate gain over the strongest physical-experiment baseline using only 20 embodiment-specific demonstrations per task. Its principal open questions concern robustness to dynamic scenes, depth-aware motion representation, and the reliability of MLLM-generated visual prompts in cluttered multi-object environments.
