Simple-to-Complex Structured Demonstrations for Vision-Language-Action Learning

Abstract: Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation by integrating visual perception, language understanding, and robot action generation. Existing research has primarily focused on improving model architectures, training strategies, and dataset scale, while little attention has been paid to how demonstrations are collected and organized. We identify demonstration organization as a fundamental yet overlooked aspect of imitation learning, as it directly affects policy learning efficiency, training stability, and policy generalization. To address this gap, we propose a simple-to-complex structured demonstration collection strategy for VLA learning using a dual-arm robotic platform. Our approach systematically organizes data through three general principles: (i) decomposing complex manipulation tasks into progressively learnable sub-skills, (ii) standardizing the interaction environment to reduce unnecessary variability, and (iii) organizing demonstrations according to progressively increasing task complexity. This structured design enables VLA models to first acquire fundamental manipulation skills before learning increasingly complex task compositions, facilitating more effective learning of long-horizon manipulation tasks. We evaluate the proposed strategy on two representative robotic manipulation tasks: block grasping and sorting, and towel folding. Experimental results show consistent improvements in task success rate and training stability compared with the baseline method of directly collecting end-to-end complete task trajectories. These findings highlight demonstration organization as a previously underexplored but important factor in VLA learning and provide practical insights into efficient skill acquisition, scalable dataset construction, and long-horizon robotic manipulation.

• The paper introduces Simple-to-Complex demonstration collection, combining capability decomposition, environment standardization, and progressive difficulty scheduling for VLA policies.
• The method improved block grasping and sorting from 0% to 80% success and enabled towel-folding completion at 25%, using the same model and training pipeline but 300 structured demonstrations instead of 200 direct ones.
• The results suggest demonstration organization is a major design factor, although small trial counts, task-specific manual staging, and unmatched demonstration quantities limit causal conclusions about structure versus data volume.

This paper investigates demonstration organization—how robotic demonstrations are collected and structured—as a design factor in Vision-Language-Action (VLA) learning, arguing that it has been largely overlooked relative to model architecture, training strategy, and dataset scale (2607.04591). The authors propose a Simple-to-Complex (S2C) structured demonstration collection strategy and evaluate it on two long-horizon dual-arm manipulation tasks, showing that demonstration structure alone—under an otherwise fixed model, training pipeline, and evaluation protocol—can determine whether a policy learns at all.

Motivation and problem statement

The central observation is that directly collected end-to-end task trajectories provide a suboptimal supervision signal for long-horizon manipulation under limited-data conditions. When a policy must jointly learn primitive motor skills, task-relevant perception, and temporal composition from the same small set of full-task trajectories, optimization becomes highly coupled and unstable. The authors demonstrate this concretely: with 200 direct demonstrations for block grasping and sorting, the trained policy exhibited severe inter-arm asymmetry (the left arm performed coarse grasping while the right arm consistently failed to initiate grasping, even when a block was manually inserted into its gripper), and for towel folding, the policy exhibited prolonged post-grasp hesitation and never completed the task. This positions the work against the dominant scaling narrative in VLA research (Open X-Embodiment, OpenVLA, $\pi_{0.5}$), noting that large-scale dataset construction is impractical for most laboratories and that real-world applications often operate with only hundreds of demonstrations.

The work also distinguishes itself from curriculum learning and task decomposition literature, which structures the training process over an already-fixed dataset; here, the collection process itself is structured.

The S2C strategy

S2C rests on three principles:

• Capability-level task decomposition: each task is decomposed into three prerequisite-ordered capability objectives—basic manipulation ($\mathcal{D}_{\text{manip}}$), object perception and state understanding ($\mathcal{D}_{\text{state}}$), and task execution ($\mathcal{D}_{\text{exec}}$)—mapped onto three cumulative demonstration stages $S_1 \subset S_2 \subset S_3$.
• Environment standardization: within each stage, variations irrelevant to the target capability (object colors, spatial layouts, initial-state diversity) are deliberately reduced so demonstrations provide focused supervision.
• Progressive complexity scheduling: new difficulty sources (multi-color scenes, dual-arm coordination, deformable-state variation) are introduced only after the preceding capability is established.

For block grasping and sorting, the stages are: (1) single-color grasping and transfer, collected independently per arm (80 demonstrations); (2) multi-color scenes requiring color-conditioned placement (160 demonstrations); (3) full dual-arm cooperative sorting in a shared workspace (60 demonstrations), totaling 300 demonstrations. For towel folding, the decomposition addresses deformable-object state uncertainty via an explicit state-normalization stage: initial-state handling from a controlled set of configurations (horizontal, vertical, slightly disordered), then a unified unfolding operation mapping diverse configurations to a standardized flat state $o_{\text{flat}}$, then rule-based folding executed exclusively from that normalized state. The state-transition formulation makes the key design choice explicit: by normalizing intermediate state, the variability propagated into the folding stage is substantially reduced.

Experimental results

All experiments use $\pi_{0.5}$ fine-tuned on a dual-arm SO-101 platform with four synchronized RGB cameras, with demonstration collection strategy as the only variable. Success rates on held-out task instances:

Task	Strategy	Demonstrations	Success rate
Block grasping and sorting	Direct	200	0% (0/5)
Block grasping and sorting	S2C	300	80.0% (4/5)
Towel folding	Direct	200	0% (0/28)
Towel folding	S2C	300	25.0% (7/28)

The block-task result is the strongest claim in the paper: a policy that completely fails under direct collection reaches 80% success purely through reorganized data collection. The towel-folding result is more modest but qualitatively important—the policy executes complete folding sequences in multiple trials rather than stalling post-grasp, indicating that state normalization makes long-horizon deformable-object learning tractable at all. A caveat stated plainly by the authors: the two conditions use different numbers of demonstrations (200 vs. 300), so the results evaluate the structured strategy as a whole rather than isolating data-quantity effects; no demonstration-matched ablation is provided.

Failure analysis

The authors categorize residual failures under S2C into grasp failure and wrong action. For the block task, the two failures split evenly (1 grasp, 1 wrong action), attributed to gripper precision and placement stability on the low-cost platform rather than to semantic misunderstanding. For towel folding, grasp failure dominates at 90.47% (19/21), driven largely by the difficulty of inserting a two-finger gripper under a towel lying flat on the tabletop; the remaining failures stem from unpredictable deformation amplifying small contact errors into out-of-distribution intermediate states. The authors' interpretation—that residual failures reflect hardware and execution-level error accumulation rather than failure of the structured collection strategy—is plausible given the failure taxonomy but rests on qualitative judgment; the small evaluation counts (5 and 28 trials) limit statistical confidence.

Limitations and open questions

The paper concedes several limitations. First, while S2C offers a general three-stage capability decomposition, each stage must still be instantiated manually for a given task; automatic stage instantiation and adaptive environment scheduling remain open. Second, the mechanism by which structured demonstrations improve learning is hypothesized to involve shared, progressively organized representations (drawing an analogy to embodiment-agnostic representations in $\pi_{0.5}$ with egocentric human data), but this is not verified—shared representations are learned implicitly, and the paper explicitly notes the model may capture statistical correlations without causal understanding of action–state–outcome dependencies. Third, the evaluation is limited to two tasks on a single low-cost platform with very small trial counts; whether the gains persist at higher demonstration volumes, on higher-precision hardware, or on tasks whose capability dependency structure differs from the manip–state–exec pattern is untested. Fourth, the demonstration-count mismatch between baseline and proposed conditions leaves the pure effect of organization versus added data unresolved.

Conclusion

This paper makes a targeted argument that demonstration organization is a first-order design variable in VLA imitation learning, comparable in importance to architecture and scale under limited-data regimes. The S2C strategy—task decomposition, environment standardization, and progressive complexity scheduling—turns two tasks that are unlearnable from direct end-to-end collection into learnable ones, most strikingly for rigid-object dual-arm sorting (0% to 80% success). The deformable-object results, while weaker, identify state normalization as a useful intermediate stage and expose hardware- and deformation-induced failures as the remaining bottleneck. The open questions—automated stage design, demonstration-matched ablations, and integration with causal world modeling to move beyond correlational policy learning—define a concrete agenda for extending structured demonstration collection beyond its current manual, task-specific instantiation (2607.04591).