SOP: A Scalable Online Post-Training System for Vision-Language-Action Models (2601.03044v1)

Abstract: Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality. Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific, limiting effective on-policy adaptation and scalable learning from real-world interaction. We introduce a Scalable Online Post-training (SOP) system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world. SOP tightly couples execution and learning through a closed-loop architecture in which a fleet of robots continuously streams on-policy experience and human intervention signals to a centralized cloud learner, and asynchronously receives updated policies. This design supports prompt on-policy correction, scales experience collection through parallel deployment, and preserves generality during adaptation. SOP is agnostic to the choice of post-training algorithm; we instantiate it with both interactive imitation learning (HG-DAgger) and reinforcement learning (RECAP). Across a range of real-world manipulation tasks including cloth folding, box assembly, and grocery restocking, we show that SOP substantially improves the performance of large pretrained VLA models while maintaining a single shared policy across tasks. Effective post-training can be achieved within hours of real-world interaction, and performance scales near-linearly with the number of robots in the fleet. These results suggest that tightly coupling online learning with fleet-scale deployment is instrumental to enabling efficient, reliable, and scalable post-training of generalist robot policies in the physical world.

• The paper introduces SOP, a system that integrates online data collection with centralized post-training to enhance the robustness of VLA models.
• It employs algorithm-agnostic modules such as HG-DAgger and RECAP to achieve significant throughput gains and improved success rates across challenging tasks.
• Empirical results demonstrate near-linear fleet scaling, with rapid policy adaptation and marked improvements in multi-task performance for real-world robotic applications.

A Scalable Online Post-Training System for Vision-Language-Action Models: Technical Summary and Analysis

Motivation and Problem Setting

The practical deployment of generalist robot policies hinges not only on broad task generalization—achievable via large-scale pretraining of Vision-Language-Action (VLA) models—but also on acquiring expert-level proficiency and robustness in real-world, multi-task scenarios. Conventional post-training paradigms for VLA models are hindered by their offline, single-robot, or task-specific nature. These settings inherently decouple experience collection from timely on-policy policy improvement, yielding compounding distribution shift and suboptimal exploitation of fleet-scale interaction data.

The presented Scalable Online Post-training (SOP) system (2601.03044) directly addresses these limitations in the context of multi-task, embodied robot learning. SOP integrates distributed online data collection, low-latency actor-learner synchronization, and algorithmic flexibility—enabling efficient, generalist post-training at fleet scale in physical environments.

Figure 1: Overview of SOP, depicting fleet-driven online post-training of VLA models, with bidirectional data and parameter streaming to a centralized learner.

SOP System Architecture and Methodology

SOP operationalizes a tightly coupled closed-loop between deployment and learning. The architecture comprises distributed robot actors collecting on-policy experience across diverse tasks, asynchronously uploading interaction data—and optional human interventions—to a centralized cloud learner. The cloud learner mixes this online buffer with a static offline dataset, applies a pluggable post-training module (e.g., HG-DAgger or RECAP), and streams updated control policies back to the robot actors, closing the data–model feedback loop.

Figure 2: Detailed architecture of SOP as an actor–learner framework, emphasizing online, multi-task adaptation with human-in-the-loop correction.

The system supports:

• Algorithmic agnosticism: The post-training module $\mathcal{G}$ can be instantiated as (interactive) imitation learning (e.g., HG-DAgger) or RL (e.g., RECAP).
• Task-level balance: Adaptive sampling ensures uniform coverage over tasks while prioritizing high-loss, online data for faster adaptation under distribution shift.
• Near-linear scaling: Policy improvement efficiency scales with the number of deployed robots, contingent on the communication and computational throughput of the centralized learner.

  Figure 3: The dual-arm robotic platform used in evaluation, representative of high-DoF, vision-language-commanded manipulators.

Task Suite and Experimental Protocol

The evaluation spans three challenging manipulation domains, each requiring both dexterous skills and semantic task comprehension:

• Grocery Restocking: Includes object retrieval, placement, and handling across 500+ objects and diverse shelf geometries, with explicit semantic disambiguation.
• Laundry Folding: Long-horizon, deformable object manipulation with sequential bimanual folds.
• Box Assembly: Multi-step transformation of flat cardboard into 3D structures, requiring precise coordination.

  Figure 4: Visualizations of the three task families—Grocery Restocking (A), Laundry Folding (B), Box Assembly (C)—demonstrating breadth of embodiment and manipulation complexity.

Success metrics are reported as policy-side episode completion rate and task throughput (episodes/hour), deliberately excluding human reset overhead to isolate policy efficacy and real time performance.

Empirical Results

Multi-task Policy Improvement

SOP, instantiated with HG-DAgger and RECAP, yields substantial post-training improvements over pretrained baselines on all tasks. The most pronounced gains are observed with SOP+HG-DAgger, which achieves success rates of 0.94–0.98 across the three task families—significantly surpassing both offline learning and SOP+RECAP, especially in tasks with heavy semantic generalization requirements.

Figure 5: Comparative analysis of success rate and throughput across approaches—SOP variants demonstrably outperform offline and non-SOP baselines across all domains.

Numerical performance exhibits a 2–4× gain in throughput for SOP, as on-policy corrections directly mitigate dominant policy failure modes (e.g., grasp errors in folding).

Fleet Scaling Properties

SOP demonstrates near-linear improvements in both final success rate and wall-clock data efficiency as the number of robots increases. For instance, quadrupling actor fleet size from one to four reduces time-to-target (reaching 0.8 success rate) from 173.6 to 71.7 minutes, with corresponding increases in final policy proficiency. These results demonstrate SOP’s capacity to effectively translate robot parallelism into learning acceleration without saturating central bottlenecks in the evaluated operational regime.

Interaction With Pretraining Quality

SOP’s online adaptation consistently improves over diverse model initializations. However, larger scale and diversity in pretraining data translate to higher post-training asymptotes—the effect of on-policy adaptation is complementary to, not a replacement for, foundational visual-language grounding. Crucially, SOP yields an order-of-magnitude more effective performance gain per operational hour than further scaling static offline datasets, reflecting the importance of learning from the true policy-induced state distribution.

Figure 6: Effect of varying offline pretraining dataset size on SOP post-training. Both initial and asymptotic success rates increase with pretraining data scale, but SOP’s online phase closes the residual proficiency gap more efficiently than additional offline finetuning.

System-Level and Algorithmic Considerations

SOP’s infrastructure enables robust operation in real cloud/fleet environments:

• Elastic horizontal scaling: Adding robots incurs no architectural modification.
• Fault-tolerant data management: All episodic data is durably persisted, and metadata is decoupled from payload to permit million-scale episode replay.
• Flexible temporal coupling: Model broadcast and rollout synchronization are optimized for minimal staleness, and actors update policies only between episodes to ensure trajectory consistency.

The adaptive sampling module preferentially rebalances toward high-loss (out-of-distribution) online samples, ensuring rapid focus on deployment-induced error states while retaining task-wide coverage.

Theoretical and Practical Implications

SOP demonstrates empirically that real-world scalable robot learning is as much a systems engineering challenge as it is algorithmic. Closed-loop integration of fleet-scale, on-policy data with low-latency model relabeling is required for reliable adaptation—mirroring lessons from large-scale RL and LLM post-training.

Practically, efficient adaptation from hours of robot interaction makes such architectures viable for real deployment, transforming fleet scaling into a direct driver of efficiency and robustness—each additional agent contributing both data diversity and instantaneous coverage of novel environment states.

The system’s architectural agnosticism to choice of post-training algorithm suggests extensibility to future methods, including reward modeling or continual learning paradigms. However, reducing reliance on human interventions and mitigating catastrophic forgetting in continual skill acquisition remain essential open challenges.

Conclusion

SOP (2601.03044) represents a significant step toward scalable, reliable, and generalizable post-training of large VLA models for robotic fleets. By resolving system-level constraints around data collection, experience replay, and low-latency learning, SOP unlocks strong multi-task proficiency, proportional fleet scaling, and algorithmic flexibility. The framework’s performance envelope and architectural choices delineate key design principles for future scalable embodied AI systems, with promising implications for persistent policy evolution, compositional skill learning, and the practical deployment of large generalist robot models.