AutoRT: Real-World Robotic Orchestration

• AutoRT is a scalable framework that orchestrates real-world robotic experiments using vision-language models for scene grounding and task synthesis.
• Its finite-state policy graph integrates exploration, instruction generation, and affordance filtering to enhance data diversity and ensure operational safety.
• The system demonstrated practical impact by collecting 77,000 episodes across varied settings, significantly advancing embodied foundation model training.

AutoRT refers to distinct systems and methodologies across several domains, principally: (1) a framework for large-scale real-world data orchestration for robotics (Ahn et al., 2024), (2) automated reinforcement learning-based red-teaming of LLMs (Liu et al., 3 Jan 2025), (3) automated REST API testing via reinforcement learning (Stennett et al., 15 Jan 2025), and (4) automated iterative radiotherapy planning at scale (AIRTP) (Gao et al., 21 Jan 2025). For clarity and relevance, the following article focuses exclusively on the prominent robotics orchestration framework, AutoRT (Ahn et al., 2024), whose technical implementation, deployment, and evaluation have established it as a reference system for embodied foundation model training.

AutoRT is a scalable system for orchestrating the deployment of physically embodied robotic agents, leveraging vision-LLMs (VLMs) and LLMs to automate instruction generation, scene grounding, safety enforcement, and data collection across heterogeneous real-world environments. It directly addresses the critical bottleneck of insufficient, low-diversity, and highly constrained data in real-robot learning by enabling the collection of large, highly diverse datasets—exemplified by 77,000 robot episodes across 20+ robots in multiple buildings with minimal human intervention.

1. System Overview and Motivation

AutoRT targets the data scarcity problem in real-world robotics, where most prior datasets are small, exhibit limited task and scene diversity, and are collected under constrained conditions. The system leverages the pre-trained knowledge encoded in foundation models—specifically VLMs for open-vocabulary scene understanding and LLMs for task synthesis and constraint enforcement—to drive autonomous fleet-scale experiments. AutoRT enables the deployment of robot fleets in previously unseen settings, generating rich, task-aligned, and operator-adjustable data for developing and fine-tuning embodied foundation models (Ahn et al., 2024).

The key objective is to exploit internet-scale, commonsense knowledge and advanced language grounding to automate environment exploration, human-aligned task creation, safe policy selection, and high-throughput data recording. By coupling these components, AutoRT produces datasets that are not only large in scale but also linguistically and visually diverse and representative of real-world operational variability.

2. System Architecture

AutoRT is structured as a finite-state policy graph executed on each robotic agent. Nodes in the graph correspond to critical operations: navigation/exploration, scene grounding using a VLM, instruction generation via an LLM, affordance filtering (policy assignment or rejection), policy execution, diversity scoring, and experiment reset. Control flow is governed by transition conditions β(v, state, data), where v is the current node.

• Exploration/Navigation: Robots maintain an open-vocabulary "natural language map" (NL map) constructed by embedding VLM detections within a dynamic SLAM-based 3D reconstruction. For exploration, each spatial cell’s relevance score is computed via

$$\mathrm{score}_i = \frac{\phi_i \cdot \phi_q - \min_j(\phi_j \cdot \phi_q)}{\max_j(\phi_j \cdot \phi_q) - \min_j(\phi_j \cdot \phi_q)}$$

where $\phi_i$ are detection embeddings and $\phi_q$ is the mean embedding of ~80 canonical object queries.

• Scene Grounding: Images from the robot’s onboard camera are passed to a VLM (e.g., PaLI, FlexCap), which outputs a scene description and a ranked object list.
• Instruction Generation: An LLM is prompted with the system role, the "Robot Constitution" (detailed rule sets), real-time VLM outputs, and a request to propose $N=10$ manipulation tasks.
• Affordance Filtering: The LLM classifies each task candidate into one of $k+1$ modes {scripted_pick, teleop, RT-2, reject}, providing a form of self-critique and constitutional compliance filtering.
• Collection Policies: At execution, the system samples among policies $$\{\pi^{\mathrm{teleop}}, \pi^{\mathrm{scripted}}, \pi^{\mathrm{RT-2}}\}$$, with their respective sampling probabilities dynamically annealed to balance autonomous throughput with human labor constraints.
• Diversity Scoring and Reset: Post-episode, an image-based diversity score is computed using a fine-tuned CLIP encoder and $k$-means clustering; resets (either autonomous or manual) enforce scene/task variety.

3. Methodologies

Scene and Task Grounding

AutoRT integrates VLM-based open-vocabulary object detection to map visual information into a unified semantic space. This enables precise language-driven referencing and goal targeting within previously unseen environments.

Task generation employs constitutional prompting where operational, safety, and embodiment rules are encoded as prompt constraints:

• Foundational Rules (F₁–F₃): Derived from modified Asimov’s laws.
• Safety Rules (S₁–S₃): Prohibit interactions with humans and hazardous objects.
• Embodiment Rules (E₁–E₂): Ensure tasks are physically feasible (e.g., weight, manipulation capabilities).
• Guidance Rule (G₁): Optionally enforces user-specified objectives.

The LLM samples diverse instructions through generation temperature and explicit requirements for task complexity and diversity.

Task and Policy Assignment

Tasks passing constitutional and affordance checks are executed via one of:

• Human-in-the-loop teleoperation,
• Scripted low-level manipulation routines,
• RT-2 vision-language-action model.

Sampling probabilities are constrained so that expected human supervision load per robot is bounded by available operator resources. Formally:

$$\mathbb{E}_{t,\pi}\left[C_{\mathrm{human}}(\pi)\right] \leq \frac{\text{available human slots}}{\text{\#robots}}$$

where $C_{\mathrm{human}}(\pi) = 1$ if teleop, $0$ otherwise.

Diversity Metrics

AutoRT directly quantifies dataset diversity:

• Language diversity: Mean pairwise L2 distance in Universal Sentence Encoder embedding space, achieving $\bar{d}_{\mathrm{lang}}=1.137$ (random baseline 1.414, prior best 1.073).
• Visual diversity: Minimum CLIP-image distance to cluster centers, with teleop data being most diverse.

4. Large-Scale Data Collection and Analysis

AutoRT enabled the collection of 77,000 episodes over 7 months, across four physical buildings and up to 20 concurrent robots. The dataset comprises 6,650 unique language instructions.

Policy-wise breakdown: | Policy | Episodes | Success Rate | |--------------|----------|--------------| | Scripted pick| 73,293 | 21% | | Teleop | 3,060 | 82% | | RT-2 | 936 | 4.7% |

A single human operator typically supervised 3–5 mobile robots, extending to 8 for stationary manipulators.

Instruction quality and guidance experiments showed that LLM-generated tasks (unguided/guided) were significantly more feasible (83% and 77% vs. 52%) and more relevant to user intent (61% guided vs. 27–28% baseline) than templated approaches.

Safety ablation measured in adversarial scenes demonstrated that constitutional prompting applied at both task-generation and affordance-filtering stages yielded 83–87% safe task proposals and 67–94% recall of unsafe tasks.

Co-fine-tuning downstream RT-1 policies on a 50/50 AutoRT/original data mix improved generalization: from 0% (RT-1 only) to 12.5% (AutoRT mix) for "pick from different heights" and from 10% to 30% for "wiping" tasks.

5. Safety and Autonomy Trade-Offs

Safety is managed via "constitutional prompting," where hard constraints are expressed as a soft cost function:

$C_{\text{safety}}(t,s) = \begin{cases} 0 & \text{if $t$ obeys F, S, E rules} \ +\infty & \text{otherwise} \end{cases}$

Affordance filtering is implemented by prompting an LLM to accept or reject tasks, with rejections accompanied by natural language justifications.

AutoRT explicitly exposes the autonomy-handover trade-off: policies are selected to maximize robotic throughput and dataset diversity without exceeding available human supervision. The autonomy constraint ensures practicality in scalable deployments.

6. Limitations and Future Directions

Notable limitations include:

• Throughput bottlenecked by the lowest-performing policy: RT-2 achieved only 4.7% success, constraining overall autonomy.
• Perception system failures (such as VLM hallucinations and motion blur) can propagate errors through the pipeline.
• High task and scene diversity result in limited example density per task, challenging standard behavioral cloning and RL algorithms for downstream policy learning.
• Constitutional prompting is not a formal safety guarantee; some unsafe tasks can bypass soft constraints, necessitating human oversight.

Planned directions involve:

• Improving autonomous success rates with more robust policies (e.g., RL-trained agents).
• Integrating active learning and uncertainty-based exploration to close the data-model improvement loop.
• Incorporating formal verification and sensor-level safety checks.
• Extending orchestration to more complex robots (e.g., bimanual, legged) and broader task domains.

7. Significance and Impact

AutoRT demonstrates that foundation models can be harnessed for scalable, real-world robotic data collection, enabling the training of embodied foundation models that were previously constrained by limited supervised data. Its modular orchestration—comprising VLMs for scene understanding, LLMs for compliant and diverse instruction synthesis, self-critical affordance filtering, and adaptive policy selection—establishes a reproducible and extensible framework for future embodied AI research. Experimental evidence shows AutoRT's datasets significantly surpass prior work in both visual and linguistic diversity, task feasibility, and operator scalability. These results suggest that the system's approach to autonomous yet safe data collection will be foundational for next-generation real-robot learning, generalization, and on-the-fly adaptability (Ahn et al., 2024).