---
title: 'AutoRT System: Modular Autonomy'
url: https://www.emergentmind.com/topics/autort-system
type: topic
---

# AutoRT System: Modular Autonomy

AutoRT denotes several systems in autonomous robotics, orchestration, formal verification, and medical planning, unified by emphasis on modular and automated reasoning about system constraints, safety, and operational objectives. Across domains—autonomous vehicles, robotic fleets, open-world manipulation, radiotherapy planning, and runtime assurance frameworks—AutoRT architectures commonly integrate rule-based or optimization-driven control, explicit interfaces for accountability, and strong extensibility to adapt to new constraints, environments, or data sources.

## 1. Modular Architectures and Core Subsystems

AutoRT systems are typically organized as modular pipelines layering domain-specific knowledge bases or constraint representations atop inference, planning, and orchestration mechanisms.

For autonomous road vehicles (“Rules of the Road Advisor” [2209.14035]), the system is partitioned into:

- **Rule Knowledge Base**: Prolog-encoded “road-rules” of the form `rule(Name, Context, Beliefs, Intentions, ActionPairs)`, allowing meta-contextualization, belief/intent grounding, and jurisdictional traceability.
- **Inference Engine**: Subset matching over context, beliefs, intentions to determine applicable rules.
- **Action Annotation**: Maps rules to “must” (hard constraint) and “should” (soft constraint) action pairs.
- **External Accountability**: Logging calls for auditability and post hoc compliance verification.
- **Jurisdictional Adaptor**: Loads rule-bases per region, preserving legal linkage.

In orchestration (“Embodied Foundation Models” [2401.12963]), the system is a finite-state “policy graph” controlling navigation, task generation, and execution, with subsystems for:

- Open-vocabulary scene mapping with VLM embeddings.
- Autonomous and teleoperation policy routing via LLM-reasoned affordance filtering.
- Real-time diversity and safety scoring, episodic data aggregation.

For runtime assurance (“Safe Autonomy Run Time Assurance” [2209.01120]), AutoRT is built around a core RTA module interface and JAX-automatic differentiation for constraints and dynamics:

```python
class RTAModule(ABC):
    @abstractmethod
    def filter(self, u_des: np.ndarray, x_sys: Any) -> np.ndarray:
        """Return u_act given u_des and system state x_sys."""
```

All differentiable primitives (barriers, dynamics, backup controllers) are JAX-compiled for high-performance online safety filtering.

## 2. Formal Representations and Constraint Handling

Rule–driven AutoRT variants employ first-order languages or Prolog facts. For instance, situations in road vehicle reasoning are defined as:

- $s = \langle C, B, I \rangle$, with context $C$ (e.g., “standard”), beliefs $B$, and intentions $I$, all drawn from a domain ontology.
- Applicability: rule $r$ applies if $beliefs(r) \subseteq beliefs(s)$, $intentions(r) \subseteq intentions(s)$, $context(r) = context(s)$.
- Actions are labeled “must” or “should” for distinction between legal constraint and recommendation.

For runtime assurance, constraints are implemented as scalar-valued barrier functions $h(x)$, composing the safe set $\mathcal C_S = \{x : h_i(x) \ge 0, i=1..M\}$ and evaluated via JAX auto-differentiation:

- Lie derivatives: $L_f h = \nabla h(x) f(x)$, $L_g h = \nabla h(x) g(x)$.
- Control barrier QPs: $\min_{u} \|u - u_{des}\|^2$ s.t. $BC_i(x, u) \ge 0$, with $BC_i(x, u) = L_f h_i(x) + L_g h_i(x) u + \alpha(h_i(x))$.

In data orchestration, diversity and safety are quantified using embedding distances:

- Navigation sampling: $\mathrm{score}_i = \left(\frac{\phi_i \cdot \phi_q - \min_j \phi_j \cdot \phi_q}{\max_j \phi_j \cdot \phi_q - \min_j \phi_j \cdot \phi_q}\right)^{\beta}$
- Visual diversity: $D_{vis}(\tau) = \min_{c \in C} \|v(\tau) - c\|_2$
- Language diversity: $\bar D_{lang} = \frac{2}{n(n-1)} \sum_{i<j} \|u_i - u_j\|_2$

## 3. Applications: Autonomous Vehicles, Robotic Fleets, and Medical RT Planning

AutoRT frameworks support:

- **Autonomous road vehicles** ([2209.14035]): Integration with AV control cycles, converting sensor-extracted beliefs and intentions into actionable constraint sets; “must” actions become non-negotiable constraints, “should” actions integrated as weighted preferences.
- **Robotic Fleet Orchestration** ([2401.12963]): VLM-driven exploration, LLM-generated task diversification, multicentric manipulation episode collection across diverse real-world scenes.
- **Open-world Transportation Manipulation** ([1810.03400]): Navigation, grasp planning, and finite-state task execution for outdoor pick-and-deliver with modular integration of SLAM, grasp detection, and human-in-the-loop oversight.
- **Automated Iterative RT Planning** ([2501.11803]): Full-stack pipeline for OAR segmentation, helper structure generation, beam geometry selection (IMRT/VMAT), convex QP dose optimization, and deliverable plan conversion for large-scale radiotherapy data generation.

Table: Representative Domains and AutoRT Implementations

| Domain                          | System Representation                                       | Key Subsystems                                                         |
|----------------------------------|-------------------------------------------------------------|------------------------------------------------------------------------|
| Autonomous Vehicles              | Prolog rule knowledge base, action advisor (RoTRA)          | Rule base, inference engine, annotation, audit interface, jurisdiction adaptor |
| Robotic Fleet Orchestration      | VLM/LLM-guided tasking and diversity scoring                | Policy graph, scene mapping, affordance filter, data collection         |
| Runtime Assurance                | JAX-based constraint QPs, Simplex/ASIF filters              | Constraint manager, dynamics adapter, backup controller, optimization   |
| Radiotherapy Planning            | Modular AI pipeline for plan generation (AIRTP)             | Segmentation, structure creation, geometry setup, QP optimizer          |
| Open-World Manipulation          | Integrated mobile manipulation pipeline (AutOTranS)          | Navigation stack, grasping stack, FSM task planner                      |

## 4. Experimental Results, Metrics, and Performance

Metrics, latency, and empirical evaluation are central in AutoRT systems.

- **Autonomous Car Simulation** ([2209.14035]): “Reactive” agents (obeying all “must” and “should”) pass canonical road scenarios safely; “Morally Bankrupt” agents (obeying only “must”) fail virtual driving tests on overtake and turn tasks. Query latency with ~300 rules <1ms.
- **Orchestration Diversity** ([2401.12963]): 77,000 episodes with 53 robots; average pairwise language diversity (USE space) up to 1.137 (FlexCap); visual diversity histogram median up to 0.55 for AutoRT teleop dataset.
- **Runtime Assurance** ([2209.01120]): Multi-agent spacecraft simulation, zero constraint violations, average ~0.7ms/step for explicit ASIF filter, high trajectory fidelity.
- **Medical Planning** ([2501.11803]): Plan generation time 0.3–1h; scorecard-based plan quality improved +10–20 points; DVH reproducibility deviations <2%.

## 5. Extensibility and Jurisdictional Adaptation

AutoRT enables adaptation across operational environments and regulatory contexts:

- **Rule sets** ([2209.14035]): Simple loading of new Prolog rule-bases allows legal adaptation; semantic vocabulary is derived from source legal text for expert traceability; consistency and conflict among “must” rules can be formally checked.
- **Constraint extension** ([2209.01120]): Users define arbitrary system dynamics and safety barrier constraints in JAX; supports batch-differentiated, high-order barrier function composition.
- **Interoperable orchestration** ([2401.12963]): Mixes scripted, teleop, and autonomous execution modes; adaptable policy-sampling probabilities; constitutional prompting for safety specification per context.

## 6. Limitations and Prospects

AutoRT systems encounter several practical and theoretical limitations:

- In AV and manipulation domains, reliance on structured sensors (2D LiDAR, monocular depth) inhibits terrain generality and detection of low-profile objects ([1810.03400]).
- Rule-based legal advisors require comprehensive ontology mapping for cross-jurisdictional robustness ([2209.14035]).
- For RTA, model mismatch, infeasible QPs due to tight barriers, and system identification errors may erode safety margins ([2209.01120]).
- In large-scale orchestration, data sparsity and foundation model hallucinations limit downstream RL generalization ([2401.12963]); constitutional prompting improves but does not fully guarantee task safety (filtered unsafe → safe task recall up to 94% at both prompt stages).
- In RT planning, patient-specific protocol variation is unaddressed, necessitating manual review for edge cases ([2501.11803]).

Planned extensions include institution-specific scorecard incorporation (AIRTP [2501.11803]), dynamic adaptation of robot constitutions ([2401.12963]), closure between data collection and model improvement, and integration of stronger physics-based grounding in embodied foundation models.

## 7. Accountability, Verification, and Data Provenance

Across applications, AutoRT architectures enforce clear separation between policy-generation and operational code, maintaining logs and formal representations for external verification:

- All advisory calls (context, beliefs, intentions → actions) are logged for audit and can be replayed for post hoc compliance review without exposing proprietary AV or robotic controller code ([2209.14035]).
- External regulators or monitors can independently verify rule adherence in given situations; each rule/action is traceable to its legal or clinical source ([2209.14035], [2501.11803]).
- In data collection, human oversight and constitutional filtering ensure alignment to safety and operational guidelines; episodic scoring and sampling probabilities are exposed for transparency ([2401.12963]).

*This unification of modular knowledge representation, high-performance optimization, external traceability, and flexible adaptation positions AutoRT and its derivatives as principal responders to the increasing demand for certifiable autonomy, explicit safety, and scalable data-driven deployment in diverse robotic and cyber-physical systems.*

Source: https://www.emergentmind.com/topics/autort-system