AutoRT System: Modular Autonomy

• AutoRT is a unified system architecture that integrates modular pipelines for rule-based reasoning, inference, and optimization across domains like autonomous vehicles, robotic fleets, and medical RT planning.
• It employs specialized subsystems such as Prolog-encoded rule bases, JAX-compiled barrier functions, and finite-state policy graphs to ensure safety, accountability, and adaptability in dynamic environments.
• The framework demonstrates high performance with sub-millisecond response times and quantifiable safety metrics while allowing extensible rule sets and jurisdiction-specific adaptations.

AutoRT denotes several systems in autonomous robotics, orchestration, formal verification, and medical planning, unified by emphasis on modular and automated reasoning about system constraints, safety, and operational objectives. Across domains—autonomous vehicles, robotic fleets, open-world manipulation, radiotherapy planning, and runtime assurance frameworks—AutoRT architectures commonly integrate rule-based or optimization-driven control, explicit interfaces for accountability, and strong extensibility to adapt to new constraints, environments, or data sources.

1. Modular Architectures and Core Subsystems

AutoRT systems are typically organized as modular pipelines layering domain-specific knowledge bases or constraint representations atop inference, planning, and orchestration mechanisms.

For autonomous road vehicles (“Rules of the Road Advisor” (Collenette et al., 2022)), the system is partitioned into:

• Rule Knowledge Base: Prolog-encoded “road-rules” of the form rule(Name, Context, Beliefs, Intentions, ActionPairs), allowing meta-contextualization, belief/intent grounding, and jurisdictional traceability.
• Inference Engine: Subset matching over context, beliefs, intentions to determine applicable rules.
• Action Annotation: Maps rules to “must” (hard constraint) and “should” (soft constraint) action pairs.
• External Accountability: Logging calls for auditability and post hoc compliance verification.
• Jurisdictional Adaptor: Loads rule-bases per region, preserving legal linkage.

In orchestration (“Embodied Foundation Models” (Ahn et al., 2024)), the system is a finite-state “policy graph” controlling navigation, task generation, and execution, with subsystems for:

• Open-vocabulary scene mapping with VLM embeddings.
• Autonomous and teleoperation policy routing via LLM-reasoned affordance filtering.
• Real-time diversity and safety scoring, episodic data aggregation.

For runtime assurance (“Safe Autonomy Run Time Assurance” (Ravaioli et al., 2022)), AutoRT is built around a core RTA module interface and JAX-automatic differentiation for constraints and dynamics:

class RTAModule(ABC): @abstractmethod def filter(self, u_des: np.ndarray, x_sys: Any) -> np.ndarray: """Return u_act given u_des and system state x_sys."""

All differentiable primitives (barriers, dynamics, backup controllers) are JAX-compiled for high-performance online safety filtering.

2. Formal Representations and Constraint Handling

Rule–driven AutoRT variants employ first-order languages or Prolog facts. For instance, situations in road vehicle reasoning are defined as:

• $s = \langle C, B, I \rangle$, with context $C$ (e.g., “standard”), beliefs $B$, and intentions $I$, all drawn from a domain ontology.
• Applicability: rule $r$ applies if $beliefs(r) \subseteq beliefs(s)$, $intentions(r) \subseteq intentions(s)$, $context(r) = context(s)$.
• Actions are labeled “must” or “should” for distinction between legal constraint and recommendation.

For runtime assurance, constraints are implemented as scalar-valued barrier functions $h(x)$, composing the safe set $\mathcal C_S = \{x : h_i(x) \ge 0, i=1..M\}$ and evaluated via JAX auto-differentiation:

• Lie derivatives: $L_f h = \nabla h(x) f(x)$, $L_g h = \nabla h(x) g(x)$.
• Control barrier QPs: $\min_{u} \|u - u_{des}\|^2$ s.t. $BC_i(x, u) \ge 0$, with $$BC_i(x, u) = L_f h_i(x) + L_g h_i(x) u + \alpha(h_i(x))$$.

In data orchestration, diversity and safety are quantified using embedding distances:

• Navigation sampling: $$\mathrm{score}_i = \left(\frac{\phi_i \cdot \phi_q - \min_j \phi_j \cdot \phi_q}{\max_j \phi_j \cdot \phi_q - \min_j \phi_j \cdot \phi_q}\right)^{\beta}$$
• Visual diversity: $D_{vis}(\tau) = \min_{c \in C} \|v(\tau) - c\|_2$
• Language diversity: $$\bar D_{lang} = \frac{2}{n(n-1)} \sum_{i<j} \|u_i - u_j\|_2$$

3. Applications: Autonomous Vehicles, Robotic Fleets, and Medical RT Planning

AutoRT frameworks support:

• Autonomous road vehicles (Collenette et al., 2022): Integration with AV control cycles, converting sensor-extracted beliefs and intentions into actionable constraint sets; “must” actions become non-negotiable constraints, “should” actions integrated as weighted preferences.
• Robotic Fleet Orchestration (Ahn et al., 2024): VLM-driven exploration, LLM-generated task diversification, multicentric manipulation episode collection across diverse real-world scenes.
• Open-world Transportation Manipulation (Zapata-Impata et al., 2018): Navigation, grasp planning, and finite-state task execution for outdoor pick-and-deliver with modular integration of SLAM, grasp detection, and human-in-the-loop oversight.
• Automated Iterative RT Planning (Gao et al., 21 Jan 2025): Full-stack pipeline for OAR segmentation, helper structure generation, beam geometry selection (IMRT/VMAT), convex QP dose optimization, and deliverable plan conversion for large-scale radiotherapy data generation.

Table: Representative Domains and AutoRT Implementations

Domain	System Representation	Key Subsystems
Autonomous Vehicles	Prolog rule knowledge base, action advisor (RoTRA)	Rule base, inference engine, annotation, audit interface, jurisdiction adaptor
Robotic Fleet Orchestration	VLM/LLM-guided tasking and diversity scoring	Policy graph, scene mapping, affordance filter, data collection
Runtime Assurance	JAX-based constraint QPs, Simplex/ASIF filters	Constraint manager, dynamics adapter, backup controller, optimization
Radiotherapy Planning	Modular AI pipeline for plan generation (AIRTP)	Segmentation, structure creation, geometry setup, QP optimizer
Open-World Manipulation	Integrated mobile manipulation pipeline (AutOTranS)	Navigation stack, grasping stack, FSM task planner

4. Experimental Results, Metrics, and Performance

Metrics, latency, and empirical evaluation are central in AutoRT systems.

• Autonomous Car Simulation (Collenette et al., 2022): “Reactive” agents (obeying all “must” and “should”) pass canonical road scenarios safely; “Morally Bankrupt” agents (obeying only “must”) fail virtual driving tests on overtake and turn tasks. Query latency with ~300 rules <1ms.
• Orchestration Diversity (Ahn et al., 2024): 77,000 episodes with 53 robots; average pairwise language diversity (USE space) up to 1.137 (FlexCap); visual diversity histogram median up to 0.55 for AutoRT teleop dataset.
• Runtime Assurance (Ravaioli et al., 2022): Multi-agent spacecraft simulation, zero constraint violations, average ~0.7ms/step for explicit ASIF filter, high trajectory fidelity.
• Medical Planning (Gao et al., 21 Jan 2025): Plan generation time 0.3–1h; scorecard-based plan quality improved +10–20 points; DVH reproducibility deviations <2%.

5. Extensibility and Jurisdictional Adaptation

AutoRT enables adaptation across operational environments and regulatory contexts:

• Rule sets (Collenette et al., 2022): Simple loading of new Prolog rule-bases allows legal adaptation; semantic vocabulary is derived from source legal text for expert traceability; consistency and conflict among “must” rules can be formally checked.
• Constraint extension (Ravaioli et al., 2022): Users define arbitrary system dynamics and safety barrier constraints in JAX; supports batch-differentiated, high-order barrier function composition.
• Interoperable orchestration (Ahn et al., 2024): Mixes scripted, teleop, and autonomous execution modes; adaptable policy-sampling probabilities; constitutional prompting for safety specification per context.

6. Limitations and Prospects

AutoRT systems encounter several practical and theoretical limitations:

• In AV and manipulation domains, reliance on structured sensors (2D LiDAR, monocular depth) inhibits terrain generality and detection of low-profile objects (Zapata-Impata et al., 2018).
• Rule-based legal advisors require comprehensive ontology mapping for cross-jurisdictional robustness (Collenette et al., 2022).
• For RTA, model mismatch, infeasible QPs due to tight barriers, and system identification errors may erode safety margins (Ravaioli et al., 2022).
• In large-scale orchestration, data sparsity and foundation model hallucinations limit downstream RL generalization (Ahn et al., 2024); constitutional prompting improves but does not fully guarantee task safety (filtered unsafe → safe task recall up to 94% at both prompt stages).
• In RT planning, patient-specific protocol variation is unaddressed, necessitating manual review for edge cases (Gao et al., 21 Jan 2025).

Planned extensions include institution-specific scorecard incorporation (AIRTP (Gao et al., 21 Jan 2025)), dynamic adaptation of robot constitutions (Ahn et al., 2024), closure between data collection and model improvement, and integration of stronger physics-based grounding in embodied foundation models.

7. Accountability, Verification, and Data Provenance

Across applications, AutoRT architectures enforce clear separation between policy-generation and operational code, maintaining logs and formal representations for external verification:

• All advisory calls (context, beliefs, intentions → actions) are logged for audit and can be replayed for post hoc compliance review without exposing proprietary AV or robotic controller code (Collenette et al., 2022).
• External regulators or monitors can independently verify rule adherence in given situations; each rule/action is traceable to its legal or clinical source (Collenette et al., 2022, Gao et al., 21 Jan 2025).
• In data collection, human oversight and constitutional filtering ensure alignment to safety and operational guidelines; episodic scoring and sampling probabilities are exposed for transparency (Ahn et al., 2024).

This unification of modular knowledge representation, high-performance optimization, external traceability, and flexible adaptation positions AutoRT and its derivatives as principal responders to the increasing demand for certifiable autonomy, explicit safety, and scalable data-driven deployment in diverse robotic and cyber-physical systems.