The Unified Autonomy Stack: Toward a Blueprint for Generalizable Robot Autonomy

Abstract: We introduce and open-source the Unified Autonomy Stack, a system-level solution that enables resilient autonomy across diverse aerial and ground robot morphologies. The architecture centers on three synergistic modules -- multi-modal perception, multi-behavior planning, and multi-layered safe navigation -- that together deliver comprehensive mission autonomy. The stack fuses data from LiDAR, radar, vision, and inertial sensing, enabling (a) robust localization and mapping through factor graph-based fusion, (b) semantic scene understanding, (c) motion and informative path planning through sampling-based techniques adaptive across spatial scales, as well as (d) multi-layered safe navigation both through planning on the online reconstructed map and deep learning-driven exteroceptive policies alongside last-resort safety filters using control barrier functions. The resulting behaviors include safe GNSS-denied navigation into unknown and perceptually-degraded regions, exploration of complex environments, object discovery, and efficient inspection planning. The stack has been field-tested and validated on both aerial (rotorcraft) and ground (legged) robots operating in a host of demanding environments, including self-similar and smoke-filled settings, with complex geometries and high obstacle clutter. These tests demonstrate resilient performance in challenging conditions. To facilitate ease of adoption, we open-source the implementation alongside supporting documentation, validation, and evaluation datasets https://github.com/ntnu-arl/unified_autonomy_stack. A video giving the overview of the paper and the field experiments is available at https://youtu.be/l8Su8OXsM-E.

• The paper presents a unified autonomy stack that integrates multi-modal SLAM, abstracted planning, and layered safety to enable robust, cross-morphology robot autonomy.
• Multi-modal perception using LiDAR, radar, vision, and IMU with factor graph optimization achieves lower trajectory errors in challenging, GNSS-denied environments.
• Swappable navigation policies, including Neural SDF-NMPC and DRL with a composite CBF safety filter, ensure near-zero crash rates under diverse operational conditions.

The Unified Autonomy Stack: Advancing Generalizable Robot Autonomy

Motivation and Scope

The Unified Autonomy Stack (UA) presents a comprehensive solution for resilient, cross-morphology robot autonomy spanning aerial and ground platforms. Its architecture is structured to address the fragmentation in existing autonomy systems, which are often specialized for particular morphologies, sensor setups, or narrowly defined mission profiles. This specialization hinders systematic field evaluation, limits reuse, and inhibits the accumulation of deployment experience across the robotics community. The UA is designed for robustness in GNSS-denied, perceptually degraded, and geometrically complex environments, integrating multi-modal sensor fusion, semantic perception, modular planning, and resourceful, multi-layered navigation safety.

System Architecture

The UA is organized around three synergistic modules—multi-modal perception, multi-behavior planning, and multi-layered safe navigation—interfaced through abstract Robot and Mission layers to maximize generalizability. Core contributions include:

• Perception: Factor-graph-based multi-modal SLAM fusing LiDAR, radar, vision, and IMU cues for reliable pose estimation and mapping in challenging environments.
• Semantic Reasoning: Integration of vision-LLMs (VLMs), enabling open-vocabulary object detection, semantic 3D mapping, and high-level binary visual question answering.
• Planning: A morphology- and mission-agnostic kernel facilitating target reaching, exploration, and inspection behaviors via dual-representation (volumetric and, for ground robots, elevation maps) and sampling-based graph methods.
• Navigation: Swappable depth-exteroceptive navigation modalities (Neural SDF-NMPC and Exteroceptive DRL) complemented with a composite Control Barrier Function (CBF) safety filter for reactive, last-resort collision avoidance.

  Figure 1: Evaluation studies covering diverse GNSS-denied and perceptually degraded environments with both aerial and ground robots using the UA stack modules.

Multi-Modal Perception and Semantic Reasoning

At the heart of the perception stack is the MIMOSA-X multi-modal SLAM system, which fuses asynchronous LiDAR, radar, camera, and IMU measurements using a windowed smoother and nonlinear factor graph optimization. The estimator explicitly maintains states for position, velocity, attitude, biases, and gravity direction, supporting robust marginalization and online gravity estimation.

The estimator tightly couples exteroceptive factors—LiDAR point-to-plane, radar Doppler, visual-inertial relative transforms—with efficient outlier rejection via Huber and Cauchy M-estimators. This configuration is resilient to modality-specific degradations such as geometric self-similarity (LiDAR), low texture/light (vision), or channel occlusion (radar), achieving robust odometry across diverse field conditions.

Figure 2: Exemplary factor graph structure in the multi-modal estimator demonstrating the integration of multiple sensory modalities.

The semantic pipeline leverages open-vocabulary detectors (e.g., YOLOe, GPT-5) to project detections into a volumetric LiDAR-derived map for 3D semantic mapping. Binary Q/A through GPT-5 augments the geometric understanding with context-driven semantic reasoning for advanced mission-level behavior and safety assessment.

Figure 3: Current VLM-based semantic scene reasoning and Q/A pipeline integrating perception outputs and supporting high-level contextual awareness.

Planning Module and Multi-Objective Behaviors

The planning module operates through a graph-based kernel, OmniPlanner, that is agnostic to robot morphology and mission objectives. It aligns configuration space constraints and sensor field-of-view/range to dynamically construct dense local and sparse global graphs. The architecture enables behaviors including:

• Target Reaching: Iterative repositioning via shortest paths in known or frontier spaces.
• Volumetric Exploration: Information-theoretic path selection maximizing expected volume gain subject to FOV/range constraints.
• Inspection: Optimal viewpoint selection and TSP-based ordering for surface inspection with camera constraints.
• Mission Endurance Awareness: Ensures the feasibility and safety of planned behaviors with return-to-home triggers on critical thresholds.

  Figure 4: Abstracted planner architecture, showing dual-representation graphs and flexible behavioral execution on arbitrary morphologies.

Multi-Layered Navigation and Safety

Conventional navigation pipelines rely on map-based planning and are therefore vulnerable to localization/mapping errors or perceptual degradations. The UA explicitly addresses this single point of failure by deploying:

• Neural SDF-NMPC: Local environment encoding as a SDF in a neural network, embedded as a constraint in a real-time NMPC controller for short-horizon, map-free collision avoidance, with provable recursive feasibility and stability.
• Exteroceptive DRL Policy: End-to-end depth and proprioceptive input policy trained via PPO to directly output acceleration & yaw-rate setpoints for collision-free navigation, using time-to-collision rewards and curriculum learning to generalize across environments.
• Composite CBF Filter: Real-time, reactive safety filter solving a quadratic program at each step to guarantee forward invariance of the robot’s safe set with respect to dynamic obstacles and unmodeled disturbances.
• Swappable Layers: Selection of navigation and safety configurations based on platform characteristics, with both model-based and learning-based options.

  Figure 5: Modular navigation pipeline with swappable exteroceptive policies and last-resort formal CBF safety buffering.

  

  Figure 6: Visualization of the CBF function used for invariant set enforcement, critical for last-resort safety filtering.

Empirical Validation and Robustness

The UA was validated extensively in real-world environments using rotary-wing aerial robots and quadruped ground robots, and included challenging settings such as geometrically self-similar tunnels, smoke-/fog-filled buildings, snow-laden forests, underground mines, and vessel cargo holds. Across these conditions, the UA’s multi-modal perception outperformed uni-modal and state-of-the-art SLAM baselines, with lower absolute and relative trajectory error rates, especially in scenarios where individual modalities fail due to environmental factors.

Ablation and stress tests in simulation confirmed that the addition of the CBF safety filter significantly reduces crash rates under control command mismatch and in cluttered environments. The modular design allows each safety and navigation layer to compensate for the other's limitations, enabling robust avoidance of both mapped and unmapped/dynamic obstacles.

Figure 7: Ablation comparing DRL and NMPC policies, with/without the CBF safety layer, under varying environment densities and command mismatches—showing critical reduction in failure rates from multi-layered safety.

Strong Numerical Results and Claims

Bold numerical outcomes from field evaluations include:

• In complex, GNSS-denied, and visually/fog-obscured test environments, the LRI fusion mode of UA’s SLAM module achieved Absolute Trajectory Error (ATE) as low as $3.9\,\mathrm{m}$ over $1275\,\mathrm{m}$ trajectories, outperforming state-of-the-art alternatives by a substantial margin.
• The CBF-supplemented Neural MPC and DRL navigation stacks consistently reduced crash rates to near-zero even under induced control mismatches where baseline learning-based policies independently failed catastrophically.
• Full-stack deployments demonstrated successful autonomous operation and mission completion (exploration, inspection, target-reaching) across multiple payloads, platforms, and sensing configurations without platform- or morphology-specific adaptations.

Theoretical and Practical Implications

The UA represents a decisive step towards reducing the fragmentation of autonomous robot deployment across platform, sensor, and mission axes. By integrating multi-modal SLAM, abstracted planning, semantics, and redundant, formally grounded safety, it provides a foundation for robust autonomy research, field deployment, and reproducibility. The design enables cross-embodiment, cross-environment, cross-mission adaptability, and further paves the way for unified data collection, benchmarking, and comparative analysis in the robotics community.

By supporting out-of-the-box configurations for both rotorcraft and legged robots, UA enables rapid extension to new morphologies and missions, and can serve as a testbed for future foundation models of robot control and cognition [schubert_generalist_2023]. The open-source release of code and datasets further accelerates community validation and extension.

Prospective Developments

Directions for future work include:

• Expanding morphology compatibility (e.g., fixed-wing UAVs, marine vehicles).
• More tightly-coupled fusion of vision cues in factor-graph SLAM for further resilience.
• Extension of the CBF safety paradigm to traversability-aware constraints on ground systems.
• Integration of real-time semantic feedback into planning and navigation loops.
• Enhanced long-horizon reasoning and hierarchical behavioral orchestration via language and vision-language-action models.
• Co-evolution with developments in generalist policy and foundation dynamics models for embodied agents [reed_generalist_2022, schubert_generalist_2023-1].

Conclusion

The UA establishes a blueprint for resilient, cross-morphology robot autonomy by unifying robust multi-modal perception, abstracted planning, and multi-layered navigation safety in a modular, open-source stack. Empirical results across challenging real-world environments validate the system’s claims of adaptivity, resilience, and generalizability. The UA positions itself as a foundation for future embodied AI research through its methodological plurality and extensibility.