Robust Stereo Visual Inertial Odometry

• Robust stereo visual inertial odometry is a real-time state estimation system that combines stereo vision and inertial data to accurately compute 6-DOF pose under adverse conditions.
• It leverages both filtering-based and optimization-based architectures to efficiently handle rapid motion, sensor noise, and environmental variances.
• The system enhances robustness using redundant feature types, geometric priors, and noise rejection techniques for applications in robotics, UAVs, and AR/VR.

Robust stereo visual inertial odometry (VIO) refers to algorithms and systems that estimate the state—mainly 6-DOF pose and velocity—of a mobile platform in real time by tightly fusing data from a stereo camera and an inertial measurement unit (IMU). Achieving robustness means that the estimation remains accurate and consistent in the presence of real-world challenges: rapid motion, textureless areas, illumination changes, rolling-shutter effects, event stream noise (for event cameras), and computation constraints. Modern robust stereo VIO systems, leveraging both classical filter-based approaches and optimization-based sliding-window methods, offer state-of-the-art accuracy and computational efficiency across a wide range of industrial, robotics, UAV, and AR/VR applications.

1. Algorithmic Building Blocks and Architectural Variants

Robust stereo VIO architectures can be categorized as either filtering-based (Extended Kalman Filter—EKF, Error-State Kalman Filter—ESKF, dual-stage filtering) or optimization-based (keyframe sliding window, bundle adjustment). Filter-based examples include DS-VIO's dual-stage EKF, where the first filter fuses gyroscope and accelerometer streams to stabilize roll and pitch, and the second fuses the full IMU state with stereo constraints on a limited sliding window (Xiong et al., 2019). The Multi-State Constraint Kalman Filter (MSCKF), as extended to stereo in S-MSCKF and Trifo-VIO, estimates a low-dimensional state vector augmented by a time-window of camera poses, incorporating feature constraints via nullspace projection to avoid latent 3D point states and enforce multi-view geometry efficiently (Sun et al., 2017, Zheng et al., 2018).

Optimization-based systems (e.g., UMS-VINS, co-planar parametrization pipelines) instead maintain a sliding window of keyframes (and optionally map features/planes), directly minimizing the sum of IMU preintegration residuals and stereo (and sometimes monocular) reprojection errors using robustified nonlinear solvers, with old states marginalized to retain real-time performance (Li et al., 2020, Jiang et al., 2023). Hybrid control-filter architectures, such as FLVIS, introduce loop gains and feedforward corrections to treat inertial integration, bias correction, and pose feedback as explicit control loops (Chen et al., 2020).

Recent event-based stereo VIO systems harness neuromorphic sensors to operate under high dynamic range and ultrafast motion (e.g., ESVIO, ESVO2), adapting both the front end (temporal/stereo event association, time-surface construction) and the state estimator (direct alignment, specialized noise handling, voxelized map point management) (Chen et al., 2022, Niu et al., 2024, Zhang et al., 29 Jun 2025, Wang et al., 2023).

2. State Representation and Process Models

The fundamental filter state for robust stereo VIO typically includes the IMU quaternion pose $q_{G}^I$, gyroscope bias $b_g$, velocity $v_I^G$, accelerometer bias $b_a$, and position $p_I^G$, often maintained in error-state form to ensure correct handling of manifold constraints (e.g., multiplicative error for quaternions, additive for bias/velocity/position). The state is further augmented with a sliding window of $N$ past camera pose clones (each with $[q_{G}^{C_i}, p_{C_i}^G]$), and sometimes explicit extrinsic calibration parameters between IMU and camera(s) (Xiong et al., 2019, Sun et al., 2017, Zheng et al., 2018).

Process models propagate the state using the IMU kinematic equations, incorporating bias random walks, and integrate using techniques such as Runge-Kutta or preintegration with closed-form Jacobians. Event-based pipelines extend this with adaptive event accumulation, time-surface or surface-of-active-events representations, and potentially use compact back-ends focused on bias and velocity parameters only (Wang et al., 2023, Niu et al., 2024).

3. Measurement Models and Feature Processing

Stereo VIO systems employ both stereo and monocular measurement models. Feature tracking is performed using corner or line detectors (e.g., FAST, ORB, LSD for lines), KLT optical flow for temporal association, and robust stereo matching (via NCC or RANSAC epipolar gating) (Li et al., 2020, Zheng et al., 2018, Jiang et al., 2023). For classical cameras, matched features are triangulated to obtain inverse-depth or 3D positions; for event cameras, time-surface representations serve as the data carrier, with 3D structure inferred either directly, in a semi-dense manner or through robust voxel selection (Chen et al., 2022, Zhang et al., 29 Jun 2025).

Measurement update equations linearize the pixel-level residuals (reprojection or event-alignment errors) with respect to the current state and, where needed, project out feature depths via nullspace or Schur complement as in MSCKF style filters. Line features, co-planar constraints, or plane-induced priors can be included to improve observability and robustness in low-texture or man-made environments (Li et al., 2020, Zheng et al., 2018).

IMU constraints are enforced via preintegration residuals between consecutive keyframes or filter time steps, encompassing position, velocity, orientation, and bias evolution, all weighted by the noise and bias-covariance models (e.g., see Forster et al. preintegration) (Jiang et al., 2023, Zhang et al., 29 Jun 2025).

4. Robustness Mechanisms

Robust stereo VIO systems employ multiple levels of resilience to real-world failure modes:

• Redundant Feature Types: Use of both points and lines (Zheng et al., 2018), or mix of monocular/stereo (2D/3D) features (Jiang et al., 2023), or incorporating event corners when image features are weak (Chen et al., 2022).
• Geometric Priors: Co-planar parametrization ties features to planar structures, reduces map parameterization, and improves conditioning, especially in environments with dominant planes (Li et al., 2020).
• Noise and Outlier Rejection: Feature associations are filtered via RANSAC (for stereo/temporal consistency), robust kernels (Huber, Cauchy, Tukey), and adaptive residual gating; event-based VIOs further apply voxel-based map filtering to exclude noisy or poorly observed points, which is effective in suppressing event noise (Zhang et al., 29 Jun 2025).
• Observability-Constrained Filtering: Explicitly preserve the unobservable subspace (global yaw, translation) by using observability-constrained EKF updates, avoiding overconfidence and drift (Xiong et al., 2019, Sun et al., 2017).
• Specialized Initialization and Fallback: Hybrid systems such as UMS-VINS provide sub-pixel feature extraction, robust multi-case initialization (vision-only, IMU-only, stereo-only), and multi-camera feature fusion to maximize initialization robustness; if visual-inertial alignment fails, they fall back to visual odometry or IMU-only propagation (Jiang et al., 2023, He et al., 2021).

Event-based designs employ tailored front-ends—motion compensation, direct TS registration, contour-based AA map sampling, and voxel-based management—to maintain performance in HDR/high-motion/low-light regimes where frame-based methods degrade (Chen et al., 2022, Niu et al., 2024, Zhang et al., 29 Jun 2025).

5. Computational Efficiency and Scalability

Filter-based approaches (dual-stage EKF, S-MSCKF, Trifo-VIO) deliver constant or bounded per-update cost, with state and covariance growth controlled by window size (typically $N=5$–$20$) and feature limits (Xiong et al., 2019, Sun et al., 2017, Zheng et al., 2018). Feature and measurement marginalization via nullspace or Schur complement ensures scalability. On typical ARM or laptop CPUs, filter-based systems achieve full pipeline runtimes of 8–12 ms/stereo-frame, even with 200 Hz IMU rates (Xiong et al., 2019).

Optimization-based pipelines exploit problem structure (e.g., Schur complement for landmarks vs. cameras, co-planar grouping) to maintain per-frame BA below 25–60 ms, with robustification and marginalization to bound memory and computation (Li et al., 2020, Jiang et al., 2023). Methods such as Voxel-ESVIO further reduce cost by selecting only high-quality, frequently observed 3D event points per voxel (Zhang et al., 29 Jun 2025). Event-based pipelines reach real-time on single CPUs even at VGA event rates (tracking ≈5 ms, mapping ≈32 ms, back-end ≈5 ms per frame) (Niu et al., 2024).

GPU-accelerated front-ends (VPI) dramatically cut resource usage for large camera arrays, freeing resources for back-end optimization and scale, supporting multi-camera VIO without bottleneck (He et al., 2021).

6. Empirical Performance and Comparative Results

The literature attests that robust stereo VIO methods attain state-of-the-art accuracy and low drift across challenging benchmarks. On the EuRoC MAV dataset:

• DS-VIO achieves top-1 or top-2 RMSE (e.g., MH_01: 0.046 m; V2_01: 0.054 m), matching or outperforming S-MSCKF, OKVIS, VINS-MONO (Xiong et al., 2019).
• Co-planar parametrization pipelines reduce ATE by up to 26–35% over Mesh-VIO and outperform ORB-SLAM2 and other stereo-SLAM baselines in adverse conditions (Li et al., 2020).
• Trifo-VIO demonstrates improved performance under low-texture/rapid motion via point+line fusion, and its loop-closure via EKF updates reduces ATE by up to 70% without global BA (Zheng et al., 2018).

Event-based stereo VIOs consistently achieve sub-meter ATE and demonstrate robust operation in aggressive motion, HDR, and large-scale outdoor sequences where standard image-based pipelines fail or degrade; for example, ESVIO obtains MPE ≈ 0.14% and MRE = 0.033°/m, surpassing image-based and monocular event VIOs (Chen et al., 2022). Voxel-ESVIO achieves ATEs down to 0.02 m and speeds up per-frame computation by 3–5x (Zhang et al., 29 Jun 2025). ESVO2 reports the lowest ATE/RPE on 24/30 benchmark sequences (Niu et al., 2024).

7. Current Trends and Research Directions

Recent advances in robust stereo VIO target:

• Improved exploitation of geometric scene structure (planes, lines, coplanarity) for conditioning and feature economy, especially in man-made environments (Li et al., 2020, Zheng et al., 2018).
• Incorporation of neuromorphic event camera streams for resilience under challenging lighting or rapid motion, motivating novel front-end and back-end architectures (Chen et al., 2022, Niu et al., 2024, Zhang et al., 29 Jun 2025).
• Efficient map management and selective feature filtering (e.g., voxel map strategies) to suppress noise and scale to high event/frame rates (Zhang et al., 29 Jun 2025).
• Multi-camera and multi-modal fusion, leveraging overlapping/non-overlapping fields of view, GPU-accelerated pre-processing, and adaptive sensor selection to enable robust, rapid initialization and redundancy (He et al., 2021).
• Hybrid approaches fusing control-theoretic feedback loops with BA filtering for real-time embedded deployment and practical flight/commercial robotics constraints (Chen et al., 2020).

Limitations remain in scenarios such as highly curved natural scenes (where plane priors are weak), low-texture or dynamic scenes (which challenge point/line-based methods), initialization under degenerate motion or blind spots, and resilience to severe sensor mis-calibration or aggressive event noise. Future directions include tighter integration of semantic priors, dense reconstruction, and learning-based front-ends for adaptive feature extraction and dynamic weighting.

---

References

• DS-VIO: Dual-stage EKF-based Stereo Visual Inertial Odometry (Xiong et al., 2019)
• S-MSCKF: Stereo Multi-State Constraint Kalman Filter (Sun et al., 2017)
• Trifo-VIO: Point and Line Fusion, EKF Loop-Closure (Zheng et al., 2018)
• Co-Planar Parametrization in VIO (Li et al., 2020)
• Event-based Stereo VIO (SEVIO, ESVIO, Voxel-ESVIO) (Chen et al., 2022, Wang et al., 2023, Niu et al., 2024, Zhang et al., 29 Jun 2025)
• GPU-enhanced Multi-Camera VIO (He et al., 2021)
• UMS-VINS: United Monocular-Stereo Feature VIO (Jiang et al., 2023)
• Control-Loop FLVIS (Chen et al., 2020)