--- title: 'Omni-LIVO: Multi-Camera LiDAR-Inertial Fusion' url: https://www.emergentmind.com/topics/omni-livo type: topic --- # Omni-LIVO: Multi-Camera LiDAR-Inertial Fusion Searching arXiv for Omni-LIVO and related LIVO papers. Omni-LIVO is a tightly coupled multi-camera LiDAR-inertial-visual odometry system designed to address the field-of-view mismatch between wide-angle LiDAR and conventional cameras. It is presented as the first tightly coupled multi-camera LIVO system and introduces three linked mechanisms: Cross-View direct tracking, photometric migration across non-overlapping views, and multi-view Error-State Iterated Kalman Filter fusion with adaptive covariance weighting. The system targets failure modes of earlier single-camera LIVO pipelines, especially when rapid rotation, occlusion, or viewpoint change causes an image patch to leave one camera and reappear in another, while LiDAR continues to observe wide-angle geometry [2509.15673]. ## 1. System configuration and synchronization model Omni-LIVO uses a 360° LiDAR, four synchronized cameras, and an IMU. In the reported experiments, the LiDAR is a LIVOX MID360 operating at 10 Hz, the cameras operate at \(1024\times768\) and 10 Hz in the custom dataset, and the IMU provides high-rate motion propagation. The four cameras are externally triggered so that their exposures are simultaneous: \[ t_\mathrm{cam}^{(1)}=t_\mathrm{cam}^{(2)}=t_\mathrm{cam}^{(3)}=t_\mathrm{cam}^{(4)}=t_\mathrm{sync}. \] The measurement pipeline first synchronizes and packages sensor data. LiDAR points are temporally segmented with respect to camera exposure, IMU measurements are collected over the frame interval, and the four camera frames are treated as one synchronized multi-view packet. A LiDAR point \(p_i\) is assigned to the nearest camera exposure time according to \[ p_i\to\mathcal{C}_j \text{ if } |\tau_i-t_\mathrm{cam}^{(j)}|=\min_k |\tau_i-t_\mathrm{cam}^{(k)}|. \] This synchronization is structurally important because the subsequent state update combines direct photometric residuals from cameras with geometric residuals from LiDAR in the same Kalman update [2509.15673]. The practical significance of this sensor setting is that Omni-LIVO is built around a wide-angle LiDAR and a conventional multi-camera rig rather than a single panoramic imager. This suggests that the method is intended for robotic platforms where dense geometry is already available from LiDAR, but visual coverage is fragmented across several ordinary cameras. ## 2. Tightly coupled multi-view ESIKF formulation Omni-LIVO builds on the FAST-LIVO2-style Error-State Iterated Kalman Filter. The filter propagates the IMU state forward, propagates covariance, iteratively linearizes nonlinear residuals around the current estimate, updates the error state with the Kalman gain, and applies the box-plus operation to update the nominal state. At iteration \(\ell\), the linearization point is \[ \check{\mathbf{x}}^{(\ell)} = \hat{\mathbf{x}}_{k|k-1} \boxplus \delta\hat{\mathbf{x}}^{(\ell-1)}. \] LiDAR and vision are not processed as separate back ends. Their residuals and Jacobians are stacked into a single measurement model: \[ \mathbf{r}^{(\ell)} = \begin{bmatrix} \mathbf{r}^{(\ell)}_{\mathrm{LIO}} \\ \mathbf{r}^{(\ell)}_{\mathrm{VIO}} \end{bmatrix}, \qquad \mathbf{H}^{(\ell)} = \begin{bmatrix} \mathbf{H}^{(\ell)}_{\mathrm{LIO}} \\ \mathbf{H}^{(\ell)}_{\mathrm{VIO}} \end{bmatrix}. \] The Kalman gain is \[ \mathbf{K}^{(\ell)}= \mathbf{P}_{k|k-1}(\mathbf{H}^{(\ell)})^T \left[ \mathbf{H}^{(\ell)}\mathbf{P}_{k|k-1}(\mathbf{H}^{(\ell)})^T + \mathbf{R}_{\mathrm{multi}} \right]^{-1}, \] and the iterated error-state update is \[ \delta\hat{\mathbf{x}}^{(\ell)}= \delta\hat{\mathbf{x}}^{(\ell-1)}+ \mathbf{K}^{(\ell)} \left( \mathbf{r}^{(\ell)}-\mathbf{H}^{(\ell)}\delta\hat{\mathbf{x}}^{(\ell-1)} \right). \] After convergence, the nominal state is updated as \[ \hat{\mathbf{x}}_{k|k}= \hat{\mathbf{x}}_{k|k-1}\boxplus \delta\hat{\mathbf{x}}^{(\ell)}. \] The resulting estimator is therefore a tightly coupled iterative filter rather than a loose sensor-by-sensor pipeline [2509.15673]. In context, this architectural decision is central: the system does not merely append a second visual module to an existing LiDAR-inertial backbone, but instead redefines the measurement model so that multi-camera photometric information and LiDAR geometry are fused in the same iterative update. ## 3. LiDAR geometric constraints and unified voxel map On the LiDAR side, Omni-LIVO inherits the FAST-LIVO2 backbone and uses a voxelized map representation to extract local planes efficiently. For each voxel \(\mathcal{V}_k\) containing points \(\mathcal{P}_k=\{p_i\}_{i=1}^{N_k}\), the covariance is \[ \mathbf{C}_k=\frac{1}{N_k}\sum_{i=1}^{N_k}(p_i-\bar{p}_k)(p_i-\bar{p}_k)^T. \] A plane is detected when the smallest eigenvalue satisfies \[ \lambda_{\min}<\epsilon_{\text{plane}}, \] with the corresponding eigenvector defining the plane normal \(\mathbf{n}_k\). The plane equation is \[ \mathbf{n}_k^T(p-\bar{p}_k)=0, \] and the LiDAR geometric residual is a point-to-plane distance, \[ r_{\mathrm{geo}}(p_w)=\mathbf{n}_k^T p_w + d_k, \qquad d_k=-\mathbf{n}_k^T\bar{p}_k. \] These constraints provide strong geometry, especially where visual information is weak [2509.15673]. A further structural element is the unified voxel map. Omni-LIVO maintains a single voxel-based map, \[ \mathcal{V}_k=\{\mathcal{G}_k,\mathcal{M}_k,\mathcal{J}_k\}, \] where \(\mathcal{G}_k\) stores geometric plane parameters from LiDAR, \(\mathcal{M}_k\) stores visual map points with multi-view observations, and \(\mathcal{J}_k\) stores the octree subdivision structure. This is explicitly not a split representation with separate LiDAR and visual maps. For each visual map point, the system stores a multi-view observation set, \[ \mathcal{F}_j= \left\{ \mathcal{P}^{(k)}_j,\, \mathbf{T}^{(k)}_{wc},\, \tau^{(k)},\, id^{(k)}_{\text{cam}} \right\}_{k=1}^{N_{\text{obs}}}, \] where \(\mathcal{P}^{(k)}_j\) is the patch pyramid representation, \(\mathbf{T}^{(k)}_{wc}\) is the camera pose, \(\tau^{(k)}\) is the exposure, and \(id^{(k)}_{\text{cam}}\) is the camera index. These points are updated when the camera pose changes or patch displacement exceeds a threshold, and Tukey weighting is used to suppress photometric outliers. The map design therefore couples geometric structure and RGB observations in one spatial data structure rather than aligning two independently maintained maps. ## 4. Cross-View direct tracking and photometric migration The central visual idea is that a patch leaving one camera’s field of view should not be treated as irretrievably lost if it can reappear in another camera. Omni-LIVO formulates this as a temporal migration problem across views rather than a standard feature-drop event. A patch \(\mathbf{p}\) is considered to migrate from camera \(i\) to camera \(j\) if it disappears from \(i\), reappears in \(j\), and satisfies visibility and temporal conditions: \[ M(\mathbf{p}, i\rightarrow j)=1 \Leftrightarrow \begin{cases} \text{cam}_{\text{ref}(\mathbf{p})}=i \\ \text{cam}_{\text{cur}(\mathbf{p})}=j \\ i\neq j \\ \text{exclusive}_{\text{visible}(\mathbf{p},j)}=1 \\ \text{temporal}_{\text{migrated}(\mathbf{p},i\rightarrow j)}=1 \end{cases}. \] This formulation turns multi-camera tracking into a continuity problem across camera boundaries [2509.15673]. Because different cameras can have different exposure times and vignetting, raw intensities are normalized before comparison: \[ I'_k(u,v)=\alpha_k \cdot V_k(r^2)\cdot I_k(u,v), \] with vignetting model \[ V_k(r^2)=1+\sum_{n=1}^{3}\beta_{k,n}\,r^{2n}. \] Here, \(\alpha_k\) is the exposure scale for camera \(k\), \(V_k(r^2)\) models vignetting, and \(r\) is the radial distance from the image center. This normalization is necessary for cross-camera photometric consistency. Patch transfer between cameras is defined through the relative pose \[ \mathbf{T}_{j\to i}(t)=\mathbf{T}^{w}_{j}(t_j)\cdot \left(\mathbf{T}^{w}_{i}(t_i)\right)^{-1}, \] and the associated local planar homography \[ \mathbf{H}_{i\rightarrow j}(\mathbf{p},t)= \mathbf{R}_{j\rightarrow i}(t) + \frac{\mathbf{t}_{j\rightarrow i}(t)\mathbf{n}_i^T}{d_i}, \] where \(\mathbf{R}_{j\rightarrow i}(t)\in SO(3)\), \(\mathbf{t}_{j\rightarrow i}(t)\in\mathbb{R}^3\), \(\mathbf{n}_i\) is the surface normal in camera \(i\), and \(d_i\) is the depth or distance to the plane. The patch warp under homography \(H\) is denoted \(W_H[\cdot]\). Within a single camera, the direct residual is \[ r_{\text{direct}}^{k}(\mathbf{p})= \gamma_{\mathrm{cur}}\cdot I_k'(\pi_k(\mathbf{T}_k^w\cdot p)) - \gamma_{\mathrm{ref}}\cdot I_{\mathrm{ref}}'(\pi_{\mathrm{ref}}(p_{\mathrm{ref}})). \] For migration across cameras, the residual is \[ r_{\mathrm{migration}}^{i\to j}(p)= \phi_{i\to j}\cdot \tau_j \cdot I_j(\pi_j(p,t_j)) - \tau_i \cdot W_H\!\left[I_i',\pi_i(p,t_i)\right], \] with \[ \phi_{i\to j}=\frac{\alpha_j}{\alpha_i}, \] and \(\tau_k\) the inverse exposure time. The current intensity in camera \(j\) is therefore compared against a warped and normalized patch from camera \(i\), with exposure compensation across the two cameras. This is the photometric migration mechanism proper. All visual constraints are stacked into a single residual vector, \[ \mathbf{r}_{\mathrm{total}}= \left[ \mathbf{r}_{\mathrm{direct}}^{1}, \mathbf{r}_{\mathrm{direct}}^{2}, \ldots, \mathbf{r}_{\mathrm{direct}}^{N}, \mathbf{r}_{\mathrm{migration}}^{1\to2}, \mathbf{r}_{\mathrm{migration}}^{1\to3}, \ldots, \mathbf{r}_{\mathrm{migration}}^{i\to j} \right]^T, \] with dimension \[ \dim(r_{\mathrm{total}})= N\cdot P\cdot S + \sum_{M\in\mathcal{A}} P_M\cdot S, \] where \(N\) is the number of cameras, \(P\) the number of tracked patches, \(S\) the patch size, \(\mathcal{A}\) the active migration pairs \((i,j)\), and \(P_M\) the number of migration points for pair \(M\). Migration bookkeeping is accumulated over time as \[ \text{Migration count}(p) = \sum_{t=0}^{T}\sum_{i,j\in\mathcal{C}} M(p,i\to j,t). \] The explicit accumulation of migration events indicates that cross-view continuity is treated as a first-class temporal quantity rather than an incidental recovery heuristic. ## 5. Adaptive covariance weighting in multi-view fusion A distinctive contribution of Omni-LIVO is adaptive multi-view covariance design. The motivation is that different cameras can have very different tracking quality at the same instant: one may observe a texture-rich scene, another a texture-poor wall, another motion blur, and a cross-view residual may vary in reliability depending on overlap or occlusion. A fixed covariance for all visual residuals would therefore treat all views equally even when their information quality is unequal. The multi-view measurement covariance is defined as \[ \mathbf{R}_{\mathrm{multi}}= \mathrm{diag}\big( \alpha_1\mathbf{R}_{\mathrm{base}}^{(1)}, \alpha_2\mathbf{R}_{\mathrm{base}}^{(2)}, \ldots, \alpha_{\mathrm{cross}}\mathbf{R}_{\mathrm{cross}} \big), \] where the scaling factor for each view is given by a function of tracking quality, \[ \alpha_i = f\left( \frac{N^{(i)}_{\mathrm{tracked}}}{N_{\mathrm{tracked}}}, \bar{r}_i^2, \mathrm{gradient}_i \right). \] The terms encode the fraction of tracked patches contributed by camera \(i\), the average squared residual as a measure of photometric fit quality, and the image gradient or texture strength as a measure of informativeness. If a camera’s tracking degrades, its covariance increases and its influence on the update decreases; if a camera has sharp and well-tracked features, it is weighted more heavily [2509.15673]. The iterated update loop is described as: start from the predicted state, compute residuals and Jacobians for LiDAR and all camera views, assemble the full measurement model, compute \(\mathbf{K}^{(\ell)}\), update the error state, and relinearize until convergence. The pseudocode is consistent with IMU propagation, LiDAR motion compensation and plane extraction, LiDAR update, selection of photometric patches, per-camera photometric residual computation, stacking all residuals, adaptive covariance construction, and Kalman update until convergence. A plausible misconception is that multi-camera fusion is primarily a matter of stacking more residuals. Omni-LIVO instead defines multi-camera fusion as quality-aware estimation, in which the reliability of each view is explicitly modeled in the covariance structure. ## 6. Evaluation, reported performance, and limitations The reported evaluation covers three datasets: the Hilti SLAM Challenge 2022, described as indoor construction-site sequences with millimeter-accurate ground truth; the Newer College multi-camera LiDAR-inertial dataset, described as including campus and underground sequences with difficult lighting and viewpoint changes; and a custom dataset using a LIVOX MID360 LiDAR, four synchronized cameras in a cross pattern, and both handheld and robot-mounted platforms. The baselines include FAST-LIVO, FAST-LIVO2, R3LIVE, FAST-LIO2, and OpenMAVIS where compatible. The paper reports ATE RMSE over 14 sequences. Omni-LIVO achieves the lowest RMSE on 11 of 14 sequences, completes all 14 sequences, shows about 34% average accuracy improvement over FAST-LIVO2 in the discussion, and the conclusion also reports 23% RMSE reduction overall, depending on the summary metric being emphasized [2509.15673]. In the Hilti sequences, FAST-LIVO fails on 5 of 7 sequences and FAST-LIO2 on 4 of 7, while Omni-LIVO completes all of them. In Newer College underground-hard, Omni-LIVO reports \(0.0547\,\mathrm{m}\), compared with \(11.0177\,\mathrm{m}\) for FAST-LIVO and \(0.2414\,\mathrm{m}\) for FAST-LIVO2. On quad-hard, Omni-LIVO reports \(0.0569\,\mathrm{m}\), compared with \(0.0825\,\mathrm{m}\) for FAST-LIVO2 and \(0.1239\,\mathrm{m}\) for OpenMAVIS. The qualitative findings isolate the roles of the main components even though the provided text does not include a formal ablation table. Cross-View temporal migration is described as the key reason the system survives sequences such as Exp06, where rapid turning causes single-camera tracking to lose patches as they leave the field of view. Adaptive covariance weighting stabilizes estimation under strong illumination variation and changing texture quality, with Underground-hard given as the representative case. Multi-camera mapping produces much denser RGB point clouds than single-camera methods, with especially large gains in scenes such as basement3; the reported RGB point cloud statistics show multi-camera colorization benefits of up to around \(3.5\times\) relative to single-camera baselines. The reported computational performance, measured on an i9-13900HX plus RTX 4070 laptop, is: FAST-LIVO at \(25.2\,\mathrm{ms}\), \(39.7\,\mathrm{Hz}\); FAST-LIVO2 at \(30.7\,\mathrm{ms}\), \(32.6\,\mathrm{Hz}\); R3LIVE at \(28.5\,\mathrm{ms}\), \(35.1\,\mathrm{Hz}\); and Omni-LIVO at \(35.5\,\mathrm{ms}\), \(28.2\,\mathrm{Hz}\). Omni-LIVO is therefore slower than FAST-LIVO2 but remains near real-time, with an overhead described as about \(1.19\times\) over FAST-LIVO2. The paper’s conclusion identifies online loop closure and multi-sensor calibration as future work. The described practical limitations are higher computation and memory than single-camera systems, dependence on good synchronization and calibration, complexity in maintaining robust cross-view patch associations, and likely sensitivity to severe blur or situations in which all cameras simultaneously have poor visual quality. Within those constraints, Omni-LIVO is positioned as a system that improves trajectory accuracy and RGB mapping completeness in aggressive motion, occlusion, poor lighting, and geometrically ambiguous environments. Source: https://www.emergentmind.com/topics/omni-livo