DL-VINS-Factory: Modular Visual-Inertial SLAM

• The paper introduces DL-VINS-Factory as a modular framework that isolates the impact of learned visual front-ends by keeping the back-end constant.
• The system architecture combines multiple learned extractors (ALIKED, RaCo, SuperPoint, XFeat) with dual tracking regimes (optical flow and LightGlue) to address diverse visual challenges.
• The framework demonstrates real-time feasibility on embedded platforms while offering actionable insights for configuring visual-inertial SLAM in varying environments.

Searching arXiv for the cited papers to ground the article in current records. arxiv_search(query="(Lim et al., 2 Jul 2026) DL-VINS-Factory learned visual front-ends visual-inertial SLAM", max_results=5, sort_by="submittedDate") arxiv_search(query="(Qin et al., 2017) VINS-Mono monocular visual-inertial state estimator", max_results=5, sort_by="submittedDate") arxiv_search(query="(Park et al., 19 Mar 2026) ROFT-VINS harsh environment", max_results=5, sort_by="submittedDate") arxiv_search(query="(Merrill et al., 2024) AB-VINS simple as A, B, VINS", max_results=5, sort_by="submittedDate") DL-VINS-Factory is a modular visual-inertial SLAM framework that integrates learned visual front-ends into a tightly coupled sliding-window back-end while preserving a common optimization core across configurations. In its published form, it combines learned feature extractors—ALIKED, RaCo, SuperPoint, and XFeat—with either Lucas–Kanade optical-flow tracking or LightGlue descriptor matching, and couples these front-ends to a Ceres-based visual-inertial optimizer, optional AnyLoc DINOv2-VLAD loop closure, and 4-DoF pose-graph optimization (Lim et al., 2 Jul 2026). The framework is explicitly designed to isolate front-end effects by keeping the back-end, loop-closure structure, and evaluation protocol fixed, thereby addressing a recurrent problem in the literature on learned VI-SLAM: comparisons are often confounded by differing estimators, hardware, and retrieval modules. Its design is situated within the lineage of VINS-Mono’s tightly coupled monocular visual-inertial estimation and 4-DoF global consistency machinery (Qin et al., 2017), while also intersecting with later efforts that use deep learning to harden front-end perception under visually adverse conditions (Park et al., 19 Mar 2026) or to restructure the SLAM state around learned dense depth (Merrill et al., 2024).

1. Historical position and design rationale

DL-VINS-Factory was introduced to evaluate the practical value of learned visual correspondences inside tightly coupled visual-inertial SLAM rather than in isolation. The stated motivation is that handcrafted front-ends such as ORB, BRISK, and GFTT are fast and effective in structured scenes but degrade under low texture, motion blur, illumination change, and large viewpoint variations, whereas learned keypoints and descriptors paired with learned matchers can produce more reliable correspondences. The framework’s central methodological move is to hold the back-end constant and make the visual front-end interchangeable, so that changes in accuracy, runtime, and robustness can be attributed more directly to the extractor–tracker or extractor–matcher combination rather than to unrelated system-level differences (Lim et al., 2 Jul 2026).

This positioning places DL-VINS-Factory in continuity with VINS-Mono, whose contribution was a robust and versatile monocular visual-inertial estimator with tightly coupled nonlinear optimization, online extrinsic calibration, loop detection, relocalization, and 4-DoF pose-graph optimization (Qin et al., 2017). DL-VINS-Factory retains the core estimator logic of that family—IMU preintegration, visual reprojection residuals, marginalization, and gravity-aligned global correction—but opens the visual front-end to learned alternatives. A plausible implication is that it should be understood less as a new estimator class than as a controlled experimental and deployment framework for studying front-end substitution inside a mature VINS architecture.

The framework also occupies a distinct position relative to other deep-learning-enhanced VINS systems. ROFT-VINS replaces Lucas–Kanade tracking with a RAFT-based dense optical-flow tracker and adds local motion-consistency filtering, but leaves the VINS-Fusion/VINS-Mono-style back-end unchanged (Park et al., 19 Mar 2026). AB-VINS, by contrast, reorganizes the state around dense monocular depth and loop closures in a memory tree, thereby departing more fundamentally from sparse landmark optimization (Merrill et al., 2024). DL-VINS-Factory differs from both by emphasizing modular interchangeability across multiple learned front-ends under a common back-end and benchmark protocol (Lim et al., 2 Jul 2026).

2. System architecture and modular decomposition

At a high level, DL-VINS-Factory consists of sensors, a selectable front-end, a shared visual-inertial back-end, and an optional loop-closure stack. The supported sensor configurations are mono/stereo cameras plus IMU. The front-end options are learned feature extractors—ALIKED for RGB, RaCo for RGB keypoints with descriptors from the ALIKED head, SuperPoint for grayscale, and XFeat for grayscale—paired with either pyramidal KLT inherited from VINS-Fusion or LightGlue matching. All front-ends output a common abstraction

$\Phi_\theta(I)=\{(p_i,d_i,s_i)\}_{i=1}^N,$

where $p_i\in \mathbb{R}^2$, $d_i\in \mathbb{R}^D$, and $s_i\in[0,1]$, allowing components to be swapped without changing downstream modules (Lim et al., 2 Jul 2026).

The back-end is a sliding-window Ceres solver that jointly minimizes visual reprojection residuals and IMU preintegration residuals. Marginalization is retained, and solver time is capped at 40 ms per optimization. Optional loop closure is implemented through AnyLoc global retrieval using DINOv2 patch tokens aggregated with a universal VLAD codebook, followed by geometric verification and insertion of 4-DoF pose-graph constraints (Lim et al., 2 Jul 2026). This architecture directly reflects the VINS-Mono design principle that local visual-inertial estimation and global drift correction can be separated: a local optimizer provides low-latency state estimates, while an asynchronous global module corrects accumulated $x,y,z,$ and yaw drift (Qin et al., 2017).

The implementation places unusual emphasis on deployment-level modularity. Pre-processing is fused into a CUDA kernel performing letterbox padding, format conversion, normalization, NCHW packing, and stereo batching. Feature extraction uses TensorRT FP16 with static shapes and a 256-keypoint cap. For ALIKED, the original DKD selection is replaced by a custom CUDA kernel so that static-shape TensorRT engines can be exported for both Arm and x86_64 platforms (Lim et al., 2 Jul 2026). This is not merely an optimization detail; it is part of the framework’s claim that learned front-ends can be evaluated not only for accuracy but also for embedded feasibility.

The tracking/matching layer admits two distinct paradigms. In the LK branch, temporal continuity is exploited through optical flow, and stereo gating accepts only cross-camera verified features that are triangulated and have small reprojection error. In the LG branch, temporal and stereo LightGlue matching are both executed, with batch-2 inference for stereo, confidence thresholding, and geometric verification via MAGSAC++ with a 1-pixel epipolar error cutoff. This bifurcation is central to the framework’s empirical conclusions because it enables direct comparison between tracking-based and matching-based learned front-ends under identical estimator conditions (Lim et al., 2 Jul 2026).

3. Estimation model and optimization backbone

DL-VINS-Factory inherits the canonical VINS state representation

$x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$

with $R_i\in SO(3)$, $p_i\in \mathbb{R}^3$, $v_i\in \mathbb{R}^3$, and gyroscope and accelerometer biases. Gravity direction is estimated, enabling gravity-aligned constraints in which roll and pitch are stabilized by the IMU and yaw remains the free rotational variable in the 4-DoF pose graph (Lim et al., 2 Jul 2026). This is consistent with the VINS-Mono formulation, where gravity aligns the world $z$-axis and reduces long-term global drift correction to position and yaw (Qin et al., 2017).

The IMU model follows standard preintegration. Between frames $p_i\in \mathbb{R}^2$0 and $p_i\in \mathbb{R}^2$1, the residual is

$p_i\in \mathbb{R}^2$2

Bias handling follows VINS-Fusion practice: preintegrated terms are corrected by first-order Jacobians with respect to bias changes, and biases follow random-walk evolution (Lim et al., 2 Jul 2026). The underlying logic is the same as in VINS-Mono’s Forster-style bias-corrected preintegration, where local-frame increments and their covariances provide computationally efficient inertial constraints inside nonlinear optimization (Qin et al., 2017).

Visual measurements are represented through reprojection residuals

$p_i\in \mathbb{R}^2$3

For stereo, additional cross-camera residuals are added using the known or estimated baseline, and observations failing disparity checks or with more than $p_i\in \mathbb{R}^2$4 px reprojection error are rejected (Lim et al., 2 Jul 2026). In a specific ablation for RaCo, the framework also supports covariance-weighted reprojection. RaCo predicts a per-keypoint $p_i\in \mathbb{R}^2$5 pixel covariance $p_i\in \mathbb{R}^2$6, which is mapped to the normalized image plane and converted into square-root information: $p_i\in \mathbb{R}^2$7 The paper reports that this covariance weighting helps stereo more than monocular estimation (Lim et al., 2 Jul 2026).

The overall sliding-window cost is written as

$p_i\in \mathbb{R}^2$8

Although the framework description says “without specifying robust loss (default quadratic),” its estimator lineage is the tightly coupled VINS family, in which visual and inertial residuals are optimized jointly in a bounded sliding window with marginalization of old states (Lim et al., 2 Jul 2026). This suggests continuity of estimator structure rather than replacement of the back-end by a learned optimizer.

4. Learned front-ends, tracking regimes, and loop closure

The framework’s front-end modularity is the central object of study. ALIKED is used as an RGB extractor with a Sparse Deformable Descriptor Head and descriptors of dimension $p_i\in \mathbb{R}^2$9 or $d_i\in \mathbb{R}^D$0; the paper uses the ALIKED N16 variant. RaCo is an RGB detector with a multi-scale covariance estimator and differentiable ranker; its descriptors are obtained from the ALIKED descriptor head conditioned on RaCo keypoints. SuperPoint is used as a grayscale detector/descriptor with $d_i\in \mathbb{R}^D$1-D descriptors and dual decoders. XFeat is an ultra-light grayscale CNN with $d_i\in \mathbb{R}^D$2-D descriptors and a reliability heatmap for keypoint filtering (Lim et al., 2 Jul 2026).

The framework evaluates these extractors under two association regimes. The first is LK optical flow, which uses VINS-Fusion’s pyramidal KLT and therefore inherits the temporal smoothness assumptions of classical VIO. The second is LightGlue matching, which performs temporal matching $d_i\in \mathbb{R}^D$3 and stereo matching $d_i\in \mathbb{R}^D$4, followed by confidence thresholding and MAGSAC++ verification (Lim et al., 2 Jul 2026). This division allows one to distinguish gains due to learned keypoints alone from gains due to learned matching. It also clarifies a common misconception in learned SLAM literature: better local descriptors do not imply that descriptor matching is always superior to optical flow inside a tightly coupled estimator. The reported results explicitly show that LK remains preferable in several low-parallax or visually degraded settings (Lim et al., 2 Jul 2026).

Loop closure is deliberately made agnostic to local descriptor choice. AnyLoc uses DINOv2 ViT-S/14 patch tokens aggregated by a universal VLAD codebook with $d_i\in \mathbb{R}^D$5, yielding an $d_i\in \mathbb{R}^D$6-normalized vector $d_i\in \mathbb{R}^D$7. Retrieval uses cosine similarity

$d_i\in \mathbb{R}^D$8

with keyframes inserted by traveled distance, the last 50 keyframes excluded to avoid self-loops, and candidate loops admitted when $d_i\in \mathbb{R}^D$9 (Lim et al., 2 Jul 2026). Verification then uses local descriptor matching with cosine nearest neighbor and Lowe ratio, 2D–2D fundamental matrix RANSAC, and 3D–2D PnP RANSAC, accepting a loop only if there are at least 25 inliers.

The resulting pose-graph constraint is 4-DoF: $s_i\in[0,1]$0 The choice of yaw plus translation follows the gravity-aligned logic already established in VINS-Mono, where roll and pitch are treated as drift-free because of IMU gravity observability (Qin et al., 2017). DL-VINS-Factory therefore preserves a computationally lightweight global correction mechanism while changing the front-end and loop retrieval machinery (Lim et al., 2 Jul 2026).

5. Benchmarks, runtime, and empirical behavior

DL-VINS-Factory is benchmarked on four datasets chosen to stress different failure modes: EuRoC for indoor structured trajectories, NTU-VIRAL for aggressive aerial motion, Botanic Garden for unstructured natural outdoor scenes, and SubT-MRS for subterranean degraded environments. EuRoC uses $s_i\in[0,1]$1 Hz grayscale stereo with $s_i\in[0,1]$2 Hz IMU. NTU-VIRAL uses $s_i\in[0,1]$3 Hz grayscale stereo with $s_i\in[0,1]$4 Hz IMU. Botanic Garden provides RGB and grayscale cameras with a $s_i\in[0,1]$5 Hz subset and $s_i\in[0,1]$6 Hz IMU. SubT-MRS uses $s_i\in[0,1]$7 Hz fisheye RGB with $s_i\in[0,1]$8 Hz IMU, and for fair comparison with LET-NET the cap is reduced to 160 keypoints (Lim et al., 2 Jul 2026).

Trajectory error is reported with ATE RMSE using the evo toolkit and SE(3) alignment: $s_i\in[0,1]$9 where $x,y,z,$0 is the alignment transform (Lim et al., 2 Jul 2026). Relative to the corresponding GFTT+LK baseline, the headline results are conditional rather than universal. On EuRoC, ALIKED+LG reduces ATE by approximately $x,y,z,$1 in monocular odometry and approximately $x,y,z,$2 in stereo with loop closure. On NTU-VIRAL, ALIKED+LG stereo reduces loop-closed ATE by approximately $x,y,z,$3. On Botanic Garden, optical flow remains preferable, but learned keypoints still improve over the GFTT baseline: SuperPoint+LK reduces grayscale camera ATE by approximately $x,y,z,$4, and RaCo+LK reduces RGB camera ATE by approximately $x,y,z,$5. On SubT-MRS, learned front-ends show varying degree of improvement depending on the individual case (Lim et al., 2 Jul 2026).

Several specific findings complicate simplistic claims about learned front-ends. EuRoC shows that GFTT+LK remains competitive, and under the 256-keypoint cap ALIKED+LG is the best learned monocular configuration at $x,y,z,$6 m, while OKVIS2 is the best stereo system at $x,y,z,$7 m due to its factor-graph formulation and higher keypoint limits. NTU-VIRAL favors learned matching under large inter-frame viewpoint changes, with ALIKED+LG stereo best loop-closed at $x,y,z,$8 m and RaCo+LG strong in monocular mode at $x,y,z,$9 m. Botanic Garden prefers optical flow, with SuperPoint+LK best for grayscale at $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$0 m and RaCo+LK best for RGB at $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$1 m; stereo often degrades accuracy there. SubT-MRS reveals brittleness of learned methods in smoke and strong low light, though some LG variants succeed in specific lowlight cases, while LK with learned detectors can recover when the keypoint cap is raised (Lim et al., 2 Jul 2026).

Runtime results are a major part of the framework’s contribution. With TensorRT acceleration on a Jetson AGX Orin, all valid configurations run in real time between $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$2–$x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$3 FPS in monocular mode and $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$4–$x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$5 FPS in stereo mode for EuRoC and NTU-VIRAL. The back-end often bounds throughput, so front-end latency is largely hidden by concurrent optimization. LG adds substantial GPU cost relative to LK. Reported Jetson module power ranges from approximately $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$6 W for GFTT to approximately $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$7 W for Ra+LG, with increases driven by the GPU demand of LG and the DINOv2 loop engine. Front-end time scales approximately linearly with pixel count, with a $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$8 cost increase from $x_i=\{R_i,p_i,v_i,b_{g,i},b_{a,i}\},$9 to $R_i\in SO(3)$0 (Lim et al., 2 Jul 2026).

Loop-closure statistics further support the descriptor-agnostic AnyLoc design. Valid loops per keyframe query are reported as $R_i\in SO(3)$1 versus $R_i\in SO(3)$2 on EuRoC, $R_i\in SO(3)$3 versus $R_i\in SO(3)$4 on NTU-VIRAL, $R_i\in SO(3)$5 versus $R_i\in SO(3)$6 on Botanic Garden, and $R_i\in SO(3)$7 versus $R_i\in SO(3)$8 on SubT-MRS, corresponding to roughly $R_i\in SO(3)$9, $p_i\in \mathbb{R}^3$0, $p_i\in \mathbb{R}^3$1, and $p_i\in \mathbb{R}^3$2 more valid loops than BRIEF+DBoW2, respectively (Lim et al., 2 Jul 2026). The paper also notes that $p_i\in \mathbb{R}^3$3 is permissive: $p_i\in \mathbb{R}^3$4–$p_i\in \mathbb{R}^3$5 of frames may pass the gate, but only $p_i\in \mathbb{R}^3$6–$p_i\in \mathbb{R}^3$7 become verified loops, so decisive filtering occurs in the geometric verification stage.

6. Robustness, limitations, and relation to factory-oriented deep VINS

A central conclusion of DL-VINS-Factory is that learned front-ends are viable for real-time embedded VI-SLAM but are not universally superior to classical tracking (Lim et al., 2 Jul 2026). This directly addresses a common misconception in the field. The paper’s practical guidance is environment-specific: aggressive aerial motion favors ALIKED+LG or RaCo+LG; low-texture or unstructured vegetation favors LK tracking with learned detectors; degraded illumination may still favor LK, while LG can be sensitive to dynamic objects. Stereo should be used only when disparity is well-conditioned. In vegetation-heavy or low-parallax scenes, stereo can degrade accuracy by polluting optimization with weak cross-camera factors (Lim et al., 2 Jul 2026).

The framework’s limitations are correspondingly concrete. XFeat+LG fails to initialize across most monocular sequences and performs poorly in stereo on some datasets, which the paper attributes to limitations in the XFeat-specific “LighterGlue” branch rather than to the extractor itself. Dynamic-object sensitivity appears in SubT-MRS Lowlight 1, where a large dynamic person dominating the field of view causes late divergence while the rig is stationary. Severe smoke causes all learned variants to fail under the 160-keypoint cap; LK variants partially recover with a 256-keypoint cap, while LG remains brittle (Lim et al., 2 Jul 2026).

These findings resonate with earlier work on harsh-environment visual-inertial estimation. ROFT-VINS showed that replacing the KLT tracker with RAFT-based dense optical flow and applying KD-tree neighbor-consistency rejection improves robustness in textureless regions and under rapid illumination changes, with EuRoC RPE mean results of MH05 $p_i\in \mathbb{R}^3$8, V103 $p_i\in \mathbb{R}^3$9, and V203 $v_i\in \mathbb{R}^3$0, compared with VINS-Fusion’s MH05 $v_i\in \mathbb{R}^3$1, V103 $v_i\in \mathbb{R}^3$2, and V203 $v_i\in \mathbb{R}^3$3 (Park et al., 19 Mar 2026). That work specifically targeted low texture, dynamic illumination, specular reflections, motion blur, and vibration—conditions often associated with industrial environments—and thus provides a concrete example of how learned tracking can strengthen the front-end while retaining a conventional VINS back-end (Park et al., 19 Mar 2026).

A different line of development is AB-VINS, which centers deep learning more deeply in the estimation problem by using MiDaS DPT Swin Tiny for monocular inverse depth, NetVLAD for place recognition, and SuperPoint for local matching, while compressing the feature state into per-keyframe affine depth parameters and correction terms (Merrill et al., 2024). AB-VINS is reported to be more robust than state-of-the-art VINS algorithms, though not as accurate, and its memory tree allows loop closures with constant relinearization impact on average (Merrill et al., 2024). Relative to AB-VINS, DL-VINS-Factory is more conservative architecturally: it does not replace sparse reprojection-based VI-SLAM with dense learned depth, but instead systematizes the substitution of learned front-ends inside a VINS-Mono/VINS-Fusion-style estimator (Lim et al., 2 Jul 2026).

For factory-facing deployments, a plausible implication of these combined results is that DL-VINS-Factory should be treated as a framework for matching front-end choice to motion regime, scene geometry, and computational budget rather than as a single best configuration. That reading is reinforced by the original VINS-Mono observation that reliable VI-SLAM in practical settings depends not only on local odometry accuracy, but also on robust initialization, relocalization, and low-overhead 4-DoF global correction (Qin et al., 2017). DL-VINS-Factory preserves that estimator backbone while turning the visual front-end into an experimentally controllable and deployment-adaptable module (Lim et al., 2 Jul 2026).