Tightly-Coupled Schmidt Kalman Filter

• Tightly-coupled SKF is a Kalman-filter variant that partitions states into actively updated and frozen blocks while preserving essential cross-covariances.
• Its methodology selectively updates only the active states and retains matured map features, ensuring both computational efficiency and consistency.
• The approach underpins advanced applications in visual–inertial SLAM and LiDAR–inertial odometry by incorporating map uncertainty without incurring full EKF cost.

Searching arXiv for tightly-coupled Schmidt Kalman filter papers relevant to the provided topic. A tightly-coupled Schmidt Kalman Filter (SKF) is a Kalman-filter variant in which the measurement model remains jointly defined over actively updated states and a set of non-updated nuisance or considered states, while the filter explicitly preserves the cross-covariances that couple the two blocks. In the tightly-coupled visual–inertial and LiDAR–inertial literature, this construction is used to incorporate map information, past poses, or other latent variables directly into the estimation equations without incurring the full computational and memory cost of a standard EKF update on all variables. In SEVIS, for example, the Schmidt-EKF VI-SLAM formulation optimally fuses IMU measurements and monocular images in a tightly-coupled manner, selectively including informative features in the state vector while treating matured features as nuisance parameters once their estimates and uncertainties have converged (Geneva et al., 2019).

1. Definition and state partitioning

The defining structural feature of the SKF is the partition of the full state into an updated block and a frozen block. In the general discrete-time formulation, the augmented state is written as

$$x_a = \begin{bmatrix}x \ p\end{bmatrix},$$

where $x$ denotes the estimated state and $p$ denotes considered states or parameters that are not updated directly during the measurement step (Ramos, 2022). In SEVIS, the full EKF state is

$x = [x_A; x_S],$

where $x_A$ contains the IMU navigation state and a sliding window of cloned past poses, while $x_S$ contains matured map features represented as 3D points (Geneva et al., 2019). In map-based visual–inertial localization, the same active–nuisance decomposition is applied to odometry variables and map-keyframe poses; in LODESTAR, the partition is expressed as updating and fixed blocks inside a sliding window of current and past states (Zhang et al., 2022).

The associated covariance is partitioned conformally. In SEVIS,

$$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$

with $P_{AS}$ encoding the cross-correlation between active states and Schmidt states (Geneva et al., 2019). The rationale is explicit: once map features have “matured,” their means are no longer updated, but their cross-covariances with the active states continue to be maintained so that their uncertainty still informs future active-state corrections. This same logic appears in the broader SKF formulation, where the considered-state gain block is forced to zero while the corresponding covariance blocks remain part of the filter algebra (Ramos, 2022).

A tightly-coupled SKF therefore differs from a loosely coupled architecture not by abandoning shared uncertainty, but by preserving it selectively. The frozen block is not ignored; it is held fixed in mean and, in standard SKF form, also fixed in its own covariance block, while its uncertainty continues to enter the innovation covariance and the active-state gain through cross-covariances. This suggests that “tightly coupled” refers to the measurement and covariance structure, not to whether all variables are explicitly corrected at each update.

2. Prediction model and partial-update equations

In tightly-coupled SKF systems, propagation generally affects only the evolving or active portion of the state. In SEVIS, the nonlinear IMU dynamics are written as

$x_{k+1|k} = f(x_{k|k}, u_k),$

and after linearization the error-state dynamics take the form

$\delta x_{k+1} \approx F_k\,\delta x_k + G_k\,n_k.$

Because the process model does not evolve Schmidt states, the transition matrix has block form

$x$0

which yields

$x$1

$x$2

$x$3

Thus the Schmidt block remains static under propagation, while the active–Schmidt correlation continues to evolve (Geneva et al., 2019).

The measurement update is the point at which the SKF departs from a full EKF. For a linearized measurement

$x$4

the standard Schmidt update imposes

$x$5

with

$x$6

The considered states are left unchanged in mean, but their uncertainty contributes to the residual covariance and hence to the active-state gain (Ramos, 2022).

SEVIS instantiates the same rule for visual measurements. After linearization,

$x$7

and the Schmidt-EKF modification sets $x$8 while using

$x$9

The state update becomes

$p$0

and only $p$1 and $p$2 are updated, while $p$3 is frozen (Geneva et al., 2019). In LODESTAR, the same pattern is written for updating and fixed blocks, with the covariance update expressed in Joseph form so that fixed states remain unchanged but still anchor the active update via $p$4 and $p$5 (Lee et al., 12 Nov 2025).

3. Tight coupling in visual–inertial SLAM and localization

In tightly-coupled visual–inertial SLAM, measurements are modeled directly against the joint state rather than through a separate pose-estimation stage. SEVIS uses a monocular feature measurement model

$p$6

where the camera-frame feature position depends jointly on the IMU pose, extrinsics, and the global 3D feature position (Geneva et al., 2019). This direct dependence on both active navigation states and map-feature states is what makes the Schmidt construction relevant: the filter can exploit map-feature uncertainty without repeatedly updating all matured map points.

The SEVIS workflow on each camera frame is explicitly organized around multiple tightly-coupled update modes. After stochastic cloning of the current pose into the sliding window, the system tracks features, extracts descriptors, and matches to last frames and keyframes. It then performs keyframe-aided 2D-to-2D matching to identify which tracked features correspond to map features, applies an MSCKF update for features that only live in the window, applies a standard EKF update for newly initialized SLAM features, and finally applies a Schmidt update for matched map features (Geneva et al., 2019). This arrangement places Schmidt updates alongside standard EKF and MSCKF updates inside a single estimation pipeline rather than treating map matching as a separate post-processing stage.

The 2022 map-based visual–inertial localization framework extends the tightly-coupled SKF idea to an invariant-EKF setting. There, the active state is represented on a Lie group, local visual features are handled in multi-state form, and map-keyframe poses are treated as nuisance states under a Schmidt update (Zhang et al., 2022). The authors explicitly note that directly combining the invariant EKF with the Schmidt filter cannot maintain the correct observability properties of the system that considers the uncertainty of the map information, and they therefore introduce an observability-constrained technique. A plausible implication is that tightly-coupled SKF design in visual–inertial localization is not solely a computational device; it also interacts with the observability structure induced by map uncertainty.

4. Cross-correlation maintenance, consistency, and observability

The distinctive probabilistic role of the SKF is the preservation of cross-correlation between active and frozen variables. In SEVIS, although $p$7 and $p$8 are not updated, $p$9 and $x = [x_A; x_S],$0 continue to be updated. The stated purpose is to ensure that future active updates remain consistent with the uncertainty of the frozen map features; the filter “knows” how uncertain the map is and how that uncertainty couples into the active IMU and past-pose block, even though the map’s own covariance is not re-linearized (Geneva et al., 2019).

This consistency interpretation also appears in the classical map-based SKF formulation. For a state $x = [x_A; x_S],$1 with covariance partitioned as

$x = [x_A; x_S],$2

the SKF sets the map part of the gain to zero, leaves $x = [x_A; x_S],$3 and $x = [x_A; x_S],$4 unchanged, and the resulting posterior covariance satisfies

$x = [x_A; x_S],$5

so SKF remains consistent (Dutoit et al., 2016). The price of this consistency in the classical form is that storing the dense map covariance $x = [x_A; x_S],$6 requires $x = [x_A; x_S],$7 memory for $x = [x_A; x_S],$8 map features (Dutoit et al., 2016).

The observability issue is treated in greater depth in the invariant map-based visual–inertial localization framework. There, when map-keyframe poses are uncertain, the augmented system ideally has 10 null-directions, but direct linearization breaks part of that structure in practice. The proposed remedy is to enforce

$x = [x_A; x_S],$9

and to replace the offending Jacobian block by an orthogonally projected form

$x_A$0

Used together with the Schmidt update, this restores the correct 10-D unobservable subspace and yields what the paper describes as a fully consistent filter despite map uncertainty (Zhang et al., 2022). This suggests that a tightly-coupled SKF must often be analyzed not only as a reduced-update estimator but also as an estimator whose linearization policy determines whether the correct gauge freedoms are preserved.

5. Computational scaling and memory-efficient variants

The principal computational motivation for the tightly-coupled SKF is the avoidance of quadratic map updates. In SEVIS, the propagation step updates $x_A$1 through multiplication by $x_A$2, and $x_A$3 is untouched. The measurement update forms the innovation covariance and updates $x_A$4 and $x_A$5 with cost $x_A$6, where in practice the number of Schmidt features matched per image is limited to a small constant $x_A$7. State and window management are implemented with $x_A$8 block copies and index swaps. The overall result is that every step scales $x_A$9, in contrast to a full EKF-SLAM whose update would be $x_S$0 because both $x_S$1 and $x_S$2 would need full updates (Geneva et al., 2019).

A limitation of the standard SKF is that even if update cost is linear, the map covariance may remain dense and expensive to store. The Cholesky-Schmidt-Kalman filter addresses this by replacing the explicit map covariance with the sparse Cholesky factor of the map Hessian. At mapping time,

$x_S$3

and the cross-covariance is factorized as

$x_S$4

Mapped-feature updates are then performed via sparse back-solves with $x_S$5, and all operations cost $x_S$6 per feature, while memory is dominated by storing sparse $x_S$7, giving $x_S$8 memory (Dutoit et al., 2016).

The same work introduces the submap relaxation sC-SKF, which partitions the map into overlapping submaps, each with its own Cholesky factor $x_S$9 and cross-covariance factor $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$0. When a feature from submap $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$1 is observed, only that submap participates in the expensive back-solve, while the remaining cross-covariance factors are updated through the active-state term. The paper states that this bounds per-update cost between linear and quadratic in the map size and remains consistent by a Schur-complement argument (Dutoit et al., 2016). A plausible implication is that tightly-coupled SKF design has two separate complexity axes: update sparsity from the Schmidt partial update, and storage/computation sparsity from factorized map representations.

6. Adaptive and factorized extensions

Recent work extends tightly-coupled SKF ideas beyond static map-feature freezing. In LODESTAR, the Adaptive Schmidt-Kalman Filter is embedded in LiDAR–inertial odometry and uses a sliding window partitioned into active states and fixed states together with the current IMU state (Lee et al., 12 Nov 2025). After each LiDAR update, the window is slid, the oldest slot is discarded, and the previous last active state is either transferred into the fixed set or removed altogether according to the condition number

$$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$2

of its associated stacked Jacobian. If $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$3, the state is deemed well-conditioned and moved into the fixed set; otherwise it is removed. The fixed states then serve as reference anchors in subsequent Schmidt updates (Lee et al., 12 Nov 2025).

LODESTAR also illustrates a different notion of tight coupling: current LiDAR point-to-plane residuals are augmented with residuals to past states, producing a Jacobian partition $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$4 that couples updating and fixed blocks directly (Lee et al., 12 Nov 2025). The method further adds degeneracy-aware data exploitation, pruning weak measurements using localizability criteria and compensating by adding selected fixed-state points when the condition number remains above threshold. This suggests that adaptive SKF variants can use the active–fixed partition itself as a mechanism for degeneracy management rather than only for computational savings.

A separate extension concerns numerical robustness. The work on square-root and U–D factorized Schmidt filters formulates SKF variants that incorporate the numerical robustness of square-root and factorized filtering forms and also allow actively attempting to update the considered states (Ramos, 2022). In the square-root SKF, the covariance is factored as $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$5 and propagated through orthonormal transformations; in the U–D factorized SKF, $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$6 and weighted modified Gram–Schmidt is used in propagation and update. The stated motivation is to handle nonlinearities, larger initial uncertainties, and poorly conditioned systems better than a typical Extended or Schmidt Kalman filter (Ramos, 2022). Within the tightly-coupled setting, such factorized forms are relevant because active–considered cross-covariance preservation can otherwise be vulnerable to numerical degradation when the augmented state is large.

7. Applications, reported performance, and recurring trade-offs

The tightly-coupled SKF appears in multiple application regimes: visual–inertial SLAM, map-based visual–inertial localization, map-based mobile-device localization, and LiDAR–inertial odometry. The recurring application goal is to incorporate prior map uncertainty or past-state information without fully re-estimating all nuisance variables at each update. In SEVIS, the result is described as a tightly-coupled visual–inertial SLAM that runs in real time on resource-limited platforms with $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$7 complexity in the number of map features (Geneva et al., 2019). In the map-based visual–inertial localization framework, the Schmidt filter is introduced specifically so that map uncertainty can be considered while the computation cost only increases linearly with the size of the map keyframes (Zhang et al., 2022).

The map-based localization literature makes the memory–accuracy–speed trade-off explicit. The C-SKF retains full SKF-consistency while replacing dense map covariance storage by a sparse Cholesky factor, and sC-SKF further reduces per-feature back-solve cost by dividing the map into overlapping submaps (Dutoit et al., 2016). On a Google Tango tablet over a 60 m loop using a map built from 75 m of data, the reported position RMSE values are 6.2 cm for C-SKF, 6.6 for sC-SKF with 2 submaps, 8.3 for “perfect map” noise-inflation, and 14.7 for no map; map-uncertainty storage for the full map is approximately 76 MB for $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$8 versus approximately 1.2 GB for $$P = \begin{bmatrix} P_{AA} & P_{AS}\ P_{SA} & P_{SS} \end{bmatrix},$$9; and the full pipeline runs at approximately 6 Hz (Dutoit et al., 2016).

The observability and consistency trade-off is emphasized in the visual–inertial localization framework based on invariant filtering. The reported comparisons across EuRoC, Kaist, 4Seasons, and YQ campus indicate NEES close to 1 for the observability-constrained Schmidt invariant filter, while variants that ignore map uncertainty or use standard EKF formulations underestimate uncertainty or diverge (Zhang et al., 2022). In LODESTAR, the adaptive SKF and degeneracy-aware data exploitation are presented as a response to ill-posed estimation in degenerate LiDAR environments, where fixed states operate as anchors and measurement selection is driven by localizability contribution and Jacobian conditioning (Lee et al., 12 Nov 2025).

A common misconception is that freezing nuisance states means discarding their information. The tightly-coupled SKF formulations discussed here do the opposite: they preserve nuisance-state uncertainty in the innovation covariance and maintain the cross-correlation terms needed for future active-state corrections (Geneva et al., 2019). Another common misconception is that SKF alone guarantees correct observability. The visual–inertial localization results indicate that when map uncertainty is modeled, direct linearization can still violate the correct unobservable subspace, requiring an observability-constrained technique in addition to the Schmidt update (Zhang et al., 2022).