Sky-GVIO: Urban GNSS/INS/Vision Navigation

• Sky-GVIO is a fully integrated GNSS/INS/vision system that uses upward-pointing sky imagery and FCN-based segmentation to identify and mitigate non-line-of-sight satellite signals.
• It employs the S-NDM method to adjust GNSS measurement covariances by classifying satellites as LOS or NLOS, enhancing positioning reliability in urban canyons.
• The system supports both SPP and RTK modes within an MSCKF-based estimator, achieving meter-level accuracy in SPP and sub-decimeter precision in RTK.

Searching arXiv for the relevant Sky-GVIO and closely related Sky-GVINS papers. Sky-GVIO is a tightly coupled GNSS/INS/Vision navigation system for urban canyon environments that augments multi-sensor fusion with sky-view-image-based non-line-of-sight detection and mitigation. The formulation presented under the name “Sky-GVIO” combines GNSS observations, IMU measurements, stereo visual features, and an upward-pointing fish-eye camera whose segmented sky mask is used to classify satellites as line-of-sight or non-line-of-sight before GNSS measurements enter the estimator (Wang et al., 2024). In the cited work, the sky segmentation module is based on a Fully Convolutional Network (FCN), the NLOS-handling method is termed S-NDM, and the fusion backbone is MSCKF-based; the system is evaluated under both SPP-related and RTK-related tightly coupled models, with the stated goal of achieving continuous and accurate positioning in severe urban canyon conditions (Wang et al., 2024). A closely related antecedent is Sky-GVINS, which addressed the same problem family using sky segmentation for NLOS rejection in a GNSS-Visual-Inertial Navigation System rather than the FCN-based GNSS/INS/Vision formulation of Sky-GVIO (Yin et al., 2023).

1. Problem setting and system identity

Sky-GVIO addresses vehicle localization in urban canyons, where high-rise buildings, trees, poles, and elevated roads degrade GNSS through blockage, NLOS reception, and multipath (Wang et al., 2024). The motivating premise is that stand-alone sensing is insufficient in such environments: GNSS is vulnerable to NLOS and multipath, INS drifts over time, and vision can degrade under low texture, dynamic scenes, lighting changes, or long-term scale and pose drift when not globally constrained (Wang et al., 2024). The system therefore adopts multi-sensor integration, with GNSS providing absolute positioning, INS providing high-rate motion propagation, and vision providing relative pose constraints (Wang et al., 2024).

Within this fusion stack, the paper explicitly positions GNSS as the only subsystem supplying absolute position information, which makes raw GNSS quality control decisive for overall estimator performance (Wang et al., 2024). The central claim is that tightly coupled GNSS/INS/vision integration alone is insufficient in urban canyons unless GNSS measurements are screened or reweighted using environment-aware visibility reasoning (Wang et al., 2024). Sky-GVIO operationalizes this by using sky-view images to determine whether the sky is visible in the direction of each satellite; once the sky/non-sky segmentation is available, each satellite’s direction is projected into the fish-eye image, and the resulting visibility classification informs GNSS measurement mitigation (Wang et al., 2024).

The paper defines Sky-GVIO as an enhanced tightly coupled GNSS/INS/Vision navigation system with FCN-based sky segmentation and sky-view-image-aided NLOS detection and mitigation for urban canyons (Wang et al., 2024). Although the acronym “GVIO” is not explicitly expanded in the manuscript, the system is described as a GNSS/INS/Vision tightly coupled estimator with odometric and navigation functionality (Wang et al., 2024). A useful contextual distinction is provided by the related Sky-GVINS paper, in which “GVINS” denotes a GNSS-Visual-Inertial Navigation System; this suggests that Sky-GVIO belongs to the same methodological family but is formulated around GNSS/INS/Vision integration and evaluated with SPP-related and RTK-related models (Yin et al., 2023).

2. Architecture and sensing pipeline

Sky-GVIO takes as inputs GNSS observations, IMU measurements, stereo-camera visual measurements, and sky-view images from an upward-pointing fish-eye camera (Wang et al., 2024). In SPP mode, the GNSS input includes pseudorange observations; in RTK-related mode, it includes double-differenced carrier-phase or RTK-related GNSS observations (Wang et al., 2024). The system also uses satellite ephemerides, derived satellite azimuth and elevation, and offline calibration between camera and IMU frames (Wang et al., 2024).

The pipeline described in the block diagram begins with GNSS preprocessing, where satellite elevation and azimuth are computed from ephemeris and SPP or RTK observation equations are prepared (Wang et al., 2024). In parallel, the fish-eye sky camera provides sky-view images that are semantically segmented into sky and non-sky regions by the FCN module, yielding a binary sky mask (Wang et al., 2024). The S-NDM stage then combines the sky mask with satellite projection locations, classifies each satellite as LOS or NLOS, and adjusts the GNSS observation model or covariance accordingly (Wang et al., 2024). IMU data undergo compensation and mechanization for state propagation, while the stereo frontend performs feature extraction, matching, and tracking (Wang et al., 2024). The MSCKF back end then tightly fuses GNSS, INS, and vision observations to produce the navigation solution (Wang et al., 2024).

The estimator output includes position, velocity, attitude, IMU bias states, a GNSS clock state in SPP mode, double-difference ambiguity states in RTK mode, and camera clone states used by MSCKF (Wang et al., 2024). The paper does not define a separate integrity variable, but LOS/NLOS classification acts directly as GNSS quality control through covariance adjustment (Wang et al., 2024). The critical dependency chain is therefore from sky segmentation to LOS/NLOS detection, then to GNSS screening or reweighting, and finally to tightly coupled fusion (Wang et al., 2024).

This architectural choice differs from the earlier Sky-GVINS design chiefly in how the sky module interacts with the rest of the system. Sky-GVINS inherits a nonlinear optimization backend from GVINS and performs hard rejection of blocked satellites before factor insertion, whereas Sky-GVIO adopts an MSCKF-based tightly coupled estimator and injects sky-derived LOS/NLOS decisions into the GNSS covariance model (Yin et al., 2023, Wang et al., 2024). This suggests a methodological shift from binary exclusion toward mitigation by reweighting, although both systems rely on upward-looking sky visibility reasoning.

3. FCN-based sky segmentation

The sky segmentation module in Sky-GVIO is based on a Fully Convolutional Network, introduced because traditional methods such as thresholding, K-means, and region growing are described as sensitive to illumination and cloud conditions, especially near building edges or under elevated roads (Wang et al., 2024). Since segmentation errors directly lead to NLOS misclassification, the paper replaces hand-crafted segmentation with a learned model (Wang et al., 2024).

The network architecture consists of a VGG19 encoder and an FCN-8s decoder (Wang et al., 2024). The downsampling backbone includes 16 convolution layers with kernel size $3 \times 3$, stride 1, and 4 max-pooling layers (Wang et al., 2024). The decoder uses transposed convolution for upsampling, combines 8x, 16x, and 32x feature maps, and uses skip connections to fuse multi-scale features (Wang et al., 2024). The input is a sky-view fish-eye image of arbitrary size, and the output is a segmentation map of the same size (Wang et al., 2024). Although the general FCN formulation allows $n+1$ channels, only two labels are needed here: sky and non-sky (Wang et al., 2024).

The paper reports an inconsistency in dataset counts. In the method section, it states that 550 training datasets were made by the authors, while the experiment section reports a training dataset of 440 images and a testing dataset of 2000 images (Wang et al., 2024). The public dataset contains urban canyon scenes from two areas in Wuhan and includes objects such as trees, tall buildings, and light poles; manual semantic labels use blue for sky and black for non-sky (Wang et al., 2024). The dataset is released at the stated repository URL (Wang et al., 2024).

Quantitatively, the segmentation comparison includes Otsu, K-means, Region Growing, and the proposed FCN (Wang et al., 2024). The reported FPS and Accuracy values are as follows.

Method	FPS	Accuracy
Kmeans	0.34	49.50%
Otsu	5.47	36.45%
Region growth	3.69	44.96%
Ours	5.96	97.76%

The paper defines Accuracy as the percentage of correctly segmented images in the total processed images and argues that 5.96 FPS is sufficient for real-time use because the tightly coupled MSCKF update cycle is 1 s, matching GNSS at 1 Hz (Wang et al., 2024). Qualitatively, the FCN is described as more robust to clouds, varying light, building boundaries, and elevated bridge scenes (Wang et al., 2024). This contrasts with Sky-GVINS, where the preferred segmentation method was not a deep model but Otsu thresholding, selected because it provided the best empirical tradeoff on that paper’s dataset, with 96.5453% IoU at 0.0105 s per image (Yin et al., 2023). The divergence reflects a difference in experimental conditions and segmentation criteria rather than a contradiction in the general sky-aware navigation principle.

4. S-NDM and satellite visibility reasoning

The paper names its sky-view-image-aided GNSS NLOS detection and mitigation method S-NDM (Wang et al., 2024). Its operational logic is direct: a binary sky mask is computed from FCN segmentation, satellite ephemeris is used to derive azimuth and elevation, a fish-eye projection model maps each satellite into the sky-view image, and the binary sky mask at that projected pixel determines LOS or NLOS status (Wang et al., 2024). LOS satellites are illustrated as blue dots and NLOS satellites as red dots in the manuscript (Wang et al., 2024).

A notable limitation is that Sky-GVIO does not provide the explicit fish-eye projection equations in the manuscript; it states that the projection model follows prior work (Wang et al., 2024). Accordingly, only the conceptual mapping is available from the paper: $$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$ The calibration assumptions include offline IMU-camera extrinsic calibration, known left-right stereo calibration, and a calibrated fish-eye camera model, but the exact fisheye projection type and explicit camera-body-GNSS lever-arm treatment are not stated (Wang et al., 2024).

The mitigation stage is covariance-based rather than purely exclusion-based. The paper gives separate covariance models for LOS and NLOS pseudorange and carrier phase observations, with nominal covariance depending on elevation angle and SNR, and NLOS observations assigned larger covariance (Wang et al., 2024). The reported formulas are garbled in formatting, but the intended structure is explicit. For carrier phase, the NLOS covariance is scaled as

$$R^{k}_{L,\text{NLOS}} = K \times R^{k}_{L,\text{LOS}},$$

with the adopted parameters

$S_1 = 50,\quad A = 30,\quad S_0 = 10,\quad a = 20,$

standard deviations

$$\sigma_p = 0.3\ \text{m}, \qquad \sigma_L = 0.03\ \text{m},$$

and scale factor

$K = 10$

(Wang et al., 2024). The paper characterizes this as making the weighting of GNSS observations “more reasonable” (Wang et al., 2024).

This is a substantive distinction from the related Sky-GVINS approach. Sky-GVINS performs a hard visibility gate: satellites projected into non-sky regions are treated as NLOS and rejected before fusion (Yin et al., 2023). Sky-GVIO instead supports detection and mitigation through covariance modeling and reweighting (Wang et al., 2024). A plausible implication is that Sky-GVIO is designed to preserve more GNSS information when visibility classification is uncertain, whereas Sky-GVINS emphasizes pre-factor exclusion.

5. Estimation model and tightly coupled fusion

Sky-GVIO is built on MSCKF, described in the paper as an EKF-style visual-inertial fusion architecture using cloned camera poses (Wang et al., 2024). The GNSS observation models are separated into SPP and RTK-related cases. For SPP, the raw observation equations are

$$P = \rho + c(t_r - t^s) + I + T + \varepsilon_P \tag{1}$$

and

$$L = \rho + c(t_r - t^s) - I + T + \lambda N + \varepsilon_L \tag{2}$$

(Wang et al., 2024). The manuscript states that for SPP, equation (1) is sufficient (Wang et al., 2024). For RTK, it uses the double-difference representation

$$\Delta \nabla P = \Delta \nabla \rho + \Delta \nabla I + \Delta \nabla T + \Delta \nabla \varepsilon_P \tag{3}$$

and states that double differencing eliminates or reduces satellite orbit errors, satellite clock errors, receiver clock errors, and tropospheric and ionospheric delays (Wang et al., 2024).

The INS mechanization equations in the body frame are

$n+1$0

and

$n+1$1

(Wang et al., 2024). The paper notes that, for the low-cost IMU model used, Coriolis and centrifugal forces due to Earth rotation are ignored (Wang et al., 2024). The linearized INS error dynamics are stated as

$n+1$2

$n+1$3

$n+1$4

$n+1$5

with 15-state INS error vector

$n+1$6

(Wang et al., 2024).

The stereo visual measurement model is written as

$n+1$7

with left and right camera transforms given by equations (9) and (10), camera clone error state in equation (11), and visual residual in equation (12) (Wang et al., 2024).

The full error state is

$n+1$8

(Wang et al., 2024). In SPP mode, the GNSS-specific state is

$n+1$9

while in RTK mode it is

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$0

(Wang et al., 2024). The paper does not include troposphere or ionosphere states, GNSS extrinsic lever-arm states, or online camera-IMU extrinsic estimation in the state vector (Wang et al., 2024).

Continuous-time propagation is expressed as

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$1

and covariance propagation as

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$2

with 4th-order Runge–Kutta used for discretization and propagation (Wang et al., 2024). State augmentation for camera clones follows the standard MSCKF procedure, with the state and covariance augmented when a new image arrives and IMU-camera extrinsics $$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$3 and $$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$4 calibrated offline (Wang et al., 2024). The joint tightly coupled measurement equation is

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$5

(Wang et al., 2024).

6. SPP-related and RTK-related operating modes

A defining feature of Sky-GVIO is that the same sky-aided NLOS mechanism is evaluated under both SPP-related and RTK-related tightly coupled formulations (Wang et al., 2024). In the SPP-related version, GNSS uses pseudorange observations, the GNSS-specific state includes the receiver clock offset, and NLOS handling affects pseudorange covariance and thereby the quality of global fusion (Wang et al., 2024). The paper frames this mode as suitable for low-cost applications, smartphone navigation, and pedestrian navigation, and states that it delivers meter-level accuracy (Wang et al., 2024).

In the RTK-related version, GNSS uses double-differenced RTK observations, the state includes double-difference carrier-phase ambiguities, and carrier phase contributes stronger geometric constraints than SPP (Wang et al., 2024). The manuscript explicitly states that “we use the RTK float solution,” so the reported results correspond to RTK float tightly coupled integration rather than an integer-fixed RTK solution (Wang et al., 2024). This distinction is consequential because the paper does not report ambiguity fixing, integer fix rate, or fixed-solution statistics (Wang et al., 2024).

The system’s “harmonizes SPP with RTK methodologies” wording refers to this shared architecture: the same sensors, the same MSCKF backbone, the same visual subsystem, the same INS propagation, and the same S-NDM quality control are used, while only the GNSS-specific state and observation model differ between SPP and RTK (Wang et al., 2024). In this sense, Sky-GVIO is not two separate systems but a common sky-aware fusion framework instantiated with different GNSS measurement models.

The contrast with Sky-GVINS is again instructive. Sky-GVINS extends a GVINS-style optimization framework with sky-based NLOS rejection and evaluates primarily global trajectory accuracy in open and dense urban scenarios (Yin et al., 2023). Sky-GVIO instead emphasizes an MSCKF-based GNSS/INS/Vision formulation and explicitly compares SPP-related and RTK-related tightly coupled variants under the same sky-aware frontend (Wang et al., 2024). This suggests a change in experimental emphasis from trajectory-level comparison against baseline GNSS-VIO systems toward mode-specific GNSS integration analysis.

7. Experimental results, comparative context, and limitations

The experimental vehicle platform in Sky-GVIO consists of a Septentrio mosaic-X5 mini GNSS receiver, a NovAtel GNSS-850 antenna, two forward-looking FLIR BFS-U3-31S4C-C cameras, one sky-pointing fish-eye camera FE185C057HA-1, a tactical-grade NovAtel SPAN-ISA-100C IMU, an ADIS-16470 MEMS IMU, and a time synchronization board; NovAtel ProPak7 is also mentioned in the hardware illustration context (Wang et al., 2024). Sampling rates are GNSS at 1 Hz, MEMS-IMU at 100 Hz, forward-looking cameras at 10 Hz, and fish-eye camera at 1 Hz, with all sensors synchronized to GPS time using GNSS PPS via the time synchronization board (Wang et al., 2024). The reference trajectory is produced by tightly coupled multi-GNSS PPK/INS bidirectional smoothing using commercial IE 8.9 software (Wang et al., 2024).

Data were collected in Wuhan on September 3, 2023 in a typical urban canyon with high-rise buildings, dense foliage, and overpasses or elevated structures (Wang et al., 2024). The route was chosen specifically because it contains severe NLOS, multipath, cycle slips, and poor satellite geometry; the LOS satellite count in some deep urban canyon or overpass scenes dropped below 4 (Wang et al., 2024). The experiments are real-world vehicle experiments rather than simulation (Wang et al., 2024).

For tightly coupled SPP/INS/Vision, the reported RMSE in East, North, and Up is

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$6

whereas tightly coupled SPP/INS/Vision/Sky yields

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$7

(Wang et al., 2024). The relative improvements are 36% in East, 29% in North, and 27% in Up (Wang et al., 2024). For tightly coupled RTK/INS/Vision, the reported RMSE is

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$8

and for tightly coupled RTK/INS/Vision/Sky it is

$$(\text{satellite ephemeris}) \rightarrow (\text{azimuth},\text{elevation}) \rightarrow (\text{fish-eye image pixel}) \rightarrow \text{sky/non-sky test}.$$9

(Wang et al., 2024). The relative improvements are 24% in East, 15% in North, and 25% in Up (Wang et al., 2024). The paper summarizes these findings by stating that Sky-GVIO achieves meter-level accuracy under SPP mode and sub-decimeter precision with RTK (Wang et al., 2024).

The baselines include TC-SPP/INS/Vision, TC-RTK/INS/Vision, their sky-aware counterparts, VINS-Mono, and GVINS (Wang et al., 2024). The paper reports GVINS RMSE as

$$R^{k}_{L,\text{NLOS}} = K \times R^{k}_{L,\text{LOS}},$$0

and notes that GVINS outperforms the authors’ baseline TC-SPP/INS/Vision because GVINS also uses Doppler measurements in addition to pseudorange, visual, and inertial constraints (Wang et al., 2024). Sky-GVIO in SPP-related mode nevertheless outperforms GVINS, with

$$R^{k}_{L,\text{NLOS}} = K \times R^{k}_{L,\text{LOS}},$$1

which the authors attribute to stricter GNSS quality control via S-NDM (Wang et al., 2024). VINS-Mono is said to accumulate drift significantly, and no RMSE is reported in Table III because the errors become too large (Wang et al., 2024).

The earlier Sky-GVINS results provide further comparative context. In dense urban campus scenarios, the Otsu-based Sky-GVINS variant improved ATE RMSE from 25.06 m to 4.27 m on Street02 relative to GVINS, while showing smaller gains or no penalty in more open conditions (Yin et al., 2023). That earlier study also showed that poor segmentation quality could severely degrade performance, as seen in the local-threshold Sky-GVINS variant (Yin et al., 2023). This reinforces a common conclusion across the two system families: sky-aware GNSS filtering is only as reliable as the sky segmentation and projection chain.

The limitations reported for Sky-GVIO are both explicit and implicit. The paper does not provide explicit sky-projection equations, uses RTK float solution only, and offers no detailed analysis of integer ambiguity fixing, fix rate, NLOS classification precision or recall, calibration sensitivity, or weather and night robustness (Wang et al., 2024). The fish-eye camera is used only for NLOS detection rather than as a direct visual measurement source in the navigation backend (Wang et al., 2024). The manuscript also indicates that performance may degrade if sky segmentation quality drops, fish-eye camera calibration is poor, the sky is heavily obscured, lighting changes severely, or very few LOS satellites remain (Wang et al., 2024). Future work is stated to include using fish-eye camera data beyond NLOS detection and incorporating prior information such as high-precision maps to target centimeter-level accuracy and improve robustness (Wang et al., 2024).

Taken together, Sky-GVIO and Sky-GVINS define a coherent line of research in sky-aware GNSS-visual-inertial navigation for urban canyons. Sky-GVINS demonstrates the value of lightweight sky segmentation and hard NLOS rejection in a GVINS-based optimization framework (Yin et al., 2023). Sky-GVIO extends the concept into an FCN-based GNSS/INS/Vision MSCKF framework with S-NDM covariance mitigation and explicit SPP-related versus RTK-related evaluation (Wang et al., 2024). The shared methodological premise is that GNSS should not be fused as if all satellites were equally visible: the local sky geometry must first be inferred from an upward-looking sensor, and that visibility information must then regulate how GNSS enters the estimator.