Scalar and Vector Airborne Platform Calibration Using Quantum and Classical Magnetometers and Inertial Sensors

Abstract: Airborne magnetometry requires rigorous calibration to isolate geomagnetic signals from sensor errors and platform magnetic fields. This magnetic compensation is needed for applications like geophysical exploration and magnetic anomaly navigation. The standard approach utilizes a quantum scalar Optically Pumped Magnetometer (OPM) and a less sensitive fluxgate vector sensor for attitude information. This configuration typically results in a scalar approximation of the platform field. Advancements in high-sensitivity Diamond Nitrogen-Vacancy (NV) vector magnetometers now enable a re-evaluation of the standard hardware configuration and full vector calibration models. We show through rigorous theoretical analysis that scalar calibration models are robust to misalignment. Vector calibration models, however, are intrinsically first-order sensitive to attitude errors, irrespective of the accuracy of the magnetic field measurements. These errors arise from inaccurate representation of the background field direction in the body frame, and can amplify small orientation errors into noticeable calibration residuals. Using realistic sensor models and flight trajectories, we evaluate different sensor configurations for magnetic calibration and assess the use of onboard Inertial Navigation Systems (INS) as an independent attitude reference to enable stable compensation. Our results suggest that quantum vector magnetometers like NV sensors are not sufficient to solve the attitude bottleneck for airborne vector magnetic calibration. However, as a single sensor capable of providing both absolute field and directional measurements, they may offer benefits regarding sensor placement, synchronization, and scalar calibration accuracy.

• The paper demonstrates that scalar calibration maintains single-digit nanotesla residuals under realistic sensor noise and environmental conditions.
• It employs comprehensive simulations and error models, comparing 1D and 3D Tolles-Lawson calibration using quantum, fluxgate, and NV magnetometers.
• The study reveals that vector calibration is highly sensitive to attitude errors, making it less effective than robust scalar approaches.

Platform Calibration with Quantum and Classical Magnetometers: A Technical Analysis

Introduction

Airborne platform calibration of magnetometers is a critical enabler for geomagnetic surveying, magnetic anomaly navigation (MagNav), and geophysical exploration. The complexity of separating geomagnetic signals from substantial platform-generated magnetic interference necessitates precise calibration protocols. Traditionally, these rely on a combination of quantum scalar optically pumped magnetometers (OPMs) for absolute field magnitude and classical fluxgate vector magnetometers for attitude determination. However, the advancement of high-sensitivity vector-capable diamond nitrogen-vacancy (NV) quantum magnetometers motivates a rigorous reevaluation of both hardware configurations and calibration algorithms. This paper conducts a comprehensive theoretical and simulation-based comparison of scalar (1D) and vector (3D) Tolles-Lawson (TL) calibrations, critically evaluates the error sensitivity of each model, and quantifies their performance using realistic sensor error models and flight profiles.

Theoretical Framework: Tolles-Lawson Calibration and Error Sensitivity

Platform calibration leverages the TL model to parameterize the aircraft's magnetic interference as the sum of permanent, induced, and eddy-current fields, all expressed in the body frame. Scalar (1D) calibration restricts compensation to the projection of the interference along the main geomagnetic field, whereas vector (3D) calibration regresses all orthogonal components.

A central analytical result is that scalar calibration exhibits inherent robustness to both sensor and reference attitude errors, with systematic errors from geometric projection and misalignment remaining on the order of $\sim 1$ nT even when platform fields are misaligned by $90^\circ$ relative to Earth's field. By contrast, the vector calibration model is fundamentally first-order sensitive to errors in the orientation of the reference background field in the body frame; a $0.1^\circ$ error in attitude estimation can induce calibration residuals on the order of $87$ nT due to the amplification by the Earth's field magnitude. These results establish the dominance of the scalar model in operational environments where precise real-time attitude knowledge is unavailable or impaired by sensor noise, drift, or reference field uncertainty.

Figure 2: Measured heading errors $\Delta F$ in the total field from a QuSpin QTFM Gen-2 during controlled rotations, highlighting sensitivity to orientation.

Quantum and Classical Sensor Error Models

The study implements detailed error models for OPMs, fluxgate magnetometers, and both field-grade and lab-grade NV quantum magnetometers. The OPM model incorporates deterministic heading errors with multi-harmonic expansions and dead-zone SNR degradation. Fluxgate errors are characterized by non-orthogonality, scale and offset drifts, and dominant Barkhausen $1/f$ noise. NV magnetometer models consider orthogonality and scaling errors, temperature-dependent zero-field splitting as a dominant drift mechanism (with compensation efficiency $\lambda_t$), and geometric SNR penalties stemming from sensor axis alignment.

Figure 4: Magnetometer noise characterizations, showing amplitude spectral densities (ASD) and noise floors for different sensor classes [Fescenko2020].

Figure 3: ASD and ADEV of simulated 20 Hz onboard magnetometer measurements incorporating altitude-dependent temperature drifts, exemplifying the impact of environmental conditions on instrument stability.

The simulation framework further propagates these errors through FIR bandwidth filters appropriate to each sensor's measurement chain. This rigorous stochastic modeling, anchored in published results and manufacturer specifications, provides credible estimates of in situ instrument behavior.

Simulation Methodology: Flight Trajectories and Signal Corruption

Test scenarios include a dynamic calibration trajectory comprising repeated pitch, roll, and yaw maneuvers, and a separate quasi-linear validation trajectory, both at realistic UAV speeds. Platform field configurations span random typical UAV profiles ($\sim$40–150 nT) and "perpendicular stress tests" with strong interference components orthogonal to Earth's field ($\sim$500–1100 nT). Ground truth magnetic fields are systematically corrupted with modeled sensor and environmental errors, including significant temperature-induced drifts for non-OPM sensors.

Figure 7: Calibration and validation flight trajectories with representative maneuvers and velocities used in simulation.

Results: Calibration Performance and Model Robustness

Idealized Sensor Cases

When perfect knowledge of the external reference frame is assumed, scalar and vector models achieve near-identical, sub-nanotesla calibration residuals. However, when realistic vector attitude references are emulated using vector magnetometer or tactical-grade INS data, the scalar model preserves single-digit nanotesla accuracy across all tested platform configurations. In contrast, vector calibration residuals escalate precipitously due to attitude reference drift and measurement uncertainty.

Figure 1: Calibration and validation residuals in ideal noise-free sensor conditions for 1D and 3D models, showing the vector model's potential under theoretically perfect reference knowledge.

Realistic Sensor Performance

Incorporating empirically motivated sensor noise and drift, the two-magnetometer (Fluxgate + OPM) state-of-the-art remains superior to both current field-grade and lab-grade NV configurations under operational assumptions. Nonetheless, lab-grade NV sensors approach and at times surpass dual-sensor performance, especially in challenging orthogonal interference regimes.

Figure 10: Performance across magnetometer configurations for 1D and 3D models using vector magnetometer attitude for $B_e^b$, illustrating persistent superiority of scalar calibration and progress in lab-grade NV sensors.

Using tactical-grade INS-derived attitude as an external reference for both regression and target confirmation further establishes that residuals are fundamentally limited by the scalar sensor's noise floor rather than the vector reference. Even with idealized (error-free) attitude references, only lab-grade NV sensors exhibit minor further improvement (<0.5 nT), confirming that calibration accuracy plateaus due to irreducible sensor/model constraints.

Figure 6: Calibration and validation for different sensor setups with tactical-grade INS-derived attitude for $90^\circ$0, confirming that model and scalar sensor performance, rather than reference accuracy, bound achievable residuals.

Implications and Future Directions

These results reinforce several critical points:

• Scalar calibration remains the most robust and reliable solution for airborne platform correction under any realistic error assumptions, achieving single-digit nanotesla residuals in all tested scenarios.
• 3D vector calibration is fundamentally limited by first-order sensitivity to reference attitude errors; absent perfect inertial or external reference, its operational use is not justified.
• NV vector magnetometers are not independently sufficient to solve the "attitude bottleneck": even with ideal vector measurements, attitude errors dominate vector model residuals.
• Integrated high-accuracy quantum vector magnetometers (e.g., lab-grade NV sensors) offer unique advantages by combining vector and absolute field measurements, potentially simplifying sensor placement, eliminating cross-sensor misalignment issues, and marginally improving calibration in situations where the sensor becomes the dominant error source.

The practical threshold for post-calibration residuals in airborne MagNav is approximately 1 nT. Existing technology, especially classical two-sensor systems, meets this threshold, and only significant further improvements in vector attitude knowledge or sensor noise are likely to shift the calibration bottleneck from attitude error to sensor/model constraints.

Future research demands extension of these simulation frameworks to encompass dynamic and nonlinear time-varying interference, as well as large-scale field validation campaigns using next-generation vector quantum sensors and advanced real-time compensation algorithms.

Conclusion

Scalar 1D Tolles-Lawson calibration, when employed with state-of-the-art sensors, constitutes a reliable, robust baseline for airborne platform magnetic calibration. Vector 3D calibration, although mathematically attractive for complete platform field removal, is rendered operationally nonviable by the acute amplification of attitude reference errors. High-accuracy lab-grade quantum NV sensors marginally advance the state-of-the-art and may enable new system architectures that simplify calibration logistics and integrate vector and scalar measurements. However, calibration accuracy remains ultimately bounded by either irreducible model errors or the scalar sensor's noise characteristics. These findings delineate the theoretical and practical limitations of vector magnetometer calibration and clarify the technical priorities for the next generation of airborne magnetic sensing and navigation systems. Future work should pivot toward robustly addressing dynamic field disturbances and further integration of quantum sensor technologies into operational geophysical platforms.