Airborne Vector Magnetic Calibration

• Airborne vector magnetic calibration is a process that converts contaminated aircraft magnetometer readings into precise geographic magnetic estimates by addressing sensor biases and platform interference.
• It employs methodologies such as intrinsic sensor calibration, coordinate transformation, and compensation for permanent, induced, and eddy-current aircraft fields using models like Tolles–Lawson.
• The method balances scalar and full vector approaches to minimize errors from attitude misalignment, temperature drift, and dynamic interference, validated via tailored in-flight maneuvers.

Airborne vector magnetic calibration is the set of procedures by which a triaxial magnetic measurement made on an aircraft or UAV is transformed from a contaminated platform-fixed observation into a geographically referenced estimate of the external field. In practice, the problem is broader than bench calibration of a magnetometer triad: it couples intrinsic sensor calibration, sensor-to-INS misalignment, coordinate transformation from magnetometer or body frame to Earth or geographic frame, compensation of aircraft-generated magnetic fields, and management of drift, temperature dependence, timing latency, and motion-induced error. A compact statement of the measurement problem is

$\mathbf{B}_t^b = \mathbf{B}_e^b + \mathbf{B}_a^b,$

where the desired background field in body coordinates is corrupted by the platform field (Hager et al., 6 May 2026). Review literature on airborne vector platforms further stresses that vector data must be corrected to the geographic coordinate system and corrected for temperature drift, and that an error of $10^{-3}{}^{\circ}$ of one inertial measurement unit angle could produce an error of $1$ nT for components after rotation from magnetometer coordinates to Earth coordinates (Liu et al., 2020).

1. Problem formulation, frames, and physical scope

Airborne vector magnetic calibration begins from the fact that an aircraft-mounted magnetometer does not observe only the geomagnetic field. In the body frame, the total field is modeled as the sum of the external field and the aircraft’s own magnetic interference, and the aircraft field is further decomposed in Tolles–Lawson form as

$$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$

where $\mathbf{p}$ is the permanent magnetization vector, $\mathbf{N}\in\mathbb{R}^{3\times 3}$ is the induced-field matrix, and $\mathbf{E}\in\mathbb{R}^{3\times 3}$ is the eddy-current matrix (Hager et al., 6 May 2026). This decomposition already shows that calibration is not merely an offset-removal task: the platform field depends on attitude through $\mathbf{B}_e^b$, and on maneuver dynamics through $\dot{\mathbf{B}_e^b}$.

The frame structure is correspondingly central. The 2026 airborne analysis distinguishes an Earth frame $\{e\}$, a body frame $10^{-3}{}^{\circ}$0, and a sensor frame $10^{-3}{}^{\circ}$1, and assumes $10^{-3}{}^{\circ}$2 for analytical convenience because static sensor misalignment is mathematically absorbable into calibration coefficients (Hager et al., 6 May 2026). By contrast, IMU-aided magnetometer calibration literature makes the misalignment explicit through a constant rotation $10^{-3}{}^{\circ}$3 between the magnetometer frame and the inertial body frame, with the calibrated vector required for downstream orientation fusion rather than only norm restoration (Kok et al., 2016).

Airborne practice therefore spans at least four coupled subproblems. The first is intrinsic vector-sensor calibration: bias, scale factor, non-orthogonality, and fixed misalignment. The second is platform magnetic compensation: permanent, induced, and eddy-current aircraft fields. The third is attitude-aware frame registration from sensor or body axes into Earth or geographic coordinates. The fourth is temporal stability: thermal drift, dynamic onboard interference, timing lag, and low-observability attitudes. The review of sensing platforms treats these jointly and notes that final magnetic measurement accuracy depends on the accuracies of the magnetometer and the attitude measurement system, respectively (Liu et al., 2020).

2. Scalar projection models and full vector formulations

The classical Tolles–Lawson derivation starts from

$10^{-3}{}^{\circ}$4

and uses the small-aircraft-field approximation to obtain

$10^{-3}{}^{\circ}$5

The practical consequence is that scalar corruption is modeled through the projection of the aircraft field onto the total-field direction, which converts aeromagnetic compensation into a linear regression problem once permanent, induced, and eddy-current terms are parameterized (Gnadt et al., 2022). In the standard reduced form, the regressors are the three direction-cosine permanent terms, six induced terms, and nine eddy-current terms, giving the familiar 18-parameter Tolles–Lawson model (Gnadt et al., 2022).

The same source also makes explicit that Tolles–Lawson originates from a vector aircraft-field model,

$10^{-3}{}^{\circ}$6

so that vector aircraft interference can be reconstructed from fitted coefficients rather than only projected away from a scalar channel (Gnadt et al., 2022). This matters because airborne vector magnetic calibration often needs the compensated vector

$10^{-3}{}^{\circ}$7

not just a corrected total-field magnitude.

A more recent analytical comparison places scalar and full-vector airborne calibration on the same footing. In 3D vector calibration, the regression target is

$10^{-3}{}^{\circ}$8

whereas the scalar approach retains only the projection of aircraft interference onto the Earth-field direction (Hager et al., 6 May 2026). The decisive conclusion is that scalar calibration errors due to misalignment and proxy-field approximations are small, typically scaling like

$10^{-3}{}^{\circ}$9

while vector calibration errors are intrinsically first-order sensitive to attitude or reference-field errors, scaling like

$1$0

At $1$1 and $1$2, this gives $1$3, whereas the comparable scalar residual with $1$4 is $1$5 (Hager et al., 6 May 2026).

The distinction can be summarized compactly.

Paradigm	Representative formulation	Dominant sensitivity
Scalar compensation	$1$6	Errors scale with $1$7 or $1$8
Full vector calibration	$1$9	Errors scale with $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$0
Vector aircraft-field reconstruction	$$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$1	Depends on vector-model validity

This has direct methodological consequences. Scalar compensation remains the robust operational choice under realistic airborne errors, while full vector calibration becomes an attitude-limited estimation problem rather than a magnetometer-noise-limited one (Hager et al., 6 May 2026).

3. Intrinsic triaxial calibration and magnetometer–INS alignment

Before aircraft compensation is attempted, the vector sensor itself must be calibrated. A rigorous IMU-aided formulation models the raw triaxial magnetometer as

$$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$2

where $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$3 is the full calibration or distortion matrix, $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$4 is the additive offset, $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$5 rotates the constant local magnetic field $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$6 into the body frame, and $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$7 is measurement noise (Kok et al., 2016). The matrix $$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$8 lumps together scale-factor error, non-orthogonality, soft-iron distortion, and fixed frame misalignment, while the corrected vector is

$$\mathbf{B}_a^b = \mathbf{p} + \mathbf{N}\mathbf{B}_e^b + \mathbf{E}\dot{\mathbf{B}_e^b},$$9

This framework is explicitly optimized for heading-relevant performance rather than only mapping an ellipsoid to a sphere (Kok et al., 2016).

The maximum-likelihood estimator is implemented with an EKF inside a nonlinear offline optimization, minimizing

$\mathbf{p}$0

over parameters that include $\mathbf{p}$1, $\mathbf{p}$2, the local magnetic field direction, gyroscope bias, and the measurement noise covariances (Kok et al., 2016). Because ellipsoid fitting identifies $\mathbf{p}$3 only up to a rotation, inertial measurements are used to recover the rotation part through a separate misalignment-identification step. The empirical result is that unit-norm restoration alone is not sufficient: on experimental data, mean heading error was reduced from $\mathbf{p}$4 with the initialization-only estimate to $\mathbf{p}$5 with the full ML estimate, and maximum heading error from $\mathbf{p}$6 to $\mathbf{p}$7 (Kok et al., 2016).

A complementary laboratory method uses a triaxial Helmholtz coil system and a remote Overhauser scalar magnetometer to calibrate triaxial magnetometers in the Earth-field range without Earth-field nulling and without physically rotating the sensor (Zikmund et al., 2019). The method estimates axis sensitivities, non-orthogonality or angular misalignment, and offsets from controlled coil-generated field changes. In the reported test, the extended calibration uncertainty was better than $\mathbf{p}$8 ppm in sensitivity and $\mathbf{p}$9 in orthogonality, with reported final sensitivities

$\mathbf{N}\in\mathbb{R}^{3\times 3}$0

and orthogonality-angle results

$\mathbf{N}\in\mathbb{R}^{3\times 3}$1

(Zikmund et al., 2019).

These intrinsic methods solve the sensor-calibration subproblem, not the full airborne one. The Helmholtz–Overhauser procedure is explicitly described as a pre-flight calibration method for the intrinsic vector-sensor parameters of a triaxial magnetometer, and its own summary states that it does not address aircraft magnetic interference, platform-current effects, dynamic eddy-current effects, or in-flight compensation (Zikmund et al., 2019). The IMU-aided ML method similarly assumes a constant local magnetic field and slow rotation, which is a strong approximation for flight segments with significant translational acceleration or spatial field variation (Kok et al., 2016).

4. Attitude bottleneck, observability, and maneuver design

The main obstacle separating scalar aeromagnetic compensation from robust full vector airborne calibration is the body-frame representation of the background field. In the 2026 analysis, full vector calibration is described as intrinsically first-order sensitive to attitude errors, irrespective of the accuracy of the magnetic field measurements, because inaccurate representation of the background field direction in the body frame leaks the large Earth field into orthogonal axes (Hager et al., 6 May 2026). This “attitude bottleneck” reframes calibration design: richer magnetometers alone do not solve the dominant error source.

The same conclusion appears in the platform review, which emphasizes that vector data must be rotated into the Earth or geographic frame and that tiny IMU angle errors can create nT-level component errors after rotation (Liu et al., 2020). The implication is that observability is not just a matter of estimating a calibration matrix; it is a matter of adequately exciting the joint magnetic–attitude model and controlling the orientation reference during estimation.

Calibration maneuvers are therefore structural, not incidental. The factor-graph in-flight calibration study uses pitch, roll, and yaw doublets on four headings roughly $\mathbf{N}\in\mathbb{R}^{3\times 3}$2 apart, with magnitudes representative of large-aircraft-compatible maneuvers (Lathrop et al., 2023). Classical Tolles–Lawson calibration similarly relies on roll, pitch, and yaw excitations, typically sinusoidal, to inject aircraft magnetic content into a frequency band that can be separated from the slowly varying Earth field (Gnadt et al., 2022). The review literature presents the same idea more generally: aircraft and UAV magnetic compensation research focuses on suppressing magnetic interference generated by aircraft maneuvers and on-board electronic equipment, with estimation methods including ridge regression, truncated singular value decomposition, recursive least-squares, partial least-squares, support vector methods, c-k class estimation, and principal component analysis (Liu et al., 2020).

Observability is also degraded by directional singularities or low-sensitivity attitudes. In airborne language this is often discussed as heading error or poor excitation; in atomic vector magnetometry the same issue appears as dead zones. The quantum Rabi-frequency systems are not airborne instruments, but their calibration summaries are unusually explicit about a transferable design rule: use multiple excitation or observation geometries so that no attitude yields total loss of observability, and downweight channels with poor calibration quality during inversion (Kiehl et al., 2024). This suggests a general airborne principle: maneuver diversity and channel weighting should be treated as integral parts of calibration architecture rather than post hoc fixes.

5. In-flight, adaptive, and machine-learning architectures

Several recent systems move calibration from a pre-flight or offline batch activity toward in-flight joint estimation. A factor-graph approach for magnetic anomaly navigation fuses a vector magnetometer, a scalar magnetometer, and inertial measurements in a single nonlinear optimization and is designed for conditions that are difficult for classical methods: large hard-iron or permanent moments and time-varying ambient field during the calibration maneuver (Lathrop et al., 2023). Its vector measurement model is

$\mathbf{N}\in\mathbb{R}^{3\times 3}$3

with scalar model

$\mathbf{N}\in\mathbb{R}^{3\times 3}$4

The graph estimates hard iron, vector bias, vector scale factors, vector non-orthogonality, external field at each time, and orientation at each time, while explicitly allowing the external field to vary through smoothness factors (Lathrop et al., 2023). On real data with a known hard-iron change, the estimated change had componentwise error

$\mathbf{N}\in\mathbb{R}^{3\times 3}$5

with norm $\mathbf{N}\in\mathbb{R}^{3\times 3}$6, compared with $\mathbf{N}\in\mathbb{R}^{3\times 3}$7 for TWOSTEP and $\mathbf{N}\in\mathbb{R}^{3\times 3}$8 for Tolles–Lawson (Lathrop et al., 2023). Its stated limitation is equally important: the current algorithm is limited to hard-iron calibration and does not estimate induced, eddy-current, or soft-iron terms (Lathrop et al., 2023).

A second line of work embeds platform-field calibration directly into navigation filtering. A fully adaptive MagNav architecture uses an augmented-state EKF in which both the 18 Tolles–Lawson coefficients and the parameters of a small neural network are estimated online, with the network constrained to a residual-learning role: $\mathbf{N}\in\mathbb{R}^{3\times 3}$9 The Kalman-filter update is interpreted as an online Natural Gradient step for the neural-network parameters, and the system is presented as having “cold-start” capability without dedicated calibration flights or offline pretraining (Hager et al., 9 Mar 2026). On the MagNav Challenge dataset, the most pronounced improvement occurs for a high-interference sensor: for Magnetometer 2, TL-only online calibration gives DRMS $\mathbf{E}\in\mathbb{R}^{3\times 3}$0 m in cold start and $\mathbf{E}\in\mathbb{R}^{3\times 3}$1 m in warm start, whereas the hybrid TL+NN gives $\mathbf{E}\in\mathbb{R}^{3\times 3}$2 m and $\mathbf{E}\in\mathbb{R}^{3\times 3}$3 m, respectively (Hager et al., 9 Mar 2026). The same paper also reports cold-start convergence over roughly $\mathbf{E}\in\mathbb{R}^{3\times 3}$4–$\mathbf{E}\in\mathbb{R}^{3\times 3}$5 km and notes that larger neural networks can reduce observability and “cannibalize” navigation signal (Hager et al., 9 Mar 2026).

A related physics-informed machine-learning approach retains Tolles–Lawson compensation as a preprocessing or calibration baseline and uses Liquid Time-Constant Networks with a Closed-form Continuous-time variant to model residual nonlinear, time-varying aircraft interference (Nerrise et al., 2024). On reported test flights, the best RMSE values are $\mathbf{E}\in\mathbb{R}^{3\times 3}$6 nT and $\mathbf{E}\in\mathbb{R}^{3\times 3}$7 nT for LTC-CfC, compared with $\mathbf{E}\in\mathbb{R}^{3\times 3}$8 nT and $\mathbf{E}\in\mathbb{R}^{3\times 3}$9 nT for the Tolles–Lawson baseline, summarized as about $\mathbf{B}_e^b$0 RMSE reduction on average versus T–L (Nerrise et al., 2024). The target remains a cleaned scalar magnetic signal, so this is better described as a hybrid multi-sensor regression model for scalar compensation than as a full vector calibration method (Nerrise et al., 2024).

In-flight calibration under poor platform cleanliness has also been demonstrated in space systems in a way that transfers directly to airborne conditions with inadequate sensor standoff. The Tianwen-1 orbiter magnetometer is calibrated in sequence by first identifying and removing spacecraft-generated dynamic magnetic fields through dual-sensor comparison, then estimating slowly varying offsets using Alfvénic intervals in which the magnetic-field magnitude remains nearly constant while the vector direction varies (Zou et al., 2023). The accepted-event criterion is

$\mathbf{B}_e^b$1

evaluated over a $\mathbf{B}_e^b$2D offset grid of $\mathbf{B}_e^b$3 with $\mathbf{B}_e^b$4 step size (Zou et al., 2023). The method does not solve scale, non-orthogonality, or misalignment matrix errors, but it provides a rigorous template for sequential treatment of dynamic interference first and slow offset drift second (Zou et al., 2023). This suggests that airborne systems with multiple standoffs or auxiliary references can exploit spatial redundancy for jump-like onboard disturbances and reserve slower model updates for thermal or configuration drift.

6. Internal references, drift tracking, and quantum analogues

A recent and conceptually important development is the treatment of the reference geometry itself as a calibrated, drifting state. In a vector optically pumped magnetometer based on microwave-driven Rabi measurements, the magnetic-field direction is inferred from its orientation relative to a calibrated microwave polarization ellipse rather than by direct componentwise sensing in an orthogonal basis (Kiehl et al., 2024). The system uses scalar free-induction decay for field magnitude and transition-resolved Rabi data for direction, with the forward model written in dressed-state form rather than a naive two-level approximation. The calibration explicitly estimates 12 distinct microwave phasors—60 microwave parameters in total—and includes off-resonant driving, nonlinear Zeeman effects, and buffer-gas shifts in the forward model (Kiehl et al., 2024).

Several features are directly transferable at the level of calibration architecture. The microwave vector reference is not assumed stable or perfectly known; instead, it is recalibrated over known orientations, drift is tracked through repeated conditions, and a residual consistency statistic

$\mathbf{B}_e^b$5

is proposed as a thresholded trigger for recalibration (Kiehl et al., 2024). Calibration residuals are propagated as uncertainty weights in operational inversion, and dead zones are mitigated by using multiple microwave polarization ellipses plus an alternate simultaneous-precession-and-Rabi mode. The reported performance is mean directional accuracy $\mathbf{B}_e^b$6 for standard Rabi inversion and $\mathbf{B}_e^b$7 for SPaR in the dead-zone region, with best directional sensitivity $\mathbf{B}_e^b$8 (Kiehl et al., 2024). The paper itself states that what transfers are the calibration architecture and estimation philosophy: use an overdetermined forward model, include physically known systematic terms in that model, calibrate the internal reference against many orientations, monitor residuals for drift, employ alternate measurement modes for singular attitudes, and weight operational inversion by calibration quality (Kiehl et al., 2024).

A second compact atomic system based on Zeeman Rabi oscillations makes the same point with a different internal reference. Here, a series of resonant RF polarization ellipses are calibrated by rotating a known DC magnetic field through $\mathbf{B}_e^b$9 random but predefined orientations, and field direction is then solved from six complementary excitation states (Menon et al., 9 Mar 2026). The fitted model includes RF Stark shifts, Bloch–Siegert shifts, and an RF-based heading-error systematic arising from the nonlinear Zeeman effect, rather than treating them as residual nuisances (Menon et al., 9 Mar 2026). The instrument achieves $\dot{\mathbf{B}_e^b}$0 mean angular accuracy over $\dot{\mathbf{B}_e^b}$1 random field directions and angular noise densities as low as $\dot{\mathbf{B}_e^b}$2, with current calibration time about $\dot{\mathbf{B}_e^b}$3 s and an estimated drift contribution of about $\dot{\mathbf{B}_e^b}$4 (Menon et al., 9 Mar 2026). The calibration logic is again the main airborne lesson: define a controllable vector reference in the sensor frame, rotate a known field through many orientations, fit a forward model that includes geometry plus systematic biases, and use redundant directional observables to avoid dead zones (Menon et al., 9 Mar 2026).

Taken together, these atomic systems do not provide ready-made equations for fluxgates, Hall sensors, SQUIDs, or conventional airborne OPM packages. They do, however, sharpen a broader conclusion that now also appears in airborne simulation and navigation work: high vector accuracy is obtained only when reference geometry, attitude representation, drift, and excitation diversity are treated as explicit states or modeled systematics, not as fixed assumptions. This suggests that airborne vector magnetic calibration is best understood not as a single matrix fit, but as a layered estimation problem combining intrinsic triad calibration, attitude-aware frame registration, platform-field compensation, residual monitoring, and periodic in situ recalibration when the internal or external reference state no longer remains consistent.