InCaRPose: In-Cabin Pose Estimation

• The paper introduces a single-step learning framework that directly predicts the full 6-DoF relative pose, including both rotation and metric translation, using a frozen self-supervised ViT backbone and a Transformer decoder.
• It employs a quaternion-based loss function to achieve superior rotational accuracy (as low as 2.75°) and ensures real-time inference (<15 ms) critical for safety-sensitive automotive applications.
• The method is trained exclusively on synthetic data and leverages aggressive domain randomization to robustly generalize to real-world, high-distortion fisheye images in automotive interiors.

Searching arXiv for the specified paper to ground the article and citation. InCaRPose is a learning-based framework for estimating the relative 6-DoF pose between two in-cabin camera views, introduced as a method for camera extrinsic calibration in automobile interiors (Stillger et al., 4 Apr 2026). It is designed for settings in which wide-angle or fisheye NIR cameras mounted on a movable rear-view mirror undergo repeated, safety-relevant pose changes, making robust recovery of both rotation and metric translation necessary for downstream perception tasks such as driver monitoring and occupant positioning. The method combines a frozen self-supervised ViT backbone with a Transformer-based decoder and a quaternion-based metric pose objective, and it is trained exclusively on synthetic data while targeting generalization to real-world cabin environments (Stillger et al., 4 Apr 2026).

1. Problem setting and calibration objective

Camera extrinsic calibration is the underlying task addressed by InCaRPose. In the in-cabin automotive monitoring setting, the challenge is not merely relative orientation recovery, but precise estimation of the full relative transform between a calibrated “standard” view and a shifted view acquired after a mount displacement. The target quantity is a relative rigid transform $T_{\mathrm{rel}} \in SE(3)$ satisfying

$T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$

where $T_{v1}$ is a known calibrated reference pose and $T_{v2}$ is the shifted pose (Stillger et al., 4 Apr 2026).

The motivating conditions are specific to ICAM. In-cabin cameras often use fisheye lenses with up to $180^\circ$ field of view, and the scene contains strong distortion, constrained baselines, and frequent occlusions from visors, hands, and headrests. According to the reported formulation, these factors degrade classical pipelines based on undistortion, feature matching, and essential-matrix decomposition, while incremental SfM cannot recover metric scale without a 3D map (Stillger et al., 4 Apr 2026).

The operational constraints are also safety-critical. Mounting variability caused by a camera attached to a movable mirror leads to unpredictable on-the-fly shifts of the extrinsic pose. For applications such as gaze estimation and airbag actuation, the relevant requirement is metric accuracy at real-world scale together with real-time inference, specifically $\leq 15\,\mathrm{ms}$ for inference within a $15$–$50\,\mathrm{ms}$ end-to-end response window in crash scenarios (Stillger et al., 4 Apr 2026). Within that context, a single-step regressor that directly predicts metric relative pose is presented as an alternative to iterative optimization, scene reconstruction, or explicit undistortion.

2. Architecture and computational design

InCaRPose is organized into three modules: a frozen DINOv3 ViT backbone, a Transformer-based decoder, and a prediction head (Stillger et al., 4 Apr 2026). The backbone, instantiated with ViT-Small, Base, or Large variants, encodes the reference and target images into patch-token sets $$X_{\mathrm{ref}}, X_{\mathrm{tgt}} \in \mathbb{R}^{N \times D}$$. The backbone remains frozen, and the stated rationale is to preserve strong self-supervised geometric features while enabling data-efficient training on only approximately $6.5\mathrm{k}$ synthetic image pairs (Stillger et al., 4 Apr 2026).

The decoder consists of $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$0 stacked decoder blocks. Each block includes multi-head self-attention over the concatenated token set, cross-attention in both directions between reference and target tokens, 2D rotary positional embeddings on queries and keys, and standard residual connections with LayerNorm (Stillger et al., 4 Apr 2026). This design is intended to capture relative geometric cues directly from paired views rather than from hand-crafted correspondences.

After the decoder, the representation passes through a residual convolutional bottleneck for dimensionality reduction, followed by global average pooling, LayerNorm, GELU, and a linear layer that produces a pose vector $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$1 (Stillger et al., 4 Apr 2026). During training, the model can optionally predict both $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$2 and its inverse in one forward pass to enforce geometric consistency. The output is described as “quaternion + translation,” indicating direct regression of orientation and metric displacement in a single inference step.

A plausible implication is that the architecture is positioned between generic image-pair regression and classical geometric estimation: it retains explicit pairwise structure through cross-attention while avoiding correspondence extraction and scale ambiguity.

3. Pose parameterization and optimization criterion

The pose representation used in InCaRPose consists of a unit quaternion $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$3 for rotation and a translation vector $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$4 in metric units, assembled as

$T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$5

Post-processing enforces $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$6 (Stillger et al., 4 Apr 2026).

Ground-truth relative pose is derived from the absolute poses $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$7 and $T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$8 through

$T_{v2} = T_{v1} \cdot T_{\mathrm{rel}},$9

with

$T_{v1}$0

This formulation expresses translation in the coordinate system of the reference camera and directly supervises the target relative motion (Stillger et al., 4 Apr 2026).

The loss is a quaternion-based pose loss,

$T_{v1}$1

where

$T_{v1}$2

and $T_{v1}$3 is set empirically to $T_{v1}$4 (Stillger et al., 4 Apr 2026). The reported interpretation is that this objective yields stable gradients on $T_{v1}$5 and directly supervises metric translation.

An ablation reported in the same source compares rotation vectors, Euler angles, quaternions, and 9-DoF matrix outputs. The quaternion representation together with the quaternion loss gives the lowest median rotation error, specifically $T_{v1}$6 versus $T_{v1}$7–$T_{v1}$8 for the alternatives (Stillger et al., 4 Apr 2026). Within the paper’s scope, that result supports the selected parameterization as the preferred representation for relative-pose regression under the stated constraints.

4. Synthetic data regime and real-world generalization

Training is performed exclusively on synthetic data. The synthetic cabin dataset comprises eight distinct car interiors modeled in Blender. For each standard view, the generation process uniformly samples rotations of $T_{v1}$9 about $T_{v2}$0 and $T_{v2}$1 about $T_{v2}$2, together with translations of $T_{v2}$3 along each camera axis, yielding approximately $T_{v2}$4 rotation-only and $T_{v2}$5 rotation-plus-translation pairs for training, plus approximately $T_{v2}$6 validation pairs (Stillger et al., 4 Apr 2026).

The rendering process randomizes cabin occupants, objects, textures, and NIR lighting to cover occlusions and varied reflectance. Fisheye intrinsics and distortion parameters are randomly sampled per render, images are rendered with full border distortion, and zero-padding rather than center-cropping is used to preserve extreme corners. ColorJitter over brightness, contrast, saturation, and hue is applied during training (Stillger et al., 4 Apr 2026).

No real images or precise intrinsics are used during training. Nevertheless, the framework is reported to generalize to real NIR fisheye interiors without undistortion or camera-specific fine-tuning (Stillger et al., 4 Apr 2026). The stated explanation is that the frozen DINOv3 backbone provides strong domain-invariant features and that aggressive domain randomization bridges the synthetic-to-real gap.

This suggests that InCaRPose should be understood not only as a pose regressor but also as an instance of synthetic-data-driven calibration, in which robustness to distortion and intrinsics mismatch is treated as a representation-learning problem rather than as a pre-calibration prerequisite.

5. Empirical performance across datasets

The principal evaluation is conducted on the real-world In-Cabin-Pose test set, which contains $T_{v2}$7 images with ARUCO marker ground truth (Stillger et al., 4 Apr 2026). The reported median errors are summarized below.

Variant	Rotation error	Translation error
InCaRPoseLarge224	$T_{v2}$8	$T_{v2}$9
InCaRPoseBase224	$180^\circ$0	$180^\circ$1
InCaRPoseSmall224	$180^\circ$2	$180^\circ$3

For the same in-cabin benchmark, InCaRPoseLarge224 also reports a median translation direction error of $180^\circ$4 (Stillger et al., 4 Apr 2026). The classical SIFT plus essential-matrix baseline recovers rotation with approximately $180^\circ$5 median error but does not recover metric translation. Reloc3r, described as a large-data baseline, also cannot recover scale and requires undistortion to approach $180^\circ$6 median direction error (Stillger et al., 4 Apr 2026). The comparison therefore distinguishes between orientation recovery and metric-scale pose estimation, with InCaRPose explicitly targeting the latter.

On the public 7-Scenes benchmark, InCaRPoseLarge512 achieves mean rotation error $180^\circ$7, median translation error $180^\circ$8, and median direction error $180^\circ$9 (Stillger et al., 4 Apr 2026). The paper states a $\leq 15\,\mathrm{ms}$0 rotation improvement relative to Reloc3r512’s $\leq 15\,\mathrm{ms}$1, and further states that all InCaRPose variants outperform Reloc3r224/512 on rotation while recovering metric translation (Stillger et al., 4 Apr 2026). This places the method outside a purely in-cabin niche and indicates competitive behavior on a public indoor relocalization benchmark.

On Cambridge Landmarks, the reported aggregated performance is approximately $\leq 15\,\mathrm{ms}$2 mean rotation error for InCaRPoseLarge224 and approximately $\leq 15\,\mathrm{ms}$3 median translation error (Stillger et al., 4 Apr 2026). Performance degrades outdoors, which the source attributes to sparser features and larger scale variation, although the method remains competitive on ShopFacade. A plausible implication is that the model’s inductive biases and training regime are particularly aligned with dense, geometrically constrained interiors rather than broad outdoor viewpoint changes.

6. Ablations, runtime, and release artifacts

Several ablation results clarify the model’s design trade-offs. Larger DINOv3 backbones, progressing from Small to Base to Large, steadily reduce rotation error on the in-cabin data; on large datasets such as 7-Scenes, the gains are smaller (Stillger et al., 4 Apr 2026). The multi-teacher DUNE-Base encoder slightly outperforms DINOv3-Base, and increasing input resolution from $\leq 15\,\mathrm{ms}$4 to $\leq 15\,\mathrm{ms}$5 pixels improves rotational accuracy while having only marginal effect on translation (Stillger et al., 4 Apr 2026). These results indicate that orientation estimation benefits more directly from representational scale and spatial detail than metric translation does under the tested conditions.

Training uses batch size $\leq 15\,\mathrm{ms}$6, the AdamW optimizer with learning rate $\leq 15\,\mathrm{ms}$7, weight decay $\leq 15\,\mathrm{ms}$8, and $\leq 15\,\mathrm{ms}$9 dropout in the MLP head; only the decoder and head are trained, while the ViT backbone remains frozen (Stillger et al., 4 Apr 2026). This training configuration is consistent with the data-efficient design choice described earlier.

Inference speed is reported on an NVIDIA RTX 4090. InCaRPoseSmall224 runs at approximately $15$0 FPS ($15$1/frame), Base224 at approximately $15$2 FPS, and Large224 at approximately $15$3 FPS. With FP16 and torch.compile, the Large variant can reach approximately $15$4 FPS (Stillger et al., 4 Apr 2026). The source concludes that all variants exceed real-time requirements for in-cabin streams.

The release artifacts include the In-Cabin-Pose test dataset and the codebase. The dataset contains $15$5 high-distortion NIR fisheye image pairs with ARUCO and COLMAP ground truth, systematically covering $15$6 translations and $15$7 rotations along each axis, and the code and synthetic data generation scripts are made available through the project repository (Stillger et al., 4 Apr 2026). The inclusion of both real-world evaluation data and synthetic-data tooling is significant because it supports replication of the central claim: a single-step, metric-scale relative-pose predictor trained purely on synthetic fisheye imagery.

7. Position within relative-pose estimation research

InCaRPose is presented as a response to several limitations of existing approaches in constrained, highly distorted environments. Classical feature-based methods are described as losing robustness under heavy lens distortion and constrained baselines, while incremental SfM is described as incapable of recovering metric scale without a 3D map (Stillger et al., 4 Apr 2026). In that sense, the method addresses not just relative pose in the abstract, but a specific conjunction of requirements: distorted fisheye input, real-time execution, absence of a prior 3D scene model, and direct recovery of metric translation.

Its defining characteristic is the combination of frozen self-supervised visual features, a lightweight Transformer decoder, and a quaternion-based metric loss to estimate 6-DoF relative pose directly from distorted image pairs (Stillger et al., 4 Apr 2026). The method is also unusual in being trained on purely synthetic renders yet evaluated on real NIR cabin data without undistortion or camera-specific fine-tuning. This suggests a broader methodological connection to synthetic-to-real transfer in geometric vision, particularly where camera models and distortion profiles vary across deployment settings.

A common misconception in this area is that rotation recovery alone is sufficient for calibration. The reported comparisons explicitly distinguish between estimating orientation and recovering metric translation: SIFT plus essential matrix is said to recover rotation but not metric translation, and Reloc3r is said not to recover scale (Stillger et al., 4 Apr 2026). InCaRPose is therefore notable less for replacing all geometric methods universally than for targeting the subset of calibration problems where absolute translation within a physically plausible adjustment range must be inferred in one forward pass.

Within that scope, the method is positioned as an efficient single-step extrinsic calibration solution for safety-critical automotive monitoring, while the reported degradation on outdoor Cambridge Landmarks indicates that its empirical strengths are most evident in dense, bounded interiors rather than unconstrained large-scale scenes (Stillger et al., 4 Apr 2026).