SenseShift6D: Sensor-Control 6D Benchmark
- SenseShift6D is an RGB-D benchmark for 6D object pose estimation that defines and tests sensor control by varying camera exposure, gain, depth modes, and illumination.
- It uses a controlled acquisition protocol with an Intel RealSense D455 and household objects to systematically capture over 1,380 unique sensor configurations per pose.
- Test-time sensor control, particularly Oracle selection, significantly boosts estimation performance compared to digital augmentation or larger real-world training sets.
Searching arXiv for the cited SenseShift6D paper and closely related benchmarking work. Use the search tool to look up "SenseShift6D" and related 6D pose benchmarking papers. SenseShift6D is an RGB-D benchmark for 6D object pose estimation that targets a specific failure mode of existing evaluation practice: robustness is usually measured under object, viewpoint, clutter, or occlusion variation, but not under physically realized changes in camera configuration and illumination. It introduces a controlled acquisition protocol in which RGB exposure, RGB gain, auto-exposure, depth capture mode, and ambient illumination are systematically varied, and it uses that benchmark to evaluate whether test-time sensor control can improve pose estimation more effectively than digital augmentation or larger real-world training sets (Han et al., 8 Jul 2025).
1. Problem setting and benchmark rationale
SenseShift6D is framed around a benchmark gap rather than a new pose-estimation architecture. The central claim is that representative 6D pose datasets such as LM-O, YCB-V, and T-LESS were collected under essentially fixed illumination and camera settings, so they do not expose how pose estimators behave under shifts in exposure, gain, depth-sensor mode, or lighting intensity. The benchmark therefore extends 6D-pose evaluation from a data-centered regime to a sensor-aware one, with explicit emphasis on test-time sensor adaptation (Han et al., 8 Jul 2025).
This positioning is distinct from earlier RGB-D benchmark efforts that emphasized viewpoint diversity, synthetic-to-real transfer, reflective and texture-less objects, or multimodal fusion, but not systematic camera-parameter sweeps. The SHREC 2020 track, for example, used mixed synthetic and real RGB-D data and concluded that methods that fully exploit color and geometric features are more robust for reflective and texture-less objects and occlusion, yet it did not define a formal sensor-shift protocol (Yuan et al., 2020). Similarly, multimodal 6D methods such as PoseFusion targeted severe hand self-occlusion and tactile sparsity in dexterous manipulation rather than exposure, gain, or depth-mode variation (Tu et al., 2023).
The motivating question is therefore operational: if a deployed pose estimator fails because the camera is operating under a suboptimal exposure, gain, or depth preset, should robustness be sought only through larger training sets and digital augmentation, or can one recover performance by changing sensor settings at inference time? SenseShift6D is designed to make that question measurable under physically realized RGB-D acquisition conditions rather than simulated perturbations (Han et al., 8 Jul 2025).
2. Dataset construction and controlled variation axes
SenseShift6D is built with an Intel RealSense D455 in a darkroom enclosed by black curtains. Illumination is provided by three Philips Hue White & Color Ambiance bulbs at a fixed color temperature of , while brightness is swept across five levels: , , , , and . The dataset contains three household objects: Spray, Pringles, and Tincase. The paper reports $101.9$k RGB images and $10$k depth images, and states that the captures provide unique sensor-lighting permutations per object pose (Han et al., 8 Jul 2025).
The controlled variation factors are the core of the benchmark.
| Factor | Settings |
|---|---|
| RGB exposure | 13 manual exposure levels |
| RGB gain | 9 gain settings |
| RGB auto-exposure | Auto |
| Depth capture mode | Default, High Accuracy, High Density, Medium Density |
| Illumination |
For RGB exposure, the RealSense D455 supports manual exposure from 0 to 1. The benchmark uses 13 logarithmically spaced exposure values, partitioned between train and test. The full manual set reported across splits is 2, with auto-exposure also included (Han et al., 8 Jul 2025).
For RGB gain, the D455 supports values from 3 to 4. The paper states that linear spacing produced more diverse visual characteristics than exponential spacing. The split-specific values reported are 5 for Train-Var and 6 for Test-Var, with auto-exposure mode also present. The text repeatedly refers to nine gain settings in total, but the complete nine-manual-value list is not fully enumerated in one place (Han et al., 8 Jul 2025).
For depth, the benchmark uses four Intel predefined presets rather than manual low-level stereo tuning. The intended characteristics are explicitly summarized: Default provides visually clean depth maps with reduced noise and good edges; High Accuracy uses stricter confidence thresholds and lower fill rate; High Density increases fill factor; Medium Density offers a trade-off between completeness and precision (Han et al., 8 Jul 2025).
The acquisition protocol is engineered for repeatability. Objects are mounted rigidly on a ChArUco board placed on a motorized turntable. Pose diversity is created by changing object orientation via the turntable, adjusting camera position, and adjusting camera tilt angle. For each scene, object and camera remain fixed while the system captures the full set of sensor configurations defined by the split. The paper also notes that, after changing lighting or sensor parameters, the system waits to allow stabilization, and that train and test scenes are assigned exclusively with no overlap (Han et al., 8 Jul 2025).
One numerical detail is explicitly unresolved in the paper. Although the dataset is described as sweeping 13 RGB exposures, 9 RGB gains, auto-exposure, 4 depth-capture modes, and 5 illumination levels, the reported total is 7 unique sensor-lighting permutations per pose. The manuscript consistently reports 8, but does not provide an explicit formula reconciling that value with the full listed factor counts (Han et al., 8 Jul 2025).
3. Splits, annotation pipeline, and evaluation protocol
The benchmark is organized into four principal splits that separate default conditions from variable conditions and separate training sensor settings from testing sensor settings (Han et al., 8 Jul 2025).
| Split | Core condition | Captured images |
|---|---|---|
| Train-Def | One RGB-D scene per pose at 9 brightness, RGB Auto, gain Auto, depth Default | RGB: 385, Depth: 385 |
| Train-Var | Physically varied RGB and lighting, depth Default | RGB: 80.0k, Depth: 1.9k |
| Test-Def | One RGB-D scene per pose at 0 brightness, RGB Auto, gain Auto, depth Default | RGB: 128, Depth: 128 |
| Test-Var | Brightness 1, manual RGB settings, 4 depth modes | RGB: 13k, Depth: 2.5k |
The split arithmetic is important because it determines how sensor-shift generalization is tested. Train-Var is reported as 2 RGB-D scenes per pose, corresponding to 3 RGB options and 4 brightness levels, while Test-Var is reported as 5 RGB-D scenes per pose, corresponding to 6 RGB options, 7 depth modes, and 8 brightness levels (Han et al., 8 Jul 2025).
Annotation combines 2D segmentation and 3D pose refinement. Object masks are initialized with Grounding DINO, manually reviewed or corrected, and refined with FastSAM. Initial camera-to-object transforms are obtained from ChArUco detection, then refined in 3D with the BOP toolkit annotation utility using High Density depth, and the final reprojection is visually inspected. Ground-truth depth consistency is evaluated using
9
where 0 is the captured depth map and 1 is the depth map rendered from the ground-truth pose. Differences above 2 are treated as outliers and removed. The reported depth-consistency statistics are mean error 3, standard deviation 4, mean absolute error 5, and median absolute error 6, which the text summarizes as roughly 7–8 and the introduction summarizes as approximately 9 error (Han et al., 8 Jul 2025).
The primary pose metric is ADD: 0 with 1 the set of 3D model points, 2 the number of model points, 3 the ground-truth pose, and 4 the predicted pose. The paper also reports AR@5, defined operationally as recall at threshold 5 of object diameter, and AUC@[0:0.1], the area under the ADD recall curve up to 6 of object diameter. The manuscript does not provide explicit equations for AR@5 or AUC@0:0.1.
The benchmark also defines sensor-control policies operationally rather than as explicit optimization problems. For RGB evaluation, AE denotes the auto-exposure baseline, Rand denotes a randomly selected manual exposure-gain configuration averaged over random choices, and Oracle denotes the best achievable per-scene result if one could choose the best sensor configuration for that scene. For multimodal RGB-D evaluation, the paper further defines Baseline, Depth-Only, RGB-Only, Oracle-Fixed, and Oracle-Dynamic, but does not write explicit optimization equations for Oracle-Fixed or Oracle-Dynamic (Han et al., 8 Jul 2025).
4. Evaluated pose estimators and training regimes
SenseShift6D evaluates three state-of-the-art 6D pose estimators under common training regimes. ZebraPose is RGB-based with a ResNet-34 backbone. GDRNPP is RGB-based with a ConvNeXt backbone; although it includes a depth-based refinement module, the experiments here exclude refinement. HiPose is the RGB-D model, with ConvNeXt for RGB, RandLA-Net for depth or point cloud, and a bidirectional interaction module for RGB-depth fusion (Han et al., 8 Jul 2025).
For each object, the paper generates 7k photorealistic synthetic images using BlenderProc. Each model is then trained under one of three regimes. The first is PBR + Train-Def, which combines synthetic data with default real images only. The second is PBR + Train-Def with augmentation, which adds digital RGB augmentation to the first regime. The third is PBR + Train-Var, which uses synthetic data plus physically varied real images and functions as the “more real, more diverse” baseline. An important implementation detail is that depth images use Default depth mode during training (Han et al., 8 Jul 2025).
The augmentation regime is explicitly enumerated. It includes Gaussian blur, sharpness enhancement, contrast enhancement, brightness enhancement, color enhancement, additive intensity, invert pixels, multiply intensity, additive Gaussian noise, linear contrast, and grayscale conversion. These are applied with overall probability 8, and each augmentation has its own activation probability (Han et al., 8 Jul 2025).
This experimental design is central to the benchmark’s argument. It isolates four distinct strategies for robustness: fixed default sensing, physically adaptive sensing at test time, digital augmentation during training, and increased real-world training diversity. Because all three strategies are tested on the same benchmark and the same models, the comparison is not between architectures but between robustness mechanisms (Han et al., 8 Jul 2025).
5. Sensor-control results and robustness findings
The primary empirical result is that performance under Test-Var depends strongly on sensor configuration, and that test-time sensor control can recover a large fraction of the lost performance. Auto-exposure is not model-optimal, random sensor changes are strongly detrimental, and Oracle sensor selection often outperforms digital augmentation and approaches or exceeds the gains obtained by substantially larger and more diverse real-world training sets (Han et al., 8 Jul 2025).
For RGB-based models, the overall Test-Var numbers under PBR + Train-Def show the pattern clearly.
| Model | AE | Oracle |
|---|---|---|
| ZebraPose | 85.41 | 96.75 |
| GDRNPP | 77.53 | 92.91 |
| HiPose | 85.03 | 92.59 |
The corresponding gains are 9 for ZebraPose, 0 for GDRNPP, and 1 for HiPose. Under PBR + Train-Def with augmentation, the Test-Var AE-to-Oracle gains remain substantial for ZebraPose (2, 3) and GDRNPP (4, 5), while shrinking for HiPose (6, 7). Under PBR + Train-Var, the same gap narrows but persists: ZebraPose 8 (9), GDRNPP $101.9$0 ($101.9$1), HiPose $101.9$2 ($101.9$3) (Han et al., 8 Jul 2025).
The paper summarizes these results at the benchmark level. The abstract states that applying sensor control during test time induces an average improvement of $101.9$4 over digital augmentation baselines, and that for RGB-based models Oracle achieves over $101.9$5 average improvement compared to AE. This is the strongest argument for the paper’s sensor-aware viewpoint: robustness is not only a property of model parameters and training data, but also of the camera operating point selected at inference (Han et al., 8 Jul 2025).
Equally important is the negative result for random control. Rand is consistently much worse than AE. Under the overall Test-Var, Train-Def setting, ZebraPose drops to $101.9$6 under Rand versus $101.9$7 under AE, and GDRNPP drops to $101.9$8 versus $101.9$9. This establishes that the benefit does not come from injecting arbitrary variability at test time; it comes from choosing a good setting (Han et al., 8 Jul 2025).
The comparison between test-time control and expanded real-world training is one of the paper’s most practical findings. On overall Test-Var, Train-Def Oracle reaches $10$0 for ZebraPose versus $10$1 for Train-Var AE; $10$2 for GDRNPP versus $10$3 for Train-Var AE; and $10$4 for HiPose versus $10$5 for Train-Var AE. The paper states that Oracle without augmentation consistently surpasses AE with augmentation and even outperforms AE trained on Train-Var in most scenarios; in the worst case, GDRNPP with Train-Def Oracle remains within less than $10$6 of GDRNPP with Train-Var AE (Han et al., 8 Jul 2025).
The appendix extends this conclusion to severe low-light domain shift. At brightness $10$7, Train-Def Oracle under Test-B5Var reaches $10$8 for ZebraPose, $10$9 for GDRNPP, and 0 for HiPose, despite Train-Def AE under Test-B5Def being much lower for GDRNPP and HiPose. The paper’s interpretation is explicit: test-time sensor control can mitigate environmental domain gaps without retraining (Han et al., 8 Jul 2025).
6. Multimodal RGB-D adaptation, interpretation, and limitations
The multimodal study uses HiPose to separate the effects of adapting RGB, adapting depth, and adapting both jointly. On Test-Var, overall AUC@[0:0.1], the reported results are as follows (Han et al., 8 Jul 2025).
| Configuration | AUC@[0:0.1] | Gain |
|---|---|---|
| Baseline | 72.17 | +0.00 |
| Depth-Only | 74.25 | +2.08 |
| RGB-Only | 77.13 | +4.95 |
| Oracle-Fixed | 72.53 | +0.53 |
| Oracle-Dynamic | 79.42 | +7.24 |
The same ordering holds per object. Spray improves from 1 to 2 under Oracle-Dynamic, Pringles from 3 to 4, and Tincase from 5 to 6. Depth-only adaptation is consistently beneficial, but RGB-only adaptation is usually larger, especially on Pringles and Tincase. Oracle-Fixed is only marginally better than Baseline, which shows that one globally optimal RGB-D configuration is not enough; the main benefit comes from per-scene dynamic adaptation (Han et al., 8 Jul 2025).
This section of the benchmark is significant because it establishes modality complementarity at the sensor-control level. The paper’s conclusion is explicit: adapting either RGB or depth sensors individually is effective, while jointly adapting multimodal RGB-D configurations yields even greater improvements. A plausible implication is that sensor selection can be treated as part of the inference algorithm, not merely as a capture-time engineering choice.
The qualitative analysis adds an important nuance. The paper notes that Oracle-selected RGB images may look visually similar to AE images to humans while producing much better model predictions. It also shows examples in which changing only the depth preset, with RGB fixed, turns a failure into a success. This directly contradicts the common assumption that perceptually best-looking sensor output is also the most informative for pose estimation (Han et al., 8 Jul 2025).
SenseShift6D also has explicit limitations. It contains only three objects and only single-object tabletop scenes. Illumination variation is limited to brightness intensity and does not include directional lighting or shadow-rich conditions. The benchmark does not include clutter- or occlusion-rich multi-object scenes. The strongest adaptation results are reported under Oracle policies rather than a learned real-time controller. The paper therefore demonstrates the potential of sensor control, but not a deployed control policy (Han et al., 8 Jul 2025).
Within the broader 6D literature, this makes SenseShift6D complementary rather than redundant. It is not a replacement for dexterous multimodal robustness work such as PoseFusion, which studies tactile sparsity and self-occlusion in object-in-hand settings (Tu et al., 2023), nor for sparse high-resolution geometric pipelines such as SDT-6D, which target depth-only multi-view bin picking under clutter and reflections (Leuze et al., 9 Dec 2025). Its distinctive contribution is to formalize sensor configuration itself as a benchmark variable for 6D pose estimation.
SenseShift6D therefore marks a shift in evaluation philosophy. Instead of assuming a fixed image distribution and asking how much robustness can be baked into a model through training, it asks how much robustness can be recovered by sensing the same scene differently. That is the benchmark’s central technical and methodological contribution (Han et al., 8 Jul 2025).