SpaceSense-Bench: Spacecraft Perception Benchmark

• The benchmark provides 136 satellite models and 70 GB of synchronized RGB, depth, and LiDAR data, enabling comprehensive spacecraft perception and pose analysis.
• It employs a detailed 7-class part-level semantic taxonomy to facilitate tasks including 2D/3D segmentation, depth estimation, and object detection with high precision.
• A strict zero-shot split and advanced simulation pipeline using UE5 and AirSim ensure robust evaluation of generalization to unseen spacecraft geometries.

SpaceSense-Bench is a large-scale multi-modal benchmark for spacecraft perception and pose estimation. It was introduced to address autonomous space operations such as on-orbit servicing and active debris removal, where a chaser spacecraft must achieve part-level semantic understanding and precise relative navigation of a target spacecraft. The benchmark provides 136 satellite models and approximately 70 GB of data, with time-synchronized 1024$\times$1024 RGB images, millimeter-precision depth maps, 256-beam LiDAR point clouds, dense 7-class part-level semantic labels at both the pixel and point level, and accurate 6-DoF pose ground truth. Its evaluation suite covers object detection, 2D semantic segmentation, RGB–LiDAR fusion-based 3D point cloud segmentation, monocular depth estimation, and orientation estimation, with a strict zero-shot split in which test satellites are entirely unseen during training (Wu et al., 10 Mar 2026).

1. Scope, rationale, and semantic taxonomy

SpaceSense-Bench was created in response to three limitations identified in prior synthetic spacecraft datasets: very few targets, single-modality sensing, and incomplete annotations. The benchmark addresses these by aggregating 136 satellite models from sources including NASA 3D Resources, NASA Eyes, and the ESA Science Satellite Fleet; standardizing them in Blender; and exposing synchronized RGB, depth, and LiDAR observations together with dense semantic labels and pose annotations (Wu et al., 10 Mar 2026).

The satellite set spans communication satellites, Earth and space science missions, planetary explorers, space telescopes, and ISS-class structures. Maximum diameter ranges from 0.27 m to 112 m. All models are normalized to a common coordinate frame with $+X$ as velocity direction, $+Y$ as starboard, and $+Z$ as nadir, then exported as textured FBX assets. This standardization is important because the benchmark evaluates generalization across targets with highly variable geometry rather than repeated views of a single canonical spacecraft (Wu et al., 10 Mar 2026).

A central design choice is the use of a spacecraft part taxonomy rather than whole-object labels alone. Each satellite is decomposed once in 3D, with a unique Stencil ID assigned to each mesh component. The seven semantic classes are main_body, solar_panel, dish_antenna, omni_antenna, payload, thruster, and adapter_ring. The class system is shared across both 2D and 3D annotations, enabling direct comparison between image-space segmentation, point-wise segmentation, and pose-conditioned analyses. This class inventory emphasizes functional spacecraft structure rather than generic object categories, and it directly supports tasks relevant to proximity operations such as locating thrusters, docking interfaces, and appendages (Wu et al., 10 Mar 2026).

2. Data modalities, annotations, and split protocol

Each frame contains outputs from three co-located sensors mounted on a virtual chaser spacecraft. The RGB camera produces 1024 $\times$ 1024 images with a 50° field of view. The depth camera shares the same intrinsics and resolution and reports perspective depth with millimeter precision. The LiDAR sensor uses 256 beams and a 40° field of view, returning point clouds in the camera frame. Because all three sensors share the same pose and are synchronized through AirSim, their measurements are geometrically and temporally aligned (Wu et al., 10 Mar 2026).

The annotation stack is unusually dense for the domain. Pixel-level 2D semantic labels are derived from the Unreal Engine 5 Custom Depth Stencil buffer, yielding 7-class segmentation masks aligned to the RGB image. Point-level 3D semantic labels are generated by projecting LiDAR returns into the image plane and reading the corresponding pixel label. Each frame also includes target pose with respect to the camera frame, conceptually represented as

$$\mathbf{T}_{\text{cam}\leftarrow\text{sat}}= \begin{bmatrix} \mathbf{R} & \mathbf{t}\ \mathbf{0}^{\top} & 1 \end{bmatrix}\in SE(3),$$

where $\mathbf{R}\in SO(3)$ is the rotation from satellite body frame to camera frame and $\mathbf{t}\in\mathbb{R}^3$ is translation in camera coordinates (Wu et al., 10 Mar 2026).

The currently used release is a sparse 90,000-frame subset, although the same generation pipeline can produce more than 2 million frames. Sampling is organized by 27 trajectory types per satellite, comprising 22 approach trajectories and 5 orbit trajectories. The benchmark adopts a strict zero-shot, out-of-distribution split: 117 satellites for training, 5 for validation, and 14 for testing, with no satellite appearing in more than one split. This protocol makes geometric novelty, rather than frame count alone, the defining generalization challenge (Wu et al., 10 Mar 2026).

The data are intentionally realistic in their imbalance. In foreground pixels, solar_panel and main_body each account for about 41%, whereas thruster is about 0.76%, adapter_ring about 1.2%, and omni_antenna about 0.20%. LiDAR density also changes sharply with range: at distances below 30 m, frames may contain tens of thousands of points, while at distances above 100 m they may contain only a few hundred. This long-tail regime is not incidental; it is part of the benchmark’s attempt to model approach and inspection conditions encountered in orbital operations (Wu et al., 10 Mar 2026).

3. Simulation environment and automated generation pipeline

SpaceSense-Bench is generated in a high-fidelity Unreal Engine 5 simulation with AirSim integration. The orbital scene models three illumination regimes: direct sunlight, Earth albedo, and eclipse or darkness. The implementation uses a directional light for the Sun, an Earth model with high-resolution textures and cloud layers, SkyAtmosphere and SkyLight for ambient illumination, hardware ray tracing for metallic reflections and sharp shadows, and a starfield backdrop based on real star catalogues. This combination is intended to produce strong photometric variation under which perception systems must still generalize (Wu et al., 10 Mar 2026).

Trajectory design is scale-adaptive through the satellite maximum diameter $d_{\max}$. Approach trajectories start at $10\times d_{\max}$ and end at $+X$0, with geometric decay such that each step is about 10% of the current distance; the benchmark uses 22 approach directions, including 6 axis-aligned and 16 diagonal directions. Orbit trajectories use radius $+X$1, angular step 1°, full 360° coverage, and 5 orbital planes. This makes sampling roughly uniform in apparent scale across CubeSat-class and ISS-class targets while still covering near-field and far-field views (Wu et al., 10 Mar 2026).

The generation workflow has four stages. In asset preparation, 3D models are imported into Blender, cleaned, reoriented into the standardized coordinate frame, decomposed into parts, assigned Stencil IDs, and exported. In scene setup, each target is inserted into the UE5 orbital environment with the co-located RGB, depth, and LiDAR sensors attached to the chaser. In trajectory execution and capture, AirSim loads the asset, runs all 22 approach and 5 orbit trajectories, and captures synchronized RGB, depth, LiDAR, semantic masks, and target pose. In ground-truth generation and export, the benchmark derives pixel and point labels and converts data into mainstream formats including YOLO for object detection, MMSegmentation format for 2D semantic segmentation, and SemanticKITTI-style format for 3D LiDAR segmentation (Wu et al., 10 Mar 2026).

Quality control is multi-stage and automatic. The pipeline validates segmentation masks, checks depth validity and coverage, inspects LiDAR point clouds for emptiness and expected spatial bounds, and verifies strict cross-modal timestamp consistency. Anomalous frames are discarded and re-captured. Because labels come from mesh decomposition and engine internals rather than manual polygon tracing, the paper characterizes annotation accuracy for both 2D and 3D labels as effectively perfect within the simulator (Wu et al., 10 Mar 2026).

4. Tasks and evaluation methodology

SpaceSense-Bench defines five benchmark tasks. Together they cover 2D recognition, 3D fusion, metric scene understanding, and orientation inference under a zero-shot protocol on unseen spacecraft (Wu et al., 10 Mar 2026).

Task	Output	Principal metrics
2D semantic segmentation	7-class pixel labels	mIoU, aAcc
Object detection	2D boxes and class labels	mAP@0.5, [email protected]:0.95
3D point cloud segmentation	7-class point labels	mIoU, mAcc
Monocular depth estimation	Dense depth map	AbsRel, SqRel, RMSE, RMSE log, $+X$2, Spearman
Orientation estimation	Satellite orientation	MAAE, median angular error, error-threshold percentages

For 2D semantic segmentation, the baselines are FCN with ResNet-50, DeepLabV3+ with ResNet-50, SegFormer with MiT-B3, and Mask2Former with Swin-Base, all initialized from ImageNet and trained for 50 epochs on 640$+X$3640 crops. The principal metric is mean Intersection over Union,

$+X$4

along with overall pixel accuracy $+X$5 (Wu et al., 10 Mar 2026).

For object detection, the benchmark uses YOLO26 in nano, small, medium, large, and xlarge scales, trained for 100 epochs. Reported metrics include [email protected] and COCO-style [email protected]:0.95. For RGB–LiDAR fusion-based 3D point cloud segmentation, the baseline is PMFNet with a ResNet-34 backbone and learning rate $+X$6, evaluated by point-wise mIoU and mAcc (Wu et al., 10 Mar 2026).

Monocular depth estimation is evaluated in zero-shot mode using Depth Anything V2 with ViT-S, ViT-B, and ViT-L encoders. Because monocular depth has scale ambiguity, predicted depth is aligned on the satellite region by least-squares affine fitting,

$+X$7

The reported metrics are Absolute Relative Error, Squared Relative Error, RMSE, RMSE log, threshold accuracy $+X$8, and Spearman rank correlation. Orientation estimation is likewise zero-shot and uses Orient Anything with a DINOv2-Large backbone. Error is reported as Mean Axis Angular Error, with geodesic rotation discrepancy

$+X$9

The benchmark focuses its pose evaluation on orientation rather than full translation, although full 6-DoF pose is provided in the dataset (Wu et al., 10 Mar 2026).

5. Empirical findings and diagnostic significance

The empirical results show a consistent division between large structural components and small specialized components. In 2D semantic segmentation on the 14-satellite zero-shot test set, Mask2Former achieves the best overall result with aAcc $+Y$0 and mIoU $+Y$1. Its per-class IoU is 71.4% for main_body, 88.6% for solar_panel, 26.3% for dish_antenna, 1.9% for omni_antenna, 19.1% for payload, 24.6% for thruster, and 33.3% for adapter_ring. SegFormer is close in aggregate, with mIoU $+Y$2, but the same long-tail pattern remains (Wu et al., 10 Mar 2026).

Object detection follows the same structure. YOLO26-XL is the strongest configuration, with precision $+Y$3 and [email protected] $+Y$4. Per-class [email protected] is 91.0% for main_body, 82.5% for solar_panel, 23.7% for dish_antenna, 8.0% for omni_antenna, 9.1% for payload, 23.3% for thruster, and 51.6% for adapter_ring. In 3D point cloud segmentation, PMFNet attains mAcc $+Y$5 and mIoU $+Y$6, with class IoU 68.8% for main_body, 85.8% for solar_panel, 51.7% for dish_antenna, 8.9% for omni_antenna, 21.9% for payload, 25.2% for thruster, and 34.2% for adapter_ring (Wu et al., 10 Mar 2026).

Depth estimation appears strong under affine alignment but is not uniformly solved. Depth Anything V2 ViT-L gives the best overall depth result with AbsRel $+Y$7, SqRel $+Y$8, RMSE $+Y$9 m, RMSElog $+Z$0, $+Z$1, and Spearman $+Z$2. The benchmark interprets this as small absolute depth error after alignment but limited preservation of fine-grained depth ordering on complex metallic structures and wide baselines. Orientation estimation with Orient Anything yields MAAE $+Z$3, median error $+Z$4, and proportions below 10°, 20°, 30°, and 45° of 53.7%, 78.2%, 91.7%, and 98.8%, respectively. Performance is better on structurally simple satellites such as GRAIL, at about $+Z$5, and worse on compact or symmetric designs such as LADEE, at about $+Z$6 (Wu et al., 10 Mar 2026).

Two findings organize the benchmark’s interpretation. First, small-scale components remain a major bottleneck. omni_antenna is the clearest example, with 2D IoU 1.9%, 3D IoU 8.9%, and detection [email protected] 8.0%. thruster also remains difficult across tasks. The paper attributes this to extremely small pixel or point footprint, high cross-satellite variability, severe distance sensitivity, and long-tail data imbalance. Second, zero-shot generalization to unseen spacecraft remains difficult even for strong modern baselines. Reported segmentation and detection scores remain in the roughly 40–46% range on unseen target geometries, and the benchmark presents this as evidence that geometry-aware generalization is still limited (Wu et al., 10 Mar 2026).

A separate scaling experiment supports the importance of target diversity. Using PMFNet for 3D segmentation, progressively increasing the number of training satellites from about 9 to all 117 raises test mIoU from 24.4% to 42.4%, a 73% relative gain, and mAcc from 35.3% to 57.5%, a 63% relative gain. The reported curve has diminishing but non-saturating returns. This indicates that adding distinct spacecraft geometries continues to improve zero-shot performance on held-out targets, a point the benchmark emphasizes as more important than increasing frames from a narrow target set (Wu et al., 10 Mar 2026).

6. Position within the benchmark landscape, limitations, and future directions

SpaceSense-Bench belongs to a different benchmark lineage from general multimodal spatial reasoning suites. SpatialBench evaluates hierarchical spatial cognition of multimodal LLMs from egocentric videos (Xu et al., 26 Nov 2025), EarthSpatialBench targets distance, direction, and topology reasoning on georeferenced Earth imagery (Xu et al., 17 Feb 2026), Space3D-Bench provides 1000 general spatial questions and answers on indoor Replica scenes (Szymanska et al., 2024), SpaCE-10 defines 10 atomic spatial capabilities and 8 compositional capabilities over 811 real indoor scenes (Gong et al., 9 Jun 2025), SpinBench centers perspective taking and rotation in vision-LLMs (Zhang et al., 29 Sep 2025), EmbSpatial-Bench studies egocentric embodied spatial understanding in indoor 3D environments (Du et al., 2024), GSR-BENCH emphasizes grounded spatial relation evaluation with boxes and depth (Rajabi et al., 2024), SCBench focuses on executable spatial competence with deterministic and simulator-based verification (Vira et al., 5 Mar 2026), and SenseBench evaluates low-level remote-sensing perception and description under physics-driven degradations (Zhong et al., 11 May 2026). By contrast, SpaceSense-Bench is specifically a spacecraft perception and pose-estimation benchmark, not a generic question-answering or multimodal reasoning suite (Wu et al., 10 Mar 2026).

Within spacecraft datasets, the benchmark is positioned against SPEED and SPEED+, URSO, Hoang et al. 2021, UESD, SPARK, and NCSTP. Earlier datasets typically use 1 to 16 satellite models, often RGB only, and usually lack either dense part-level labels, LiDAR, or pose. SpaceSense-Bench is presented as the first dataset that jointly provides synchronized RGB–depth–LiDAR triplets, supports 3D point cloud segmentation with point-level semantics, and is designed explicitly for zero-shot generalization to unseen spacecraft (Wu et al., 10 Mar 2026).

The paper also identifies several limitations. It remains a synthetic dataset, so real on-orbit imagery may include sensor noise, lens effects, and structural degradation not fully captured in simulation. The semantic taxonomy has only 7 part classes, which may be insufficient for missions requiring finer distinctions such as multiple antenna subclasses, valves, or thermal radiators. The current release uses 90,000 frames rather than the pipeline’s larger $+Z$7 million-frame capacity. These constraints do not invalidate the benchmark’s intended use, but they delimit its realism and granularity (Wu et al., 10 Mar 2026).

Future directions are correspondingly concrete. The authors propose expanding target diversity using 3D generative models, conducting sim-to-real validation with laboratory testbeds and real orbital imagery, and exploring tasks already enabled by the dataset such as multi-view 3D reconstruction, cross-modal generation between LiDAR, depth, and RGB, and visual navigation and control tasks. The dataset, code, and toolkit are publicly available at https://github.com/wuaodi/SpaceSense-Bench, with ready-to-use conversions for YOLO, MMSegmentation, and SemanticKITTI-style pipelines (Wu et al., 10 Mar 2026).