nuScenes-FG: Foreground-Centric Evaluation

• nuScenes-FG is an informal umbrella term for nuScenes research that focuses on detecting, segmenting, and tracking foreground ('thing') objects.
• It leverages comprehensive sensor data from lidar, cameras, and radar with detailed 3D box and point-wise annotations to support various tasks.
• Foreground-centric methodologies improve robustness and address class imbalance through refined metrics, advanced tracking, and semantic occupancy evaluations.

nuScenes-FG is best understood as a de facto foreground-centric reading of the nuScenes benchmark ecosystem rather than an official dataset split or challenge. The survey literature explicitly states that there is no dataset, benchmark, or task with the exact name “nuScenes-FG”; instead, nuScenes organizes foreground research through 3D object detection, semantic and panoptic segmentation, panoptic tracking, image-level instance annotation, and derived tasks such as semantic occupancy that emphasize “thing” classes including cars, pedestrians, cyclists, barriers, and traffic cones (Fong et al., 2 Dec 2025). In that sense, “nuScenes-FG” denotes the collection of nuScenes tasks, annotations, and evaluation protocols that target foreground objects, their identities, their temporal persistence, and their separation from background or “stuff” structure.

1. Terminology and conceptual scope

The relevant survey literature distinguishes between official nuScenes benchmarks and the informal usage implied by “nuScenes-FG.” It states that the paper “nuScenes Revisited: Progress and Challenges in Autonomous Driving” does not reference a dataset, benchmark, or task with the exact name “nuScenes-FG.” The closest concepts are 3D and 2D detection of foreground objects, segmentation with a thing/stuff distinction, Panoptic nuScenes, nuImages, and occupancy tasks derived from nuScenes that often focus on foreground class occupancy in 3D (Fong et al., 2 Dec 2025).

Foreground in this context corresponds to “thing” classes. A high-level split reported for Panoptic nuScenes comprises 10 foreground classes and 6 background classes. The foreground classes are car, truck, bus, trailer, construction vehicle, bicycle, motorcycle, pedestrian, traffic cone, and barrier, while the background classes are drivable surface, other flat surface, sidewalk, terrain, manmade, and vegetation (Fong et al., 2 Dec 2025). This thing/stuff decomposition is central to how nuScenes foreground research is structured.

A useful clarification is that some later occupancy papers use language such as “nuScenes-FG: nuScenes ForeGround task” when discussing semantic occupancy evaluation on nuScenes (Zuo et al., 12 Jun 2025). This suggests an informal, task-local shorthand rather than an official benchmark identity, because the broader nuScenes survey still states that no official “nuScenes-FG” split exists (Fong et al., 2 Dec 2025).

2. Dataset substrate and annotation basis

The foreground-centric use of nuScenes rests on the original nuScenes dataset and its extensions. nuScenes is described as the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars, and 1 lidar, all with full 360 degree field of view. It comprises 1000 scenes, each 20 seconds long, and is fully annotated with 3D bounding boxes for 23 classes and 8 attributes (Caesar et al., 2019). The dataset includes 40,000 annotated keyframes, 1.4 million images, 400,000 lidar point clouds, and 1.3 million radar point clouds, with scenes collected in Boston and Singapore under day, night, and rain conditions (Caesar et al., 2019).

For foreground research, the key annotation unit in the original benchmark is the 3D box. Each object is annotated with position, dimensions, and yaw, and detection tasks use 10 main classes with enough data: car, pedestrian, bicycle, bus, motorcycle, truck, trailer, construction vehicle, traffic cone, and barrier (Caesar et al., 2019). The dataset also provides object attributes and 2D velocity where measurable (Caesar et al., 2019).

Panoptic nuScenes extends this substrate with dense point-wise annotation. It covers 1.1 billion annotated points across 40,000 LiDAR keyframes from 1,000 scenes. Each LiDAR point is assigned a semantic class label, and “thing” points also receive instance IDs that are temporally consistent across frames (Fong et al., 2021). The full label space contains 32 classes, with 23 “thing” classes and 9 “stuff” classes, and challenge protocols merge similar or rare categories into 10 thing and 6 stuff classes (Fong et al., 2021). The annotation protocol initializes thing labels from 3D boxes, manually refines boundaries, and labels overlapping box regions as noise; the survey reports that overlapping regions account for less than 0.8% of all points (Fong et al., 2 Dec 2025).

nuImages supplies the 2D image-side complement. It supports 2D detection and segmentation with instance masks for foreground classes and annotates foreground object attributes such as flashing lights and sitting pedestrians (Fong et al., 2 Dec 2025).

3. De facto foreground benchmarks

The foreground interpretation of nuScenes is distributed across several official and unofficial tasks.

Task	Output	Foreground role
3D object detection	3D boxes, class labels, attributes	Largely foreground-object centric
LiDAR semantic/panoptic segmentation	Point-wise semantic and instance labels	Explicit thing/stuff split
LiDAR panoptic tracking	Semantic labels with temporally consistent IDs	Tracks thing instances over time
nuImages	2D detection, segmentation, instance masks	Dense foreground instance annotation
Semantic occupancy derived from nuScenes	Voxel-wise occupancy semantics	Often emphasizes foreground occupancy

The official nuScenes Detection Challenge focuses on 23 object classes, largely foreground objects, and evaluates 3D bounding box prediction for the major detection categories (Fong et al., 2 Dec 2025). Panoptic nuScenes provides the point-level semantic and instance labeling needed for foreground/background separation and for instance-level temporal analysis (Fong et al., 2021). nuImages extends the same foreground emphasis to dense image annotation (Fong et al., 2 Dec 2025).

The survey literature also identifies unofficial community benchmarks that use nuScenes for out-of-domain object detection, few-shot object detection, and semantic occupancy, including Occ3D-nuScenes, where foreground class occupancy becomes a natural object-centric evaluation target (Fong et al., 2 Dec 2025). This suggests that “nuScenes-FG” is most accurate as an umbrella term for these foreground-driven research uses rather than as a single benchmark.

4. Metrics and evaluation logic

Foreground evaluation in nuScenes is task-dependent. For 3D detection, nuScenes departs from IoU-based matching and uses center distance on the ground plane, averaged over distance thresholds $\mathbb{D} = \{0.5, 1, 2, 4\}$ meters across detection classes (Caesar et al., 2019). The canonical aggregate score is the nuScenes Detection Score:

$$\text{NDS} = \frac{1}{10} \left[ 5 \times \text{mAP} + \sum_{i=1}^5 \left(1 - \min(1, \nu_i)\right) \right]$$

where the five mean true-positive metrics are Average Translation Error, Average Scale Error, Average Orientation Error, Average Velocity Error, and Average Attribute Error (Fong et al., 2 Dec 2025).

For semantic segmentation and occupancy, mean Intersection-over-Union is the standard summary:

$$\text{mIoU} = \frac{1}{N} \sum_{c=1}^{N} \frac{|P_c \cap T_c|}{|P_c \cup T_c|}$$

for point segmentation in Panoptic nuScenes (Fong et al., 2 Dec 2025), and occupancy papers on nuScenes use the same IoU logic voxel-wise while excluding the empty class from the mean (Wu et al., 2 Jun 2026).

For panoptic segmentation, Panoptic Quality is defined as

$$\text{PQ} = \frac{\sum_{(p, g) \in TP} \text{IoU}(p,g)}{|TP|} \times \frac{|TP|}{|TP| + \frac{1}{2}|FP| + \frac{1}{2}|FN|}$$

which couples segment quality with recognition quality (Fong et al., 2 Dec 2025). For panoptic tracking, Panoptic nuScenes introduces the instance-centric PAT metric,

$PAT = \frac{2 \times PQ \times TQ}{PQ + TQ},$

to address fragmentation and identity-consistency failures that are especially consequential for foreground agents (Fong et al., 2021).

These metrics show that nuScenes foreground evaluation is not a single-number problem. Detection emphasizes localization and attributes, segmentation emphasizes class overlap, and panoptic tracking emphasizes temporal identity continuity.

5. Methodological evolution in foreground-centric research

Foreground-driven nuScenes research has repeatedly motivated architectural changes. In LiDAR detection, InfoFocus models the non-uniform distribution of point clouds through density-guided refinement and reports 37.5 mAP on the nuScenes test set, versus 30.5 for the PointPillars baseline, with 31 FPS; with increased training it reaches 39.5 mAP (Wang et al., 2020). The method explicitly targets informative and visible regions, which the paper links to better performance on partially occluded or sparse objects (Wang et al., 2020).

Class imbalance and point-level foreground supervision are another recurrent theme. “Structure Aware and Class Balanced 3D Object Detection on nuScenes Dataset” augments CBGS with an auxiliary network trained by foreground segmentation and center estimation. On the reported 10% split, CBGS attains 18.96 mAP and 26.8 NDS, while the best SA-CBGS configuration reaches 20.67 mAP and 29.36 NDS (Nagesh et al., 2022). The paper further notes that foreground segmentation alone still provides a performance benefit relative to the baseline (Nagesh et al., 2022).

Recent multimodal detectors push the same foreground emphasis into fusion design. FocalFormer3D introduces Hard Instance Probing to reduce false negatives and reports 70.5 mAP and 73.9 NDS on the nuScenes detection benchmark, along with 72.1 AMOTA on the tracking benchmark (Chen et al., 2023). IS-Fusion, which combines Hierarchical Scene Fusion and Instance-Guided Fusion, reports 73.0% mAP and 75.2% NDS on the nuScenes test set, with particularly strong results on small or difficult classes such as motorcycle and trailer (Yin et al., 2024). Fusion4CA, built on BEVFusion, reports 69.7% mAP and 72.1 NDS with only 6 training epochs (Luo et al., 5 Mar 2026).

Foreground-centric rendering has also entered detection. OcRFDet argues that whole-scene radiance-field supervision produces strong background responses and instead uses Object-centric Radiance Fields that render only foreground regions. On the nuScenes test benchmark it reports 57.2% mAP and 64.8% NDS (Ji et al., 30 Jun 2025). This suggests that foreground isolation is not only an evaluation concern but also a training-stability concern.

6. Challenges, extensions, and occupancy-derived interpretations

Several persistent difficulties shape any account of nuScenes-FG. nuScenes exhibits severe class imbalance: the original dataset analysis reports a most-to-least frequent class ratio of approximately 1:10,000, and camera-based detection drops much more at night than lidar-based detection (Caesar et al., 2019). The later survey adds that less than 7% of lidar points, and a similar area in images, are foreground, and notes that the lidar/radar-centric design may create “unfair” cases for camera-only approaches for certain far or partially visible foreground objects (Fong et al., 2 Dec 2025).

Controlled robustness studies amplify this line of work. “Occluded nuScenes” extends the original benchmark with parameterized occlusions for cameras, radar, and LiDAR while preserving the original annotations and file structure (Kumar et al., 21 Oct 2025). The resource is intended for repeatable robustness evaluation under partial sensor failures and environmental interference (Kumar et al., 21 Oct 2025). This broadens foreground research from nominal recognition to resilience under sensing degradation.

Foreground reasoning also appears in nuScenes-derived semantic occupancy. UnsOcc evaluates 3D semantic occupancy on nuScenes using 17 classes, a voxel grid of size $[200, 200, 16]$ with voxel size $[0.5\ \text{m}, 0.5\ \text{m}, 0.5\ \text{m}]$, and reports 28.1 mIoU, outperforming prior camera-only, LiDAR-only, and fusion baselines (Wu et al., 2 Jun 2026). The paper highlights strong gains on challenging, rare, or small classes such as pedestrian, traffic cone, and bicycle (Wu et al., 2 Jun 2026). OccFusion similarly frames nuScenes occupancy as a multimodal robustness problem and reports 27.3% mIoU for camera + lidar + radar fusion on the SurroundOcc labels (Ming et al., 2024). Although these are occupancy benchmarks rather than canonical foreground benchmarks, the emphasis on rare and safety-critical object classes makes them part of the broader foreground-centered interpretation of nuScenes.

The main misconception surrounding “nuScenes-FG” is therefore terminological. It is not an official split. It is the aggregate of nuScenes benchmarks, extensions, and derived tasks in which the primary scientific question is how reliably a system can localize, segment, track, and reason about foreground objects against sparse geometry, long-tail class statistics, adverse conditions, and multimodal sensing constraints (Fong et al., 2 Dec 2025).