---
title: 'nuScenes-FG: Foreground-Centric Evaluation'
url: https://www.emergentmind.com/topics/nuscenes-fg
type: topic
---

# nuScenes-FG: Foreground-Centric Evaluation

nuScenes-FG is best understood as a de facto foreground-centric reading of the nuScenes benchmark ecosystem rather than an official dataset split or challenge. The survey literature explicitly states that there is no dataset, benchmark, or task with the exact name “nuScenes-FG”; instead, nuScenes organizes foreground research through 3D object detection, semantic and panoptic segmentation, panoptic tracking, image-level instance annotation, and derived tasks such as semantic occupancy that emphasize “thing” classes including cars, pedestrians, cyclists, barriers, and traffic cones [2512.02448]. In that sense, “nuScenes-FG” denotes the collection of nuScenes tasks, annotations, and evaluation protocols that target foreground objects, their identities, their temporal persistence, and their separation from background or “stuff” structure.

## 1. Terminology and conceptual scope

The relevant survey literature distinguishes between official nuScenes benchmarks and the informal usage implied by “nuScenes-FG.” It states that the paper “nuScenes Revisited: Progress and Challenges in Autonomous Driving” does not reference a dataset, benchmark, or task with the exact name “nuScenes-FG.” The closest concepts are 3D and 2D detection of foreground objects, segmentation with a thing/stuff distinction, Panoptic nuScenes, nuImages, and occupancy tasks derived from nuScenes that often focus on foreground class occupancy in 3D [2512.02448].

Foreground in this context corresponds to “thing” classes. A high-level split reported for Panoptic nuScenes comprises 10 foreground classes and 6 background classes. The foreground classes are car, truck, bus, trailer, construction vehicle, bicycle, motorcycle, pedestrian, traffic cone, and barrier, while the background classes are drivable surface, other flat surface, sidewalk, terrain, manmade, and vegetation [2512.02448]. This thing/stuff decomposition is central to how nuScenes foreground research is structured.

A useful clarification is that some later occupancy papers use language such as “nuScenes-FG: nuScenes ForeGround task” when discussing semantic occupancy evaluation on nuScenes [2506.10977]. This suggests an informal, task-local shorthand rather than an official benchmark identity, because the broader nuScenes survey still states that no official “nuScenes-FG” split exists [2512.02448].

## 2. Dataset substrate and annotation basis

The foreground-centric use of nuScenes rests on the original nuScenes dataset and its extensions. nuScenes is described as the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars, and 1 lidar, all with full 360 degree field of view. It comprises 1000 scenes, each 20 seconds long, and is fully annotated with 3D bounding boxes for 23 classes and 8 attributes [1903.11027]. The dataset includes 40,000 annotated keyframes, 1.4 million images, 400,000 lidar point clouds, and 1.3 million radar point clouds, with scenes collected in Boston and Singapore under day, night, and rain conditions [1903.11027].

For foreground research, the key annotation unit in the original benchmark is the 3D box. Each object is annotated with position, dimensions, and yaw, and detection tasks use 10 main classes with enough data: car, pedestrian, bicycle, bus, motorcycle, truck, trailer, construction vehicle, traffic cone, and barrier [1903.11027]. The dataset also provides object attributes and 2D velocity where measurable [1903.11027].

Panoptic nuScenes extends this substrate with dense point-wise annotation. It covers 1.1 billion annotated points across 40,000 LiDAR keyframes from 1,000 scenes. Each LiDAR point is assigned a semantic class label, and “thing” points also receive instance IDs that are temporally consistent across frames [2109.03805]. The full label space contains 32 classes, with 23 “thing” classes and 9 “stuff” classes, and challenge protocols merge similar or rare categories into 10 thing and 6 stuff classes [2109.03805]. The annotation protocol initializes thing labels from 3D boxes, manually refines boundaries, and labels overlapping box regions as noise; the survey reports that overlapping regions account for less than 0.8% of all points [2512.02448].

nuImages supplies the 2D image-side complement. It supports 2D detection and segmentation with instance masks for foreground classes and annotates foreground object attributes such as flashing lights and sitting pedestrians [2512.02448].

## 3. De facto foreground benchmarks

The foreground interpretation of nuScenes is distributed across several official and unofficial tasks.

| Task | Output | Foreground role |
|---|---|---|
| 3D object detection | 3D boxes, class labels, attributes | Largely foreground-object centric |
| LiDAR semantic/panoptic segmentation | Point-wise semantic and instance labels | Explicit thing/stuff split |
| LiDAR panoptic tracking | Semantic labels with temporally consistent IDs | Tracks thing instances over time |
| nuImages | 2D detection, segmentation, instance masks | Dense foreground instance annotation |
| Semantic occupancy derived from nuScenes | Voxel-wise occupancy semantics | Often emphasizes foreground occupancy |

The official nuScenes Detection Challenge focuses on 23 object classes, largely foreground objects, and evaluates 3D bounding box prediction for the major detection categories [2512.02448]. Panoptic nuScenes provides the point-level semantic and instance labeling needed for foreground/background separation and for instance-level temporal analysis [2109.03805]. nuImages extends the same foreground emphasis to dense image annotation [2512.02448].

The survey literature also identifies unofficial community benchmarks that use nuScenes for out-of-domain object detection, few-shot object detection, and semantic occupancy, including Occ3D-nuScenes, where foreground class occupancy becomes a natural object-centric evaluation target [2512.02448]. This suggests that “nuScenes-FG” is most accurate as an umbrella term for these foreground-driven research uses rather than as a single benchmark.

## 4. Metrics and evaluation logic

Foreground evaluation in nuScenes is task-dependent. For 3D detection, nuScenes departs from IoU-based matching and uses center distance on the ground plane, averaged over distance thresholds $\mathbb{D} = \{0.5, 1, 2, 4\}$ meters across detection classes [1903.11027]. The canonical aggregate score is the nuScenes Detection Score:
$$
\text{NDS} = \frac{1}{10} \left[ 5 \times \text{mAP} + \sum_{i=1}^5 \left(1 - \min(1, \nu_i)\right) \right]
$$
where the five mean true-positive metrics are Average Translation Error, Average Scale Error, Average Orientation Error, Average Velocity Error, and Average Attribute Error [2512.02448].

For semantic segmentation and occupancy, mean Intersection-over-Union is the standard summary:
$$
\text{mIoU} = \frac{1}{N} \sum_{c=1}^{N} \frac{|P_c \cap T_c|}{|P_c \cup T_c|}
$$
for point segmentation in Panoptic nuScenes [2512.02448], and occupancy papers on nuScenes use the same IoU logic voxel-wise while excluding the empty class from the mean [2606.03581].

For panoptic segmentation, Panoptic Quality is defined as
$$
\text{PQ} = \frac{\sum_{(p, g) \in TP} \text{IoU}(p,g)}{|TP|} \times \frac{|TP|}{|TP| + \frac{1}{2}|FP| + \frac{1}{2}|FN|}
$$
which couples segment quality with recognition quality [2512.02448]. For panoptic tracking, Panoptic nuScenes introduces the instance-centric PAT metric,
$$
PAT = \frac{2 \times PQ \times TQ}{PQ + TQ},
$$
to address fragmentation and identity-consistency failures that are especially consequential for foreground agents [2109.03805].

These metrics show that nuScenes foreground evaluation is not a single-number problem. Detection emphasizes localization and attributes, segmentation emphasizes class overlap, and panoptic tracking emphasizes temporal identity continuity.

## 5. Methodological evolution in foreground-centric research

Foreground-driven nuScenes research has repeatedly motivated architectural changes. In LiDAR detection, InfoFocus models the non-uniform distribution of point clouds through density-guided refinement and reports 37.5 mAP on the nuScenes test set, versus 30.5 for the PointPillars baseline, with 31 FPS; with increased training it reaches 39.5 mAP [2007.08556]. The method explicitly targets informative and visible regions, which the paper links to better performance on partially occluded or sparse objects [2007.08556].

Class imbalance and point-level foreground supervision are another recurrent theme. “Structure Aware and Class Balanced 3D Object Detection on nuScenes Dataset” augments CBGS with an auxiliary network trained by foreground segmentation and center estimation. On the reported 10% split, CBGS attains 18.96 mAP and 26.8 NDS, while the best SA-CBGS configuration reaches 20.67 mAP and 29.36 NDS [2205.12519]. The paper further notes that foreground segmentation alone still provides a performance benefit relative to the baseline [2205.12519].

Recent multimodal detectors push the same foreground emphasis into fusion design. FocalFormer3D introduces Hard Instance Probing to reduce false negatives and reports 70.5 mAP and 73.9 NDS on the nuScenes detection benchmark, along with 72.1 AMOTA on the tracking benchmark [2308.04556]. IS-Fusion, which combines Hierarchical Scene Fusion and Instance-Guided Fusion, reports 73.0% mAP and 75.2% NDS on the nuScenes test set, with particularly strong results on small or difficult classes such as motorcycle and trailer [2403.15241]. Fusion4CA, built on BEVFusion, reports 69.7% mAP and 72.1 NDS with only 6 training epochs [2603.05305].

Foreground-centric rendering has also entered detection. OcRFDet argues that whole-scene radiance-field supervision produces strong background responses and instead uses Object-centric Radiance Fields that render only foreground regions. On the nuScenes test benchmark it reports 57.2% mAP and 64.8% NDS [2506.23565]. This suggests that foreground isolation is not only an evaluation concern but also a training-stability concern.

## 6. Challenges, extensions, and occupancy-derived interpretations

Several persistent difficulties shape any account of nuScenes-FG. nuScenes exhibits severe class imbalance: the original dataset analysis reports a most-to-least frequent class ratio of approximately 1:10,000, and camera-based detection drops much more at night than lidar-based detection [1903.11027]. The later survey adds that less than 7% of lidar points, and a similar area in images, are foreground, and notes that the lidar/radar-centric design may create “unfair” cases for camera-only approaches for certain far or partially visible foreground objects [2512.02448].

Controlled robustness studies amplify this line of work. “Occluded nuScenes” extends the original benchmark with parameterized occlusions for cameras, radar, and LiDAR while preserving the original annotations and file structure [2510.18552]. The resource is intended for repeatable robustness evaluation under partial sensor failures and environmental interference [2510.18552]. This broadens foreground research from nominal recognition to resilience under sensing degradation.

Foreground reasoning also appears in nuScenes-derived semantic occupancy. UnsOcc evaluates 3D semantic occupancy on nuScenes using 17 classes, a voxel grid of size $[200, 200, 16]$ with voxel size $[0.5\ \text{m}, 0.5\ \text{m}, 0.5\ \text{m}]$, and reports 28.1 mIoU, outperforming prior camera-only, LiDAR-only, and fusion baselines [2606.03581]. The paper highlights strong gains on challenging, rare, or small classes such as pedestrian, traffic cone, and bicycle [2606.03581]. OccFusion similarly frames nuScenes occupancy as a multimodal robustness problem and reports 27.3% mIoU for camera + lidar + radar fusion on the SurroundOcc labels [2403.01644]. Although these are occupancy benchmarks rather than canonical foreground benchmarks, the emphasis on rare and safety-critical object classes makes them part of the broader foreground-centered interpretation of nuScenes.

The main misconception surrounding “nuScenes-FG” is therefore terminological. It is not an official split. It is the aggregate of nuScenes benchmarks, extensions, and derived tasks in which the primary scientific question is how reliably a system can localize, segment, track, and reason about foreground objects against sparse geometry, long-tail class statistics, adverse conditions, and multimodal sensing constraints [2512.02448].

Source: https://www.emergentmind.com/topics/nuscenes-fg