nuView: View-Aware Scene Captioning

• nuView is a view-aware scene captioning corpus that delivers detailed, per-camera descriptions of environmental and situational contexts for autonomous driving.
• It generates six distinct captions per timestamp from nuScenes, enabling rigorous evaluation of view-conditioned language generation and multimodal grounding.
• Balanced across weather, lighting, and regional diversity, nuView fosters the development of models with robust environmental perception and spatial precision.

nuView is a large-scale, view-aware scene captioning corpus derived from nuScenes, designed explicitly for advancing and benchmarking multimodal models in autonomous driving that require fine-grained, per-camera environmental and situational understanding. Built to address significant limitations in existing 3D scene captioning datasets, nuView provides model developers with granular, richly annotated training and evaluation data to foster research on view-conditioned language generation and cross-modal grounding (Brandstaetter et al., 25 Jul 2025).

1. Motivation and Rationale

Existing autonomous driving captioning datasets—most notably nuCaption—lack explicit view-conditioning and are strongly biased toward descriptions of traffic agents, vehicles, and dynamic interactions while severely under-representing contextual environmental details such as weather, scene illumination, road surface characteristics, and broader infrastructural context. Moreover, all prior benchmarks provide only a single caption per timestamp, overlooking the multi-camera nature of real-world AD perception stacks, and do not require or evaluate the capacity to generate camera-specific descriptions. nuView was introduced to address these deficiencies by generating six distinct captions per timestamp (one per camera), each explicitly annotated for environmental and viewpoint attributes.

2. Dataset Construction and Core Statistics

nuView leverages the native nuScenes sensor rig comprising six 1600×900 RGB surround cameras (CAM_FRONT, CAM_FRONT_LEFT, CAM_FRONT_RIGHT, CAM_BACK, CAM_BACK_LEFT, CAM_BACK_RIGHT) and a 32-beam LiDAR. For every timestamp in nuScenes, nuView provides six samples—one per camera view—culminating in a total of 205,067 captioned instances spanning 34,178 unique driving moments (205,067 = 34,178 × 6) and ∼2.85 hours of real-world driving coverage. The dataset is officially split into 169,000 training and 36,067 validation samples; test captions are withheld for future benchmarking (Brandstaetter et al., 25 Jul 2025).

Each sample contains:

• view_id: identifying one of the six camera poses
• image: RGB frame from the specific camera
• timestamp and sample_token: unique identifiers as in nuScenes
• caption: a 20–60 word English paragraph emphasizing environmental, lighting, and infrastructural features in addition to traffic agents

Captions were auto-generated with Llava-1.6 (Llama-3-7B backbone) using tailored, view-specific prompts and lightly filtered for coherence.

Environmental and temporal balance is maintained by stratified sampling across Boston, Cambridge, and Singapore, covering variable weather patterns, architectural styles, and urban densities.

3. Emphasis on Environmental and Viewpoint Diversity

nuView distinctly annotates and balances for a wide range of environmental phenomena and camera perspectives:

• Weather/Lighting Conditions: Captions explicitly reference five major conditions—rain (12%), wet-road reflections (8%), overcast/cloudy skies (35%), direct sunlight/shadows (25%), nighttime (5%)—alongside mixed and less common scenarios (e.g., fog).
• Viewpoint Coverage: All six camera IDs are represented at roughly equal frequencies (16–17% each), ensuring no viewpoint dominates or is underrepresented.
• Geography: Distribution is balanced among the three major nuScenes cities to provide variation in visual backgrounds and scene types.

This design enforces that novel models must perform robustly across an array of environmental and positional settings, not merely in canonical daylight, traffic-dense, or front-facing scenarios.

4. Annotation Schema and Data Format

Annotations follow a concise JSON Lines format, e.g.:

{ "timestamp": "...", "sample_token": "...", "view_id": "CAM_FRONT_RIGHT", "caption": "The road appears wet as recent rain reflects light from street lamps. I see two cars ahead and several buildings lining the street." }

• Captions are generally phrased in first-person or neutral descriptive prose.
• Prompts for Llava-1.6 were view-specific; e.g., instructing the model to focus on objects "to your left" or "straight ahead" in the corresponding view sector.
• Annotation post-processing was limited to light coherence filtering, not human-intensive curation.

5. Benchmarking Protocol and Evaluation Metrics

nuView establishes a rigorous protocol for evaluating view-conditioned 3D captioning:

• Input at inference: six camera images, one LiDAR sweep, and an explicit prompt specifying the target camera view (e.g., "Focus on CAM_FRONT_LEFT. Describe what you see.").
• Task: For the specified view, generate a paragraph that richly describes visible weather, lighting, road surface, infrastructure, and traffic scene.
• Metrics:
  • BLEU-n (n=1..4): computes brevity-penalized n-gram precision
  • ROUGE-L: F1 score for longest common subsequence
  • BERT-F1: cosine similarity in BERT-base embedding space (as in [Zhang et al., 2020])
  • All metrics are computed per instance and macro-averaged over the split.

This protocol is uniquely tailored to penalize models unable to focus on the designated view or to describe salient environmental cues, thus promoting true multimodal grounding across appearance, geometry, and language.

6. Comparison with Related Datasets

A summary comparison of nuView with nuCaption and GroundView:

Dataset	#Samples (train/val)	3D Caption Focus	Grounding Focus	Main Theme
nuCaption	240,000 (169k/70k)	Traffic agents	55k samples	Traffic
nuView	205,067 (169k/36k)	Environment	(none)	Weather/Views
GroundView	7,400 (6k/1.4k)	Objects	51k samples	Grounding

Key distinctions:

• nuCaption supplies a single LiDAR-focused caption per timestamp, skewed toward vehicle and pedestrian traffic.
• nuView supplies six camera-aligned, view- and environment-specific captions per timestamp, providing rich descriptions suitable for evaluating per-view reasoning, weather perception, and positional encoding techniques.
• GroundView, by contrast, provides terse, template-based object count annotations aimed at geometric grounding rather than scene or environmental description.

nuView is thus complementary, trading off pure sample quantity for crucial gains in viewpoint granularity and ecological realism.

7. Significance and Applications

nuView has become a reference benchmark for developing and evaluating models—such as BEV-LLM—that must fuse multi-camera images, LiDAR, spatial encoding, and view-conditioning to generate grounded, interpretable scene descriptions (Brandstaetter et al., 25 Jul 2025). Its focus on environmental awareness and explicit view annotation facilitates research into LLMs and multimodal systems that must reason about lighting, weather, and real-world nuances, which are essential for safe, transparent autonomous navigation and human-aligned AI in dynamic, unstructured environments.

By explicitly specifying both "where to look" (camera selection) and "what to describe" (environment, surface, context), nuView imposes requirements on models for both spatial localization and compositional, contextually-aware language generation, catalyzing advances across scene understanding, vision-and-language fusion, and real-world robotic interpretation.