DeepSense 6G Dataset for Integrated Wireless Research

Updated 5 January 2026

DeepSense 6G is a comprehensive, real-world collection of synchronized multi-modal sensor data including radar, camera, LiDAR, GPS, and mmWave for 6G research.
It enables studies on sensing-aided communication, ISAC waveform design, and precise localization across over 40 dynamic scenarios.
Benchmark results show that machine learning models leveraging this dataset achieve high beam prediction accuracy and efficient blockage prediction with reduced overhead.

DeepSense 6G is a large-scale, real-world dataset designed to advance deep learning research at the intersection of multi-modal sensing, communication, and positioning, specifically targeting challenges in 6G-class wireless systems. It comprises over one million data samples containing synchronized measurements from co-located radar, camera, LiDAR, GPS, and mmWave communication devices. By facilitating reproducible studies in sensing-aided communication, integrated sensing and communication (ISAC) waveform design, positioning, and object detection, DeepSense 6G establishes a comprehensive resource for evaluating machine learning approaches in realistic wireless deployment scenarios (Alkhateeb et al., 2022).

1. Dataset Objectives and Scope

DeepSense 6G was developed to provide a large-scale foundation for machine-learning-based research in highly-mobile and dynamic wireless environments. The principal objectives include supplying diverse, time-synchronized data from varied modalities, enabling studies in:

Sensing-aided communication (e.g., beam prediction, blockage forecasting, and hand-off optimization).
ISAC waveform design and resource allocation.
Communication-aided positioning and scene understanding via sensor fusion.

Data were collected in over forty scenarios, spanning urban, suburban, and indoor settings with vehicular, pedestrian, drone, and RIS-assisted deployments. Notable use cases include vehicle-to-infrastructure (V2I) beam selection, proactive blockage prediction and link recovery, high-accuracy localization via GPS-RTK and mmWave channel fusion, and multi-modal tracking in cluttered scenes (Alkhateeb et al., 2022, Demirhan et al., 2021).

2. Sensors and Modalities

DeepSense 6G implements a modular testbed concept. Each unit, whether stationary or mobile, carries a subset of the following sensors:

mmWave Communication Measurement: 60 GHz phased-array transceiver (Sivers Semiconductors) with a 16-element uniform linear array (ULA) receiver and quasi-omni transmitter. Beam-training performed with a static 64-beam codebook covering 90° field of view; sampling at 10 Hz yields a 64-element received-power vector per sweep.
GPS-RTK Receiver: Sub-10 cm accuracy with 10 Hz updates.
RGB Stereo Camera (ZED2): 110° horizontal FoV, 1920×1080 at 30 fps.
3D LiDAR (Ouster OS1-32): 32 vertical × 1024 horizontal channels, 120 m max range, 20 Hz.
FMCW mmWave Radar (TI AWR2243BOOST): 76–81 GHz, 750 MHz bandwidth, up to 20 Hz, 4 RX antennas, and 1 TX antenna.

Each measurement is time-synchronized, calibrated using mechanical fixtures and software tools, with overlapping physical and field-of-view configurations to guarantee cross-modality alignment. Environmental metadata (weather, time of day, location) and sensor calibration parameters are included per scenario (Alkhateeb et al., 2022, Demirhan et al., 2021).

3. Data Structure, Formats, and Synchronization

Data is organized hierarchically at the scenario level:

Directory Layout
- /scenario_{ID}/
- config.json: Scenario metadata (location, time, modalities)
- calibration/: YAML/JSON files for intrinsic and extrinsic sensor parameters
- timestamps.csv: UTC-aligned frame records
- data/frame_{N}/: Contains sensor files for each sample
Per-Frame Files
- RadarRaw ∈ ℂ^{{4×256×128}} (complex64), raw radar I/Q samples
- BeamPowers ∈ ℝ^{64}, received power per predefined beam
- CameraImage ∈ ℝ^{H×W×3}, RGB image
- lidar.pcd: Point cloud (when available)
- gps.csv: Position log
Synchronization: UTC timestamps and hardware triggers assure cross-modality temporal alignment. Typical inter-modal jitter is under 50 ms. Each frame consists of radar + beam sweep + synchronized camera/LiDAR/GPS measurements. All calibration is performed offline; for beam prediction, radar and mmWave beam sweep are simply paired by frame index, with no explicit calibration required (Demirhan et al., 2021).

4. Annotation, Ground Truth, and Labeling

Beam Labeling: For each frame, the ground-truth beam index $n^* \in \{0, ..., 63\}$ is assigned as the argmax of the received power vector:

$n^* = \underset{n}{\arg\max} \; \mathsf{BeamPowers}[n]$

Top-K Annotation: For ranking-based tasks (e.g., Top-3, Top-5 accuracy), annotations are derived by sorting BeamPowers.
Object Detection / Position Reference: When camera and LiDAR data are provided, bounding boxes (YOLOv3 for images), LiDAR/radar clustering, and GPS-RTK serve as ground truth for position localization. No explicit object boxes are provided in Scenario 9; inference of object position from radar data is left to the user (Alkhateeb et al., 2022, Demirhan et al., 2021).

5. Radar Data Preprocessing and Feature Extraction

Three canonical radar feature variants are specified:

Range–Angle Map ( $X_{RA}$ ):

Range FFT across ADC samples per chirp.
Mean-removal across chirps.
Angle FFT across RX antennas (zero-padded to $M_F$ ).
Sum over all chirps:

$X_{RA} = \Psi_P^{RA}(X) = \sum_{a=1}^A F_{2D}(X_{:,\,\cdot,\,a})$

Result: Real-valued $[M_F \times S]$ range-angle matrix.

Range–Velocity Map ( $X_{RV}$ ):

Range FFT across ADC samples.
Doppler FFT across chirps.
Sum over RX antennas:

$X_{RV} = \Psi_P^{RV}(X) = \sum_{m=1}^{M_r} F_{2D}(X_{m,\,\cdot,\,\cdot})$

Result: $[S \times A]$ range-velocity matrix.

Radar Cube ( $X_{RC}$ ):
- Full 3D FFT across antennas, samples, chirps:
$X_{RC} = \Psi_P^{RC}(X) = F_{3D}(X)$

Result: $[M_r \times S \times A]$ cube.

Feature standardization is performed by zero-mean, unit-variance normalization. Input maps are subsequently processed by a convolutional neural network as described in the respective benchmark papers (Demirhan et al., 2021).

6. Loading Pipelines and Machine Learning Workflows

The dataset includes open-source loading scripts and Python/PyTorch examples:

import h5py, numpy as np
with h5py.File('snapshot123.h5','r') as f:
    X = f['RadarRaw'][...]        # shape: (4,256,128), complex64
    y = np.argmax(f['BeamPowers'][...])
X_fft_range = np.fft.fft(X, axis=1)
X_clutter_removed = X_fft_range - X_fft_range.mean(axis=2, keepdims=True)
X_angle = np.fft.fft(X_clutter_removed, n=M_F, axis=0)
X_RA = X_angle.sum(axis=2)       # feed to CNN±FC architecture (see Table II)

Labels are one-hot encoded (

\mathbb{R}^{64}

) for cross-entropy loss. Scenario-specific ML pipelines can be constructed by selecting feature extractors and modality combinations per benchmark specification (Demirhan et al., 2021, Alkhateeb et al., 2022).

7. Benchmarks and Representative Applications

Published results on DeepSense 6G include:

Radar-aided Beam Prediction: CNN-based models using radar features achieve $\sim90\%$ Top-5 beam prediction accuracy and require only $7\%$ of the original beam training overhead in vehicular scenarios (Demirhan et al., 2021).
Multi-modal Beam Prediction: DNNs integrating camera bounding boxes and GPS data yield over $90\%$ Top-3 beam accuracy scenario-specifically, with cross-scenario generalization at $>70\%$ Top-3 (Alkhateeb et al., 2022).
Blockage Prediction & Handoff: LSTM networks consuming sequences of beam vectors and radar heatmaps output "blockage imminent" labels.
Localization Refinement: Regression DNNs using GPS and mmWave fingerprints can reduce V2I LoS localization error to sub-$0.2$ m.
Object Detection/Classification: Fused radar, LiDAR, and camera annotations support evaluation of object detection ([email protected] IoU).
Data access: All code, data loaders, and baseline models are freely available at https://deepsense6g.net for reproducibility.

A plausible implication is that the DeepSense 6G dataset standardizes research practices for integrated sensing and communication in practical wireless systems, enabling new advances in multi-modal fusion and situational awareness for 6G deployments (Alkhateeb et al., 2022, Demirhan et al., 2021).