---
title: DeepSense 6G Dataset for Integrated Wireless Research
url: https://www.emergentmind.com/topics/deepsense-6g-dataset
type: topic
---

# DeepSense 6G Dataset for Integrated Wireless Research

DeepSense 6G is a large-scale, real-world dataset designed to advance deep learning research at the intersection of multi-modal sensing, communication, and positioning, specifically targeting challenges in 6G-class wireless systems. It comprises over one million data samples containing synchronized measurements from co-located radar, camera, LiDAR, GPS, and mmWave communication devices. By facilitating reproducible studies in sensing-aided communication, integrated sensing and communication (ISAC) waveform design, positioning, and object detection, DeepSense 6G establishes a comprehensive resource for evaluating machine learning approaches in realistic wireless deployment scenarios [2211.09769].

## 1. Dataset Objectives and Scope

DeepSense 6G was developed to provide a large-scale foundation for machine-learning-based research in highly-mobile and dynamic wireless environments. The principal objectives include supplying diverse, time-synchronized data from varied modalities, enabling studies in:

- Sensing-aided communication (e.g., beam prediction, blockage forecasting, and hand-off optimization).
- ISAC waveform design and resource allocation.
- Communication-aided positioning and scene understanding via sensor fusion.

Data were collected in over forty scenarios, spanning urban, suburban, and indoor settings with vehicular, pedestrian, drone, and RIS-assisted deployments. Notable use cases include vehicle-to-infrastructure (V2I) beam selection, proactive blockage prediction and link recovery, high-accuracy localization via GPS-RTK and mmWave channel fusion, and multi-modal tracking in cluttered scenes [2211.09769, 2111.09676].

## 2. Sensors and Modalities

DeepSense 6G implements a modular testbed concept. Each unit, whether stationary or mobile, carries a subset of the following sensors:

- **mmWave Communication Measurement**: 60 GHz phased-array transceiver (Sivers Semiconductors) with a 16-element uniform linear array (ULA) receiver and quasi-omni transmitter. Beam-training performed with a static 64-beam codebook covering 90° field of view; sampling at 10 Hz yields a 64-element received-power vector per sweep.
- **GPS-RTK Receiver**: Sub-10 cm accuracy with 10 Hz updates.
- **RGB Stereo Camera (ZED2)**: 110° horizontal FoV, 1920×1080 at 30 fps.
- **3D LiDAR (Ouster OS1-32)**: 32 vertical × 1024 horizontal channels, 120 m max range, 20 Hz.
- **FMCW mmWave Radar (TI AWR2243BOOST)**: 76–81 GHz, 750 MHz bandwidth, up to 20 Hz, 4 RX antennas, and 1 TX antenna.

Each measurement is time-synchronized, calibrated using mechanical fixtures and software tools, with overlapping physical and field-of-view configurations to guarantee cross-modality alignment. Environmental metadata (weather, time of day, location) and sensor calibration parameters are included per scenario [2211.09769, 2111.09676].

## 3. Data Structure, Formats, and Synchronization

Data is organized hierarchically at the scenario level:

- **Directory Layout**
  - `/scenario_{ID}/`
    - `config.json`: Scenario metadata (location, time, modalities)
    - `calibration/`: YAML/JSON files for intrinsic and extrinsic sensor parameters
    - `timestamps.csv`: UTC-aligned frame records
    - `data/frame_{N}/`: Contains sensor files for each sample

- **Per-Frame Files**
  - `RadarRaw` ∈ ℂ^{4×256×128} (complex64), raw radar I/Q samples
  - `BeamPowers` ∈ ℝ^{64}, received power per predefined beam
  - `CameraImage` ∈ ℝ^{H×W×3}, RGB image
  - `lidar.pcd`: Point cloud (when available)
  - `gps.csv`: Position log

- **Synchronization**: UTC timestamps and hardware triggers assure cross-modality temporal alignment. Typical inter-modal jitter is under 50 ms. Each frame consists of radar + beam sweep + synchronized camera/LiDAR/GPS measurements. All calibration is performed offline; for beam prediction, radar and mmWave beam sweep are simply paired by frame index, with no explicit calibration required [2111.09676].

## 4. Annotation, Ground Truth, and Labeling

- **Beam Labeling**: For each frame, the ground-truth beam index $n^* \in \{0, ..., 63\}$ is assigned as the argmax of the received power vector:  
  $$ n^* = \underset{n}{\arg\max} \; \mathsf{BeamPowers}[n] $$
- **Top-K Annotation**: For ranking-based tasks (e.g., Top-3, Top-5 accuracy), annotations are derived by sorting `BeamPowers`.
- **Object Detection / Position Reference**: When camera and LiDAR data are provided, bounding boxes (YOLOv3 for images), LiDAR/radar clustering, and GPS-RTK serve as ground truth for position localization. No explicit object boxes are provided in Scenario 9; inference of object position from radar data is left to the user [2211.09769, 2111.09676].

## 5. Radar Data Preprocessing and Feature Extraction

Three canonical radar feature variants are specified:

- **Range–Angle Map ($X_{RA}$)**:
  1. Range FFT across ADC samples per chirp.
  2. Mean-removal across chirps.
  3. Angle FFT across RX antennas (zero-padded to $M_F$).
  4. Sum over all chirps:
     $$ X_{RA} = \Psi_P^{RA}(X) = \sum_{a=1}^A F_{2D}(X_{:,\,\cdot,\,a}) $$
     Result: Real-valued $[M_F \times S]$ range-angle matrix.

- **Range–Velocity Map ($X_{RV}$)**:
  1. Range FFT across ADC samples.
  2. Doppler FFT across chirps.
  3. Sum over RX antennas:
     $$ X_{RV} = \Psi_P^{RV}(X) = \sum_{m=1}^{M_r} F_{2D}(X_{m,\,\cdot,\,\cdot}) $$
     Result: $[S \times A]$ range-velocity matrix.

- **Radar Cube ($X_{RC}$)**:
  - Full 3D FFT across antennas, samples, chirps:
    $$ X_{RC} = \Psi_P^{RC}(X) = F_{3D}(X) $$
    Result: $[M_r \times S \times A]$ cube.

Feature standardization is performed by zero-mean, unit-variance normalization. Input maps are subsequently processed by a convolutional neural network as described in the respective benchmark papers [2111.09676].

## 6. Loading Pipelines and Machine Learning Workflows

The dataset includes open-source loading scripts and Python/PyTorch examples:

```python
import h5py, numpy as np
with h5py.File('snapshot123.h5','r') as f:
    X = f['RadarRaw'][...]        # shape: (4,256,128), complex64
    y = np.argmax(f['BeamPowers'][...])
X_fft_range = np.fft.fft(X, axis=1)
X_clutter_removed = X_fft_range - X_fft_range.mean(axis=2, keepdims=True)
X_angle = np.fft.fft(X_clutter_removed, n=M_F, axis=0)
X_RA = X_angle.sum(axis=2)       # feed to CNN±FC architecture (see Table II)
```
Labels are one-hot encoded ($\mathbb{R}^{64}$) for cross-entropy loss. Scenario-specific ML pipelines can be constructed by selecting feature extractors and modality combinations per benchmark specification [2111.09676, 2211.09769].

## 7. Benchmarks and Representative Applications

Published results on DeepSense 6G include:

- **Radar-aided Beam Prediction**: CNN-based models using radar features achieve $\sim90\%$ Top-5 beam prediction accuracy and require only $7\%$ of the original beam training overhead in vehicular scenarios [2111.09676].
- **Multi-modal Beam Prediction**: DNNs integrating camera bounding boxes and GPS data yield over $90\%$ Top-3 beam accuracy scenario-specifically, with cross-scenario generalization at $>70\%$ Top-3 [2211.09769].
- **Blockage Prediction & Handoff**: LSTM networks consuming sequences of beam vectors and radar heatmaps output "blockage imminent" labels.
- **Localization Refinement**: Regression DNNs using GPS and mmWave fingerprints can reduce V2I LoS localization error to sub-$0.2$ m.
- **Object Detection/Classification**: Fused radar, LiDAR, and camera annotations support evaluation of object detection (AP@0.5 IoU).
- **Data access**: All code, data loaders, and baseline models are freely available at https://deepsense6g.net for reproducibility.

A plausible implication is that the DeepSense 6G dataset standardizes research practices for integrated sensing and communication in practical wireless systems, enabling new advances in multi-modal fusion and situational awareness for 6G deployments [2211.09769, 2111.09676].

Source: https://www.emergentmind.com/topics/deepsense-6g-dataset