DeepSense 6G: Multi-Modal ISAC Dataset

• DeepSense 6G is a large-scale real-world dataset integrating synchronized modalities like radar, camera, LiDAR, GPS, and IMU for multi-modal sensing and communication research.
• It employs a scenario-centric design to support tasks such as beam prediction, proactive blockage prediction, and sensor fusion with demonstrated high accuracy and reduced overhead.
• The dataset underpins studies in integrated sensing and communication, edge AI, and networked sensing, serving as a benchmark for reproducible and transferable 6G research.

Searching arXiv for papers on DeepSense 6G and closely related 6G ISAC work. First search: DeepSense 6G dataset and core papers. Searching for DeepSense 6G on arXiv. DeepSense 6G denotes a large-scale real-world multi-modal sensing-and-communication dataset and, in later 6G literature, a broader research paradigm in which synchronized sensing and communication measurements are used to train deep learning models for beam prediction, blockage prediction, localization, and integrated sensing and communication workflows. The dataset was built to advance deep learning research in the intersection of multi-modal sensing, communication, and positioning, and it has been used in studies spanning radar-aided beam prediction, proactive blockage prediction, multimodal fusion, and system-level beam-management evaluation (Alkhateeb et al., 2022, Demirhan et al., 2022, Demirhan et al., 2021, Orimogunje et al., 3 Jan 2026).

1. Terminology, scope, and historical placement

The term DeepSense 6G has two distinct but related uses in the literature. In the dataset lineage, it refers to “a large-scale real-world multi-modal sensing and communication dataset” built around co-existing sensing and communication measurements for 6G-oriented machine learning tasks such as sensor-aided communication and positioning (Demirhan et al., 2022). In a broader conceptual sense, later work uses the expression to denote a deep learning-driven, multi-modal sensing approach for 6G integrated sensing and communication, in which heterogeneous inputs such as radar, mmWave beams, camera, LiDAR, GPS RTK, and IMU are fused to support inference and control (Muthineni et al., 31 Mar 2025).

This usage must be distinguished from the older DeepSense dynamic spectrum sensing line. In the reading report on AI/ML for dynamic spectrum sensing and Open RAN, “DeepSense” refers to a foundational DSS framework that operates on raw in-phase and quadrature samples or time–frequency views for interference detection and spectral-hole identification; the paper explicitly states that it “does not introduce or define a separate ‘DeepSense 6G’ dataset or program” (Barker, 5 Feb 2025). The resulting ambiguity is substantive rather than merely lexical: one branch centers on spectrum awareness and ORAN deployment, whereas the DeepSense 6G dataset branch centers on synchronized multi-modal measurements for beam management, blockage prediction, localization, and ISAC evaluation.

As a dataset resource, DeepSense 6G occupies a specific position between synthetic frameworks and deployment-oriented testbeds. The dataset paper presents it as real-world, synchronized, and scenario-centric, explicitly contrasting that role with synthetic platforms such as DeepMIMO and ViWi (Alkhateeb et al., 2022). This makes DeepSense 6G important not only as a source of training data but also as a reproducibility substrate for cross-scenario generalization studies and physically grounded evaluation.

2. Dataset architecture, sensing stack, and collection methodology

DeepSense 6G is reported as containing more than one million data points across 40+ scenarios, collected in two countries across 15+ indoor/outdoor locations, with scenarios spanning vehicle-to-infrastructure, vehicle-to-vehicle, pedestrian, drone communication, fixed wireless, and RIS testbeds (Alkhateeb et al., 2022). The collection design is scenario-centric: each scenario is a single, typically long, data collection session, structured as a sequence of synchronized per-time-step data groups. This organization is intended to support both within-scenario learning and cross-scenario transfer.

At the communication layer, the canonical DeepSense 6G configuration uses a mmWave band centered at 60 GHz with a base-station unit built around a 16-element uniform linear phased array and a 64-beam combining codebook covering a 90° field of view. The user equipment employs a quasi-omni 60 GHz transmit antenna. At each time instant, the base station sweeps the 64 beams and records a 64-element receive-power vector; the ground-truth optimal beam index is obtained by the argmax over measured power (Alkhateeb et al., 2022). In later system-level work, this same 64-dimensional receive-power vector is treated not only as a label source but also as an in-band sensing feature, with the oracle beam defined as

$b_t^\star = \arg\max_{i\in\{1,\dots,64\}} p_{t,i}$

(Orimogunje et al., 3 Jan 2026).

The sensing stack is explicitly multi-modal. Reported modalities include RGB camera, 2D/3D LiDAR, radar, GPS RTK, and, in downstream experimental work, IMU and LiDAR ground truth. Capture rates used to synchronize modalities are 10 Hz beam sweeping for the mmWave receiver, 30 fps for the ZED2 RGB camera, 20 Hz sweeps for the Ouster 3D LiDAR, 10 Hz for GPS RTK, and radar acquisition in the TI FMCW family, reported as 76–81 GHz in the dataset overview and as 77 GHz FMCW in the ISAC task paper (Alkhateeb et al., 2022, Demirhan et al., 2022). Synchronization relies on UTC timestamps for all modalities, with camera exposure triggered immediately after each beam sweep and post-hoc alignment scripts used to form coherent per-time-step groups (Alkhateeb et al., 2022).

The collection methodology emphasizes overlapping fields of view and scenario-level alignment. Mechanical and geometric alignment is performed so that the phased array, camera, LiDAR, and radar observe consistent regions. The dataset paper also reports an in-house GUI visualizer for detecting misalignment, corrupted samples, or missing data (Alkhateeb et al., 2022). This instrumentation focus is central to the dataset’s role: it supports the conversion of heterogeneous raw measurements into benchmark-ready tuples that preserve both spatial context and communication labels.

3. Benchmark tasks and reported performance

DeepSense 6G has been used for several distinct task families, with radar-aided beam prediction and proactive blockage prediction constituting the earliest demonstrated use cases, and multimodal beam prediction and positioning extending that agenda in later work.

Task	Inputs	Reported result
Beam prediction in V2I	77 GHz radar range–angle maps to 60 GHz 64-beam codebook	Approximately 44% top-1 and up to 93% top-5 accuracy; about 10% top-1 gain over a classical solution (Demirhan et al., 2022)
Real-world radar-aided beam prediction	Raw radar cube, RA/RV/RC features, 64-beam labels	6319 labeled samples; ~45% top-1, top-5 ≈ 93.5%, and ≈ 93% beam-training overhead saving (Demirhan et al., 2021)
Proactive blockage prediction	Sequences of 77 GHz radar observations	F1-score more than 85% for 330 ms, 550 ms, 770 ms, and 990 ms horizons (Demirhan et al., 2022)
ISAC + IMU positioning	Range, Doppler, IMU, LiDAR ground truth	3.0 cm average 2D error and 5.5 cm 90th percentile error for the proposed DNN fusion (Muthineni et al., 31 Mar 2025)
Scenario-33 beam prediction	mmWave power, camera, LiDAR, radar, GPS	Top-5 = 0.98 for mmWave alone and mmWave+GPS/Radar/LiDAR; mmWave+camera = 0.94 (Orimogunje et al., 3 Jan 2026)

The original real-world radar-aided beam prediction demonstration is especially consequential because it ties DeepSense 6G directly to the beam-management problem in highly mobile mmWave/sub-THz systems. In Scenario 9, simultaneous FMCW radar measurements and mmWave beam training measurements were collected by the same stationary unit; after manual filtering to keep frames where the target car is in sight, the final dataset comprised 6319 labeled samples with a 70/30 train/test split (Demirhan et al., 2021). Compact CNNs operating on range–angle or range–velocity representations achieved around 90% top-5 beam prediction accuracy while saving 93% of the beam-training overhead, and an implementation based on range–angle maps with $M_F=4$ reported total preprocessing plus inference of approximately 15 ms (Demirhan et al., 2021).

The ISAC survey that uses DeepSense 6G formalizes two sensing-aided communication roles. The first is sensing-aided beam prediction, where pre-processed radar range–angle maps are used to predict top-$k$ beams from the 64-beam codebook in a vehicle-to-infrastructure scenario with nearly six thousand samples. The second is sensing-aided blockage and proactive hand-off prediction, where a CNN + LSTM consumes radar observation sequences to predict future LOS link blockages at 330 ms, 550 ms, 770 ms, and 990 ms before blockage (Demirhan et al., 2022). These two demonstrations anchor the dataset’s early identity as an ISAC benchmark rather than a generic sensor repository.

Later work extends this benchmark role. An experimental 27.4 GHz OFDM ISAC proof-of-concept fuses network sensing and IMU data for indoor target positioning in a $3.5\,\mathrm{m}\times 3.5\,\mathrm{m}$ area of interest, using a two-stage cascaded MLP architecture trained with LiDAR ground truth; the reported average 2D errors are 8.2 cm for DNN (ISAC), 5.6 cm for EKF fusion, and 3.0 cm for the proposed DNN (ISAC + IMU) (Muthineni et al., 31 Mar 2025). A system-level beam prediction study on DeepSense-6G Scenario-33 then reframes the 64-dimensional receive-power vector itself as an ISAC sensing stream and shows that mmWave alone is a strong standalone predictor, with fusions such as mmWave+GPS, mmWave+Radar, and mmWave+LiDAR preserving 98% Top-5 accuracy and very small rate loss (Orimogunje et al., 3 Jan 2026).

A further extension appears in the BD-RIS literature, where DeepSense 6G Scenario 8 is used with RGB camera images and GPS location data for quantum-enhanced beam prediction. That paper reports training and validation curves, confusion matrices, and beam-index distributions, and states that Q-ResNet34 significantly outperforms both Q-ViT and ViT in distance-based accuracy and cross-entropy loss, but it does not provide exact numeric top-$k$ accuracies or scenario split details (Khan et al., 29 Dec 2025). This establishes DeepSense 6G as a testbed not only for conventional deep learning but also for hybrid quantum–classical beam-management models.

4. DeepSense 6G within 6G ISAC, networked sensing, and edge intelligence

The broader 6G literature places DeepSense 6G within a transition from communication-centric wireless systems to systems in which sensing, communication, and learning are jointly designed. One line of work describes 6G ISAC as integrating sensing, learning, and communication across infrastructure sensors, user-equipment sensors, and radio sensing that reuses communication waveforms, with the fused output organized as an “ISAC map” that acts as a digital–physical twin of the environment (González-Prelcic et al., 2024). Another line describes networked sensing, where base stations share extracted range, angle, and Doppler features to jointly estimate target positions and velocities, emphasizing geometry diversity, multistatic sensing, and Fisher-information aggregation (Liu et al., 2022).

DeepSense 6G matches these architectural needs at the data level. Its co-existing sensing and communication measurements provide synchronized observations of the kind required for sensing-aided communication, communication-aided sensing, and multi-modal fusion. The ISAC roles paper explicitly uses the dataset to demonstrate sensing-aided beam prediction and sensing-aided blockage and hand-off prediction, while also positioning it as an initial step toward investigating waveform optimization, resource optimization, system security, network operation, communication-aided sensing optimization, and communication-aided network sensing (Demirhan et al., 2022). This suggests that the dataset functions both as a benchmark for current models and as a scaffold for broader ISAC role exploration.

The edge-AI perspective sharpens this interpretation. Integrated Sensing and Edge AI defines a task-oriented design that jointly optimizes sensing, communications, and edge AI computation to maximize end-to-end sensing-task performance under latency, reliability, energy, and privacy constraints (Liu et al., 12 Jan 2025). In that framework, DeepSense 6G is naturally aligned with split inference, cooperative multi-modal fusion, federated learning, and edge–cloud orchestration, because its synchronized modalities expose the tradeoffs between feature quality, sensing overhead, and downstream task performance. The dataset does not itself define an edge architecture, but the research it enables is structurally compatible with ISEA.

5. Relation to the DeepSense DSS and Open RAN lineage

A separate, but often conflated, branch of the literature connects DeepSense to dynamic spectrum sensing and ORAN deployment rather than to the DeepSense 6G dataset. In that branch, DeepSense is a real-time wideband DSS framework that operates directly on raw I/Q samples, adapts CNNs from computer vision to RF data, and targets interference detection and spectral-hole identification while minimizing preprocessing at the base station (Barker, 5 Feb 2025). Successor systems introduce explicit scalability mechanisms: DeepSweep divides wideband spectrum into subbands processed in parallel by lightweight CNNs, while Wideband Signal Stitching performs semantic spectrum segmentation at the I/Q level using a U-Net adaptation with non-local blocks and a scalable stitching pipeline (Barker, 5 Feb 2025).

The reported quantitative gains in this DSS lineage are specific: DeepSweep is reported to achieve below 1 ms end-to-end inference latency, 98% detection accuracy for narrowband interference, and approximately 10× lower computational demand than non-parallel baselines (Barker, 5 Feb 2025). The same paper argues that DeepSense/DeepSweep/WSS models can be packaged as xApps on the near-real-time RAN Intelligent Controller via the E2 interface, enabling spectrum monitoring, scheduling and resource-allocation support, and mitigation actions such as frequency reassignments or beam adjustments. It further situates these functions in ORAN control loops involving RF telemetry, DSS inference, policy decisions, RAN actuation, performance monitoring, adaptation, and digital-twin rehearsal before deployment (Barker, 5 Feb 2025).

The conceptual connection to DeepSense 6G is therefore architectural rather than taxonomic. DeepSense 6G provides synchronized multi-modal data for learning sensing-aided communication and ISAC tasks; the DeepSense DSS lineage provides a model of how learned RF perception can be embedded into ORAN-compatible, near-real-time 5G/6G control systems. The terminology overlap has generated understandable confusion, but the cited literature treats them as distinct research trajectories.

6. Limitations, interpretive cautions, and future directions

Several limitations recur across DeepSense 6G studies. At the dataset level, measurements center on mmWave at approximately 60 GHz, additional bands such as sub-6 GHz and THz are not included in current scenarios, and not all scenarios include every modality (Alkhateeb et al., 2022). The scenario-centric structure is powerful for transfer studies, but it also means that sensor availability, weather coverage, and environmental balance vary across scenarios. Position labels rely on GPS RTK, which is higher fidelity than standard GPS but still subject to practical errors (Alkhateeb et al., 2022).

At the task level, reproducibility is uneven. Some studies report detailed counts and splits—such as the 6319 labeled-sample Scenario 9 beam-prediction experiment with a 70/30 train/test split—whereas others omit dataset sizes, optimizer settings, or split protocols (Demirhan et al., 2021, Khan et al., 29 Dec 2025). The multimodal Scenario-33 study provides a contiguous 70/15/15 train/validation/test split and detailed modality preprocessing, but it does not report the number of sessions, frames, or samples (Orimogunje et al., 3 Jan 2026). The ISAC role paper explicitly notes that datasets and labels are crucial, and that the choice of supervised, unsupervised, or semi-supervised learning depends on annotation availability (Demirhan et al., 2022).

Methodological limitations also remain visible. The ISAC + IMU positioning demonstration uses a single indoor area of interest, a single wide-beam setup with limited angular resolution, and a single recorded trajectory, and it identifies the need for more trajectories, motion profiles, and multi-node ISAC data for stronger generalization (Muthineni et al., 31 Mar 2025). The DSS/ORAN report highlights labeling complexity in overlapping spectrum, broader dataset diversity needs, dynamic RF conditions that strain static classifiers, and missing standardized detection metrics such as ROC/AUC and $P_d/P_{fa}$ in the reported results (Barker, 5 Feb 2025).

The future directions are correspondingly broad. Reported priorities include transfer, meta, federated, and continual learning for cross-site adaptation; graph neural networks and attention for multi-node cooperative ISAC fusion; reinforcement learning for autonomous spectrum optimization; scaling to higher bands including terahertz; multi-RAT/NTN spectrum sharing; digital twins for safe validation; and standardized multi-modal datasets and benchmarks with synchronized measurements, calibration metadata, and diverse environments (Demirhan et al., 2022, Muthineni et al., 31 Mar 2025, Liu et al., 2022, Barker, 5 Feb 2025). A plausible implication is that DeepSense 6G will remain important less as a fixed benchmark than as a template for how 6G research should collect, align, and operationalize multi-modal wireless data.