Rascene: High-Fidelity 3D Scene Imaging with mmWave Communication Signals

Abstract: Robust 3D environmental perception is critical for applications such as autonomous driving and robot navigation. However, optical sensors such as cameras and LiDAR often fail under adverse conditions, including smoke, fog, and non-ideal lighting. Although specialized radar systems can operate in these environments, their reliance on bespoke hardware and licensed spectrum limits scalability and cost-effectiveness. This paper introduces Rascene, an integrated sensing and communication (ISAC) framework that leverages ubiquitous mmWave OFDM communication signals for 3D scene imaging. To overcome the sparse and multipath-ambiguous nature of individual radio frames, Rascene performs multi-frame, spatially adaptive fusion with confidence-weighted forward projection, enabling the recovery of geometric consensus across arbitrary poses. Experimental results demonstrate that our method reconstructs 3D scenes with high precision, offering a new pathway toward low-cost, scalable, and robust 3D perception.

• The paper introduces a monostatic mmWave sensing framework using commodity hardware to achieve high-fidelity 3D scene imaging with robust multi-frame fusion.
• It employs OFDM signals and phased-array antennas to convert channel impulse responses into precise point clouds, mitigating signal sparsity and multipath interference.
• Empirical evaluations show improved depth accuracy and occlusion resilience, outperforming LiDAR and existing radar methods in diverse indoor environments.

High-Fidelity 3D Scene Imaging with mmWave Communication Signals: An Expert Overview of "Rascene"

Motivation and Context

The proliferation of mmWave communication technologies such as 5G and high-frequency Wi-Fi has enabled widespread access to bandwidths and antenna arrays previously confined to radar systems. The limitations of conventional optical sensors (cameras, LiDAR) in adverse conditions—specifically, their dependency on lighting and susceptibility to smoke, fog, and occlusions—motivates the exploration of radio frequency (RF) modalities for environmental perception. Traditional radar-based solutions offer robustness but are hindered by hardware costs, spectrum licensing, and integration challenges. The integrated sensing and communication (ISAC) paradigm leverages existing communication infrastructure, but prior approaches have been largely bistatic and low-resolution. The Rascene framework proposes monostatic operation using commodity mmWave devices to enable robust, scalable, and high-fidelity 3D scene reconstruction.

Figure 1: Illustration of monostatic sensing in a mmWave communication system. The mmWave device simultaneously transmits and receives OFDM communication signals for sensing.

Monostatic mmWave Sensing Architecture

Rascene utilizes OFDM signals, standardized in 5G and Wi-Fi, for monostatic sensing. The system exploits full-duplex capability in commodity mmWave devices, enabled by directional phased-array antennas and short carrier wavelengths, to suppress self-interference and maintain Tx/Rx isolation. Channel Impulse Response (CIR) measurements are synchronized and locally acquired, allowing precise ranging and angular resolution analogous to FMCW radar without bespoke sensing hardware or licensed spectrum allocations. The raw radio data acquired are inherently sparse and multipath-corrupted, which precludes direct high-resolution reconstruction.

Figure 2: Illustration of angular estimation on a mmWave device.

Radio-frame Generation and 3D Representation

RF measurements are processed by transforming CIR estimates to point clouds in spherical and Cartesian coordinates, with spatial projection enabled by phased-array antenna geometry. Beamforming is used to estimate reflection strength at each voxel across azimuth, elevation, and range bins. The resulting radio point clouds encode object returns and are thresholded to suppress noise, but single-frame imaging remains fundamentally ill-posed under signal sparsity and multipath ambiguity.

Figure 3: Examples of generated radio point clouds, demonstrating the spatial sparsity and varying thresholds.

Multi-frame Confidence-weighted 3D Fusion

To overcome single-frame limitations, Rascene introduces a multi-frame, spatially adaptive fusion mechanism. Multiple radio frames, acquired at known poses via IMU, are encoded and warped into a reference frame. A confidence-aware forward projection is employed, leveraging predicted reliability and geometric proximity to fuse latent feature volumes. A coarse-to-fine 3D decoder densifies the sparse representations and outputs a reconstructed voxel grid and depth map. The entire imaging pipeline is optimized end-to-end using binary cross-entropy loss for voxels and L1 loss for depth, aggregating supervision across frames.

Figure 4: Overview of the multi-frame 3D RF imaging network, illustrating fusion, warping, decoding, and supervised output.

Prototyping, Data Collection, and Hardware Integration

A prototype Rascene device was constructed using an AMD/Xilinx RFSoC FPGA and Sivers mmWave transceiver, supporting 16 × 16 phased-array antennas, 60 GHz operation, and 1.2 GHz bandwidth. The platform, mounted on a cart with an Ouster LiDAR and IMU, was deployed across 20 indoor environments for large-scale dataset collection. Pairing RF and LiDAR data enabled calibration and ground truth extraction, permitting rigorous quantitative evaluation and alignment of modalities.

Figure 5: Rascene data collection platform showing integration of mmWave ISAC device, LiDAR, and IMU.

Empirical Evaluation and Numerical Results

Rascene demonstrates strong cross-scene generalization, achieving average absolute relative error (AbsRel) of 9.4%, MAE of 20.2 cm, and normalized Chamfer Distance CD$_{\text{Diag}}$ of 2.3% in unseen environments. Multi-frame fusion exhibits marked improvement over single-frame inference; for example, transitioning from 1 to 5 frames decreases AbsRel from 14.1% to 9.4% and CD from 31.6 cm to 19.7 cm. The system outperforms contemporary baselines (CartoRadar, PanoRadar), showing clear gains in depth and voxel reconstruction fidelity.

Figure 6: Qualitative results comparing Rascene predictions with LiDAR ground truth, highlighting reconstruction in optically challenging regions.

Figure 7: Qualitative comparison of single-frame and 5-frame predictions, demonstrating reduced artifacts and improved geometry with multi-frame fusion.

Figure 8: Cumulative distribution functions illustrating absolute and relative depth estimation errors, indicating a median absolute error of 6.1 cm and robust performance.

Figure 9: Depth estimation accuracy across azimuth and altitude, revealing field of view-dependent variance and highest accuracy near boresight.

Figure 10: Depth error as a function of range, showing multi-frame fusion's efficacy in mitigating long-distance degradation.

Occlusion Resilience and Robustness

Rascene is resilient to occlusion by common materials such as paper and styrofoam, with negligible degradation in depth and voxel metrics. Qualitative examples illustrate its ability to reconstruct geometry where LiDAR fails due to optical occlusion, validating RF sensing's practical robustness.

Figure 11: Representative examples of occlusion resilience, comparing LiDAR and Rascene in corridor scenes.

Practical and Theoretical Implications

Rascene demonstrates that high-fidelity 3D imaging is achievable using ubiquitous mmWave communication signals without dedicated sensing hardware or spectrum. The monostatic configuration ensures coherent ranging and geometric consistency fundamental to robust perception. The adaptive multi-frame fusion strategy addresses multipath ambiguity and signal sparsity, establishing a scalable ISAC paradigm for 3D scene understanding. Practical implications include deployment on commodity devices, energy efficiency, and sensing in optically challenging environments. Theoretically, the work establishes a pathway for dual-functionality in wireless networks—enabling perception in addition to communication—and suggests directions for adaptive signal processing, more sophisticated fusion algorithms, and cross-modal learning.

Figure 12: Prototyped monostatic ISAC device, integrating communication and sensing in a compact form factor.

Figure 13: Illustration of simultaneous video streaming communication during sensing data collection.

Figure 14: Sample trajectory segments from datasets, visualized on ground truth LiDAR point clouds.

Figure 15: Example snapshots from 20 distinct indoor environments, showing the diversity of the data corpus.

Conclusion

Rascene offers a rigorous ISAC framework for 3D environmental perception using mmWave communication signals, integrating monostatic sensing, full-duplex CIR acquisition, and multi-frame confidence-weighted fusion. Empirical results establish its superiority over baseline radar methods, enhanced generalization, occlusion resilience, and fidelity approaching LiDAR in challenging conditions. The integration of sensing and communication on commodity hardware foreshadows a new generation of scalable, robust, and cost-effective perception systems. Future directions may include outdoor deployment, real-time inference, mobile device integration, and fusion with additional sensing modalities to further improve environmental perception under adverse conditions.

(2604.02603)