---
title: 'RAD-GAN: Radar-Aware Speech Reconstruction'
url: https://www.emergentmind.com/topics/radar-aware-dual-conditioned-generative-adversarial-network-rad-gan
type: topic
---

# RAD-GAN: Radar-Aware Speech Reconstruction

Searching arXiv for the RAD-GAN paper and closely related radar GAN work.
arXiv search query: "Radar-Aware Dual-conditioned GAN mmWave radar speech reconstruction"
Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN) denotes a two-stage speech reconstruction pipeline for mmWave radar in which a radar-derived low-band, low-SNR representation is transformed into an intelligible full-band \(8\) kHz waveform through dual-conditioned adversarial training. In its named form, RAD-GAN was introduced for speech reconstruction from **mmWave FMCW radar** captures obtained **through a glass wall**, under **global SNR roughly \(-5\) dB to \(-1\) dB**, with a design centered on a **Residual Fusion Gate (RFG)**, a **Multi-Mel Discriminator (MMD)**, and staged training that first learns bandwidth extension from synthetically clipped clean speech and then adapts to real radar-derived inputs [2602.22431].

## 1. Definition and problem setting

RAD-GAN addresses a radar-conditioned speech reconstruction problem rather than conventional radar imaging or radar scene synthesis. The input is not an optical recording, nor a clean microphone spectrogram corrupted by additive noise, but a radar-derived observation whose information content is both **band-limited** and **noisy**, and whose relation to the target waveform is indirect because mmWave radar measures vibration rather than air pressure. The target is a **full-band \(8\) kHz waveform**, while the reliable conditioning content is concentrated below about **\(1\) kHz**; the method therefore treats the task as **speech reconstruction via bandwidth extension** [2602.22431].

The paper formulates two data-acquisition regimes. In **Task 1**, the radar captures **speaker diaphragm vibrations directly** through a glass wall. In **Task 2**, the radar captures vibrations from an **aluminum foil** placed near the speaker diaphragm, which yields worse segmental SNR. Across both tasks, the task difficulty is defined by three forms of mismatch: the radar input is **low-SNR**, **spectrally limited**, and collected under physically difficult sensing conditions. The paper therefore frames RAD-GAN not as a generic enhancement model, but as a constrained low-band-to-full-band reconstruction system that must preserve whatever low-frequency content is recoverable while inferring plausible upper-band structure [2602.22431].

A common misunderstanding is to treat RAD-GAN as a generic “radar GAN” for automotive perception, point-cloud generation, or radar-map translation. That interpretation is not supported by the named RAD-GAN paper. In this usage, RAD-GAN is specifically an **mmWave radar-to-speech** model. This distinguishes it from earlier radar GAN work on point-cloud scene synthesis [2410.13526], radar signal restoration [2407.13949], one-bit PMCW range–Doppler reconstruction [2503.12841], or spectral recovery in UWB SAR [1812.04744].

## 2. Dual conditioning and generator architecture

The architecture is organized as a **two-stage radar-to-speech reconstruction pipeline**. A noisy radar-derived waveform is converted into a mel representation. In parallel, the same noisy waveform is processed by **WaveVoiceNet (WVN)** to produce an auxiliary enhanced waveform, which is also converted into a mel spectrogram. These two mel representations are then fused by the **Residual Fusion Gate**, and the fused mel spectrogram conditions a **HiFi-GAN-based generator** that synthesizes the reconstructed waveform [2602.22431].

The paper is explicit that the **generator architecture itself is not modified from HiFi-GAN**. It uses the **original HiFi-GAN generator without architectural modification**, with **transposed-convolution upsampling** and **Multi-Receptive Field (MRF) residual fusion blocks**. The generator input is an **80-bin mel spectrogram** derived with \(n_{\text{fft}} = 1024\), hop size \(=128\), window size \(=512\), Hann window, \(n_{\text{mels}}=80\), \(f_{\min}=0\) Hz, and \(f_{\max}=1000\) Hz. There is **no stochastic noise input**; the waveform is generated directly from mel conditioning [2602.22431].

The term **“dual-conditioned”** has a specific meaning in this paper. It does **not** mean that the HiFi-GAN generator contains two separate conditioning branches injected into its internal blocks. Instead, dual conditioning occurs **before** the generator: the conditioning mel is constructed from two sources, the **noisy radar mel** \(\mathbf{M}_n\) and the **WVN-enhanced mel** \(\mathbf{M}_w\). The fused mel \(\mathbf{M}_f\) is then supplied as the single conditioning input to the unchanged generator. This resolves a second common misconception: RAD-GAN is dual-conditioned at the **input-representation level**, not via a custom dual-stream generator body [2602.22431].

The **Residual Fusion Gate** is defined as
\[
\begin{gathered}
\mathbf{G}=\sigma\left(\mathrm{Conv}_{1\times1}\left([\mathbf{M}_{n};\ \mathbf{M}_{w}-\mathbf{M}_{n}]\right)\right),\\
\mathbf{M}_{f}=\mathbf{M}_{n}+\sigma(a)\,\mathbf{G}\odot\left(\mathbf{M}_{w}-\mathbf{M}_{n}\right),
\end{gathered}
\]
where \([\cdot;\cdot]\) denotes channel-wise concatenation, \(\odot\) is element-wise multiplication, \(a\) is a learnable scalar logit, and \(\sigma(\cdot)\) is the sigmoid function [2602.22431].

This fusion rule gives the architecture its main radar-aware inductive bias. \(\mathbf{M}_n\) acts as the **carry baseline**, while \(\mathbf{M}_{w}-\mathbf{M}_{n}\) is the **residual correction** proposed by WVN. The gate \(\mathbf{G}\) decides where that correction should be trusted. Because the input to the gate is \([\mathbf{M}_n;\mathbf{M}_w-\mathbf{M}_n]\), and \(F=80\), the gate performs a **\(160 \rightarrow 80\)** pointwise mapping. The paper states that this is applied **frame-wise**, performing **cross-frequency mixing across mel bins without temporal smoothing**. Both the gate bias and the scalar \(a\) are initialized to **\(-2.0\)** so that early training remains conservative and does not over-trust the auxiliary WVN branch [2602.22431].

## 3. Discriminators, losses, and training procedure

RAD-GAN augments the standard HiFi-GAN waveform adversaries with a mel-domain discriminator. The model retains **MPD** and **MSD** from HiFi-GAN, then adds a **Multi-Mel Discriminator (MMD)** to judge realism directly in the time-frequency domain. In the accessible description, “multi-mel” refers not to multiple mel resolutions, but to **two parallel mel discriminators** with different normalization strategies: one uses **spectral normalization** and one uses **weight normalization** [2602.22431].

Each MMD branch processes mel inputs of shape \((B,1,n_{\text{mels}},T)\), with \(n_{\text{mels}}=80\), through the layer progression
\[
1 \rightarrow 32 \rightarrow 64 \rightarrow 128 \rightarrow 256 \rightarrow 1,
\]
using \(3\times 3\) convolutions, padding \(1\times 1\), strides \((1,2,2,2,1)\), and LeakyReLU activations after each layer. The final output is a **patch-level score map**, and intermediate feature maps are retained for **feature matching loss** [2602.22431].

The generator is trained with radar-aware reconstruction losses in addition to adversarial terms. The paper defines a high-frequency-weighted mel loss:
\[
\mathcal{L}_{G}^{\mathrm{mel}}
=
\lambda_{\mathrm{mel}}
\, \mathbb{E}
\Bigg[
\frac{1}{n_{\text{mels}}\, T}
\sum_{m,t}
w_m
\left|
\phi(x)_{m,t} - \phi(\hat{x})_{m,t}
\right|
\Bigg],
\]
with \(\lambda_{\mathrm{mel}} = 45.0\), where \(w_m = 5.0\) for mel bins above a cutoff \(f_c\), and \(w_m = 1\) otherwise. This weighting is intended to counter the tendency of the model to underfit upper-band reconstruction when the radar observation is informative mainly below \(1\) kHz [2602.22431].

A second reconstruction term is the multi-resolution STFT loss
\[
\mathcal{L}_{G}^{\mathrm{mrstft}}
=
\lambda_{\mathrm{stft}}
\, \mathrm{MRSTFT}(x,\hat{x}),
\]
with \(\lambda_{\mathrm{stft}} = 5.0\). The MR-STFT implementation uses FFT sizes \(\{256,512,1024\}\), hop sizes \(\{64,128,256\}\), window lengths \(N\), and spectral weights \((w_{\mathrm{sc}}, w_{\mathrm{logmag}}, w_{\mathrm{linmag}}) = (1.0,1.0,0.0)\) [2602.22431].

The training procedure is explicitly staged. In **Stage 1**, the generator is pretrained alone on synthetically clipped clean speech using
\[
\mathcal{L}_{G}^{\mathrm{pre}}
=
\mathcal{L}_{G}^{\mathrm{mel}}
+
\mathcal{L}_{G}^{\mathrm{mrstft}},
\]
with **no discriminators**. In **Stage 2**, the pretrained generator is fine-tuned adversarially on real radar-derived fused mel spectrograms using
\[
\mathcal{L}_{G}
=
\gamma \sum_{D \in \mathcal{D}} \mathcal{L}_{G,D}^{\mathrm{adv}}
+
\mathcal{L}_{G}^{\mathrm{fm}}
+
\mathcal{L}_{G}^{\mathrm{mel}}
+
\mathcal{L}_{G}^{\mathrm{mrstft}},
\qquad
\mathcal{L}_{D}
=
\sum_{D \in \mathcal{D}} \mathcal{L}_{D}^{\mathrm{adv}},
\]
where \(\mathcal{D}\) contains **MSD**, **MPD**, and **MMD**. The text does not report the numerical value of \(\gamma\) [2602.22431].

Optimization details are given separately for the three training components. The generator pretraining uses **AdamW**, \(\beta=(0.9,0.99)\), initial learning rate \(10^{-4}\), exponential scheduler \(\gamma=0.999\) each epoch, batch size \(16\), and **66k steps**. WVN is trained separately using **Adam**, learning rate \(10^{-3}\), batch size \(8\), gradient accumulation \(8\), and **30 epochs**. RAD-GAN fine-tuning then uses **AdamW**, \(\beta=(0.9,0.99)\), initial learning rate \(10^{-4}\), exponential scheduler \(\gamma=0.999\), batch size \(16\), and **100k steps**. The full model contains **\(87{,}000{,}536\) trainable parameters** [2602.22431].

## 4. Data regime and empirical performance

The experiments use the **RASE 2026 Challenge** paired radar–speech dataset collected with a **TI AWR2243BOOST mmWave FMCW radar**. The radar captures are obtained **through a glass wall**. **Task 1** contains **\(6{,}093\)** paired samples, with **\(5{,}334\)** for training and **\(759\)** for validation. **Task 2** contains **\(5{,}978\)** paired samples, with **\(5{,}229\)** for training and **\(749\)** for validation. The sampling rate is **8 kHz**, the average duration is **6.4 s**, and all samples are clipped to **4 s** segments for training. The total paired duration is about **42 h** [2602.22431].

The paper compares RAD-GAN against six baselines: **WaveVoiceNet (M0)**, **HiFi-GAN (M1)**, **DCCTN (M2)**, **AP-BWE (M3)**, **DiffWave (M4)**, and **CDiffuSE (M5)**. Evaluation uses **PESQ**, **ESTOI**, **CSMFCC**, and **DNSMOS**, with normalized task scores defined by
\[
\widetilde{\mathrm{PESQ}}_i = \frac{\mathrm{PESQ}_i - 1}{4.5 - 1},
\qquad
\widetilde{\mathrm{DNSMOS}}_i = \frac{\mathrm{DNSMOS}_i - 1}{5 - 1},
\]
\[
Task_i =
\frac{
\widetilde{\mathrm{PESQ}}_i +
\widetilde{\mathrm{DNSMOS}}_i +
\mathrm{CSMFCC}_i +
\mathrm{ESTOI}_i
}{4},
\qquad
\mathrm{Weighted\ score} = 0.4\,\mathrm{Task\ 1} + 0.6\,\mathrm{Task\ 2}.
\]
Task 2 is weighted more heavily because it is the harder setting [2602.22431].

RAD-GAN attains the best overall score among the reported systems. Its metrics are **PESQ \(1.310\)**, **ESTOI \(0.190\)**, **CSMFCC \(0.669\)**, **DNSMOS \(2.688\)**, **Task 1 score \(0.387\)**, **Task 2 score \(0.297\)**, and **Weighted score \(0.333\)**. The directly compared baselines obtain weighted scores of **0.260** for WaveVoiceNet, **0.288** for HiFi-GAN, **0.172** for DCCTN, **0.165** for AP-BWE, **0.106** for DiffWave, and **0.119** for CDiffuSE [2602.22431].

The ablation study is organized as **B0** through **B3**. **B0** is original HiFi-GAN and yields **Weighted score \(0.288\)**. **B1**, adding **MMD + MR-STFT**, yields **0.290**. **B2**, further adding **pretraining**, yields **0.312**. **B3**, further adding **WVN conditioning**, yields **0.333**. The paper interprets this as showing that **pretraining gives the largest single jump**, while the fusion-based auxiliary conditioning provides a further gain. It also notes that individual metrics do not improve monotonically, whereas the weighted score increases steadily across the ablation ladder [2602.22431].

Qualitatively, the paper reports that RAD-GAN reconstructs **clearer upper-band harmonics**, preserves a **silence region around 2.6–3.2 s with less leakage**, follows the **clean waveform envelope more closely**, and produces **sharper onsets/offsets and stronger peaks** than WVN in the harder Task 2 examples. This suggests that the model is not merely suppressing noise but performing structurally guided bandwidth extension from weak low-band cues [2602.22431].

## 5. Position within radar generative modeling

RAD-GAN belongs to a broader family of radar-domain adversarial models, but its combination of **speech reconstruction**, **dual conditioning**, and **radar-aware mel-domain discrimination** is distinct. Earlier work established several partial precedents. **SARGAN** learned recovery of missing spectral information in **UWB SAR** by combining a masked Fourier-domain content loss with an adversarial prior, but its sole test-time condition is the corrupted signal itself and it does not provide explicit dual conditioning [1812.04744]. **BRSR-OpGAN** performs blind restoration of raw complex **I/Q radar waveform segments** with a **dual domain loss in the temporal and spectral domains**, yet this is better understood as single-input conditional restoration with dual-domain supervision rather than a dual-conditioned design [2407.13949].

In automotive radar, **“Generative Adversarial Synthesis of Radar Point Cloud Scenes”** established an unconditional **PointNet++-based GAN** for full radar point cloud scene synthesis, with a global discriminator and six segment-wise discriminators encoding range-dependent density structure. That work is explicitly **unconditioned**, and its relevance to RAD-GAN lies in radar-aware adversarial priors rather than dual conditioning [2410.13526]. Likewise, **“High-Resolution Range-Doppler Imaging from One-Bit PMCW Radar via Generative Adversarial Networks”** addressed reconstruction of high-quality range–Doppler maps from **one-bit PMCW** measurements, but the described methods are a hybrid radar-processing-plus-GAN denoiser and an end-to-end learned RD reconstruction GAN rather than a named RAD-GAN system [2503.12841].

A concise comparison is as follows.

| Paper | Primary target | Conditioning character |
|---|---|---|
| RAD-GAN [2602.22431] | mmWave radar-to-speech reconstruction | Dual-conditioned through fused noisy mel and WVN mel |
| BRSR-OpGAN [2407.13949] | Blind restoration of raw radar I/Q segments | Conditional on corrupted waveform; dual-domain loss |
| Radar point-cloud GAN [2410.13526] | Automotive radar scene synthesis | Unconditional |
| One-bit PMCW RD GAN [2503.12841] | Range–Doppler reconstruction | Conditional reconstruction from degraded radar measurements |
| SARGAN [1812.04744] | Missing-spectrum UWB SAR recovery | Conditional on corrupted radar data |

This lineage shows that RAD-GAN should not be read as an isolated invention detached from prior radar GAN work. A plausible implication is that it inherits two well-established radar-GAN themes—**adversarial priors for ill-posed recovery** and **conditioning on degraded radar observations**—while shifting them into the radar-to-speech domain and making the conditioning explicitly dual through pre-generator fusion [2602.22431].

## 6. Limitations, interpretation, and future directions

Several limits are explicit. The system is built around a **very specific low-band radar regime** in which useful information is concentrated below **\(1\) kHz**. The dataset is relatively small compared with standard speech corpora, the method is tied to a **glass-wall** mmWave capture setup, and the full model is large at about **87M parameters**. The paper does **not** report latency, so real-time deployment remains open [2602.22431].

The authors identify two direct future directions: reporting and optimizing **real-time latency**, and **model compression through distillation** for edge inference. They also emphasize that the system was trained on a **limited dataset**, with **no pre-trained modules** for RAD-GAN itself and **no data augmentations**, while still outperforming the compared approaches for the specific task. That claim should be interpreted in the scope provided by the paper: the result concerns the reported **RASE 2026 Challenge** setting rather than a broad claim over all radar-conditioned speech reconstruction problems [2602.22431].

A final interpretive point concerns the meaning of **“radar-aware.”** In RAD-GAN, radar awareness does not come from an explicitly physics-embedded generator. The generator is the original HiFi-GAN. Radar awareness instead appears in the **problem formulation** and the **conditioning and loss design**: low-band mel conditioning with \(f_{\max}=1000\) Hz, high-frequency-weighted reconstruction loss, conservative residual fusion of noisy and WVN-derived mel features, and mel-domain adversarial supervision tailored to spectral errors that dominate under radar capture. This suggests a broader definition of radar-aware adversarial modeling: not necessarily a bespoke radar backbone, but a generative system whose conditioning pathway, discriminators, and optimization are explicitly matched to the information bottlenecks of the radar observation [2602.22431].

Source: https://www.emergentmind.com/topics/radar-aware-dual-conditioned-generative-adversarial-network-rad-gan