Residual Fusion Gate in RAD-GAN

• Residual Fusion Gate (RFG) is a fusion module that combines noisy radar mel-spectrograms with enhanced residual corrections for improved speech reconstruction.
• It employs a 1x1 convolution and a trainable sigmoid gate to merge dual conditioning streams while preserving the original 80-bin input dimensionality.
• Its integration within RAD-GAN yields better perceptual metrics such as increased PESQ and ESTOI scores by adaptively balancing reliable baseline data with auxiliary enhancements.

Residual Fusion Gate (RFG) denotes, in its explicit usage, the fusion module introduced within RAD-GAN for mmWave radar speech reconstruction. It fuses two mel-spectrogram conditioning streams during the fine-tuning radar stage: the mel spectrogram from the noisy radar waveform, $\mathbf{M}_n$, and the mel spectrogram from the WaveVoiceNet output, $\mathbf{M}_w$. The module treats $\mathbf{M}_n$ as a baseline “carry” path, treats $\mathbf{M}_w-\mathbf{M}_n$ as a residual correction, and applies a trainable, bounded gate so that the generator continues to receive a single 80-bin mel input rather than a changed conditioning dimensionality (Karani et al., 25 Feb 2026).

1. Conceptual definition and motivation

Within RAD-GAN, the RFG is the dedicated module that fuses two mel-spectrogram conditioning streams during the fine-tuning radar stage. Both $\mathbf{M}_n$ and $\mathbf{M}_w$ lie in $\mathbb{R}^{B \times F \times T}$, with $F=80$. The core problem is that neither stream is fully reliable: the noisy radar mel retains more direct physical correspondence to the radar measurement, but is extremely noisy and band-limited; the WVN mel is a learned magnitude-domain enhancer, but can hallucinate or introduce artifacts, and its phase quality is not reliable enough to be used as a standalone vocoder input (Karani et al., 25 Feb 2026).

The module is motivated by the inadequacy of simple fusion rules such as channel concatenation $[\mathbf{M}_n;\mathbf{M}_w]$ or averaging. In the RAD-GAN setting, such rules would expose the HiFi-GAN generator to a new input dimensionality and statistics after pretraining, provide no mechanism to suppress WVN where it is wrong, and fail to exploit the residual nature of the correction $\mathbf{M}_w-\mathbf{M}_n$. RFG therefore formulates dual conditioning not as direct multi-stream generation, but as a residual-gated correction on the noisy mel.

This formulation is explicitly inspired by highway networks, residual learning, and gated fusion mechanisms. A plausible implication is that RFG belongs to a broader design pattern in which one representation is treated as conservative and trustworthy, while an auxiliary representation is used only insofar as a gate allows it.

2. Formal definition and architecture

The RFG is defined by

$\mathbf{M}_w$0

where $\mathbf{M}_w$1 is channel-wise concatenation along the mel-channel dimension, $\mathbf{M}_w$2 maps $\mathbf{M}_w$3, $\mathbf{M}_w$4 is the sigmoid, $\mathbf{M}_w$5 is a learnable scalar logit, and $\mathbf{M}_w$6 is elementwise multiplication (Karani et al., 25 Feb 2026).

The interpretation of each term is fixed by the RAD-GAN design. $\mathbf{M}_w$7 is the “carry” path; $\mathbf{M}_w$8 is the residual correction proposed by WVN at each time-frequency bin; $\mathbf{M}_w$9 is a local per-bin mask; $\mathbf{M}_n$0 is a global confidence scaling of WVN corrections, shared across all bins; and $\mathbf{M}_n$1 is the final fused mel fed to the generator.

Architecturally, the gate uses a pointwise Conv1D with $\mathbf{M}_n$2, that is, $\mathbf{M}_n$3, operating frame-wise and learning cross-frequency mixing across mel bins without temporal smoothing. The convolution input has shape $\mathbf{M}_n$4, and the output has shape $\mathbf{M}_n$5. This means that the gate performs cross-frequency mixing within each time frame but does not span across time.

Initialization is deliberately conservative. The conv bias and the scalar $\mathbf{M}_n$6 are initialized to $\mathbf{M}_n$7, so $\mathbf{M}_n$8. At initialization, the overall correction factor $\mathbf{M}_n$9, making $\mathbf{M}_w-\mathbf{M}_n$0. The stated purpose is that RFG starts as almost identity on noisy mel and learns to trust WVN gradually.

The layer-wise computation is correspondingly simple: compute the residual $\mathbf{M}_w-\mathbf{M}_n$1; concatenate $\mathbf{M}_w-\mathbf{M}_n$2; apply the pointwise Conv1D; obtain the local gate $\mathbf{M}_w-\mathbf{M}_n$3 via sigmoid; compute the global gate $\mathbf{M}_w-\mathbf{M}_n$4; and form the fused mel $\mathbf{M}_w-\mathbf{M}_n$5.

3. Position in the RAD-GAN pipeline

RAD-GAN uses a two-stage training pipeline. In Phase 1, the model is pretrained on synthetically band-limited clean speech in the $\mathbf{M}_w-\mathbf{M}_n$6–$\mathbf{M}_w-\mathbf{M}_n$7 kHz range, conditioned only on a single 80-bin mel spectrogram derived from that band-limited audio. The generator at this stage is a plain HiFi-GAN generator, with no RFG and no dual conditioning (Karani et al., 25 Feb 2026).

RFG appears only in Phase 2, the fine-tuning stage on real radar data. Here, the radar waveform is processed along two branches: direct radar to mel, producing $\mathbf{M}_w-\mathbf{M}_n$8, and radar to WVN to waveform to mel, producing $\mathbf{M}_w-\mathbf{M}_n$9. RFG fuses these into $\mathbf{M}_n$0, and generator conditioning in Phase 2 is exactly $\mathbf{M}_n$1, still an 80-bin mel. The pretrained generator architecture and input dimensionality therefore remain unchanged.

This placement is essential to the meaning of the module. RFG is the only fusion module in RAD-GAN, and it is placed before the HiFi-GAN generator in Phase 2. It acts as a front-end conditioning fusion layer, not as a skip-connection internal to the generator. A common misunderstanding is to treat RFG as a multi-stream generator architecture; the RAD-GAN design instead realizes dual conditioning as

$\mathbf{M}_n$2

The practical significance of this arrangement is explicit in the source description: the generator continues to reuse the pretrained low-to-high frequency mapping learned in Phase 1, while Phase 2 introduces dual conditioning only through the fused mel input.

4. Optimization, learning behavior, and empirical evidence

RFG changes the generator’s conditioning distribution, but the loss formulations remain identical to HiFi-GAN-style objectives. Let $\mathbf{M}_n$3 be the clean target waveform and $\mathbf{M}_n$4 the generated waveform. The discriminator set is $\mathbf{M}_n$5. Phase 1 uses only mel loss and MR-STFT loss, while Phase 2 uses the full objective with adversarial, feature-matching, mel, and MR-STFT terms. The mel loss uses $\mathbf{M}_n$6, $\mathbf{M}_n$7 for mel bins above cutoff $\mathbf{M}_n$8 and $\mathbf{M}_n$9 otherwise, with $\mathbf{M}_w$0; the MR-STFT term uses $\mathbf{M}_w$1. There is no explicit regularization term or loss that directly involves $\mathbf{M}_w$2, $\mathbf{M}_w$3, or $\mathbf{M}_w$4; the gate is learned end-to-end solely through the generator loss on $\mathbf{M}_w$5 (Karani et al., 25 Feb 2026).

Although the paper does not include an explicit RFG-only ablation, it identifies “WVN conditioning” as the ablation step realized via RFG. The progression is B0: original HiFi-GAN; B1: B0 + MMD + MR-STFT; B2: B1 + pretraining; B3: B2 + WVN conditioning (via RFG).

Metric	B2	B3
PESQ	1.286	1.310
ESTOI	0.179	0.190
CS (MFCC cosine sim.)	0.621	0.669
DNSMOS	2.639	2.688
W (weighted)	0.312	0.333

The reported improvements from B2 to B3 are: PESQ $\mathbf{M}_w$6, ESTOI $\mathbf{M}_w$7, MFCC cosine similarity $\mathbf{M}_w$8, DNSMOS $\mathbf{M}_w$9, and weighted score $\mathbb{R}^{B \times F \times T}$0 $\mathbb{R}^{B \times F \times T}$1 (Karani et al., 25 Feb 2026).

Qualitatively, on Task 2, which is described as more challenging and involving a secondary surface with lower SNR, RAD-GAN with RFG reconstructs clearer upper-band harmonics than WVN alone, preserves silence regions with less leakage, and yields a waveform envelope that more closely tracks the clean reference, with sharper transients. The source text attributes this to the fact that WVN is already strong at magnitude-domain enhancement but weaker in phase, whereas RFG-fused conditioning allows HiFi-GAN to correct and refine the WVN output while preserving low-frequency cues from the radar mel.

5. Relation to adjacent residual-gated fusion mechanisms

The term “Residual Fusion Gate” is not standardized across the literature. The explicit name appears in RAD-GAN, but several other architectures are described as structurally or functionally close. CRFN for audio-visual navigation introduces bidirectional residual interactions between audio and visual streams via a shared interaction vector and learnable scalar gates $\mathbb{R}^{B \times F \times T}$2 and $\mathbb{R}^{B \times F \times T}$3; its Fusion Controller can be interpreted as an RFG-type mechanism, although the paper does not use that term (Wang et al., 11 Jan 2026). Review Residuals in transformers define $\mathbb{R}^{B \times F \times T}$4 with $\mathbb{R}^{B \times F \times T}$5, which is close in spirit because the residual update is gated as a function of both the current state and the proposed update (Kramer, 30 Jun 2026).

In vision, related patterns also recur. RFBNet uses a Residual Fusion Block whose Gated Fusion Unit employs input and output gates to regulate information flow between RGB, depth, and an interaction stream; DRFN uses a dynamic residual feature fusion module in which high-dimensional features are added to gated fusion features to preserve category semantic information flow; and RFN-Nest uses residual fusion networks at multiple scales to replace fixed fusion rules, even though it does not define an explicit gate variable $\mathbb{R}^{B \times F \times T}$6.

These comparisons clarify two points. First, residual fusion typically preserves a base path and applies a controlled correction rather than replacing the original feature entirely. Second, “gate” can mean different capacities: a per-bin mask plus a global scalar in RAD-GAN, learnable scalars in CRFN, spatial gates in RFBNet, channel-wise attention in DRFN, or implicit gating through residual CNNs in RFN-Nest.

A further distinction concerns architectural scope. In RAD-GAN, the RFG modulates the input conditioning before generation; in Review Residuals, gating acts on layer updates within the residual stream; in CRFN and RFBNet, fusion is bidirectional across modalities; and in DRFN it governs decoder-side multi-scale fusion. This suggests that “Residual Fusion Gate” is best understood as a family resemblance centered on residual correction plus controlled fusion, not as a single universally fixed blueprint.

6. Limitations, misconceptions, and broader applicability

The RAD-GAN paper does not list explicit limitations of RFG, but several implications follow from its formulation. RFG depends on having two reasonably aligned and complementary mel streams. If the WVN residual $\mathbb{R}^{B \times F \times T}$7 is mostly noise or misaligned, the gate must learn to suppress it; in extreme cases, the fused mel may revert essentially to $\mathbb{R}^{B \times F \times T}$8, providing little benefit. Only one global scalar, $\mathbb{R}^{B \times F \times T}$9, is used to scale WVN corrections, so more fine-grained global control is not modeled. The module also operates purely at the mel-magnitude level; any mismatches in phase information must be handled by the vocoder implicitly (Karani et al., 25 Feb 2026).

Several misconceptions are directly addressed by the design. RFG is not an attention mechanism over time or frequency; it uses a simple $F=80$0 convolution and sigmoid. It is not a feature-wise modulation of generator activations in the FiLM sense; it modulates the input conditioning. It is not used in RAD-GAN pretraining; it appears only in Phase 2. Nor is it a generic concatenation layer, because its central operation is the residual correction $F=80$1 rather than absolute fusion.

The generalization claims in the source are broader. The design is described as modality-agnostic: any setting with a baseline feature and an auxiliary “enhanced” or “predicted” feature can use the same residual-gated formulation. The paper lists potential applications in audio, including fusion of multiple enhancement models such as denoiser plus bandwidth extension; in vision, including fusion of raw and enhanced images or different sensor modalities; and in multi-conditional generative models where one conditional branch is noisy but trustworthy and another is strong but potentially hallucinating.

Taken together, these properties locate RFG at a specific point in the design space of fusion modules: it is deliberately minimal, preserves input dimensionality, biases early training toward the conservative stream, and learns end-to-end whether, where, and how strongly an auxiliary residual should modify a baseline conditioning signal.