PatchDSU: Patchwise Uncertainty in Keyword Spotting

• PatchDSU is a patchwise uncertainty-modeling method that improves keyword spotting robustness by simulating local spectral perturbations during training.
• It partitions spectrogram feature maps into non-overlapping patches to compute independent Gaussian statistics, addressing the sparsity and locality of speech signals.
• Empirical results show that PatchDSU enhances performance across in-domain, noisy, and cross-dataset evaluations when compared to baseline and global DSU methods.

Searching arXiv for PatchDSU and related DSU / keyword spotting papers. arxiv_search(query="PatchDSU uncertainty modeling out of distribution generalization keyword spotting", max_results=5) arxiv_search(query="(Chernyak et al., 5 Aug 2025)", max_results=10) PatchDSU is a training-time uncertainty-modeling method for out-of-distribution generalization in keyword spotting. It extends Domain Shifts with Uncertainty (DSU) from vision to speech by replacing global feature-statistic perturbation with patchwise perturbation on spectro-temporal feature maps. In a CNN layer with input $x \in \mathbb{R}^{B \times C \times H \times W}$, PatchDSU partitions the $(H,W)$ plane into non-overlapping patches, estimates means and variances for each patch, samples shifted statistics from patch-specific Gaussian models, and re-normalizes the features accordingly. The method is motivated by the observation that speech spectrograms are sparse and locally structured, so a single global Gaussian can yield skewed estimates and ineffective sampling, whereas local patch statistics better reflect speech variability (Chernyak et al., 5 Aug 2025).

1. Motivation in out-of-distribution keyword spotting

Out-of-distribution shifts are common in speech systems because recording devices change, background noise appears, and speaker accents differ. Standard deep models normally assume that train and test are drawn from the same distribution and can exhibit large accuracy drops when that assumption fails. PatchDSU is designed for this setting, specifically for keyword spotting under dataset shift, noise corruption, and cross-corpus evaluation (Chernyak et al., 5 Aug 2025).

The immediate precursor is DSU, introduced for vision by Li et al. at ICLR 2022. DSU improves out-of-domain generalization by modeling each layer’s feature statistics, specifically means and variances, as multivariate Gaussians and then sampling new shifted statistics during training. The sampled statistics are used to re-normalize features, thereby simulating domain shifts.

Direct transfer of that mechanism to speech is problematic. A spectrogram is a temporal representation of frequency over time, not a static visual image, and its sparsity is structurally important. The reported analysis shows that in keyword spectrograms the energy is concentrated in the lower half of the frequency axis, and that Google Speech Commands, LibriSpeech, and TED-LIUM exhibit substantially different average patterns. Under those conditions, global feature statistics can be dominated by large zero-regions, so a single global Gaussian cannot capture local variability. PatchDSU addresses this by fitting independent Gaussian models per patch rather than over the whole feature map. This suggests that PatchDSU is not merely a spatial refinement of DSU, but a response to the mismatch between speech spectrogram structure and image-style global statistic modeling.

2. Patchwise uncertainty modeling

PatchDSU operates on the $\ell^{\text{th}}$ CNN layer input

$x \in \mathbb{R}^{B \times C \times H \times W},$

where $B$ is the batch size, $C$ the number of channels, and $H,W$ the spatial dimensions. The feature map is split along the $(H,W)$ axes into $K$ non-overlapping patches of size $(H_k \times W_k)$. With integers $(H,W)$0,

$(H,W)$1

and $(H,W)$2 (Chernyak et al., 5 Aug 2025).

For each example $(H,W)$3, channel $(H,W)$4, and patch $(H,W)$5, PatchDSU computes the patch mean and variance:

$(H,W)$6

$(H,W)$7

As in DSU, each patch-statistic vector $(H,W)$8 is treated as drawn from a Gaussian with diagonal covariance $(H,W)$9, whose diagonal entries are estimated from the mini-batch:

$\ell^{\text{th}}$0

Similarly, for the standard-deviation vector $\ell^{\text{th}}$1,

$\ell^{\text{th}}$2

During training, with probability $\ell^{\text{th}}$3, PatchDSU replaces each patch’s statistics with samples obtained by the reparameterization trick:

$\ell^{\text{th}}$4

$\ell^{\text{th}}$5

where $\ell^{\text{th}}$6. It then applies instance normalization per patch:

$\ell^{\text{th}}$7

Here $\ell^{\text{th}}$8 are the re-normalized scaling-and-shifting parameters, and the sampled statistics themselves become $\ell^{\text{th}}$9 and $x \in \mathbb{R}^{B \times C \times H \times W},$0. With probability $x \in \mathbb{R}^{B \times C \times H \times W},$1, the input is left unmodified.

The method is applied layer-wise: a PatchDSU module is inserted immediately prior to every convolutional layer, patch means and variances are recomputed at that layer’s feature-map resolution, new statistics are sampled, and normalization is repeated. A plausible implication is that uncertainty is injected not only at the acoustic front end but throughout the learned hierarchy, allowing perturbations to track progressively more abstract feature representations.

3. Architectural integration and optimization

The backbone used for PatchDSU evaluation is ResNet-15, following Tang and Lin (ICASSP 2018). The model contains 13 convolutional layers with $x \in \mathbb{R}^{B \times C \times H \times W},$2 kernels and dilations, organized into 6 residual blocks, with each convolution followed by ReLU and BatchNorm. The front end uses 40-dim MFCCs with FFT size 512, window 480, and hop 160. Patch-level DSU is applied immediately before each $x \in \mathbb{R}^{B \times C \times H \times W},$3 convolution’s input (Chernyak et al., 5 Aug 2025).

Two patch configurations were validated on the in-domain validation set: $x \in \mathbb{R}^{B \times C \times H \times W},$4 and $x \in \mathbb{R}^{B \times C \times H \times W},$5. The application probability $x \in \mathbb{R}^{B \times C \times H \times W},$6 was tuned per dataset, with examples in the range $x \in \mathbb{R}^{B \times C \times H \times W},$7–$x \in \mathbb{R}^{B \times C \times H \times W},$8. Optimization used batch size 100 and SGD with momentum $x \in \mathbb{R}^{B \times C \times H \times W},$9, weight decay $B$0, and initial learning rate $B$1 with linear warm-up followed by cosine decay. Training lasted 200 epochs for Speech Commands and TED-LIUM, and 300 epochs with early stopping for LibriSpeech.

These implementation details matter because the paper’s reported gains are tied to a specific integration strategy: PatchDSU is not an auxiliary loss or a data-preprocessing transform, but an internal feature-space perturbation module inserted before every convolution. This suggests that its behavior depends jointly on patch geometry, layer resolution, and the probability with which perturbation is applied.

4. Evaluation protocol

The evaluation covers three datasets and several shift regimes: in-domain testing, time augmentation, additive noise, and cross-dataset transfer (Chernyak et al., 5 Aug 2025).

Dataset	Class setting	Notes
Google Speech Commands V2	12-class setting: Down, Up, Left, Right, On, Off, Yes, No, Go, Stop + Unknown + Silence	Used for clean, noise, and cross-dataset tests
LibriSpeech	11 classes: same ten keywords + Unknown, unbalanced	train-100+train-360, test on train-other-500
TED-LIUM	8 classes: eight keywords, lower data volume	train/val/test splits extracted via wav2vec2 alignment

Noise robustness was evaluated by augmenting test sets with White Gaussian Noise and MUSAN music noise at SNRs from $B$2 to $B$3. Out-of-domain generalization was evaluated via cross-dataset testing, including train-on-Libri/test-on-Speech-Commands-and-TED, train-on-Speech-Commands/test-on-Libri-and-TED, and train-on-TED/test-on-Speech-Commands-and-Libri. The reported metrics are average F1-score over all classes and F1 over the keyword classes only, with Unknown excluded from the latter.

The protocol is notable because it distinguishes between several types of distribution shift. Additive noise probes corruption robustness, time augmentation probes invariance to temporal displacement, and cross-dataset testing probes corpus-level mismatch involving speaker populations, recording conditions, and label-set structure.

5. Empirical results across clean, noisy, and cross-dataset settings

On clean in-domain test sets, PatchDSU and DSU generally outperform the baseline and Freq-MixStyle, with the largest gains on LibriSpeech and TED-LIUM (Chernyak et al., 5 Aug 2025). For Google Speech Commands, results are close: the baseline reaches 98.09 F1, DSU 98.42, and PatchDSU with $B$4 patches 98.43. For LibriSpeech, the baseline reaches 94.62, DSU 95.19, and PatchDSU $B$5 95.91; under time-shift augmentation, the corresponding LibriSpeech values are 93.56, 93.70, and 94.65. For TED-LIUM, the baseline yields 74.62 clean and 78.80 with augmentation, whereas DSU yields 76.31 and 81.27, and PatchDSU $B$6 yields 76.33 and 82.07. Freq-MixStyle degrades performance on TED-LIUM.

Under additive noise, the relative ranking depends on dataset and SNR. On Speech Commands, at high SNR, DSU and PatchDSU are approximately tied and both outperform the baseline; at low SNR, specifically $B$7 and $B$8, PatchDSU $B$9 exceeds DSU by approximately 1 percentage point, and both substantially outperform Freq-MixStyle. On LibriSpeech, under White Gaussian Noise at $C$0, only DSU salvages performance, while under MUSAN noise PatchDSU $C$1 is strongest at mid-SNR. On TED-LIUM, the results vary and exhibit high standard deviation across runs; PatchDSU $C$2 often leads at higher SNR, while PatchDSU $C$3 and DSU trade places at low SNR. MUSAN is reported as slightly easier than White Gaussian Noise.

The ablation on application probability $C$4 shows marked dataset dependence. On Speech Commands, any $C$5 helps, while $C$6 may over-perturb; PatchDSU $C$7 is the most stable. On LibriSpeech, PatchDSU $C$8 outperforms across all tested $C$9, whereas DSU under-performs for $H,W$0 under time-shift. On TED-LIUM, PatchDSU $H,W$1 degrades approximately linearly as $H,W$2 grows, PatchDSU $H,W$3 favors low $H,W$4, and DSU is best at approximately $H,W$5 on clean data but prefers low $H,W$6 under augmentation.

Cross-dataset evaluation reveals both strong transfer cases and notable exceptions. From LibriSpeech to TED-LIUM, PatchDSU $H,W$7 leads overall with 66.61 F1 versus approximately 62.7 for DSU, though DSU is best on time-shift. From LibriSpeech to Speech Commands, Freq-MixStyle exceeds PatchDSU and DSU by approximately 0.5 percentage points on keywords. From Speech Commands to LibriSpeech, DSU and PatchDSU are essentially tied at approximately 80.7 F1, both roughly 10 percentage points above the baseline. From Speech Commands to TED-LIUM, the baseline outperforms all augmentation methods, described as an unusual case in which data sparsity in TED hurts perturbations. From TED-LIUM to LibriSpeech, PatchDSU $H,W$8 reaches 78.4 versus 73.0 for the baseline. From TED-LIUM to Speech Commands, Freq-MixStyle again excels despite being worst in-domain on TED-LIUM.

6. Interpretation, limitations, and relation to adjacent methods

The central interpretation is that patchwise operation avoids the skew that arises when one Gaussian is fit to an entire sparse speech input. Local energy patterns, including formant tracks in the low-frequency band, are preserved more effectively, and the sampled perturbations better reflect plausible speech variability (Chernyak et al., 5 Aug 2025). This is consistent with the observed pattern that PatchDSU is more consistent than global DSU across the evaluated scenarios, especially on LibriSpeech and TED-LIUM.

At the same time, the evaluation does not support a universal dominance claim. DSU sometimes outperforms PatchDSU under pure White Gaussian Noise on Speech Commands and at mid-to-low SNR on LibriSpeech, presumably because global shifts are more aligned with those noise distributions. Freq-MixStyle can dominate in specific cross-dataset matches, notably LibriSpeech to Speech Commands and TED-LIUM to Speech Commands, yet it fails catastrophically at low SNR. These exceptions are important because they show that the effectiveness of uncertainty modeling depends on the structure of the target shift: local spectro-temporal perturbation, global statistic perturbation, and frequency-only perturbation each appear to capture different mismatch regimes.

The main reported limitations are operational rather than conceptual. PatchDSU requires tuning of the application probability $H,W$9 and the patch sizes per dataset. The paper identifies adaptive selection of $(H,W)$0 or of the patch grid as a direction that would improve usability. It also proposes overlapping or multi-scale patches to capture richer statistical structure across fine and coarse time scales. The surprising success of Freq-MixStyle in some cross-dataset tests further suggests that hybrid methods combining frequency-only and full patch perturbations could yield additional gains.

Within the reported experiments, PatchDSU is presented as a localized uncertainty-modeling extension of DSU for speech, with consistent improvements in in-domain generalization and strong robustness across a wide range of out-of-distribution scenarios and noise conditions. Its empirical profile is therefore best understood not as uniformly superior to all alternatives, but as a method whose patchwise statistics align particularly well with the sparsity and locality of keyword spectrogram representations (Chernyak et al., 5 Aug 2025).