---
title: 'DWFormer: Speech SER & UDC Restoration Models'
url: https://www.emergentmind.com/topics/dwformer
type: topic
---

# DWFormer: Speech SER & UDC Restoration Models

Searching arXiv for DWFormer and closely related entries to ground the article in the relevant papers.
arXiv search query: "DWFormer"
DWFormer is an overloaded model name in arXiv literature. Most prominently, it denotes **Dynamic Window transFormer**, a speech emotion recognition architecture that uses attention-derived temporal importance to split utterances into variable-length windows and then alternates local and global modeling over those windows [2303.01694]. The same name is also used for a **Transformer-style under-display camera restoration network** whose core block replaces multi-head self-attention with depth-wise convolution and augmented channel attention in a U-Net-like hierarchy [2209.11455]. By contrast, the 2012 finite-temperature QCD paper that may appear in string-based retrieval under “DWFormer” is a **domain wall fermion** study and not a transformer or machine-learning model [1205.3526].

## 1. Nomenclature and disambiguation

The label “DWFormer” does not identify a single architecture family. In the material considered here, it refers to distinct systems in speech emotion recognition and image restoration, while some retrieval matches are unrelated.

| Usage | Domain | Defining description |
|---|---|---|
| Dynamic Window transFormer | Speech emotion recognition | Dynamically splits speech into windows using temporal importance |
| DWFormer | Under-display camera restoration | Uses depth-wise convolution and augmented channel attention |
| Domain wall fermion study | Finite-temperature QCD | Not a transformer or machine-learning model |

The speech model is explicitly introduced as **Dynamic Window transFormer (DWFormer)** for speech emotion recognition [2303.01694]. The image-restoration model is a separate **Transformer-style network named DWFormer** for under-display camera images [2209.11455]. The finite-temperature QCD study concerns **2+1-flavor QCD using Domain Wall Fermions (DWF) with the Dislocation Suppressing Determinant Ratio (DSDR)** and is unrelated to transformer architectures [1205.3526]. Likewise, the DAFT-based waveform design framework unifying OFDM, OCDM, and AFDM is **not specifically about DWFormer** [2406.02262].

## 2. Dynamic Window transFormer in speech emotion recognition

In speech emotion recognition, DWFormer is proposed to address a specific limitation of existing transformer-based SER systems: emotionally salient evidence is often concentrated in **local temporal regions** and appears at **different temporal scales** across utterances [2303.01694]. The motivation is that a vanilla transformer can model long-range dependencies, but its **global self-attention** may become noisy and diffuse, while prior local-window transformers typically rely on **fixed window sizes** that are too rigid for variable-duration emotional cues such as laughter, sighs, pauses, and intonation changes.

The speech pipeline begins with **audio feature extraction** using a pretrained **WavLM-Large** encoder. The output from the **12th transformer layer** is taken as the acoustic representation, producing a sequence
$$
x_0 \in \mathbb{R}^{T \times D},
$$
where \(T\) is the number of time tokens and \(D\) is the feature dimension [2303.01694]. This sequence is first passed through a vanilla transformer encoder layer, yielding hidden features \(x_{11}\) and attention weights
$$
W \in \mathbb{R}^{H \times T \times T},
$$
with \(H\) attention heads. Those attention matrices are reused by the **Importance Calculation (IC) module** to estimate token-level temporal salience.

The IC module converts attention into a token-importance vector through head averaging and softmax normalization. This importance signal is not only diagnostic; it directly governs the subsequent segmentation of the utterance into windows. The central architectural claim is therefore that **window boundaries should depend on temporal importance rather than on a fixed segmentation rule** [2303.01694].

This importance-driven formulation is meant to separate regions with **strong emotional correlation** from regions with **weak emotional correlation**. A plausible implication is that DWFormer treats temporal localization as part of the model’s representational problem, rather than as a preprocessing choice.

## 3. Dynamic windowing, local-global interaction, and benchmark behavior

Each DWFormer block contains two submodules: the **Dynamic Local Window Transformer (DLWT)** and the **Dynamic Global Window Transformer (DGWT)** [2303.01694]. In the DLWT, **dynamic window splitting (DWS)** uses the importance values from the IC module, with the threshold set to the **median** of the importance scores. Tokens above and below that threshold are grouped **chronologically** into windows, so the resulting windows are not equal in length. The implementation uses an **attention mask** \(M\) that permits attention only within the same window and blocks cross-window attention during the local stage.

Within each dynamic window, self-attention is applied locally. The model then distinguishes **strong emotional correlation windows** from **weak emotional correlation windows**. Features in weak windows are multiplied by a factor \(\lambda \le 1\), and the experiments use \(\lambda = 0.85\) [2303.01694]. This creates an explicit inductive bias that less informative temporal regions should contribute less strongly without being removed.

The DGWT restores utterance-level context after the local stage. Each window is compressed into a single representative token through an **importance-weighted sum**, producing
$$
wt \in \mathbb{R}^{(A+B) \times D},
$$
where \(A+B\) is the total number of dynamic windows [2303.01694]. A transformer encoder is then applied over the sequence of window tokens, allowing **cross-window interaction**. The window-level outputs are **upsampled** back to token resolution by copying the window token across the positions belonging to that window. The local and global streams are fused through
$$
x_{(a+1)1} = x_{a2} + x_{a3}.
$$
The model also updates the importance estimates across blocks, so the segmentation can be refined iteratively.

The full speech architecture is: pretrained WavLM features \(\rightarrow\) initial transformer encoder \(\rightarrow\) IC module \(\rightarrow\) a stack of \(N\) DWFormer blocks \(\rightarrow\) temporal average pooling \(\rightarrow\) MLP classifier [2303.01694]. The paper uses **3 DWFormer blocks for IEMOCAP** and **2 blocks for MELD**, with **8 attention heads**, **ReLU** activations, **batch size 32**, **SGD** for **120 epochs**, **cosine decay with cosine warm-up**, and **cross-entropy loss**.

On **IEMOCAP**, under the common 4-class setup with happy/excited merged, DWFormer reports **72.3% WA** and **73.9% UA**. On **MELD**, it reports **48.5% WF1** [2303.01694]. The paper also reports that these results exceed its vanilla-transformer and fixed-window-transformer baselines. The ablation study shows that removing **DLWT** or **DGWT** reduces performance on both datasets, which supports the interpretation that the model depends on both dynamic local partitioning and global re-interaction rather than on either component alone.

## 4. Fine-tuning DWFormer for continuous emotion scoring in IERP Challenge 2024

A later study uses DWFormer as a **pre-trained SER backbone** for **Track 1 of the IERP Challenge 2024**, where the task is to predict **continuous emotion intensity scores** rather than discrete classes [2508.11371]. The target space consists of **8 emotion categories**—sadness, happiness, relaxation, surprise, anger, fear, disgust, and neutrality—with a required score from **1 to 5** for each category, and evaluation is based on **RMSE**.

The system architecture is summarized as: **audio feature extraction**, **data augmentation by adding noise**, **Vanilla Transformer Encoder Block**, **Importance Calculation module**, **\(N\) stacked DWFormer blocks**, **classifier**, and **score fusion** [2508.11371]. In this setting, the paper describes the DWFormer block at a high level as consisting of **DLWT** and **DGWT**, where DLWT dynamically segments speech into windows of varying scales to capture **local emotional information**, and DGWT recalculates the significance of each window for **global information fusion**.

The pre-trained DWFormer used in this study is originally a **five-class model** for happiness, sadness, neutrality, anger, and fear, with a classifier composed of **three fully connected layers with ReLU activations** [2508.11371]. For the challenge, the final output layer is modified so that the last fully connected layer produces **8 scores** instead of 5. The adaptation strategy is deliberately conservative: **all layers are frozen**, and **only the final fully connected layer of the classifier is trained**. The paper explicitly states that the **Vanilla Transformer Block, IC Module, DWFormer Block, and Classifier are unchanged**, apart from the updated final layer.

Three feature types extracted from large self-supervised speech models are evaluated: **WavLM-Large**, **Chinese-HuBERT-Large**, and **Chinese-Wav2vec2-Large** [2508.11371]. Training uses **MSE loss**,
$$
\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2,
$$
with **batch size 4**, **initial learning rate 0.0001**, **Adam**, **weight decay \(1 \times 10^{-4}\)**, and **ReduceLROnPlateau**. The dataset contains **2608 audio features** for training/validation, **480 audio features** for testing, **163 subjects**, and **16 emotion-inducing videos**.

Noise augmentation uses the **noise subset of the MUSAN dataset**. Noise is truncated or repeated to match the encoded-feature length and is added probabilistically. The tested probabilities are **0.3**, **0.5**, and **0.8**, with **0.3** reported as the best setting overall; **0.8** degrades performance [2508.11371]. Among single-feature models, the reported test RMSEs are **1.8320** for WavLM-Large, **1.9016** for Chinese-HuBERT-Large, and **1.7905** for Chinese-Wav2vec2-Large.

The paper then applies **score fusion** across feature models and across different parameter files of the same feature model. The fusion rules are **simple averaging**, **weighted averaging**, and **maximum value**:
$$
S_{\text{ave}} = \frac{1}{n}\sum_{i=1}^{n} S_i,
$$
$$
S_{\text{weighted}} = \sum_{i=1}^{n} w_i S_i.
$$
For multi-feature fusion, the reported RMSEs are **1.7867** for average, **1.8078** for weighted average, and **1.7849** for max. For fusion across different parameter files using **Chinese-Wav2vec2-Large** only, the reported RMSEs are **1.77157** for average, **1.77192** for weighted average, and **1.77042** for max [2508.11371]. Because of the submission limit, the official leaderboard submission used **average fusion** with **RMSE 1.77157**, which achieved **first place**, while the paper identifies **1.77042** from **maximum value fusion** as the best internal result.

## 5. DWFormer as an under-display camera restoration network

In under-display camera restoration, DWFormer is a different model with a different objective. The problem is that under-display cameras capture images degraded by **brightness attenuation**, **blur**, and **noise corruption**, summarized in the model
$$
y = (\gamma \cdot x)\otimes k + n,
$$
where \(x\) is the clean image, \(y\) is the degraded image, \(\gamma\) is channel-dependent brightness attenuation, \(k\) is the blur kernel, and \(n\) is zero-mean signal-dependent noise [2209.11455]. The paper introduces **MPGNet** as a modular GAN-based degradation simulator and **DWFormer** as the restoration network trained on real data alone or on a mixture of real and synthetically degraded data.

The restoration DWFormer is described as a **U-Net-like hierarchical network with skip connections** [2209.11455]. A \(3 \times 3\) convolution first extracts features
$$
F_0 \in \mathbb{R}^{h \times w \times c}.
$$
The network then proceeds through **five stages** of stacked **DWFormer Blocks (DWB)** with convolutional downsampling and sub-pixel upsampling. A final \(3 \times 3\) convolution predicts a residual image
$$
r \in \mathbb{R}^{h \times w \times 3},
$$
and the restored output is
$$
\hat{x} = y + r.
$$

The DWB is a modified Transformer block defined by
$$
\begin{aligned}
\hat{z}^l &= H_{P_2}\big(H_{ACA}(\delta(H_D(H_{P_1}(\mathrm{BN}(z^l)))))\big) + z^l, \\
z^{l+1} &= \mathrm{MLP}(\mathrm{BN}(\hat{z}^l)) + \hat{z}^l.
\end{aligned}
$$
Here \(H_{P_1}\) and \(H_{P_2}\) are pixel-wise \(1\times1\) convolutions, \(H_D\) is **depth-wise convolution**, \(\delta\) is **ReLU**, \(\mathrm{BN}\) is **BatchNorm**, and \(H_{ACA}\) is the **augmented channel attention** module [2209.11455]. The paper also states that **FrozenBN** is used to avoid train-test inconsistency.

The architectural substitution is explicit: the block is analogous to a Transformer block, but **token mixing is replaced by depth-wise convolution plus ACA**, and the feed-forward part remains MLP-like [2209.11455]. The rationale is task-specific. The paper argues that **blur and noise** are mainly local phenomena, so local spatial aggregation is well served by depth-wise convolution, while **brightness attenuation** behaves as a global, channel-dependent effect, motivating ACA for global information aggregation. ACA differs from standard squeeze-excitation by learning spatial weights before pooling into a global channel descriptor.

DWFormer itself is trained with a simple supervised objective,
$$
\mathcal{L}_{DWFormer} = \| \hat{x} - x \|_1,
$$
while the synthetic data used to augment training are produced by MPGNet under a separate adversarial-plus-supervised loss [2209.11455]. The restoration paper gives several implementation details: **five stages**, DWB counts of **\(\{8,8,8,6,6\}\)**, **Adam** with \(\beta_1=0.9\) and \(\beta_2=0.999\), **\(2\times10^5\)** training iterations, **initial learning rate \(10^{-4}\)**, cosine annealing to **\(10^{-6}\)**, **batch size 64**, and augmentation by **random flips and rotations**.

## 6. Empirical results, ablations, and recurring misconceptions

On the **UDC benchmark**, the restoration DWFormer reports **33.21 dB PSNR / 0.960 SSIM** on **P-OLED** and **38.96 dB PSNR / 0.984 SSIM** on **T-OLED** [2209.11455]. The paper states that this surpasses the previous state of the art by **+0.22 dB** on P-OLED and **+0.12 dB** on T-OLED. It also reports **1.21M parameters** and **13.46G MACs**, emphasizing that the model is relatively small. When trained with both **MPGNet-generated and real data**, performance increases to **34.22 dB** on P-OLED and **39.55 dB** on T-OLED, corresponding to gains of **+1.01 dB** and **+0.59 dB** over training on real data only [2209.11455].

The ablations isolate the contribution of the restoration model’s components. Removing **ACA** reduces PSNR by about **0.59 dB on P-OLED** and **0.51 dB on T-OLED**. Replacing ACA with **SE** remains worse than full DWFormer by **0.21 dB** on P-OLED and **0.24 dB** on T-OLED. Replacing the DWB with a **Swin Transformer block** reduces performance by **0.14 dB** on P-OLED and **0.12 dB** on T-OLED while increasing MACs. Replacing **BN** with **LN** yields smaller but still negative changes of **0.10 dB** and **0.06 dB** [2209.11455]. These results support the paper’s claim that the combination of depth-wise convolution and ACA is better aligned with low-level UDC restoration than attention-heavy alternatives.

A recurring misconception is to treat all occurrences of “DWFormer” as referring to one architecture. The evidence does not support that interpretation. In speech emotion recognition, DWFormer is specifically a **dynamic-window transformer** built around **DLWT** and **DGWT** [2303.01694]. In under-display camera restoration, DWFormer is a **Transformer-style restoration network** centered on **depth-wise convolution** and **augmented channel attention** [2209.11455]. The finite-temperature QCD study retrieved under similar strings is about **Domain Wall Fermions** and the **finite temperature transition with 2+1 flavors using DWF with DSDR**, not about machine learning [1205.3526]. This suggests that “DWFormer” functions as a reused label rather than as a single coherent research lineage.

Taken together, the speech and image-restoration usages share only a high-level naming convention. Their internal operators, objectives, datasets, and evaluation protocols are different: **WA/UA/WF1** for classification-style SER benchmarks, **RMSE** for continuous emotion scoring, and **PSNR/SSIM** for image restoration [2303.01694][2508.11371][2209.11455]. The commonality lies mainly in the attempt to replace undifferentiated global processing with structures better matched to the signal: **importance-driven variable windows** for emotional speech and **local spatial mixing plus global channel aggregation** for under-display camera images.

Source: https://www.emergentmind.com/topics/dwformer