DWFormer: Speech SER & UDC Restoration Models

• DWFormer is a term that denotes multiple architectures, notably a dynamic window transformer for speech emotion recognition and a transformer-style network for under-display camera restoration.
• In speech emotion recognition, it dynamically segments audio using an importance calculation module to separate emotionally salient regions and merge local-global information.
• In image restoration, DWFormer employs a U-Net-like design that replaces global self-attention with depth-wise convolution and augmented channel attention to enhance PSNR and SSIM.

Searching arXiv for DWFormer and closely related entries to ground the article in the relevant papers. arXiv search query: "DWFormer" DWFormer is an overloaded model name in arXiv literature. Most prominently, it denotes Dynamic Window transFormer, a speech emotion recognition architecture that uses attention-derived temporal importance to split utterances into variable-length windows and then alternates local and global modeling over those windows (Chen et al., 2023). The same name is also used for a Transformer-style under-display camera restoration network whose core block replaces multi-head self-attention with depth-wise convolution and augmented channel attention in a U-Net-like hierarchy (Zhou et al., 2022). By contrast, the 2012 finite-temperature QCD paper that may appear in string-based retrieval under “DWFormer” is a domain wall fermion study and not a transformer or machine-learning model (Cheng, 2012).

1. Nomenclature and disambiguation

The label “DWFormer” does not identify a single architecture family. In the material considered here, it refers to distinct systems in speech emotion recognition and image restoration, while some retrieval matches are unrelated.

Usage	Domain	Defining description
Dynamic Window transFormer	Speech emotion recognition	Dynamically splits speech into windows using temporal importance
DWFormer	Under-display camera restoration	Uses depth-wise convolution and augmented channel attention
Domain wall fermion study	Finite-temperature QCD	Not a transformer or machine-learning model

The speech model is explicitly introduced as Dynamic Window transFormer (DWFormer) for speech emotion recognition (Chen et al., 2023). The image-restoration model is a separate Transformer-style network named DWFormer for under-display camera images (Zhou et al., 2022). The finite-temperature QCD study concerns 2+1-flavor QCD using Domain Wall Fermions (DWF) with the Dislocation Suppressing Determinant Ratio (DSDR) and is unrelated to transformer architectures (Cheng, 2012). Likewise, the DAFT-based waveform design framework unifying OFDM, OCDM, and AFDM is not specifically about DWFormer (Zhang et al., 2024).

2. Dynamic Window transFormer in speech emotion recognition

In speech emotion recognition, DWFormer is proposed to address a specific limitation of existing transformer-based SER systems: emotionally salient evidence is often concentrated in local temporal regions and appears at different temporal scales across utterances (Chen et al., 2023). The motivation is that a vanilla transformer can model long-range dependencies, but its global self-attention may become noisy and diffuse, while prior local-window transformers typically rely on fixed window sizes that are too rigid for variable-duration emotional cues such as laughter, sighs, pauses, and intonation changes.

The speech pipeline begins with audio feature extraction using a pretrained WavLM-Large encoder. The output from the 12th transformer layer is taken as the acoustic representation, producing a sequence

$x_0 \in \mathbb{R}^{T \times D},$

where $T$ is the number of time tokens and $D$ is the feature dimension (Chen et al., 2023). This sequence is first passed through a vanilla transformer encoder layer, yielding hidden features $x_{11}$ and attention weights

$W \in \mathbb{R}^{H \times T \times T},$

with $H$ attention heads. Those attention matrices are reused by the Importance Calculation (IC) module to estimate token-level temporal salience.

The IC module converts attention into a token-importance vector through head averaging and softmax normalization. This importance signal is not only diagnostic; it directly governs the subsequent segmentation of the utterance into windows. The central architectural claim is therefore that window boundaries should depend on temporal importance rather than on a fixed segmentation rule (Chen et al., 2023).

This importance-driven formulation is meant to separate regions with strong emotional correlation from regions with weak emotional correlation. A plausible implication is that DWFormer treats temporal localization as part of the model’s representational problem, rather than as a preprocessing choice.

3. Dynamic windowing, local-global interaction, and benchmark behavior

Each DWFormer block contains two submodules: the Dynamic Local Window Transformer (DLWT) and the Dynamic Global Window Transformer (DGWT) (Chen et al., 2023). In the DLWT, dynamic window splitting (DWS) uses the importance values from the IC module, with the threshold set to the median of the importance scores. Tokens above and below that threshold are grouped chronologically into windows, so the resulting windows are not equal in length. The implementation uses an attention mask $M$ that permits attention only within the same window and blocks cross-window attention during the local stage.

Within each dynamic window, self-attention is applied locally. The model then distinguishes strong emotional correlation windows from weak emotional correlation windows. Features in weak windows are multiplied by a factor $\lambda \le 1$, and the experiments use $\lambda = 0.85$ (Chen et al., 2023). This creates an explicit inductive bias that less informative temporal regions should contribute less strongly without being removed.

The DGWT restores utterance-level context after the local stage. Each window is compressed into a single representative token through an importance-weighted sum, producing

$wt \in \mathbb{R}^{(A+B) \times D},$

where $T$0 is the total number of dynamic windows (Chen et al., 2023). A transformer encoder is then applied over the sequence of window tokens, allowing cross-window interaction. The window-level outputs are upsampled back to token resolution by copying the window token across the positions belonging to that window. The local and global streams are fused through

$T$1

The model also updates the importance estimates across blocks, so the segmentation can be refined iteratively.

The full speech architecture is: pretrained WavLM features $T$2 initial transformer encoder $T$3 IC module $T$4 a stack of $T$5 DWFormer blocks $T$6 temporal average pooling $T$7 MLP classifier (Chen et al., 2023). The paper uses 3 DWFormer blocks for IEMOCAP and 2 blocks for MELD, with 8 attention heads, ReLU activations, batch size 32, SGD for 120 epochs, cosine decay with cosine warm-up, and cross-entropy loss.

On IEMOCAP, under the common 4-class setup with happy/excited merged, DWFormer reports 72.3% WA and 73.9% UA. On MELD, it reports 48.5% WF1 (Chen et al., 2023). The paper also reports that these results exceed its vanilla-transformer and fixed-window-transformer baselines. The ablation study shows that removing DLWT or DGWT reduces performance on both datasets, which supports the interpretation that the model depends on both dynamic local partitioning and global re-interaction rather than on either component alone.

4. Fine-tuning DWFormer for continuous emotion scoring in IERP Challenge 2024

A later study uses DWFormer as a pre-trained SER backbone for Track 1 of the IERP Challenge 2024, where the task is to predict continuous emotion intensity scores rather than discrete classes (Wang et al., 15 Aug 2025). The target space consists of 8 emotion categories—sadness, happiness, relaxation, surprise, anger, fear, disgust, and neutrality—with a required score from 1 to 5 for each category, and evaluation is based on RMSE.

The system architecture is summarized as: audio feature extraction, data augmentation by adding noise, Vanilla Transformer Encoder Block, Importance Calculation module, $T$8 stacked DWFormer blocks, classifier, and score fusion (Wang et al., 15 Aug 2025). In this setting, the paper describes the DWFormer block at a high level as consisting of DLWT and DGWT, where DLWT dynamically segments speech into windows of varying scales to capture local emotional information, and DGWT recalculates the significance of each window for global information fusion.

The pre-trained DWFormer used in this study is originally a five-class model for happiness, sadness, neutrality, anger, and fear, with a classifier composed of three fully connected layers with ReLU activations (Wang et al., 15 Aug 2025). For the challenge, the final output layer is modified so that the last fully connected layer produces 8 scores instead of 5. The adaptation strategy is deliberately conservative: all layers are frozen, and only the final fully connected layer of the classifier is trained. The paper explicitly states that the Vanilla Transformer Block, IC Module, DWFormer Block, and Classifier are unchanged, apart from the updated final layer.

Three feature types extracted from large self-supervised speech models are evaluated: WavLM-Large, Chinese-HuBERT-Large, and Chinese-Wav2vec2-Large (Wang et al., 15 Aug 2025). Training uses MSE loss,

$T$9

with batch size 4, initial learning rate 0.0001, Adam, weight decay $D$0, and ReduceLROnPlateau. The dataset contains 2608 audio features for training/validation, 480 audio features for testing, 163 subjects, and 16 emotion-inducing videos.

Noise augmentation uses the noise subset of the MUSAN dataset. Noise is truncated or repeated to match the encoded-feature length and is added probabilistically. The tested probabilities are 0.3, 0.5, and 0.8, with 0.3 reported as the best setting overall; 0.8 degrades performance (Wang et al., 15 Aug 2025). Among single-feature models, the reported test RMSEs are 1.8320 for WavLM-Large, 1.9016 for Chinese-HuBERT-Large, and 1.7905 for Chinese-Wav2vec2-Large.

The paper then applies score fusion across feature models and across different parameter files of the same feature model. The fusion rules are simple averaging, weighted averaging, and maximum value:

$D$1

$D$2

For multi-feature fusion, the reported RMSEs are 1.7867 for average, 1.8078 for weighted average, and 1.7849 for max. For fusion across different parameter files using Chinese-Wav2vec2-Large only, the reported RMSEs are 1.77157 for average, 1.77192 for weighted average, and 1.77042 for max (Wang et al., 15 Aug 2025). Because of the submission limit, the official leaderboard submission used average fusion with RMSE 1.77157, which achieved first place, while the paper identifies 1.77042 from maximum value fusion as the best internal result.

5. DWFormer as an under-display camera restoration network

In under-display camera restoration, DWFormer is a different model with a different objective. The problem is that under-display cameras capture images degraded by brightness attenuation, blur, and noise corruption, summarized in the model

$D$3

where $D$4 is the clean image, $D$5 is the degraded image, $D$6 is channel-dependent brightness attenuation, $D$7 is the blur kernel, and $D$8 is zero-mean signal-dependent noise (Zhou et al., 2022). The paper introduces MPGNet as a modular GAN-based degradation simulator and DWFormer as the restoration network trained on real data alone or on a mixture of real and synthetically degraded data.

The restoration DWFormer is described as a U-Net-like hierarchical network with skip connections (Zhou et al., 2022). A $D$9 convolution first extracts features

$x_{11}$0

The network then proceeds through five stages of stacked DWFormer Blocks (DWB) with convolutional downsampling and sub-pixel upsampling. A final $x_{11}$1 convolution predicts a residual image

$x_{11}$2

and the restored output is

$x_{11}$3

The DWB is a modified Transformer block defined by

$x_{11}$4

Here $x_{11}$5 and $x_{11}$6 are pixel-wise $x_{11}$7 convolutions, $x_{11}$8 is depth-wise convolution, $x_{11}$9 is ReLU, $W \in \mathbb{R}^{H \times T \times T},$0 is BatchNorm, and $W \in \mathbb{R}^{H \times T \times T},$1 is the augmented channel attention module (Zhou et al., 2022). The paper also states that FrozenBN is used to avoid train-test inconsistency.

The architectural substitution is explicit: the block is analogous to a Transformer block, but token mixing is replaced by depth-wise convolution plus ACA, and the feed-forward part remains MLP-like (Zhou et al., 2022). The rationale is task-specific. The paper argues that blur and noise are mainly local phenomena, so local spatial aggregation is well served by depth-wise convolution, while brightness attenuation behaves as a global, channel-dependent effect, motivating ACA for global information aggregation. ACA differs from standard squeeze-excitation by learning spatial weights before pooling into a global channel descriptor.

DWFormer itself is trained with a simple supervised objective,

$W \in \mathbb{R}^{H \times T \times T},$2

while the synthetic data used to augment training are produced by MPGNet under a separate adversarial-plus-supervised loss (Zhou et al., 2022). The restoration paper gives several implementation details: five stages, DWB counts of $W \in \mathbb{R}^{H \times T \times T},$3, Adam with $W \in \mathbb{R}^{H \times T \times T},$4 and $W \in \mathbb{R}^{H \times T \times T},$5, $W \in \mathbb{R}^{H \times T \times T},$6 training iterations, initial learning rate $W \in \mathbb{R}^{H \times T \times T},$7, cosine annealing to $W \in \mathbb{R}^{H \times T \times T},$8, batch size 64, and augmentation by random flips and rotations.

6. Empirical results, ablations, and recurring misconceptions

On the UDC benchmark, the restoration DWFormer reports 33.21 dB PSNR / 0.960 SSIM on P-OLED and 38.96 dB PSNR / 0.984 SSIM on T-OLED (Zhou et al., 2022). The paper states that this surpasses the previous state of the art by +0.22 dB on P-OLED and +0.12 dB on T-OLED. It also reports 1.21M parameters and 13.46G MACs, emphasizing that the model is relatively small. When trained with both MPGNet-generated and real data, performance increases to 34.22 dB on P-OLED and 39.55 dB on T-OLED, corresponding to gains of +1.01 dB and +0.59 dB over training on real data only (Zhou et al., 2022).

The ablations isolate the contribution of the restoration model’s components. Removing ACA reduces PSNR by about 0.59 dB on P-OLED and 0.51 dB on T-OLED. Replacing ACA with SE remains worse than full DWFormer by 0.21 dB on P-OLED and 0.24 dB on T-OLED. Replacing the DWB with a Swin Transformer block reduces performance by 0.14 dB on P-OLED and 0.12 dB on T-OLED while increasing MACs. Replacing BN with LN yields smaller but still negative changes of 0.10 dB and 0.06 dB (Zhou et al., 2022). These results support the paper’s claim that the combination of depth-wise convolution and ACA is better aligned with low-level UDC restoration than attention-heavy alternatives.

A recurring misconception is to treat all occurrences of “DWFormer” as referring to one architecture. The evidence does not support that interpretation. In speech emotion recognition, DWFormer is specifically a dynamic-window transformer built around DLWT and DGWT (Chen et al., 2023). In under-display camera restoration, DWFormer is a Transformer-style restoration network centered on depth-wise convolution and augmented channel attention (Zhou et al., 2022). The finite-temperature QCD study retrieved under similar strings is about Domain Wall Fermions and the finite temperature transition with 2+1 flavors using DWF with DSDR, not about machine learning (Cheng, 2012). This suggests that “DWFormer” functions as a reused label rather than as a single coherent research lineage.

Taken together, the speech and image-restoration usages share only a high-level naming convention. Their internal operators, objectives, datasets, and evaluation protocols are different: WA/UA/WF1 for classification-style SER benchmarks, RMSE for continuous emotion scoring, and PSNR/SSIM for image restoration (Chen et al., 2023, Wang et al., 15 Aug 2025, Zhou et al., 2022). The commonality lies mainly in the attempt to replace undifferentiated global processing with structures better matched to the signal: importance-driven variable windows for emotional speech and local spatial mixing plus global channel aggregation for under-display camera images.