Dynamic Multiscale Fusion Network (DMF2Mel)

• The paper presents a novel EEG-driven mel spectrogram reconstruction framework that decodes minute-level imagined speech, achieving up to 48% improvement over baselines.
• It integrates four core modules—DC-FAM, HAMS-Net, SplineMap with AGKAN, and convMamba—to balance transient feature preservation with long-range temporal modeling.
• Efficiency is enhanced through linear state space modeling and subject-specific adaptation via an Embedding Strength Modulator, ensuring robust EEG–audio alignment.

Searching arXiv for the cited and related papers so the article can reference them with arXiv IDs. Dynamic Multiscale Fusion Network (DMF2Mel) is an EEG-driven mel spectrogram reconstruction architecture introduced to address minute-level, continuous imagined speech decoding under the coupled constraints of low signal-to-noise ratio, non-stationarity, and long-range temporal dependence. The model is organized around four core components—the Dynamic Contrastive Feature Aggregation Module (DC-FAM), the Hierarchical Attention-Guided Multi-Scale Network (HAMS-Net), the SplineMap attention mechanism with Adaptive Gated Kolmogorov-Arnold Network (AGKAN), and the bidirectional state space module convMamba—and is paired with subject adaptation through an Embedding Strength Modulator (ESM). Its stated purpose is to reconcile efficient long-sequence decoding with retention of transient speech-related information that is easily masked by background EEG activity (Fan et al., 10 Jul 2025).

1. Research setting and target representation

The paper situates DMF2Mel in the problem of decoding speech from brain signals, with particular emphasis on the precise reconstruction of minute-level continuous imagined speech. The core technical difficulty is presented as a trade-off: long sequences require efficient temporal dependency modeling, but EEG-driven speech reconstruction also depends on preserving transient, fine-grained information that conventional architectures often lose. CNNs are described as effective for local detail preservation but limited in global context modeling; attention models capture long-range structure but are computationally heavy and brittle under EEG non-stationarity; recurrent models are characterized as vulnerable to vanishing information over minute-scale durations (Fan et al., 10 Jul 2025).

The prediction target is a mel spectrogram rather than a waveform. The ground-truth mel spectrogram is computed by applying an STFT to the audio signal, mapping the resulting power spectrum through mel filterbank energies, and then applying a log transform. With audio signal $x[n]$, window function $w[n]$, FFT size $N$, hop size $H$, and STFT frame index $t$, the paper gives the standard definitions:

$$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$

$M_m(t) = \sum_k H_m(k)\, |X_t(k)|^2,$

and

$$\widetilde{M}_m(t) = \log(M_m(t) + \epsilon) \quad \text{or} \quad \widetilde{M}_m(t) = 10 \log_{10}(M_m(t) + \epsilon).$$

The paper explicitly notes that specific STFT and mel hyperparameters are not detailed, but that these definitions are consistent with its use of mel spectrograms as targets (Fan et al., 10 Jul 2025).

A common misunderstanding is to treat the system as direct speech synthesis from EEG. The formulation given in the paper is narrower and more precise: DMF2Mel reconstructs mel spectrograms from EEG, with evaluation based on correlation between predicted and target spectrogram representations rather than on waveform fidelity or intelligibility measures.

2. Data, preprocessing, and evaluation protocol

The experimental basis is the SparrKULee dataset, described as EEG from 85 participants with normal hearing who listen to fluent Flemish stories, with each subject contributing 8–10 trials of about 15 minutes. In the pipeline description, the same dataset is summarized as comprising 85 Dutch-speaking participants. Raw EEG is first cleaned with multichannel Wiener filtering for artifact removal, then re-referenced to the common average reference (CAR), and finally downsampled to 64 Hz (Fan et al., 10 Jul 2025).

Training and inference are both organized around 5-second segments. During training, 5-second EEG/mel pairs are randomly cropped. During inference, the test signal is divided into 5-second windows, processed independently, and concatenated to produce the full reconstruction. This segmentation strategy is part of the tractability argument for long recordings and should not be conflated with end-to-end full-minute decoding in a single forward pass (Fan et al., 10 Jul 2025).

Two split regimes are reported. In the held-out stories setting, 70 subjects are used for training and testing is performed on the same 70 subjects with new audio stimuli. In the held-out subjects setting, 15 subjects are unseen during training, while the audio stimuli come from the training set. Evaluation is based on Pearson correlation coefficient $r$, and the composite score is defined as

$$\mathrm{score} = \frac{2}{3} \sum_{s \in S_1} \frac{\mathrm{Pearson}\, r_s}{|S_1|} + \frac{1}{3} \sum_{s \in S_2} \frac{\mathrm{Pearson}\, r_s}{|S_2|},$$

where $w[n]$0 denotes the 70 known subjects and $w[n]$1 the 15 unknown subjects (Fan et al., 10 Jul 2025).

This protocol makes the generalization challenge explicit. Performance on held-out stories reflects robustness to new auditory content for known subjects, whereas performance on held-out subjects probes cross-individual transfer under subject-specific neural variability.

3. Architectural organization and processing pipeline

DMF2Mel is a hierarchical pipeline from preprocessed EEG to mel spectrogram output. After preprocessing, subject-specific adaptation is performed by the Embedding Strength Modulator. ESM encodes subject IDs into embedding vectors and fuses them with raw EEG features using multi-head attention, feed-forward networks, and layer normalization. The paper states that this adaptively modulates EEG representation to each subject’s neural idiosyncrasies, yielding a more suitable input for downstream modules (Fan et al., 10 Jul 2025).

Feature extraction then proceeds in two branches. One branch applies DC-FAM to separate speech-related foreground features from noisy background features by combining local convolution and global attention. The other branch applies HAMS-Net, a U-Net-like encoder–decoder with hierarchical attention, to fuse high-level semantics and low-level details across scales. These two feature streams are subsequently fused by SplineMap attention powered by AGKAN, which is described as balancing local spline-based fitting with global context modeling through a gated combination. The fused representation is passed to convMamba, a convolution-enhanced bidirectional state space model that captures long-range temporal dependencies with linear complexity. A final linear layer maps the resulting features to the target mel spectrogram (Fan et al., 10 Jul 2025).

The overall design can be read as a decomposition of the reconstruction problem into four interacting subproblems: subject adaptation, denoising and transient extraction, multiscale semantic–detail fusion, and efficient long-range sequence modeling. This suggests that the model’s contribution is not a single operator but a coordinated architecture that assigns different failure modes of EEG decoding to different modules.

4. Core modules and formal mechanisms

DC-FAM is tailored to EEG denoising and transient feature enhancement. It uses a local convolution branch and a global attention branch, followed by Contrast-Driven Feature Aggregation (CDFA). Let $w[n]$2 denote local features and $w[n]$3 global features. The module computes foreground and background attention maps as

$w[n]$4

where $w[n]$5 and $w[n]$6 are learnable projections. With

$w[n]$7

local and global contrastive aggregation are defined over local blocks $w[n]$8 as

$w[n]$9

followed by reshaping:

$N$0

The intended effect is explicit foreground–background decoupling, in which local convolution emphasizes transient foreground features while global attention suppresses long-range background interference (Fan et al., 10 Jul 2025).

HAMS-Net is described as a U-Net-style encoder–decoder with skip connections and hierarchical attention. Its encoder produces progressively higher-level representations through convolutional downsampling, while Adaptive Dual-Attention Feedback (ADAF) blocks dynamically combine spatial attention, channel attention, and inter-layer feedback:

$N$1

with

$N$2

In the decoder, upsampled features are concatenated with matching-scale encoder features via skip connections and re-enhanced by ADAF. The paper characterizes this as coarse-to-fine reconstruction of multi-scale EEG features, with semantic coherence restored across scales (Fan et al., 10 Jul 2025).

SplineMap attention modifies standard multi-head attention by replacing linear projections for $N$3, $N$4, and $N$5 with spline-based local approximations and fusing them with a global attention path through AGKAN. For spline basis functions $N$6, the mapping is

$N$7

and the projected variables are

$N$8

AGKAN then combines a global attention path and a local KAN/spline path using a gate $N$9. For one head $H$0,

$H$1

$H$2

$H$3

The progressive interaction strategy begins with $H$4 from DC-FAM as the query and $H$5 from HAMS-Net as keys and values:

$H$6

after which subsequent steps alternate queries, keys, and values among $H$7 to refine fusion (Fan et al., 10 Jul 2025).

convMamba is built on a bidirectional state space model and augmented with pre- and post-FFNs. The discrete-time state space equations are

$H$8

Within convMamba, forward and backward recurrences are parameterized as

$H$9

$t$0

Forward and backward outputs are then aligned and fused. The paper states that applying the state space model through its convolutional kernel yields $t$1 complexity over sequence length $t$2, in contrast to the $t$3 complexity of attention, and that bidirectionality mitigates information decay by incorporating both past and future context (Fan et al., 10 Jul 2025).

5. Objectives, optimization, and implementation

Training is based on a composite objective for mel spectrogram reconstruction:

$t$4

The Pearson term aligns temporal structure by maximizing correlation between prediction $t$5 and target $t$6, equivalently minimizing

$t$7

with

$t$8

The paper also gives the empirical form

$t$9

The reconstruction term is

$$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$0

A symmetric InfoNCE term with cosine similarity is used to improve cross-modal EEG–speech alignment and feature discriminability (Fan et al., 10 Jul 2025).

The implementation details reported are specific and limited. The framework is PyTorch. Optimization uses Adam with initial learning rate $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$1 and StepLR with decay factor $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$2 every 50 epochs. Training runs for 1000 epochs on 5-second randomly cropped EEG/mel pairs; inference uses sliding-window 5-second segmentation and concatenation. The paper does not report batch size, dropout, weight decay, activations, normalization layers beyond those named in ESM, or parameter counts (Fan et al., 10 Jul 2025).

Ablation over architectural quantities identifies the best configuration as 2 convMambas, 4 DC-FAMs, 6 HAMS-Nets, and 6 SplineMap Attention blocks, which achieves the top correlation of 0.074. Loss-weight ablations indicate that $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$3 for $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$4 and $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$5 for $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$6 give the best result, and that the composite loss combining Pearson, L1, and InfoNCE is superior to the alternatives reported (Fan et al., 10 Jul 2025).

6. Empirical results, ablations, and limitations

On SparrKULee, DMF2Mel is reported to achieve a Pearson correlation coefficient of 0.074 for mel spectrogram reconstruction on held-out stories and 0.048 on held-out subjects. The paper states that these correspond to a 48% improvement over the baseline for known subjects and a 35% improvement over the baseline for unknown subjects (Fan et al., 10 Jul 2025).

Setting	DMF2Mel	Compared systems
Mel spectrogram, held-out stories	0.074	VLAAI 0.05; HappyQuokka 0.052; SSM2Mel 0.066
Mel spectrogram, held-out subjects	0.048	VLAAI 0.0355; HappyQuokka 0.038; SSM2Mel 0.041
Mel spectrogram, composite score	0.0653	VLAAI 0.0452; HappyQuokka 0.0473; SSM2Mel 0.0577
Speech envelope, held-out stories	0.223	VLAAI 0.1527; Sea-Wave 0.1741; HappyQuokka 0.185; SSM2Mel 0.208
Speech envelope, held-out subjects	0.128	VLAAI 0.1008; Sea-Wave 0.1123; HappyQuokka 0.109; SSM2Mel 0.116
Speech envelope, composite score	0.1913	Highest among compared

The paper further reports violin plots with higher medians and narrower interquartile ranges, together with significant improvements marked by $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$7-values of *: $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$8 and **: $$X_t(k) = \sum_{n=0}^{N-1} x[n + tH]\, w[n]\, e^{-j 2\pi kn / N},$$9. This suggests that the gains are not limited to a few subjects but are reflected in distributional shifts of per-subject performance (Fan et al., 10 Jul 2025).

The ablation study attributes the largest degradation to removing DC-FAM: performance on held-out stories falls from 0.074 to 0.040, a reduction of 45.9%. Removing convMamba reduces performance to 0.048, removing SplineMap attention to 0.054, removing HAMS to 0.056, removing ADAF to 0.059, and removing ESM to 0.069. The paper interprets these results respectively as evidence for the importance of foreground/background decoupling, long-range temporal modeling, dynamic local/global fusion, semantic–detail linkage, multiscale attention optimization, and cross-individual alignment (Fan et al., 10 Jul 2025).

The efficiency claim is centered on convMamba. Its state space formulation yields linear complexity in sequence length for the temporal core, and the 5-second inference windowing further improves tractability for long recordings. Memory and runtime figures are not reported. The paper therefore argues for scalability primarily through asymptotic analysis and pipeline structure rather than through explicit empirical efficiency benchmarks (Fan et al., 10 Jul 2025).

Several limitations are identified. Generalization to unseen subjects remains difficult, despite improvement to 0.048, because of individual neural variability and EEG non-stationarity. EEG–audio alignment can be imperfect, and transient features may still be masked despite DC-FAM and ADAF. Future directions proposed in the paper include adaptive temporal modeling architectures for continuous discourse, and additional opportunities noted in the detailed summary include end-to-end alignment learning, multimodal fusion such as visual and auditory contexts, and self-supervised pretraining on unlabeled EEG. Ethical considerations—privacy, consent, and responsible use—are explicitly noted as essential for BCI applications (Fan et al., 10 Jul 2025).