Multi-Frame Complex Filtering

• Multi-frame complex filtering is a signal processing approach that jointly exploits temporal and spatial or frequency correlations to improve denoising, source separation, and feature extraction.
• It integrates classical methods like MVDR and Wiener filtering with deep learning architectures to estimate optimal complex filter coefficients in diverse multiframe scenarios.
• Empirical results demonstrate significant gains in speech enhancement, echo cancellation, and image denoising, highlighting improved performance metrics and computational efficiency.

Multi-frame complex filtering is a class of signal processing methodologies characterized by the joint exploitation of temporal and (often) spatial or frequency correlations over multiple frames or consecutive time-frequency segments. These approaches generalize conventional single-frame, single-channel filtering to multidimensional contexts—incorporating, for instance, several STFT frames, multiple sensors, or collections of image/video patches—to improve tasks such as denoising, source separation, dereverberation, echo cancellation, or feature extraction. The field encompasses both classical linear methods grounded in covariance estimation and constrained optimization (e.g., MVDR, Wiener), as well as recent deep-learning frameworks that embed or estimate such models in an end-to-end fashion. Multi-frame complex filtering has achieved notable impact in speech enhancement, hearing aids, acoustic beamforming, echo cancellation, and computational imaging.

1. Theoretical Foundations and Mathematical Frameworks

Multi-frame complex filtering leverages the rich intra- and inter-frame structure of signals. The canonical linear case is the multi-frame minimum variance distortionless response (MFMVDR) filter, which, at a particular time-frequency bin, applies a length-$N$ complex-valued vector $\mathbf{w} \in \mathbb{C}^N$ to $N$ stacked STFT coefficients: $\widehat{X}_l = \mathbf{w}_l^H\,\mathbf{y}_l$ where $\mathbf{y}_l = [Y_l, Y_{l-1}, \dots, Y_{l-N+1}]^T$ is the multi-frame observation vector. The filter $\mathbf{w}_l$ is found by solving

$$\min_{\mathbf{w}_l}\; \mathbf{w}_l^H \mathbf{R}_n \mathbf{w}_l \quad \text{s.t.}\quad \mathbf{w}_l^H \mathbf{r}_s = 1$$

with $\mathbf{R}_n$ the noise covariance and $\mathbf{r}_s$ the speech interframe correlation vector (Tammen et al., 2020, Schröter et al., 2023, Tammen et al., 2022).

For multichannel or binaural setups (e.g., two microphones), the formalism generalizes: stacking both channels’ histories, constructing spatio-temporal correlation vectors, and solving a block-constrained MVDR or Wiener problem (Tammen et al., 2022). The optimal solution has the form

$$\mathbf{w}_{\mathrm{MVDR}} = \frac{\mathbf{R}_n^{-1}\mathbf{r}_s}{\mathbf{r}_s^H\mathbf{R}_n^{-1}\mathbf{r}_s}$$

Extensions further address multi-band (filterbank) and tensor-structured filtering. In matrix factorization frameworks, the global analysis-synthesis operation is represented by an $N \times N$ matrix of complex-valued periodic functions, $H(z)$, acting in the $z$-domain and factorized into elementary (lifting) steps (Jorgensen et al., 2014).

Multi-frame complex filtering is also foundational to modern image processing techniques—most prominently, multi-frame patch-based denoising (Bodduna et al., 2020) and directional wavelet/framelet transforms (Han et al., 2013).

2. Model-Based Methods: MVDR, Wiener, and Lifting Frameworks

Classical model-based schemes estimate signal and noise covariances over sliding multi-frame windows, applying analytic solutions at each instance. In speech applications, STFT coefficients from past frames are collected, and the covariance matrices are computed recursively or via batch averaging. Voice activity detection or oracle knowledge is required to isolate noise-dominated regions for reference (Schröter et al., 2023).

The multi-frame MVDR filter imposes a distortionless constraint on the current frame and minimizes output noise, yielding analytic solutions with $O(N^3)$ complexity for general covariance matrices. The multi-frame Wiener filter, targeting mean-square error minimization between the output and the reference, leverages joint signal-plus-noise covariances and interpolates between subspace projections (Schröter et al., 2023).

In multi-phase and multi-band settings, filterbanks are constructed via algebraic lifting and matrix factorization. Each channel corresponds to a frequency band, with analysis and synthesis operations compactly encoded in factorizations of $H(z)$ into triangular polynomials or L-infinity functions (Jorgensen et al., 2014). This unifies wavelet lifting, polyphase, and Cuntz-algebraic frameworks, supporting perfect reconstruction and in-place computation for large $N$.

In image/video denoising, multi-frame extensions of BM3D and non-local Bayes (NLB) filters employ either separable averaging/filtering or cross-frame patch stacking, all typically preceded by frame registration via robust optical flow (Bodduna et al., 2020).

3. Deep Learning Approaches: End-to-End Multi-frame Filtering

Deep learning approaches have emerged as highly effective extensions of model-based methods for multi-frame complex filtering. Rather than relying on explicit, potentially brittle, statistical estimators for correlations and covariances, these architectures embed the analytic filtering operation inside a differentiable network. Temporal convolutional networks (TCNs), BLSTMs, or convolutional recurrent networks (CRNs) estimate either the analytical parameters (covariances, correlation vectors) or directly predict the optimal complex FIR filter coefficients (Tammen et al., 2020, Schröter et al., 2023, Tammen et al., 2022, Cheng et al., 2022).

A prototypical architecture consists of:

• Input features (real and imaginary parts, log-magnitude, phase) extracted from $N$-frame STFT segments;
• Multiple stacks of dilated temporal convolutions capturing long-range dependencies (128–512 ms receptive field, tens of layers, millions of parameters);
• Outputs representing either the parameters (e.g., real-valued Cholesky factors of the inverse noise covariance, correlation vectors with unit-gain normalization) or the entire filter vector;
• Integration of the classical MVDR or Wiener formula as a computation graph node;
• End-to-end training using mean spectral absolute error (MSAE), scale-invariant signal-to-distortion ratio (SI-SDR), or similar objectives (Tammen et al., 2020, Tammen et al., 2022).

In speech enhancement, binaural noise reduction, and echo cancellation, the TCN-based filter estimator or the analytic MVDR layer is jointly optimized, yielding robust performance and lower distortion relative to both analytic and direct-filter baselines.

In acoustic echo cancellation, multi-frame complex filtering is paired with deep CRNs predicting $L$ complex-valued FIR taps convolved with far-end signal histories, with demonstrated gains in echo return loss enhancement (ERLE) as $L$ increases (Cheng et al., 2022).

4. Algorithmic Variants and Domain-Specific Implementations

Algorithmic variants are adapted to varying constraints and application requirements. For hearing aids, low-latency and computational resource limitations motivate the design of deep filters with small BLSTM or TCN backbones (30–50 k parameters, <8 ms total latency), as opposed to computationally intensive covariance estimation (Schröter et al., 2023). The performance-complexity trade-off is evident: classical MVDR and Wiener solutions yield sub-100 µs inference times but degrade under rapid nonstationarity, while deep filtering attains higher PESQ/STOI/SDR metrics by learning implicit non-stationary adaptation.

Multi-frame patch-based image filters offer several options:

• Separate temporal averaging then spatial filtering (AF, "average–then–filter"): fast, robust, sharp spatial detail, best overall trade-off;
• Separate spatial filtering then temporal averaging (FA, "filter–then–average"): computationally more expensive, typically falls behind AF;
• Multi-frame patch stacking ("multiple reference frame", MF) with collaborative 3D filtering: highest denoising theoretically in the perfectly registered case, but sensitive to motion errors and slow (Bodduna et al., 2020).

In multi-band frequency processing, matrix factorizations into upper/lower triangular blocks enable highly parallelizable implementations, with lifting algorithms furnishing in-place filterbanks for very large $N$ (e.g., OFDM, image subband decomposition) (Jorgensen et al., 2014).

Directionally-sensitive image denoising leverages tensor product complex tight framelets (TPCTF$_n$) that generalize the dual-tree complex wavelet transform, with exact Parseval frame conditions, high directionality (up to 14 orientations for $n=6$), and low redundancy (Han et al., 2013).

5. Empirical Results and Comparative Performance

Empirical evaluations consistently demonstrate the advantages of multi-frame complex filtering:

• Multi-frame (e.g., $N=3$–$5$) filters yield substantial improvements over single-frame approaches in speech denoising, with PESQ gains of up to $1.1$ MOS and FWSSNR gains of 8 dB in binaural setups (Tammen et al., 2022);
• End-to-end deep MFMVDR outperforms state-of-the-art time-domain networks (Conv-TasNet) in both PESQ and STOI, while maintaining comparable runtime (real-time factor $<$0.2) (Tammen et al., 2020);
• In hearing aids, deep multi-frame filtering improves PESQ by $0.2$–$0.3$ and SDR by $1$–$2$ dB over analytic MVDR/Wiener at minimal computational cost (Schröter et al., 2023);
• In multi-frame image denoising, separable "average–then–filter" filters achieve state-of-the-art PSNR (e.g., 39.06 dB for ten noisy frames) and run $5$–$10\times$ faster than 3D or multiple-reference schemes (Bodduna et al., 2020);
• Acoustic echo cancellation benefits from increased multi-frame memory (e.g., filter length $L=10$) with ERLE gains up to 7 dB vs. single-frame models (Cheng et al., 2022).

A summary table of speech enhancement results (PESQ/STOI) from (Schröter et al., 2023):

Method	PESQ	STOI	SDR (dB)	DSP Runtime (µs)	Parameters
MF–MVDR	2.01	0.78	7.2	70	0
MF–Wiener	2.15	0.81	8.1	85	0
Deep MF	2.32	0.84	9.3	120	28k

6. Multi-Frame Complex Filtering in High-Dimensional and Structured Domains

Beyond audio, multi-frame complex filtering is pivotal in multiband and multidimensional domains. Matrix factorizations on $N \times N$ filterbanks underpin polyphase, OFDM, and wavelet transforms, with the entire filtering chain encoded as manipulation of Laurent polynomials or L-infinity functions (Jorgensen et al., 2014). Factorization into alternating upper and lower triangular blocks corresponds to stepwise analysis, downsampling, upsampling, and synthesis (lifting schemes).

Tensor-product complex tight framelets (TPCTF$_n$) provide structured, exact tight frames for images, with framelet banks achieving strong directionality and low redundancy. TPCTF$_4$ yields six directions per scale with $2.25$ redundancy—outperforming the dual-tree complex wavelet transform on key benchmarks (Han et al., 2013). These constructions can be hybridized, e.g., used as the first stage of DTCWT to enhance direction selectivity.

Practical design guidelines for large $N$ include the in-place implementation of lifting schemes for arbitrary band number, exploitation of parallelism, and optimal selection of filter lengths and frequency partitioning for desired resolution versus redundancy trade-offs (Jorgensen et al., 2014, Han et al., 2013).

7. Practical Considerations, Limitations, and Domain Guidance

Model-based filtering is computationally efficient and offers theoretical optimality under Gaussian and stationarity assumptions, but suffers from performance degradation in rapidly non-stationary noise or with poor parameter estimation (e.g., VAD errors) (Schröter et al., 2023). Deep learning approaches provide robustness and superior performance by learning non-linear dependencies and compensating for parameter estimation errors; the cost is increased model size and the need for representative training data (Tammen et al., 2020, Tammen et al., 2022).

In image denoising, the separable average–then–filter approach is identified as optimal for balancing denoising strength, spatial detail preservation, robustness to misregistration, and computational efficiency (Bodduna et al., 2020). Fully 3D and multi-reference strategies offer, at best, marginal PSNR gains but suffer in runtime and are less robust in the presence of motion artifacts.

In structured filterbank and lifting architectures, complexity grows only linearly with band number when using matrix factorizations, and parallel computation is naturally supported (Jorgensen et al., 2014).

Multi-frame complex filtering, as a field, represents the intersection of classical signal processing, algebraic operator theory, and modern data-driven inference methodologies. Its rigorous mathematical underpinnings and robust empirical successes underpin its centrality in cutting-edge acoustic and image enhancement.