MIMO Temporal Wavelet Transform

• Multi-input multi-output temporal wavelet transform is a method that decomposes multiple input signals into temporally ordered low-pass and high-pass subbands for coding, prediction, and enhancement.
• It leverages learned motion-compensated lifting and integrates spatial 2D transforms to ensure perfect reconstruction and preserve inter-channel relationships.
• Applications in video coding, speech enhancement, and video prediction demonstrate its scalability and improved performance through multiresolution analysis.

Searching arXiv for the cited papers to ground the article in published work. arXiv search query: (Meyer et al., 2023) OR (Han et al., 27 Jun 2025) OR (Jin et al., 2020) A multi-input multi-output temporal wavelet transform is a temporal analysis–synthesis construction in which multiple input signals are decomposed into multiple temporally ordered subband outputs, typically low-pass and high-pass components across dyadic scales, and then either reconstructed or further processed for coding, prediction, or enhancement. In recent arXiv work, this concept appears in learned motion-compensated temporal filtering for video coding, where a group of pictures is mapped to a hierarchy of temporal subbands (Meyer et al., 2023); in MIMO speech enhancement, where multichannel recordings are processed with wavelet-based multi-resolution representations while preserving inter-channel spatial cues (Han et al., 27 Jun 2025); and in spatial-temporal multi-frequency video prediction, where temporal discrete wavelet transforms operate directly on frame sequences and are fused with spatial decomposition and recurrent modeling (Jin et al., 2020). Across these formulations, the defining features are temporal multiresolution, explicit multi-branch outputs, and operator structures that expose cross-input dependencies rather than collapsing them into a single residual stream.

1. Conceptual scope and canonical realizations

The most explicit formulation of a temporal wavelet transform as a MIMO system is the learned motion-compensated temporal filtering framework for video coding. There, the recursion over dyadic temporal levels yields “a MIMO mapping from multiple temporal inputs (the whole GOP) to multiple subband outputs: all $h^{(j,t)}$ and the final $l^{(J,0)}$,” with temporal low-pass and high-pass branches produced by lifting steps implemented with learned prediction and update operators (Meyer et al., 2023). The same work extends the temporal transform to a spatio-temporal transform by applying a learned $2$D wavelet transform to each temporal subband image and by supporting both YUV $4{:}2{:}0$ and RGB $4{:}4{:}4$.

In WTFormer, the MIMO aspect is not a GOP-to-subband coder but a multichannel enhancement system whose outputs are “enhanced signals for all $M$ input channels,” with preservation of inter-channel relationships such as ITD, IPD, ILD, and relative transfer functions (Han et al., 27 Jun 2025). Temporal wavelet processing is embedded inside a time-frequency encoder–decoder through Haar-based wavelet transform convolutions operating on STFT feature maps, so the transform is internal to feature extraction rather than a standalone reconstruction stage.

In the video prediction model based on spatial-temporal multi-frequency analysis, the paper’s implemented pipeline is “multi-input single-output (per step), with multi-output achieved by rollout,” because it predicts the next frame autoregressively from $m$ past frames (Jin et al., 2020). The same source, however, gives a “proposed extension” in which future temporal subbands for $T_{\text{out}}$ steps are predicted jointly and reconstructed by inverse transforms. This suggests a broader use of the term: a MIMO temporal wavelet transform may denote either an explicit filter bank with multiple temporal outputs or a temporal decomposition stage that feeds a joint multi-output estimator.

Realization	Input–output structure	Temporal wavelet mechanism
Learned MCTF video coding (Meyer et al., 2023)	whole GOP $\rightarrow$ all $h^{(j,t)}$ and final $l^{(J,0)}$0	motion-compensated lifting
WTFormer speech enhancement (Han et al., 27 Jun 2025)	$l^{(J,0)}$1 microphones $l^{(J,0)}$2 enhanced signals for all channels	Haar WTConv inside STFT-domain network
Spatial-temporal video prediction (Jin et al., 2020)	$l^{(J,0)}$3 past frames $l^{(J,0)}$4 next frame; MIMO extension proposed	multi-level DWT on temporal dimension

2. Operator-theoretic structure and perfect reconstruction

In the lifting-based video formulation, a single temporal analysis stage acts on two frames $l^{(J,0)}$5 and $l^{(J,0)}$6 under an even–odd split. The forward transform is

$l^{(J,0)}$7

$l^{(J,0)}$8

or in simplified form,

$l^{(J,0)}$9

Here $2$0 is a differentiable, bilinear warping operator parameterized by motion vectors estimated by SPyNet and entropy-coded with a hyperprior + dual spatial prior, while $2$1 and $2$2 are learned CNN predictors and updaters applied after motion compensation and denoising. The outputs $2$3 and $2$4 are temporal low-pass and high-pass subbands. The inverse transform reverses step order and signs,

$2$5

and achieves perfect reconstruction prior to quantization because the lifting steps are triangular with identity diagonals (Meyer et al., 2023).

The same construction admits a $2$6 operator-matrix form. With $2$7 denoting warp-then-predict and $2$8 the update branch, the analysis transform maps $2$9 to $4{:}2{:}0$0 through off-diagonal motion-dependent learned operators, making the two-channel filter bank explicitly MIMO rather than a separable scalar transform. For lossless compression, integer rounding of $4{:}2{:}0$1 and $4{:}2{:}0$2 outputs yields an integer-to-integer lifting stage in the sense of Calderbank–Daubechies–Sweldens (Meyer et al., 2023).

A second, more classical formulation appears in the temporal DWT descriptions used for video prediction and as the underlying $4{:}2{:}0$3-D case for WTFormer’s separable $4{:}2{:}0$4-D wavelet convolutions. If $4{:}2{:}0$5, then level-$4{:}2{:}0$6 analysis is

$4{:}2{:}0$7

with inverse reconstruction by upsampling and synthesis filtering,

$4{:}2{:}0$8

In the video-prediction paper, Haar is stated as an assumption rather than a specified choice, whereas WTFormer explicitly uses Haar wavelet bases in WTConv (Jin et al., 2020, Han et al., 27 Jun 2025). The shared filter-bank interpretation is that temporal approximation coefficients aggregate slower motion content while detail coefficients isolate faster temporal changes.

3. Multi-level dyadic decomposition and the MIMO mapping

The dyadic recursion in learned MCTF is indexed by level $4{:}2{:}0$9 with stride $4{:}4{:}4$0. Let $4{:}4{:}4$1. Then analysis at level $4{:}4{:}4$2 is defined by

$4{:}4{:}4$3

$4{:}4{:}4$4

with $4{:}4{:}4$5 and recursion on the low-pass branch. For a GOP of $4{:}4{:}4$6 frames, the coding order is level $4{:}4{:}4$7, then level $4{:}4{:}4$8, then level $4{:}4{:}4$9, and finally the deepest low-pass $M$0. The system is open-loop: motion estimation is performed on original frames rather than decoded reconstructions, which avoids drift (Meyer et al., 2023).

As temporal distance doubles with level, displacement magnitudes increase and prediction becomes harder. To address this, MCTF-DS downsamples frames by $M$1 before motion estimation at levels $M$2, codes lower-resolution motion latents, and upsamples the motion vectors before motion compensation and inverse motion compensation. The paper states that this reduces motion bitrate and often improves robustness to large motions with minimal quality loss (Meyer et al., 2023).

The temporal DWT used in the video-prediction model is also multilevel, but it operates directly on raw frames along the time axis. The paper states that it conducts “multi-level DWT on temporal dimension (DWT-T) on the input sequence,” continues “until the number of low-frequency sub-bands or high-frequency sub-bands equals two,” and shows “three DWT-T as an example.” The resulting temporal subbands are concatenated and passed to a small CNN, the T-WAM module, and then fused with historical information from an LSTM cell (Jin et al., 2020). In that setting, faster motions are primarily represented in higher-frequency temporal subbands, while slower motions accumulate in lower-frequency subbands and the final approximation.

WTFormer uses a different decomposition locus: the time-domain signals from the microphone array are first transformed by STFT, and wavelet transform convolutions with a Haar basis then operate on the $M$3-D time-frequency feature maps. The paper states that WTConv yields multi-resolution subbands $M$4, $M$5, $M$6, and $M$7, expands the receptive field, and captures both long-term temporal dependencies and local transients (Han et al., 27 Jun 2025). A plausible implication is that the MIMO property here is distributed across temporal scale, frequency bins, and channels rather than tied to a single explicit temporal filter bank over the raw signals.

4. Extension across space, color, and channels

In learned video coding, temporal analysis is followed by a learned $M$8D wavelet transform, iWave++, applied in horizontal and vertical dimensions to each temporal subband image. iWave++ uses CNN-based prediction and update filters in lifting, four spatial decomposition levels, scalar quantization with a trainable parameter, and a CNN-based context entropy model with a Gaussian mixture likelihood. The result is an explainable and invertible transform with spatial scalability. For color, the coder supports YUV $M$9 and RGB $m$0; in YUV $m$1, motion vectors are computed on the luma channel and reused for chroma, and subbands for each color channel are coded independently, with no explicit cross-channel coupling in the entropy model (Meyer et al., 2023).

The spatial-temporal video-prediction model uses a closely related but task-specific spatial construction. Each frame is decomposed by a $m$2D DWT into $m$3, $m$4, $m$5, and $m$6 subbands, and the paper states that three S-WAMs are cascaded, each performing a $m$7D DWT per frame followed by shallow CNNs per subband and residual fusion with RRDB features (Jin et al., 2020). The stated purpose is to enrich structural information, reserve fine details, and preserve anisotropic information: $m$8 for horizontal, $m$9 for vertical, and $T_{\text{out}}$0 for diagonal detail. Temporal DWT and spatial DWT are then fused in a unified encoder–decoder with CNN and LSTM components.

In WTFormer, cross-channel structure is central. The observed mixture is modeled in the STFT domain as

$T_{\text{out}}$1

and the system estimates early reverberant speech by a learned MIMO complex masking filter. The architecture uses three WTBlocks in the encoder, two cascaded Conformer blocks for time–frequency dependencies, and multidimensional collaborative attention in place of U-Net skip connections. MCA computes attention over spatial, time, and frequency dimensions, averages the three attention outputs, gates them with a sigmoid, and fuses them with encoded features (Han et al., 27 Jun 2025).

The output definition makes the MIMO constraint explicit: enhanced signals are generated for all $T_{\text{out}}$2 channels, so inter-channel relationships can be preserved end-to-end. The paper defines common spatial cues as

$T_{\text{out}}$3

with ITD estimated from time-domain cross-correlations or phase slopes across frequency (Han et al., 27 Jun 2025). This places the temporal wavelet transform within a broader MIMO framework in which preserving spatial covariance is as important as denoising.

5. Optimization, inference control, and learning objectives

The learned MCTF video coder is trained end-to-end with a staged curriculum. Stage $T_{\text{out}}$4 trains motion estimation and denoising on distortion only with $T_{\text{out}}$5; stage $T_{\text{out}}$6 adds the motion vector rate; stage $T_{\text{out}}$7 trains all components including iWave++ on the full rate–distortion objective

$T_{\text{out}}$8

stages $T_{\text{out}}$9 and $\rightarrow$0 add multiple MCTF stages for GOP sizes $\rightarrow$1 and $\rightarrow$2 with $\rightarrow$3 and $\rightarrow$4, randomly sampling frame distances per batch, training MCTF stages first and then all modules jointly. The paper reports that attempting to train inverse MCTF with four frames per batch caused instability and degraded RD, whereas two-frame training with multi-stage scheduling stabilized optimization (Meyer et al., 2023).

The same work introduces content-adaptive inference, MCTF-CA, to mitigate error propagation and ghosting under strong motion and occlusion without retraining. For each $\rightarrow$5-frame unit, the system evaluates five options—GOP8 with standard MCTF, GOP8 with downsampling strategy, split into two GOP4 with and without downsampling, and split into four GOP2—and chooses the minimum of

$\rightarrow$6

The choice is signaled with $\rightarrow$7 bits per $\rightarrow$8-frame unit (Meyer et al., 2023).

WTFormer uses a different optimization principle: a multi-task loss with learnable uncertainty weights combining denoising and spatial preservation,

$\rightarrow$9

where $h^{(j,t)}$0 is SI-SNR-based noise-suppression loss and $h^{(j,t)}$1 is a MUSIC-based spatial spectrum loss (Han et al., 27 Jun 2025). For a ULA with spacing $h^{(j,t)}$2 cm and speed of sound $h^{(j,t)}$3 m/s, the MUSIC pseudo-spectrum is

$h^{(j,t)}$4

and the spatial loss is the mean-squared error between input and enhanced spectra averaged over $h^{(j,t)}$5 narrowband frequencies and $h^{(j,t)}$6 scanned angles. The implementation uses LibriSpeech train-360 split $h^{(j,t)}$7, Adam with learning rate $h^{(j,t)}$8 halved on plateau, batch size $h^{(j,t)}$9, $l^{(J,0)}$00 epochs, and automatic mixed precision (Han et al., 27 Jun 2025).

In the spatial-temporal video-prediction model, the implemented objective combines image-domain loss and adversarial loss,

$l^{(J,0)}$01

$l^{(J,0)}$02

$l^{(J,0)}$03

with $l^{(J,0)}$04 and $l^{(J,0)}$05 set to $l^{(J,0)}$06 for KTH and $l^{(J,0)}$07 for BAIR and KITTI/Caltech (Jin et al., 2020). The same source also gives a proposed MIMO extension in which future temporal subbands are predicted jointly and supervised by pixel-domain, perceptual, temporal-consistency, and wavelet-domain losses. Because this part is explicitly presented as a proposed extension, it should be interpreted as a framework-level generalization rather than the paper’s base implementation.

6. Empirical behavior, scalability, and limitations

The learned MCTF framework provides intrinsic temporal scalability because the dyadic temporal hierarchy yields a base layer in the low-pass subbands and enhancement layers in the high-pass subbands; spatial DWT adds spatial scalability. On UVG, MCTF-CA achieves average Bjøntegaard-Delta rate savings of $l^{(J,0)}$08 for GOP4 and $l^{(J,0)}$09 for GOP8 over HM LD-P; on HEVC Class E, $l^{(J,0)}$10 for GOP4 and $l^{(J,0)}$11 for GOP8; and on MCL-JCV, $l^{(J,0)}$12 for GOP4 and near parity at GOP8 with $l^{(J,0)}$13. The paper states that at high rates, MCTF-CA consistently surpasses HM and state-of-the-art learned coders such as DCVC-HEM, while also noting that DCVC-HEM performs better at low rates on some sets (Meyer et al., 2023). Complexity remains substantial: for $l^{(J,0)}$14p inputs, DCVC is $l^{(J,0)}$15 MB and $l^{(J,0)}$16 kMAC/px, DCVC-HEM is $l^{(J,0)}$17 MB and $l^{(J,0)}$18 kMAC/px, and MCTF-CA is $l^{(J,0)}$19 MB and $l^{(J,0)}$20 kMAC/px. Most complexity in MCTF-CA is attributed to temporal subband coding by iWave++, with MCTF modules accounting for approximately $l^{(J,0)}$21 of model size and approximately $l^{(J,0)}$22 of kMAC/px for GOP8 (Meyer et al., 2023).

WTFormer reports PESQ $l^{(J,0)}$23, STOI $l^{(J,0)}$24, eSTOI $l^{(J,0)}$25, and SI-SNR $l^{(J,0)}$26 dB, together with spatial-retention metrics $l^{(J,0)}$27ITD $l^{(J,0)}$28s, $l^{(J,0)}$29IPD $l^{(J,0)}$30 rad, and $l^{(J,0)}$31ILD $l^{(J,0)}$32 dB (Han et al., 27 Jun 2025). The paper states that these results improve spatial metrics over MIMO-UNet and EaBNet at a much smaller model size of approximately $l^{(J,0)}$33M parameters. Ablation results attribute notable PESQ gains and ITD retention to WTConv, identify MCA as critical for both PESQ and ITD, and show that removing the MUSIC-based spatial loss degrades ILD and ITD (Han et al., 27 Jun 2025). The stated limitations are practical: Conformer and MCA introduce latency, causal or streaming variants may be needed for real time, MUSIC-based loss is array-geometry dependent, and highly non-stationary noise may challenge covariance stability (Han et al., 27 Jun 2025).

The spatial-temporal multi-frequency video-prediction model reports, on KTH $l^{(J,0)}$34, PSNR $l^{(J,0)}$35, SSIM $l^{(J,0)}$36, and LPIPS $l^{(J,0)}$37; on KTH $l^{(J,0)}$38, PSNR $l^{(J,0)}$39, SSIM $l^{(J,0)}$40, and LPIPS $l^{(J,0)}$41; on BAIR, PSNR $l^{(J,0)}$42, SSIM $l^{(J,0)}$43, and LPIPS $l^{(J,0)}$44; and FVD $l^{(J,0)}$45 on KTH and $l^{(J,0)}$46 on BAIR (Jin et al., 2020). The paper states that removing S-WAM or T-WAM degrades performance and removing both degrades the most, supporting the claimed role of multi-frequency analysis in fidelity and temporal consistency. At the same time, its native temporal wavelet design remains autoregressive at inference for multi-frame outputs, so the full MIMO interpretation depends on the proposed extension rather than the base predictor (Jin et al., 2020).

Taken together, these systems delineate three closely related meanings of multi-input multi-output temporal wavelet transform. In video coding, it is an explicit invertible temporal filter bank with multiple subband outputs and perfect reconstruction before quantization. In MIMO speech enhancement, it is a multiresolution transform embedded in a channel-preserving network whose outputs remain multichannel and spatially coherent. In video prediction, it is a multi-frequency temporal decomposition that can support MIMO forecasting when future subbands are predicted jointly. The common thread is not a single standardized architecture, but a class of multiscale temporal operators that retain structure across inputs instead of reducing temporal interaction to a single residual or single-channel output.