---
title: 'Time-Mixing MLPs: Efficient Temporal Modeling'
url: https://www.emergentmind.com/topics/time-mixing-mlps
type: topic
---

# Time-Mixing MLPs: Efficient Temporal Modeling

Time-mixing MLPs (multi-layer perceptrons) are a class of deep learning architectures designed to model dependencies across the temporal (sequence) dimension by leveraging pure MLP blocks instead of recurrent mechanisms or self-attention. These models reorganize mixing operations along the time axis to capture temporal relationships in sequential, time series, or spatiotemporal data, combining high efficiency with domain-appropriate inductive biases. Time-mixing MLPs have demonstrated efficacy across diverse domains, notably in sequential recommendation, large-scale traffic forecasting, multivariate time series prediction, and video recognition.

## 1. Core Design Patterns in Time-Mixing MLPs

Time-mixing MLPs are instantiated via a variety of architectural patterns, each characterized by explicit mixing along the temporal axis—typically via linear or nonlinear feedforward layers with weight sharing and masking strategies.

- **MLP-mixer principle:** The canonical approach, inherited from vision MLP-Mixer networks, decomposes mixing operations into two stages: one MLP operates along the "token" (time) axis, and another along the "channel" (feature) axis. In time-mixing variants, the temporal axis takes the role of "token" [2303.06053].
- **Causal and masked mixing:** Auto-regressive or causal modeling is enforced by explicitly masking weights in the temporal MLP to eliminate anti-chronological information flow. The TriMLP’s Triangular Mixer, for example, applies a strictly upper-triangular mask to the token-mixing MLP, followed by row-wise softmax to implement causal "attention-like" mixing [2305.14675].
- **Grouped or blockwise time mixing:** To balance receptive field and computational load, inputs are often decomposed into temporal groups or patches, enabling both local and long-range temporal interactions, as in Grouped Time Mixing (GTM) [2206.06292] or window aggregation [2512.07854, 2412.13607, 2412.17176]. Overlapping and hierarchical grouping further modulate the temporal receptive field.
- **Specialized nonlinearities:** Non-standard nonlinear transformations increase expressiveness; e.g., KAN-based time-mixing replaces MLPs with Kolmogorov–Arnold layers, learning per-coordinate spline-basis functions [2502.18410].

Table 1 below summarizes selected representative time-mixing MLP architectures and their key design strategies.

| Model          | Temporal Mixing Mechanism              | Inductive Bias/Scope          |
|----------------|---------------------------------------|-------------------------------|
| TriMLP         | Triangular mask + softmax             | Causal, global + local        |
| TSMixer        | Linear MLP across time, residual      | Global, linear/nonlinear      |
| PreMixer       | Patchwise MLP, patch-embedding        | Patch-global, linear          |
| HSTMixer       | Windowed aggregation, hierarchical    | Hierarchical, local/global    |
| WPMixer        | Wavelet decomposition + patch mixing  | Multi-resolution, hierarchical|
| MLP-3D         | Grouped Time Mixing (GTM)             | Patch/block, video temporal   |
| RPMixer        | Blockwise random projections          | Diversity, ensemble-like      |
| TSKANMixer     | KAN splines replacing MLP             | Highly nonlinear, universal   |

## 2. Mathematical Formulations and Operations

The essential operation in time-mixing MLPs is a fully connected layer applied along the temporal axis. In its most generic form, given an input $X \in \mathbb{R}^{L \times F}$ (look-back length $L$, $F$ features), a single time-mixing MLP applies:

$$
X^{(t)} = X W^{(t)} + b^{(t)}, \qquad W^{(t)} \in \mathbb{R}^{L \times L}
$$

Enhancements include:

- **Masked Linear Mixing:** Setting $W^{(t)}_{ij} = 0$ for $j > i$ ensures causal processing [2305.14675].
- **Softmax Normalization:** Applying a row-wise softmax to masked $W$ emulates attention-like probability mixing.
- **Patchwise Mixing:** Temporal windows or patches ($X \in \mathbb{R}^{N \times t \times C}$) are vectorized and embedded via learned linear (or two-layer MLP) projections, followed by mixing [2412.13607, 2412.17176].
- **Hierarchical Aggregation and Upsampling:** In HSTMixer, bottom-up windowed aggregation reduces temporal resolution (window size $p$), followed by top-down re-expansion through learned upsampling [2512.07854].
- **Random Projections:** RPMixer injects fixed, block-specific random matrices, projecting along the spatial (node) dimension to encourage diversity and ensemble effects [2402.10487].

In Table 2, these formulation types are mapped to representative operations and intended modeling effects.

| Mixing Technique      | Equation/Process                             | Modeling Implication                    |
|----------------------|----------------------------------------------|-----------------------------------------|
| Masked linear mixing | $X' = (M \odot W) X$                         | Enforces chronology (causality)         |
| Patchwise embedding  | $z = W_2 \sigma(W_1 x + b_1) + b_2$          | Local context, efficient computation    |
| Hierarchical         | $E_\ell \to$ agg./propagation $H_\ell$       | Multi-resolution, compression/expansion |
| Random projections   | $Z = P\,\text{ReLU}(X^T)$, $S = W_2 Z$       | Block diversity, ensemble averaging     |

## 3. Variants and Hybridizations

Several time-mixing MLP models combine multiple mixing levels, adaptive parameterizations, and specialized nonlinearities to improve expressiveness and scalability.

- **Alternating global/local mixing:** TriMLP alternates full-sequence (global) and local window block mixing, using separate triangular masks to enable both long-range dependencies and short-term pattern capture. A two-stage serial application is empirically superior to any parallel or concatenated mixing [2305.14675].
- **Grouped Time Mixing (GTM):** MLP-3D introduces four core GTM blocks (short-range, long-range, shift-window, shift-token), enabling tunable windows and strided temporal grouping; Toeplitz block parametric sharing is used for further parameter efficiency [2206.06292].
- **Hierarchical spatiotemporal mixing:** HSTMixer employs a sequence of bottom-up (temporal aggregation) and top-down (temporal upsampling) mixers, combined with adaptive region-wise mixers for spatial specificity [2512.07854].
- **Wavelet and multi-resolution:** WPMixer decomposes each variable’s time series into wavelet sub-bands, mixes each at its temporal resolution with per-branch MLPs, and reconstructs by inverse wavelet transform [2412.17176].
- **Nonlinear, universal mixing (KANs):** TSKANMixer replaces MLP with Kolmogorov–Arnold networks, parameterizing each edge via B-splines; this structure deploys per-coordinate learnable non-linearities to capture intricate multi-lag interactions at scale, at the expense of higher computational cost [2502.18410].
- **Pre-training enhanced mixers:** PreMixer incorporates patchwise masked MLP pre-training for robust representation learning, combining reconstruction and contrastive loss in the patch-embedding stage, followed by global time-mixing in downstream forecasting [2412.13607].

## 4. Comparative Efficiency, Scalability, and Performance

Time-mixing MLP architectures are designed to offer competitive or superior prediction accuracy with substantially reduced computational complexity compared to attention-based (transformer) or convolutional models, particularly for long sequences.

- **Parameter and compute complexity:** TriMLP’s mixer implements $O(T^2d)$ MACs per layer (for $T=128$, $d=128$, two weight matrices totalling $\approx$0.03M parameters), compared to SASRec’s 0.4M parameters and $\sim$26G MACs per batch [2305.14675]. WPMixer achieves sub-linear GFLOPs relative to TimeMixer or self-attention due to patching and downsampling [2412.17176].
- **Linear scaling:** HSTMixer and PreMixer exhibit $O(NTD)$ complexity (nodes, time steps, embedding dim), versus $O((NT)^2)$ for full attention; HSTMixer achieves 4.5h end-to-end training on GBA (2352 nodes) versus 7–59h for transformer/GNN baselines [2512.07854, 2412.13607].
- **Empirical benchmarks:** On sequential recommendation, TriMLP gains mean 14.9% Hit Rate & NDCG over SASRec/NextItNet with 8.7% reduced inference time. On large-scale traffic forecasting, HSTMixer outperforms 20 baselines by 4.41% MAE/RMSE, and RPMixer surpasses graph neural network models on GBA (MAE 19.06 vs 20.71) and CA (MAE 17.50 vs 19.86 for TSMixer baseline) [2305.14675, 2512.07854, 2402.10487].
- **Trade-offs:** Pure MLP time-mixing is highly efficient but may lack data-adaptive weighting of remote time steps, although hybridizations (e.g., adaptive region mixing, KANs, random projections) partially address this expressivity limitation.

## 5. Domain-Specific Adaptations and Applications

Time-mixing MLPs have been extensively deployed in:

- **Time series forecasting:** TSMixer, PreMixer, HSTMixer, WPMixer, and TSKANMixer operate on short/long univariate or multivariate time-series windows—using patching, global token mixing, hierarchical windows, or universal nonlinear mixing—consistently out-performing or matching transformer baselines at a fraction of compute [2303.06053, 2412.13607, 2512.07854, 2412.17176, 2502.18410].
- **Sequential recommendation:** TriMLP leverages causal triangular mixing to recover both user’s fine-grained item preferences and long-range behavioral trends under the standard auto-regressive paradigm [2305.14675].
- **Large-scale spatial-temporal prediction:** HSTMixer and RPMixer apply adaptive region/time mixing and blockwise diversity via random projections to accurately predict urban traffic at city-scale node graphs ($N \sim 10^4$) [2512.07854, 2402.10487].
- **Video recognition:** Grouped Time Mixing enables 3D-MLP architectures (MLP-3D) to process video clips with reduced parameter count compared to 3D CNNs or transformers, leveraging blockwise/strided mixing to capture both short-term dynamics and inter-frame semantics; accuracy is competitive with SoTA vision models [2206.06292].

## 6. Practical Considerations and Limitations

- **Positional encoding:** Masked or patchwise time mixing often obviates the need for explicit positional embeddings; TriMLP and some mixer models rely solely on mask-induced chronology [2305.14675].
- **Initialization:** Proper initialization (e.g., "1-0 init" for triangular masks) and normalization (e.g., row-wise softmax, batch/layer norm) are critical for stable training [2305.14675, 2412.17176].
- **Dropout and regularization:** High dropout rates and residual connections help mitigate overfitting, particularly for uniformly shared time-mixing weights [2303.06053, 2502.18410].
- **Expressivity:** Fixed weight time-mixing MLPs are inherently less expressive than data-dependent attention, although hybrid architectures (e.g., KANs, adaptive region mixers) increase representational power at increased cost [2502.18410].
- **Efficiency–accuracy trade-off:** For long look-back or large node counts, time-mixing MLPs outperform attention mechanisms computationally; however, for modalities demanding dynamic time-step relevance, attention may retain some advantage.

## 7. Future Directions and Research Frontiers

Research on time-mixing MLPs continues along several axes:

- **Adaptive mixing mechanisms:** Ongoing work explores model structures enabling dynamically learned attention-like mechanisms atop static MLP mixing, as exemplified by adaptive region mixers and spline-based nonlinearities [2512.07854, 2502.18410].
- **Multi-resolution and wavelet integration:** Techniques combining multi-resolution signal processing (wavelet decomposition) with patch mixing are likely to enable even more scalable handling of ultra-long sequences [2412.17176].
- **Hybridization with non-MLP modules:** Random projections, ensemble structures, and KANs suggest that time-mixing MLPs can be further hybridized for efficiency–expressivity balance [2402.10487, 2502.18410].
- **Application diversity:** Time-mixing MLPs have demonstrated generality from traffic, industrial, and retail forecasting to video understanding and sequential recommendation, motivating further study into architecture adaptation for emerging temporal domains.

In summary, time-mixing MLPs constitute a highly active research area that extends the MLP-mixer paradigm to temporal and spatiotemporal data, offering attractive trade-offs of scalability, simplicity, and competitive accuracy via explicit, often masked or grouped, mixing along the time axis. Their design continues to evolve in response to domain-specific challenges and the need for modeling increasingly complex, large-scale sequential phenomena [2305.14675, 2303.06053, 2502.18410, 2512.07854, 2206.06292, 2412.13607, 2412.17176, 2402.10487].

Source: https://www.emergentmind.com/topics/time-mixing-mlps