---
title: 'TriMLP: Efficient MLP for Sequential Recommendation'
url: https://www.emergentmind.com/topics/trimlp
type: topic
---

# TriMLP: Efficient MLP for Sequential Recommendation

TriMLP is an MLP-like architecture designed for sequential recommendation tasks, featuring a novel Triangular Mixer mechanism for efficient cross-token communication and rigorous prevention of information leakage under auto-regressive training. By imposing structured triangular masks on token-mixing weights and organizing mixing operations into global and local branches, TriMLP excavates long- and short-term user preference patterns without the complexity of self-attention or convolution, achieving a superior accuracy/efficiency trade-off across diverse benchmarks [2305.14675].

## 1. Architecture and Layer-Wise Design

TriMLP processes a user's historical sequence \( X \) of length \( n \), padded or truncated as necessary. The layerwise workflow is:

- **Embedding Layer:** Each token \( x_i \) is mapped to \( e_i \in \mathbb{R}^d \), yielding the embedded sequence \( X \in \mathbb{R}^{n \times d} \).
- **Triangular Mixer:** Two alternating MLP-style token-mixing layers:
  - *Global Mixing:* Captures long-range dependencies across the entire sequence.
  - *Local Mixing:* Extracts short-range patterns within fixed-length sub-sessions.
- **Classifier:** A linear output layer followed by softmax over the item vocabulary for every time step.
- **Training Mode:** Auto-regressive protocol, with input \( X[1..n-1] \) used to predict \( X[2..n] \).

The forward pipeline (omitting batch dimension) is:

```
X (n×d)      →      Embedding      →      TriangularMixer      →      Y (n×d)
                                                      ↓
                                  Linear + Softmax      →      P (n×|I|)
```

## 2. Triangular Mixer Mechanism

The Triangular Mixer introduces two essential learnable kernels: masked global and local mixing matrices, systematically preventing anti-chronological connections.

### 2.1 Masked Global Kernel

For sequential information preservation, the global mixer weight \( W^G \in \mathbb{R}^{n \times n} \) is masked to exclude tokens from future positions:

\[
(M_G)_{j,i} =
  \begin{cases}
    W^G_{j,i}, & j \leq i \\
    -\infty, & j > i
  \end{cases}
\]

The resulting kernel enforces strictly chronological cross-token aggregation, with column-wise softmax ensuring:

\[
A^G_{j,i} = \frac{\exp((M_G)_{j,i})}{\sum_{k=1}^i \exp((M_G)_{k,i})}
\]

### 2.2 Masked Local Kernel

Local mixing focuses on intra-session relationships. Let \( s \) be the number of sessions (each of length \( l \)), such that \( n = s \cdot l \), and \( p(j) = \lceil j / l \rceil \) denotes the session index:

\[
(M_L)_{j,i} =
  \begin{cases}
    W^L_{j,i}, & p(j) = p(i) \ \text{and} \ j \leq i \\
    -\infty, & \text{otherwise}
  \end{cases}
\]

By applying softmax in each session, local dependencies are captured without cross-session interference.

## 3. Mathematical Formulation

Let \( X_G = X^{\top} \in \mathbb{R}^{d \times n} \). The full mixing operations are:

\[
Y_G = \text{Act}(X_G \cdot \text{Softmax}(M_G)), \qquad
Y_L = \text{Act}(Y_G \cdot \text{Softmax}(M_L))
\]
\[
Y = (Y_L)^{\top}
\]

Where \( \text{Act}(\cdot) \) is a nonlinearity, e.g., GELU or SiLU. All softmaxes are performed across the token dimension.

This yields the compact formula:

\[
\text{TriMix}(X) = \left( \text{Act}(X^{\top} \text{Softmax}(M_G)) \cdot \text{Softmax}(M_L) \right)^{\top}
\]

## 4. Computational Complexity and Efficiency

TriMLP employs two masked MLP layers, each with \( O(n^2 d) \) multiply-accumulate operations (MACs). Relative to prevailing baselines:

| Model      | Complexity     | MACs   | Params | GPU Mem | Inf.Time | HR@5   |
|------------|---------------|--------|--------|---------|----------|--------|
| GRU4Rec    | \( O(nd^2) \) | 13.04G | 0.19M  | 2.5GB   | 1.17 s   | 0.1084 |
| NextItNet  | \( O(knd^2) \)| 26.03G | 0.40M  | 1.8GB   | 1.67 s   | 0.1277 |
| SASRec     | \( O(n^2d) \) | 25.94G | 0.40M  | 1.8GB   | 2.50 s   | 0.1172 |
| FMLP4Rec   | \( O(nd\log(nd)) \) | 17.18G | 0.26M | 2.1GB | 2.29 s | 0.0632 |
| TriMLP     | \( O(n^2d) \) | 2.15G  | 0.03M  | 1.4GB   | 0.89 s   | 0.1390 |

TriMLP exhibits an 83.5% reduction in MACs, 84% reduction in parameters, 23% reduction in GPU memory footprint, and 24% lower inference time compared to the strongest baseline, with an 8.9% gain in HR@5 [2305.14675].

## 5. Experimental Methodology and Performance

TriMLP was empirically validated on 12 datasets spanning Amazon, MovieLens, Tenrec, and LBSN, representing 50K to 10M interactions. Preprocessing excludes users with fewer than 20 interactions and items with fewer than 10, and all sequences are padded/truncated to a fixed length of \( n \) (range: 32–256). Evaluation metrics include HR@5/10 and NDCG@5/10, alongside averaged inference-time measurements.

Hyperparameters: embedding dim = 128, mixer hidden dim = \( n \), dropout = 0.5, Adam learning rate = 0.001, early stopping after 10 epochs without improvement. The experiments are conducted on a single 3090 GPU.

Across datasets, TriMLP yields up to 14.88% higher HR@10/NDCG@10 over state-of-the-art baselines, while reducing average inference time by 8.65%. Example scale-wise results:

| Scale  | SOTA HR@10        | TriMLP HR@10 | Δ      | Inf.Time ↓ |
|--------|-------------------|--------------|--------|------------|
| Tiny   | 0.1202 (FMLP4Rec) | 0.1256       | +4.5%  | −2.3%      |
| Small  | 0.0572 (SASRec)   | 0.0611       | +6.8%  | −0.4%      |
| Base   | 0.2111 (NextItNet)| 0.2343       | +11.1% | −6.7%      |
| Large  | 0.1907 (NextItNet)| 0.2027       | +6.3%  | −23.7%     |

## 6. Training Algorithm and Pseudocode

The core masking and mixing logic is encapsulated as follows:

```python
def generate_masks(n, l, s):
    M_G = torch.triu(torch.ones(n,n))       # upper-triangular mask
    session_mask = torch.zeros(n,n)
    for i in range(s):
        session_mask[i*l:(i+1)*l, i*l:(i+1)*l] = 1
    M_L = M_G * session_mask                # block-diagonal triangles
    return nn.Parameter(M_G), nn.Parameter(M_L)

class TriangularMixer(nn.Module):
    def __init__(self, n, l, s, act=GELU):
        super().__init__()
        self.M_G, self.M_L = generate_masks(n,l,s)
        self.act = act()
    def forward(self, X):
        X_G = X.permute(0,2,1)             # [B, d, n]
        A_G = (X_G @ self.M_G).softmax(dim=-1)
        Y_G = self.act(A_G)
        A_L = (Y_G @ self.M_L).softmax(dim=-1)
        Y_L = self.act(A_L)
        return Y_L.permute(0,2,1)          # [B, n, d]
```

High-level training loop:

```python
for each batch of sequences X[1..n]:
    E = Embed(X)                  # [B,n,d]
    Y = TriangularMixer(E)        # [B,n,d]
    P = Softmax(Y @ W + b)        # [B,n,|I|]
    loss = cross_entropy(P[:, :-1], targets[:,1:])
    backprop update
# Inference: use P[:, n-1] for next-item recommendations (top-K).
```

## 7. Properties, Limitations, and Prospective Extensions

**Strengths:**
- Restores chronological inductive bias to MLPs by triangular masking, eliminating information leakage.
- Two-branch global/local mixing structure enables modeling of both long- and short-term user preferences.
- Lightweight, memory-efficient, and highly parallelizable (no Q/K/V matrices or FFN structure).
- Omits explicit positional embeddings, relying entirely on masked mixing.

**Limitations:**
- Static mixing weights may underfit highly heterogeneous or irregular sequences due to input-agnostic design.
- Requires tuning of session-size hyperparameters (\( s \) or \( l \)) for optimal local mixing.
- Fixed input length \( n \); adaptation to variable-length streams necessitates additional modification.

**Future Directions:**
- Incorporate temporal intervals or contextual information into the masking schema.
- Explore data augmentation and pre-training strategies for sparse domains.
- Investigate dynamic (input-adaptive) masking techniques to enhance representation flexibility.
- Develop methods for handling variable-length sequences without mask retraining.

TriMLP revitalizes the simplicity and efficiency of MLP architectures for sequential recommendation, achieving parity or superiority with state-of-the-art methods while delivering substantial improvements in computational efficiency [2305.14675].

Source: https://www.emergentmind.com/topics/trimlp