---
title: Markovian Pre-trained Transformer (MPT)
url: https://www.emergentmind.com/topics/markovian-pre-trained-transformer-mpt
type: topic
---

# Markovian Pre-trained Transformer (MPT)

The Markovian Pre-trained Transformer (MPT) is a universal, transferable recommendation model that achieves state-of-the-art next-item recommendation performance through a novel pre-training regime on synthetic Markov chains and the use of a lightweight adaptor. MPT exploits the empirical finding that advanced sequential recommenders primarily utilize the most recent interaction—reflecting a Markovian, first-order dependency—while earlier interactions serve as non-sequential signals for inferring user identity. By pre-training a Transformer backbone on synthetic Markov chain data and adapting with minimal parameters, MPT combines scalability, universality, and competitive accuracy for sequential recommendation tasks [2601.08275].

## 1. Markovian Principle in Sequential Recommendation

A crucial empirical insight guiding MPT is the “Markovian” nature observed across advanced sequential recommendation models. Given a user interaction sequence $[v_1,\ v_2,\dots,v_t]$, typical models estimate $\Pr(v_{t+1}\mid v_t, v_{t-1},\dots,v_1;\Theta)$. However, experiments demonstrate that shuffling all but the last interaction $\{v_1,\dots,v_{t-1}\}$ hardly degrades next-item prediction accuracy; in contrast, shuffling the entire sequence reduces performance, but the result still relies disproportionately on the final item.

This empirical property is formalized as:
$$
\Pr(v_{t+1}\mid v_t,v_{t-1},\ldots,v_1) \approx \Pr\bigl(v_{t+1}\mid v_t,\{v_1,\dots,v_{t-1}\}\bigr),
$$
i.e., next-item prediction is nearly first-order Markovian, where $v_t$ captures short-term intent and $\{v_1,\dots,v_{t-1}\}$ aggregates into a user profile. Thus, sequence modeling in recommendation decomposes into two capabilities:
1. Summarizing general user preferences from historical interactions (profile).
2. Emphasizing the last interaction for immediate next-item prediction [2601.08275].

## 2. Model Architecture and Pre-training

### 2.1 Transformer Backbone

The MPT employs a standard Transformer architecture:
- **Input Encoding:** Each state $s_t$ is projected into a continuous embedding space.
- **Positional Encoding:** Standard absolute or rotary embeddings inject sequence position.
- **Multi-Head Self-Attention (MHSA):**
  $$
  \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V
  $$
  used in $L$ stacked layers with $h$ attention heads.
- **Feed-Forward Network (FFN):** Per-layer FFN comprised of two linear projections and a nonlinearity (e.g., GeLU).
- **Residual/RMSNorm:** Each block is equipped with RMSNorm and residual connections.
- **Typical hyperparameters:** $L = 4$, $d = 256$, $h = 2$ [2601.08275].

### 2.2 Markov Chain Pre-training

MPT is pre-trained exclusively on synthetic Markov chains:
- **Trajectory Generation:** Infinite-length trajectories $(s_1,\dots,s_T)$ are generated via random $|\mathcal S|\times|\mathcal S|$ transition matrices $\mathbf P$; each row $\mathbf p_i$ follows $\text{Dir}(\alpha,\dots,\alpha)$ with $\alpha\approx 0.05$ for sparsity.
- **Trajectory length:** Fixed $T=1024$.
- **Input diversity:** Each trajectory employs a fresh random orthogonal projection from one-hot states to vector inputs.
- **Objective:** Next-State Prediction (NSP),
  $$
  \mathcal{L}_{\mathrm{NSP}}(\Theta) = \mathbb{E}_{\mathbf P\sim\mathrm{Dir}(\alpha)}\; \mathbb{E}_{(s_t)\sim \mathbf P} \left[-\sum_{t=1}^{T-1} \log \Pr(s_{t+1}|s_t,\dots,s_1;\Theta)\right],
  $$
  effectively enforcing the model to estimate state transition probabilities and to sharply attend to the current state.

Pre-training on synthetic Markov data offers architectural universality and exposes the model to unlimited, controlled trajectories [2601.08275].

## 3. Fine-tuning via Lightweight Adaptor

### 3.1 Adaptor Module

For alignment with recommendation tasks and datasets, MPT introduces a lightweight input adaptor while keeping the pre-trained backbone frozen. The adaptor, inserted at each time step, consists of:
- RMSNorm
- Linear $\rightarrow$ LeakyReLU
- Linear

This adaptor maps item semantic features (e.g., text embeddings) into the token space of MPT. The additional parameter count $\phi$ is only a few percent of the frozen parameter set $\Theta$ [2601.08275].

### 3.2 Fine-tuning and Prediction

Fine-tuning maximizes the likelihood of the next item:
$$
\mathcal{L}_{\mathrm{NIP}}(\Theta,\phi) = -\sum_{(u, v_1, \dots, v_t)\in\mathcal{D}_\mathrm{rec}} \log \Pr\left(v_{t+1}\mid v_t,\dots,v_1,u; \Theta,\phi\right)
$$
where $\Pr(\cdot)$ is implemented as a cosine similarity with a learnable temperature. With $\Theta$ frozen, only $\phi$ is trained, encouraging universal adaptation to varying item spaces while retaining generalizable sequence summarization and last-item emphasis [2601.08275].

## 4. Synthetic Data Generation

MPT’s pre-training regime leverages the following synthetic Markov chain configuration:
- **State space:** $|\mathcal S|\approx 30$
- **Dirichlet $\alpha$:** $0.05$ to induce sparse transitions
- **Trajectory Length:** $T=1024$
- **Input transformation:** New random orthogonal projections per trajectory to prevent memorization
- **Scale:** On-the-fly trajectory generation for approximately 10 billion tokens.

This pipeline enables unlimited, diverse trajectories, controlling transition sparsity and state-space cardinality to match real-world recommendation environments [2601.08275].

## 5. Empirical Performance and Analysis

### 5.1 Datasets and Evaluation Metrics

MPT is evaluated on five public datasets:

| Dataset        | Users   | Items   | Interactions | Avg. Length |
|----------------|---------|---------|--------------|-------------|
| Beauty         | 22,363  | 12,101  | 198,502      | 8.9         |
| Toys           | 19,412  | 11,924  | 167,597      | 8.6         |
| Sports         | 35,598  | 18,357  | 296,337      | 8.3         |
| Yelp           | 77,277  | 45,638  | 2,103,896    | 27.2        |
| Online Retail  | 16,520  | 3,469   | 519,906      | 31.5        |

Metrics: HR@$K$, NDCG@$K$ ($K$ = 10, 20) [2601.08275].

### 5.2 Baselines and Results

Benchmarked methods:
- **From-scratch:** GRU4Rec, SASRec, FMLPRec, HSTU, SASRec+
- **Pre-trained recommenders:** UniSRec, RecFormer
- **Language models:** E4SRec (LLaMA-2-based), Qwen2.5-7B

MPT+Adaptor achieves, on average,
- $\approx 12\%$ relative gain over the strongest from-scratch model (SASRec+)
- $\approx 35\%$ gain over the best pre-trained baseline

For instance, Beauty (NDCG@10):

| Model        | NDCG@10  |
|--------------|----------|
| SASRec+      | 0.0591   |
| UniSRec      | 0.0382   |
| MPT+Adaptor  | 0.0673   |

(+13.8% vs. SASRec+, +43.3% vs. UniSRec). Comparable gains are reported across Toys, Sports, Yelp, and Online Retail [2601.08275].

## 6. Ablations and Behavioral Analysis

Ablation experiments support the centrality of Markovian pre-training:
- **Pre-training effect:** Removing pre-training reduces the model to SASRec+, while including it yields consistent $10$–$15\%$ improvements.
- **Fine-tuning:** Adaptor-only tuning suffices in most datasets. LoRA-based strategies may overfit, except on Yelp where LoRA offers marginal benefit.
- **Synthetic sequence parameters:** $T\approx 1024$ (trajectory length) and $|\mathcal S|\approx30$ (state size) are empirically optimal. Sparsity ($\alpha=0.05$) matches real-world data.
- **Attention analysis:** Lower layers of MPT maintain sharp diagonal attention—prioritizing the most recent interaction, in contrast to pre-trained LLMs, which produce diffuse, less effective attention for recommendation [2601.08275].

## 7. Contributions, Limitations, and Prospects

MPT’s primary contributions:
- Empirical demonstration that advanced sequential recommenders display first-order, Markovian behavior.
- Introduction of MPT: a Transformer pre-trained solely on synthetic Markov chains, yielding universal sequence summarization and robust short-term preference modeling.
- State-of-the-art performance with minimal adaptation, exceeding both specialist recommenders and large LLM-based systems at lower cost.

Noted limitations include:
- Restriction to first-order dynamics and absence of higher-order or side-channel information in pre-training.
- Occasional requirement for more flexible parameter-efficient adaptation (e.g., LoRA) in complex domains.
- Underfitting in domains with highly specialized or extreme user behaviors.

Future research directions include:
- Augmenting synthetic data to reflect higher-order dependencies, rich side-info, or variable Markov orders.
- Synthetic user simulation via LLM-based agents for more realistic data.
- Extending pre-training to multi-task collaborative settings (ratings, multi-behavior, reviews).
- Theoretical investigation of scaling laws and Bayes-optimality beyond first-order Markov processes [2601.08275].

Source: https://www.emergentmind.com/topics/markovian-pre-trained-transformer-mpt