---
title: Minimal Predictive Sufficiency SSM
url: https://www.emergentmind.com/topics/minimal-predictive-sufficiency-ssm-mps-ssm
type: topic
---

# Minimal Predictive Sufficiency SSM

The Minimal Predictive Sufficiency State Space Model (MPS-SSM) is a sequence modeling framework whose content-selective state gating is derived from a first-principle information-theoretic criterion. MPS-SSM builds on the principle that the model’s hidden state should be a minimal sufficient statistic of the past for predicting the future. This results in a model that maximally compresses historical context, learns to ignore non-causal information, and exhibits robustness and accuracy across long-horizon and noisy forecasting scenarios [2508.03158].

## 1. Principle of Predictive Sufficiency

The central theoretical construct underlying MPS-SSM is the principle of predictive sufficiency. For a sequence $(U_{1:t}, Y_{t:t+\tau})$ where $U_{1:t}$ is the observed history and $Y_{t:t+\tau}$ denotes a segment of future targets, MPS-SSM demands that the hidden state $h_t$ at every time $t$ satisfies two criteria:
- **Predictive sufficiency**: The hidden state $h_t$ must retain all information in $U_{1:t}$ relevant for predicting $Y_{t:t+\tau}$; formally, $I(h_t;Y_{t:t+\tau}) = I(U_{1:t};Y_{t:t+\tau})$.
- **Minimality**: Among all statistics satisfying sufficiency, $h_t$ should minimize $I(U_{1:t};h_t)$, i.e., $I(U_{1:t};h_t) \le I(U_{1:t};h_t')$ for any $h_t'$ also satisfying sufficiency.

Collectively, these constraints characterize $h_t$ as a minimal predictive sufficient statistic and can be formalized by the optimization problem:
\[
\min_{p(h_t\mid U_{1:t})} I(U_{1:t};h_t) \quad \text{s.t.} \quad I(h_t;Y_{t:t+\tau}) = I(U_{1:t};Y_{t:t+\tau}).
\]
This setup ensures that the hidden state captures only the causal structure necessary for accurate sequence prediction and discards spurious or non-predictive variability.

## 2. MPS-SSM Objective Function Derivation

Directly enforcing the constraint $I(h_t;Y_{t:t+\tau}) = I(U_{1:t};Y_{t:t+\tau})$ is intractable, so MPS-SSM introduces a relaxed Lagrangian objective. The predictive sufficiency criterion is represented by a standard prediction loss,
\[
\mathcal{L}_\mathrm{Pred} = \frac{1}{T\tau} \sum_{t=1}^T \sum_{i=1}^\tau \mathbb{E}_{p(h_t \mid U_{1:t})} [\|\hat{y}_{t+i}(h_t) - y_{t+i}\|^2],
\]
while the minimality term is realized as an information-theoretic regularizer,
\[
\mathcal{L}_\mathrm{Min} = \frac{1}{T} \sum_{t=1}^T I(U_{1:t};h_t).
\]
The total objective becomes
\[
\mathcal{L}_\mathrm{Total} = \mathcal{L}_\mathrm{Pred} + \lambda\,\mathcal{L}_\mathrm{Min}
\]
with $\lambda>0$ balancing prediction performance and information compression.

As direct computation of $I(U_{1:t};h_t)$ is intractable, MPS-SSM employs a variational upper bound using a decoder $q_\theta(u_t|h_t)$:
\[
I(U_{1:t};h_t) \leq \mathbb{E}_{p(U_{1:t},h_t)}[-\log q_\theta(u_t | h_t)] + \mathrm{const},
\]
enabling practical and stable optimization via backpropagation with
\[
\mathcal{L}_\mathrm{Min} \approx \frac{1}{T} \sum_{t=1}^T [-\log q_\theta(u_t | h_t)].
\]

## 3. Architecture and Training Methodology

MPS-SSM extends a content-selective SSM backbone—such as Mamba—by integrating several key modules:
- **Selection Gate $G_\phi(u_t)$**: Computes adaptive state-space parameters $(\Delta_t, B_t, C_t)$ conditioned on each input $u_t$.
- **SSM Recurrence**: The core transition follows
  \[
  h_t = A(\Delta_t) h_{t-1} + B_t u_t, \quad \hat{y}_{t+1} = C_t h_t
  \]
  where $A(\Delta_t) = \exp(\Delta_t A)$ is approximated via zero-order hold or NPLR techniques.
- **Minimality Module**: A lightweight decoder $q_\theta(u_t|h_t)$ reconstructs $u_t$ from $h_t$ to facilitate the variational information regularization.
- **Prediction Head**: Projects $h_t$ into target predictions $\{\hat{y}_{t+i}\}_{i=1}^\tau$.

Training is conducted over entire unrolled sequences, jointly optimizing $\phi$ (gate), $\theta$ (decoder), and SSM matrices to minimize $\mathcal{L}_\mathrm{Total}$ with standard first-order methods. The entire process is efficiently scalable and practical for large-scale time-series tasks.

**Training Workflow Table**

| Step                   | Operation                                   | Output           |
|------------------------|---------------------------------------------|------------------|
| Selection Gate         | $(\Delta_t,B_t,C_t) \leftarrow G_\phi(u_t)$ | Adaptive params  |
| SSM Recurrence         | $h_t \leftarrow A(\Delta_t)h_{t-1} + B_tu_t$| Hidden state     |
| Prediction             | $\hat{y}_{t+i} \leftarrow C_t h_t$          | Forecasted values|
| Minimality Module      | $-\log q_\theta(u_t|h_t)$                   | Info loss        |
| Backpropagation        | $\nabla \mathcal{L}_\mathrm{Total}$         | Parameter update |

## 4. Empirical Results and Robustness Analysis

MPS-SSM has been evaluated on established sequence modeling and forecasting benchmarks, including ETT (ETTh1/2, ETTm1/2), Weather, Electricity, Traffic, and Exchange, across forecast horizons (96, 192, 336, 720) and measured via MSE and MAE.

Key findings include:
- **Optimal Regularization ($\lambda$) Sensitivity**: Each dataset and horizon displays a “sweet-spot” $\lambda$ (e.g., ETTh1: $\lambda \approx 2.0$; Weather: $\lambda \approx 0.5$; ETTm2: $\lambda \approx 100$), and the optimal $\lambda$ increases with forecast length.
- **State-of-the-Art Accuracy**:
  - On ETTh1 (96), MPS-SSM achieves MSE = 0.375, second only to PatchTST (0.360).
  - On ETTm2 (96), MSE = 0.165, outperforming PatchTST (0.224).
  - On Electricity (96), MSE = 0.151 (vs. next-best 0.225).
  - On long horizons, MPS-SSM routinely ranks best or second-best.
- **Robustness to Noise**: Under impulse noise perturbations to inputs, increasing $\lambda$ monotonically reduces forecast error degradation; at $\lambda=100$, degradation is approximately threefold lower than at $\lambda=0$. This empirically validates the theoretical prediction that MPS-SSM is resilient to non-causal spurious input patterns.

## 5. Generalization to a Regularization Framework

The MPS principle is not restricted to SSMs and can be instantiated as a model-agnostic regularizer for any sequential architecture. This extension involves:
1. Selecting an internal representation $z_t$ (e.g., an SSM state, Transformer embedding, or linear hidden vector).
2. Attaching a lightweight decoder $q(u_t|z_t)$.
3. Adding the minimality regularization term $\mathcal{L}_\mathrm{Min} = \frac{1}{T} \sum_t [-\log q(u_t | z_t)]$ to the base task loss.

This general regularization strategy takes the form:
\[
\mathcal{L}_\mathrm{Total} = \mathcal{L}_\mathrm{Task}(f) + \lambda \frac{1}{T}\sum_{t=1}^T[-\log q_\psi(u_t|z_t)]
\]
where $f$ denotes the original task model.

Empirical evidence demonstrates utility across architectures such as Mamba (MPS-Mamba), linear models (MPS-DLinear), and Transformers (MPS-PatchTST), with consistent improvements on ETT and other datasets (e.g., MPS-PatchTST achieves ETTh1/96 MSE=0.328 compared to 0.360 for vanilla PatchTST).

## 6. Significance and Implications

MPS-SSM is the first selective SSM whose gating is derived from the information-theoretic requirement that hidden states encode the minimal predictive sufficient statistic. The resulting mutual information penalty confers both empirical state-of-the-art generalization and robustness properties, notably resistance to non-causal and spurious noise. Furthermore, the principle’s generality enables its adoption as an effective regularizer in architectures beyond SSMs, including popular sequence models such as Transformers and linear models [2508.03158]. A plausible implication is the emergence of a new paradigm for designing sequential models grounded in first principles rather than heuristic mechanism design.

Source: https://www.emergentmind.com/topics/minimal-predictive-sufficiency-ssm-mps-ssm