---
title: Physics-Guided Tiny-Mamba Transformer
url: https://www.emergentmind.com/topics/physics-guided-tiny-mamba-transformer-pg-tmt
type: topic
---

# Physics-Guided Tiny-Mamba Transformer

The Physics-Guided Tiny-Mamba Transformer (PG-TMT) is a compact, tri-branch encoder architecture designed for reliability-aware early fault warning in rotating machinery under nonstationary conditions, domain shifts, and severe class imbalance. PG-TMT integrates physically guided priors—explicit temporal-to-spectral mappings aligned with mechanical defect frequencies—into a fusion of depthwise-separable convolution, state-space modeling, and attention-based resonance capture. Decision reliability is ensured through extreme-value theory (EVT) calibrated thresholds and hysteretic alarm logic. Evaluation across public and industrial datasets demonstrates competitive precision-recall metrics, timeliness, robust transfer, and deployment feasibility [2601.21293].

## 1. Tri-Branch Encoder Architecture

PG-TMT processes online windows of multichannel vibration signals, $\mathbf{x}_t\in\mathbb{R}^{C\times L}$, to produce a calibrated anomaly score $s_t\in[0,1]$ at each time $t$ (hop $h\ll L$, batch-size 1). The encoder is organized into three complementary branches:

- **Depthwise-Separable Convolutional Stem (Micro-Transients):**  
A cascade of causal 1D depthwise convolutions (kernel size $k$, optional dilation $\delta$) is followed by per-channel pointwise ($1\times1$) convolutions. At each layer $\ell$, for input $\mathbf{z}^{(\ell)}\in\mathbb{R}^{C\times L}$,
  $$
  \tilde{\mathbf{z}}^{(\ell)}_{c,*} = \mathrm{Conv1D}\bigl(\mathbf{z}^{(\ell)}_{c,*};\,k,\delta\bigr),\quad
  \mathbf{z}^{(\ell+1)}_{f,*} = \sum_{c=1}^C w^{(\ell)}_{f,c}\,\tilde{\mathbf{z}}^{(\ell)}_{c,*}.
  $$
  The receptive field, $\mathrm{RF}_{\mathrm{stem}} = 1 + \sum_{\ell=1}^{L_{\text{stem}}} (k-1)\delta_\ell$, is tuned for sub-millisecond impact-like transients. Output: $\mathbf{y}^{\rm conv}_t\in\mathbb{R}^{d_c}$.

- **Tiny-Mamba State-Space Branch (Long-Range Dynamics):**  
A gated, linear state-space model captures near-linear degradation over hundreds or thousands of timesteps:
  $$
  \mathbf{h}_{t+1} = \mathbf{A}(\mathbf{g}_t)\,\mathbf{h}_t + \mathbf{B}(\mathbf{g}_t)\,\mathbf{u}_t,\quad
  \mathbf{y}^{\rm ssm}_t = \mathbf{C}\,\mathbf{h}_t.
  $$
  Here, $\mathbf{u}_t$ is a channel-reduced input, $\mathbf{h}_t$ is the latent state, and $\mathbf{g}_t$ are learned gates. Stability is enforced via $\mathbf{A}_c = -\operatorname{softplus}(\boldsymbol{\eta})$ with discretization:
  $$
  \mathbf{A} = \exp\bigl(\Delta\,\mathbf{A}_c\bigr),\quad
  \mathbf{B} = \bigl(\int_{0}^{\Delta} e^{\tau\,\mathbf{A}_c}\,d\tau\bigr)\,\mathbf{B}_c,
  $$
  ensuring $\Re(\lambda(\mathbf{A}_c))<0 \Rightarrow \rho(\mathbf{A})<1$. Output: $\mathbf{y}^{\rm ssm}_t\in\mathbb{R}^{d_s}$.

- **Local Transformer (Cross-Channel Resonances):**  
Self-attention is restricted to a causal window $\mathcal{N}_W(t) = \{t-W,\dots,t\}$ for each head $h$:
  $$
  \alpha_{t,i}^{(h)} =
  \frac{
      \exp\bigl(\langle \mathbf{q}_t^{(h)},\mathbf{k}_i^{(h)} \rangle / \sqrt{d_{\rm head}} \bigr)
  }{
      \sum_{j\in\mathcal{N}_W(t)}
      \exp\bigl(\langle \mathbf{q}_t^{(h)},\mathbf{k}_j^{(h)} \rangle/\sqrt{d_{\rm head}} \bigr)
  },\quad i\in\mathcal{N}_W(t),
  $$
  producing $\mathbf{y}^{\rm att}_t\in\mathbb{R}^{d_a}$ for cross-channel resonance encoding.

Branch outputs are concatenated, $\mathbf{z}_t = [\mathbf{y}^{\rm conv}_t \parallel \mathbf{y}^{\rm ssm}_t \parallel \mathbf{y}^{\rm att}_t]$, and fused by a gated residual:
$$
\boldsymbol{\gamma}_t = \sigma(\mathbf{W}_\gamma\,\mathbf{z}_t),\quad
\mathbf{r}_t = \boldsymbol{\gamma}_t\odot(\mathbf{W}_f\,\mathbf{z}_t) + (1-\boldsymbol{\gamma}_t)\odot\mathbf{z}_t.
$$
A local attention distribution $p_t(i)$, Jensen–Shannon discrepancy term, and a final score $s_t = \sigma(\kappa e_t + \beta)$ (with $e_t$ incorporating evidence and discrepancy) complete the inference pipeline.

## 2. Physically Guided Temporal–Spectral Alignment

PG-TMT imposes explicit temporal-to-spectral mapping by analytically connecting learned temporal attention to classical fault-order bands—frequencies determined by bearing geometry and shaft speed.

- **Spectral Attention:**  
Let $F_s$ be sampling rate, $f_k \in [0, F_s/2]$. Spectral attention is computed as
  $$
  \tilde A_t(f_k) = \Big| \sum_{i\in\mathcal{N}_W(t)} p_t(i) e^{-j2\pi f_k (i-i_0)/F_s} \Big|^2;\quad
  A_t(f_k) = \frac{ \tilde A_t(f_k) }{ \sum_\ell \tilde A_t(f_\ell) }.
  $$

- **Fault Orders and Band Mask:**  
Classical bearing defect frequencies:
  $$
  \begin{align*}
  \mathrm{BPFI} & = \frac{N_b}{2} f_r \left( 1 + \frac{d}{D_p} \cos \theta \right),\\
  \mathrm{BPFO} & = \frac{N_b}{2} f_r \left( 1 - \frac{d}{D_p} \cos \theta \right),\\
  \mathrm{BSF}  & = \frac{D_p}{2d} f_r \left[ 1 - \left(\frac{d}{D_p} \cos \theta\right)^2 \right],\\
  \mathrm{FTF}  & = \frac{1}{2} f_r \left( 1 - \frac{d}{D_p} \cos \theta \right).
  \end{align*}
  $$
For each primary order $f_j$, side-bands, and windowing parameters, a Gaussian mixture $M_t(f_\ell)$ masks the frequencies of interest.

- **Alignment Loss and Band-Alignment Score:**  
Smoothed spectral and mask distributions $\bar A_t$, $\bar M_t$ yield a physics-based alignment penalty:
  $$
  \mathcal{L}_{\rm align} = \lambda_{\rm align}\,\mathrm{KL}\bigl(\bar M_t \| \bar A_t\bigr) + \lambda_{\rm lap}\,\sum_k |\nabla_f \bar A_t(f_k)|
  $$
and a band-alignment score
  $$
  \rho_t = \sum_{k} A_t(f_k) M_t(f_k),\quad 0 \le \rho_t \le 1,
  $$
quantifying the physics-grounded plausibility of the model’s attention.

## 3. EVT-Calibrated Reliability-Aware Decision Logic

PG-TMT translates raw anomaly scores into calibrated, reliability-guaranteed alarms using an EVT-based extremal modeling of healthy-score exceedances.

- **Peaks-Over-Threshold Extreme-Value Modeling:**  
On calibration segments, scores above a high quantile $u$ are modeled via the generalized Pareto distribution (GPD), $Y = s_t - u | s_t > u \sim \mathrm{GPD}(\xi, \beta)$. Exceedance times approximate a Poisson process of rate $\lambda_u$. The on-threshold $\tau_{\rm on}$ meeting false alarm intensity $\lambda_{\rm FA}$ is
  $$
  \boxed{
  \tau_{\rm on} = u + \frac{\beta}{\xi} \Big[ (\lambda_u / \lambda_{\rm FA})^{\xi} - 1 \Big]
  }
  $$
with the limiting case $\xi \to 0$ yielding the logarithmic form.

- **Dual-Threshold Hysteresis and Hold Time:**  
To suppress spurious frame-level alarms, $\tau_{\rm off} = \tau_{\rm on} - \delta$, with minimal episode duration $T_{\min}$ and merging of episodes separated by less than $\Delta T_{\rm merge}$. The resulting alarm logic produces episodes whose empirical rate $\widehat\lambda_{\rm FA}$ tracks the prescribed $\lambda_{\rm FA}$, including under speed drift when $\tau_{\rm on}$ is RPM-adapted.

## 4. Experimental Design and Evaluation Protocols

Evaluation follows strict leakage-free, right-censored streaming protocols emphasizing reliable, domain-robust deployment.

- **Streaming Protocol:**  
Sliding windows of length $L$, hop size $h\ll L$, batch=1. A burn-in period $T_{\rm burn}$ initializes state. Data splits are disjoint at machine, load, speed, and sensor level, with no window crossing of split boundaries. Per-channel normalization is trained only.

- **Timeliness and Right-Censoring:**  
Detection time is censored if no alarm occurs before run end. Timeliness $\Delta^{(r)}= t_0^{(r)}-t_{\rm phys}^{(r)}$ is computed using Kaplan–Meier estimators, reporting mean/median MTTD with confidence intervals.

- **Datasets:**  
  - CWRU bearing data (speeds, loads, rigs)
  - Paderborn University (seeded faults, speed$\times$torque, cross-rig)
  - XJTU-SY run-to-failure (chronological splits)
  - Industrial pilot (in-service rotating machinery)

- **Metrics:**  
  - Precision–Recall AUC (PR-AUC) under severe class imbalance
  - ROC AUC
  - Mean time-to-detect (MTTD) at matched $\lambda_{\rm FA}$
  - Alarm intensity (episodes/hour, hysteresis+merge logic)
  - Cross-domain transfer: AUC and MTTD retention and gain under directed shifts and few-shot adaptation, using
    $$
    \mathrm{Retention}_{\rm AUC} = \frac{\mathrm{AUC}_t}{\mathrm{AUC}_s},\quad
    \mathrm{Retention}_{\rm MTTD} = \frac{\mathrm{MTTD}_s}{\mathrm{MTTD}_t}
    $$

## 5. Key Results and Ablation Findings

- **Detection Performance:**  
  - PR-AUC approximately 0.96–0.94 and ROC AUC 0.99–0.97 across CWRU/Paderborn/XJTU-SY (graceful degradation to 0 dB SNR).
  - Mean MTTD $\approx$ 28–33 s (clean), increasing to 49–61 s at SNR = 0 dB, at $\lambda_{\rm FA}\approx 0.2$ events/hour.
  - Empirical false-alarm intensity $\widehat\lambda_{\rm FA}$ remains within $\pm 0.05$ events/hour of target, stable under RPM drift.

- **Transfer Across Domains:**  
  - AUC retention $>$ 0.95 for cross-load/speed; MTTD retention $\approx$ 0.9; transfer across sensor/rig/dataset is robust.
  - Few-shot adaptation (1–5% labels) recovers nearly oracle performance.

- **Ablation and Latency:**  
  - Removing any encoder branch or physics prior degrades PR-AUC, increases FAR, or worsens MTTD.
  - Excluding EVT/hysteresis disrupts intensity matching and increases chatter.
  - Latency: median inference $\leq$ 10 ms (p50), $\leq$12 ms (p90/p99) on CPU/Jetson; model size 0.8M parameters, 0.28 GFLOPs.

## 6. Significance, Applications, and Interpretation

PG-TMT combines physically aligned representation learning with calibrated, interpretable, and operationally robust early fault warnings for reliability-centric prognostics and health management. Its fusion of transient detection, slow-trend modeling, cross-channel resonance capture, and analytic attention-band alignment is directly interpretable in terms of vibrational fault physics. The EVT-calibrated, hysteretic alarm logic provides explicit guarantees on false-alarm rates and episode integrity under nonstationary and imbalanced conditions. Demonstrated performance across public benchmarks and real-world pilots, together with robustness to domain shifts and low-SNR conditions, establishes PG-TMT as a deployment-ready solution for industrial rotating machinery monitoring [2601.21293].

Source: https://www.emergentmind.com/topics/physics-guided-tiny-mamba-transformer-pg-tmt