Temporal Pseudo-Gaussian Augmented Self-Attention

• The paper introduces TPS, which augments standard self-attention by fusing dot-product and pseudo-Gaussian kernels to capture temporal structure.
• It leverages both parameterized and clock-based mechanisms to enforce smoothness, causality, and near-diagonal alignment in sequence models.
• Empirical results show TPS improves accuracy by up to 11 points on multivariate time series benchmarks and enhances alignment in speech and video tasks.

Temporal Pseudo-Gaussian Augmented Self-Attention (TPS) is an architectural modification for self-attention mechanisms that explicitly introduces smooth, causal, and near-diagonal alignment biases into neural sequence models. TPS augments standard scaled dot-product attention with content-dependent or feature-driven Gaussian-like kernels—referred to as pseudo-Gaussian kernels or meeting kernels—enabling intrinsic modeling of temporal structure and relative positions. The design space includes both parameterized variants suitable for multivariate time series classification (MTSC) and parameter-free, clock-based mechanisms for sequence-to-sequence alignment in domains such as speech, video, and temporal signal processing (Abbasi et al., 2023, Soh et al., 18 Sep 2025).

1. Mathematical Formulation and Core Mechanisms

TPS operates by fusing two forms of attention matrices: the standard scaled dot-product and a pseudo-Gaussian kernel.

For a sequence of length $N$ and feature dimension $d$:

• Compute the projections:

$Q = F W_Q,\quad K = F W_K,\quad V = F W_V,$

with $F\in\mathbb R^{N\times d}$, $W_Q, W_K, W_V\in\mathbb R^{d\times d}$.

• Standard attention computes:

$$A_{\rm dot} = \mathrm{Softmax}\left(\tfrac{1}{\sqrt d}QK^{T}\right).$$

• The pseudo-Gaussian matrix is constructed by learning, for each position $i$:

$$\hat\sigma_i = |\mathbf{W}' v_i| + b,\quad \sigma_i = |\mathbf{W} v_i| + b,$$

where $v_i\in\mathbb{R}^d$ is the $i$-th row of $V$, $\mathbf{W}, \mathbf{W}'\in\mathbb{R}^{1\times d}$, $b\geq 0$. The kernel is then

$$A_{\rm pg}[i,j]= \begin{cases} \exp\left(-\tfrac{(i-j)^2}{2\hat\sigma_i^2}\right)& j<i,\ \exp\left(-\tfrac{(i-j)^2}{2\sigma_i^2}\right)& j\geq i, \end{cases}$$

row-normalized such that $\sum_j A_{\rm pg}[i, j]=1$.

• The final TPS matrix is

$$A_{\rm TPS} = \mathrm{RowNormalize}\left(\tfrac{1}{2}(A_{\rm dot}+A_{\rm pg})\right),$$

and the output is $O_{\rm TPS} = A_{\rm TPS} V$.

This construction provides an explicit mechanism for learning content-aware, asymmetric bandwidths that encode how far a given position integrates temporal context.

In contrast, the clock-based TPS formulation (Soh et al., 18 Sep 2025) bypasses learned positional parameters by computing monotonic "clocks" $\lambda$ for source and target sequences using a fixed positive nonlinearity (e.g., softplus):

• For a sequence indexed by $s\in[0,S]$, the normalized clock is

$$\lambda^X_s = \frac{\int_{0}^{s}\phi(\eta^X_{u})\,du}{\int_{0}^{S}\phi(\eta^X_{u})\,du},$$

with $\eta^X_s = W_q x_s$.

• The meeting kernel that governs attention is computed as a Gaussian in clock space:

$$S_{s,t} = -\alpha \frac{(\lambda^x_s - \lambda^y_t )^2}{2\Sigma_{s,t}^2},$$

where $\Sigma_{s,t}^2$ encodes variance contributions (e.g., from Brownian bridge surrogates).

This formally encodes monotonicity, continuity, and (optionally) causality into the attention weights, with only minimal changes to standard implementations.

2. Relative Position Injection and Inductive Biases

TPS deliberately modifies the way relative positional information enters the attention mechanism:

• In parameterized TPS (Abbasi et al., 2023), the two learned projections $(\mathbf{W},\mathbf{W}')$ from value-vectors $v_i$ serve as adaptive, content-dependent forward and backward bandwidths, encoding how widely a given position's attention kernel spreads over its context. This establishes a strong, asymmetric bias in favor of local (or near-diagonal) temporal dependencies.
• The pseudo-Gaussian kernel is inherently asymmetric and relative—it depends on $i-j$ and the content of $v_i$. Backward and forward context widths can be independently learned.
• In clock-based TPS (Soh et al., 18 Sep 2025), position is encoded as a monotonic, nonnegative clock driven by softplus-positive projections of hidden states, eliminating the need for externally imposed sinusoidal or learned absolute positional embeddings.

Intrinsic biases introduced by TPS include:

• Strict monotonicity in temporal alignment via the construction of the clocks.
• Causality, enforced by masking and the construction of $\lambda$ (especially in autoregressive/unnormalized clock modes).
• Smoothness, as the kernel penalizes large jumps in clock-space.
• Preference for near-diagonal alignments, naturally favoring temporal correspondence and discouraging nonlocal attention unless strongly supported by feature content.

3. Algorithmic Workflow and Integration with Transformers

TPS is implemented within Transformer architecture as a drop-in replacement or augmentation for multi-head attention:

• Per head, separate projection matrices are learned for the dot-product and pseudo-Gaussian streams.
• Each TPS block blends the dot-product matrix $A_{\rm dot}$ and the pseudo-Gaussian $A_{\rm pg}$, normalizes the result, and computes output via matrix multiplication with $V$.
• In multi-head attention, per-head TPS matrices are computed, outputs concatenated, and subjected to a linear projection, followed by the standard post-attention feed-forward and normalization steps.
• Standalone TPS blocks can be added atop the feature maps produced by backbone CNNs (e.g., FCN, ResNet, InceptionTime) or used in isolation as the core of a self-attentive classifier (Abbasi et al., 2023).

For clock-based TPS, normalized (parallel decoding) and unnormalized (autoregressive) clocks can be selected according to the modeling regime, using only standard query/key projections and softplus nonlinearity; no additional parameters are introduced beyond standard Transformer attention.

4. Computational Complexity and Scalability

• The computational complexity of TPS remains $O(N^2 d)$, with the additional $O(N^2)$ cost for assembling the pseudo-Gaussian or meeting-kernel matrix, and $O(N d)$ for bandwidth calculation; the latter is negligible in moderate-to-high $d$ regimes (Abbasi et al., 2023).
• Clock-based TPS similarly introduces only minor overhead compared to standard softmax attention, as squared clock distances and variance profiles can be efficiently computed via vectorized operations (Soh et al., 18 Sep 2025).
• Importantly, TPS can be integrated into existing Transformer implementations with minimal architectural disruption and no requirement for auxiliary losses or bespoke positional regularizers.

5. Empirical Performance and Theoretical Observations

When applied to MTSC and sequence alignment problems, TPS demonstrates consistently improved empirical accuracy:

Model/Setting	Baseline (%)	+TPS (%)	Δ (%)
FCN (MTSC backbone)	71.3	74.9	+3.6
ResNet (MTSC backbone)	71.2	73.3	+2.1
InceptionTime	75.1	77.4	+2.3
Standalone Transformer:
Dot-product SA	61.7	-	-
+learnable PE	67.2	-	-
+TPS (no PE)	70.4	-	+8.7
+TPS + PE	72.7	-	+11.0

TPS thus offers improvements of 2–4 accuracy points on strong CNN backbones and a substantial 11-point boost relative to standard self-attention in standalone setups on the 30-dataset UEA benchmark (Abbasi et al., 2023). Ablation studies confirm the necessity of both the content-injected pseudo-Gaussian component and, optionally, explicit positional encodings.

Clock-based TPS consistently yields sharper, smoother, and more stable alignments, maintaining robustness to global time-scaling and providing intelligible outputs in autoregressive settings even where standard dot-product based aligners degrade (Soh et al., 18 Sep 2025).

6. Practical Applications and Modeling Regimes

TPS is applicable to a broad class of sequence modeling domains where ordered or continuous temporal structures are intrinsic:

• Multivariate time series classification, where TPS serves as a drop-in enhancement to backbone CNNs or as a lightweight replacement for full positional attention in Transformer models (Abbasi et al., 2023).
• Sequence-to-sequence alignment problems in continuous domains, such as text-to-speech, audio-to-frame, and video alignment, where explicit monotonicity and near-diagonal inductive biases are beneficial (Soh et al., 18 Sep 2025).
• TPS supports both parallel (normalized clock) and autoregressive (unnormalized clock) decoding, seamlessly meeting requirements for global and local alignment constraints.

Empirical work has demonstrated the specific utility of TPS in Transformer text-to-speech testbeds, including improved alignment stability, robustness to duration scaling, and match or surpass state-of-the-art accuracy relative to classical scaled dot-product attention with much less parameter overhead (Soh et al., 18 Sep 2025).

7. Extensions and Future Directions

Several extensions and open questions are articulated in the literature:

• Hierarchical and multi-scale clocks for capturing heterogeneous tempos or multi-resolution alignment.
• Application of the meeting-kernel as a guidance term in continuous-time generative modeling frameworks (e.g., diffusion flows).
• Broader generalization to event streams, multimodal signals, and time-warped data where local continuity may interplay with global structure.
• Use of monotonic biases in text generation tasks to enforce coherence over local sequence windows.
• Further exploration of the trade-offs between explicit, parameterized bandwidths and the implicit monotonic structure of parameter-free clock-based mechanisms (Soh et al., 18 Sep 2025).

In summary, Temporal Pseudo-Gaussian Augmented Self-Attention constitutes a principled, computationally efficient augmentation of standard attention mechanisms, providing a structurally robust and empirically validated inductive bias for temporally ordered modeling tasks in both classification and generative alignment domains (Abbasi et al., 2023, Soh et al., 18 Sep 2025).