Temporal Attention Pattern Predictability Analysis

• TAPPA is a framework that formalizes attention evolution using query self-similarity and rotary positional embeddings to reveal temporal patterns.
• It decomposes query and key channels to identify distinct attention motifs including vertical, sequential, and seasonal patterns.
• The methodology enables targeted compression and pruning workflows by quantifying pattern predictability through explicit similarity metrics.

Temporal Attention Pattern Predictability Analysis (TAPPA) is a principled framework for the analysis and exploitation of regularities in temporal attention mechanisms across deep learning models, most notably Transformers with rotary positional embeddings (RoPE) and recurrent neural networks (RNNs) operating on multivariate time series. TAPPA provides a unifying mathematical explanation for the emergence of diverse attention pattern shapes observed in LLMs and temporal sequence models, and enables targeted model compression and pruning workflows by quantifying pattern predictability through explicit query self-similarity measures (Shih et al., 2018, Yang et al., 29 Jan 2026).

1. Formal Definition of Temporal Attention Patterns

TAPPA formalizes the notion of a temporal attention pattern as the structured evolution—across sequence position—of an attention mechanism’s alignment between queries and keys. Let $q_t\in\mathbb{R}^d$ denote the query at decoding step $t$, and $K=[k_1, ..., k_T]^\top\in\mathbb{R}^{T\times d}$ the matrix of key vectors. With rotary positional embeddings, the raw attention logits are given by

$a_{t,j} = q_t^\top R_{t-j} k_j,\quad j=1,\ldots,T,$

where $R_{t-j}$ is the RoPE rotation corresponding to the relative offset $(t-j)$. The temporal attention pattern is defined by the sequence $\{a_t\}$, which may exhibit:

• Predictable (stable) patterns: indices of the top-$k$ attended positions change smoothly with $t$.
• Unpredictable (random) patterns: attended indices shift erratically from step to step.

Central to TAPPA is the query self-similarity (q-similarity), measuring the cosine similarity between consecutive queries:

$$s_t = \frac{q_t^\top q_{t+1}}{\|q_t\|\,\|q_{t+1}\|}, \qquad S = \frac{1}{T-1} \sum_{t=1}^{T-1} s_t,$$

where $S$ quantifies the overall stability of query evolution. High $S$ correlates with predictable, temporally coherent patterns, while low $S$ is associated with retrieval-focused or highly stochastic heads (Yang et al., 29 Jan 2026).

2. Mathematical Mechanisms Generating Temporal Patterns

The joint structure of queries $q_t$, keys $k_j$, and the RoPE operation underlies TAPPA’s taxonomy of temporal attention patterns. Each $q_t$ and $k_j$ is decomposed channel-wise ($d=2M$) as $q_t = \bigoplus_{m=1}^M q_t^{(m)}$, $k_j = \bigoplus_{m=1}^M k_j^{(m)}$ with $q_t^{(m)},k_j^{(m)}\in\mathbb{R}^2$. RoPE applied per-channel rotates by frequency $\theta_m$; so

$$a_{t,j} = \sum_{m=1}^M \|q_t^{(m)}\|\|k_j^{(m)}\| \cos(\phi_{t,j}^{(m)} + (j-t)\theta_m),$$

where $\phi_{t,j}^{(m)}$ is the angle between $q_t^{(m)}$ and $k_j^{(m)}$. Predictable patterns arise when one (possibly low-frequency) channel dominates, and $q_t$, $k_j$ evolve smoothly. RoPE’s relative position encoding preserves offset structure, leading to three main types:

• Vertical (re-access): High q-similarity and a dominant low-frequency channel yield persistent focus on fixed memory positions.
• Diagonal (sequential): Both queries and keys evolve slowly and RoPE invariance induces a progression along $i = t - k$.
• Seasonal (periodic): Periodicity in both $q$ and $k$ with RoPE frequency resonance ($L\theta_m \approx 2\pi k$) induces regularly repeating attention stripes (Yang et al., 29 Jan 2026).

Unpredictable patterns occur when the difference $\|q_{t+1} - q_t\|$ is large, making $a_{t+1}$ differ substantially from $a_t$.

3. Theoretical Results: Predictability Criteria and Pattern Guarantees

TAPPA establishes several formal results characterizing when and why specific temporal attention patterns occur:

• Proposition (Unpredictable patterns): If $\|\Delta q\|$ is large and unaligned with $R_{t+1-j}k_j$, then the logit vector changes satisfy

$$\|a_{t+1} - a_t\|_\infty \geq c_1 \|\Delta q\| - c_2,$$

so the argmax of attention may jump significantly.

• Theorem (Vertical stability / re-access): For small $\|q_{t+1} - q_t\| \leq \varepsilon$ and a dominant low-frequency channel,

$\max_i |a_{t+1,i} - a_{t,i}| \to 0,$

ensuring vertical pattern persistence.

• Theorem (Diagonal/sequential pattern): Slowly varying queries and keys,

$|a_{t+1,i+1} - a_{t,i}| \leq C\varepsilon,$

induce attention ridges along $(+1,+1)$.

• Theorem (Periodic/seasonal pattern): If $\|q_{t+L} - q_t\|\leq \varepsilon_q$, $\|k_{i+L} - k_i\|\leq \varepsilon_k$, and RoPE is resonant, then

$$|a_{t+L,i} - a_{t,i}| \leq O(\varepsilon_q + \varepsilon_k) + O(\delta),$$

leading to repeating attention bands at period $L$.

The mathematical explanation is that under slow drift in $(q,k)$ and channel dominance, the primary cosine term varies minimally, preserving regularity in attention maps (Yang et al., 29 Jan 2026).

4. Pattern Predictability in Multivariate Temporal Modeling

In multivariate time series, TAPPA generalizes the attention paradigm by using CNN-based temporal pattern extraction in combination with attention over the resulting pattern features. Given $X\in\mathbb{R}^{n\times T}$, a set of $k$ convolutional filters $W_f^{(j)}\in\mathbb{R}^{n\times\ell}$ ($j=1,...,k$) produce a feature map $P\in\mathbb{R}^{n\times k}$ via

$$P_{i,j} = (X * W_f^{(j)} + b_j)_i = \sum_{p=1}^n \sum_{\ell'=1}^{\ell} W_f^{(j)}(p, \ell') X_{p, T-\ell+\ell'} + b_j,$$

which encodes time-invariant temporal patterns for each variable. The attention mechanism computes scores

$s(P_i, h_{t-1}) = P_i^\top W_a h_{t-1},$

normalized and combined to produce a context vector $c_t$ that informs the forecast. Attending "by position" over $P$ rows and integrating via

$h'_t = W_h h_t + W_c c_t, \quad y_t = W_o h'_t,$

enables the model to leverage both local temporal motifs and long-range, frequency-domain structure. This multi-scale approach systematically filters out noisy variables and extends memory to capture periodicities beyond the reach of ordinary RNNs (Shih et al., 2018).

5. Applications: Compression and Pruning via Pattern Predictability

TAPPA provides q-similarity ($S_l$) as a robust metric for discriminating layers with stable, regular attention patterns (hence more predictable and redundant representations) from retrieval-type layers requiring high-fidelity retention. This distinction enables compression and pruning algorithms:

• KV Cache Compression: Under a fixed total budget $B_{\mathrm{total}}$, per-layer budgets $B_l$ are assigned according to a composite score $P'_l = P_l + \alpha (1 - S_l)$, where $P_l$ is an entropy-variance baseline and $1 - S_l$ flags unpredictability. Budgets are normalized and used to govern value retention.
• LLM Pruning: For structural pruning, a base "Block Influence" metric $BI_l$ is augmented as $BI'_l = BI_l + \beta(1 - S_l)$. Layers with high $S_l$ (stable, redundant) are pruned more aggressively, preserving unpredictable, semantic-sensitive attention heads.

Both workflows have explicit pseudocode and yield measurable improvements when $S_l$ is incorporated (Yang et al., 29 Jan 2026).

6. Empirical Evaluation and Observable Pattern Taxonomy

Extensive experiments demonstrate TAPPA’s framework and metrics increase accuracy and efficiency on both sequence modeling and LLM tasks:

• On KV cache compression, using TAPPA’s q-similarity consistently improves average accuracy compared to baseline CAKE by $+0.15$ to $+0.30$ across long-context tasks on Llama-3.1-8B and Qwen-2.5-7B.
• For LLM pruning, tapping $S_l$ alongside block influence achieves +1.34 to +5.60 points over ShortGPT at similar pruning ratios on benchmark tasks.
• In multivariate time series, TAPPA attains state-of-the-art or tied-best results in RSE, RAE, and correlation on complex datasets including electricity, solar, traffic, and exchange rates, and outperforms both stepwise attention and classical autoregressive methods (Shih et al., 2018).

Ablation analyses confirm the necessity of the CNN pattern extraction, the attention normalization, and the pattern-centric, not just step-centric, attention. The learned convolutional filters in TAPPA empirically align with the dominant frequencies in real data, such as 6h, 8h, 12h, and 24h periodicities in traffic records, supporting the theoretical interpretation that frequency-domain pattern basis enhances long-horizon prediction.

7. Cross-Domain Relevance and Theoretical Impact

TAPPA offers a unifying temporal perspective that bridges the theory and practice of attention in RNNs, Transformers, and deep sequence models. Through explicit quantification of pattern predictability and formal analysis of the interaction between query/key dynamics and position encoding, TAPPA:

• Explains vertical (re-access), sequential (diagonal), and seasonal (periodic) heads observed in modern LLMs.
• Enables principled, interpretable model compression and pruning by exposing the regularity of information flow.
• Establishes general conditions for the emergence and persistence of attention motifs across modeling paradigms.

A plausible implication is that future architectures leveraging temporal pattern predictability could further close the gap between model accuracy, compression, and interpretability by harmonizing frequency-domain and temporal-domain representations (Shih et al., 2018, Yang et al., 29 Jan 2026).