Arrow Attention: Temporal Bias in Transformers

• Arrow Attention is a variant of self-attention that incorporates adaptive temporal kernels to enforce a monotonic temporal bias reflecting the arrow of time.
• It enhances sample efficiency by biasing attention toward temporally proximate events while preserving the capacity for long-range interactions with minimal computational overhead.
• Empirical evaluations on EHR systems show measurable improvements in AUPRC and AUROC, establishing its effectiveness in temporally structured prediction tasks.

Arrow Attention is a modification of the self-attention mechanism in Transformer architectures that integrates learnable, adaptive temporal kernels to inject an explicit temporal prior reflecting the “arrow of time.” Introduced in "Self Attention with Temporal Prior: Can We Learn More from Arrow of Time?" (Kim et al., 2023), Arrow Attention is motivated by the observation that many real-world time series—especially in domains such as electronic health records (EHR)—exhibit stronger correlations among temporally proximate events. Standard self-attention, lacking any inherent temporal bias, does not readily encode this inductive bias unless provided with vast quantities of data. Arrow Attention directly incorporates a monotonic bias for short-term temporal dependencies by applying learnable kernel functions based on the temporal lag between events, enabling accurate modeling with greater sample efficiency while preserving the capability to attend over arbitrary temporal horizons.

1. Motivation and Temporal Inductive Bias

Transformer-based self-attention mechanisms are highly effective at modeling long-range dependencies in sequential data but are inherently permutation invariant with respect to sequence order. They lack a built-in preference for temporally proximate events, an inductive bias that is often essential in modeling real-world time series. Empirical evidence demonstrates that, for many prediction tasks on EHR datasets, a vanilla Transformer—absent a temporal prior—fails to learn strong short-term temporal dependencies (manifested as a diagonal bias in attention matrices) unless supplied with excessive amounts of data. The Arrow Attention mechanism addresses this deficit by integrating the “arrow of time” through parameterized temporal kernels, thereby:

• Encouraging systematically larger attention weights for events close together in time (monotonic bias)
• Allowing long-range interactions where data supports them
• Achieving these with negligible additional computational or parameter overhead
• Improving both sample efficiency and task performance on domains such as EHR where temporal proximity is salient (Kim et al., 2023)

2. Mathematical Formulation

Arrow Attention modifies the scaled dot-product calculation of self-attention via learnable temporal kernels that functionally depend on the time-lag $h=|i-j|$ between events:

• Vanilla attention computes logits: $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$.
• Temporal kernelizing: Introduces two learnable kernels:
  • Exponential kernel: $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$, with $\alpha_e, \beta_e > 0$
  • Periodic kernel: $$\Phi_p(h) = \exp\left(-2{\alpha_p}^2 \sin^2(\frac{\pi h}{\beta_p})\right)$$, with $\alpha_p, \beta_p > 0$
• Integration in attention: Let $C^{(e)}_{ij}$ and $C^{(p)}_{ij}$ be kernel matrices derived from $\Phi_e$ and $\Phi_p$, respectively. Then, the queries and keys are elementwise-modulated:
  • $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$0
  • $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$1
• Arrow-aware logits: $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$2
• Attention weights: $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$3, and output: $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$4

The kernels $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$5 and $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$6 depend only on the time difference and can be efficiently precomputed and broadcast across feature dimensions, making their application computationally negligible compared to the core dot-product operation (Kim et al., 2023).

3. Architectural Integration

Arrow Attention replaces standard self-attention modules within both encoder and decoder stacks of the Transformer architecture. Each multi-head attention layer assigns its own pair of learnable kernel parameters $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$7 per head, enabling head- and layerwise diversity in the temporal bias profile. All other components, including residual connections, layer normalization, and position-wise feed-forward networks, remain unchanged. After kernelized attention is computed for each head, the standard concatenation and output projection are performed exactly as in the original Transformer formulation. The additional cost per layer is restricted to two $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$8 element-wise multiplications per attention head, insignificant relative to the $S_{ij} = \frac{(QK^\top)_{ij}}{\sqrt{d_k}}$9 cost of the core attention operation.

4. Optimization and Hyperparameters

Adam is used as the optimizer for both model and kernel parameters. Distinct learning rates are employed:

• LR$$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$0 for the main Transformer weights
• LR$$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$1 LR$$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$2 for kernel parameters, with typical $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$3

Kernel parameters $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$4 and $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$5 are strictly positive, achieved via $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$6, $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$7, initialized from small positive constants or $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$8. Dropout in the range $$\Phi_e(h) = \exp\left(- (\alpha_e h)^{\beta_e} \right)$$9–$\alpha_e, \beta_e > 0$0 is applied post-attention and in feed-forward sublayers. Training employs early stopping on validation AUPRC, with differing patience values depending on the dataset. Mini-batches are balanced for class imbalance (50/50 event vs. non-event). Hyperparameter regimes explored include LR$\alpha_e, \beta_e > 0$1, batch sizes $\alpha_e, \beta_e > 0$2, layers $\alpha_e, \beta_e > 0$3, heads $\alpha_e, \beta_e > 0$4, hidden units $\alpha_e, \beta_e > 0$5, and dropout $\alpha_e, \beta_e > 0$6 (Kim et al., 2023).

5. Experimental Evaluation on Electronic Health Records

Arrow Attention was empirically validated across several EHR-based tasks, directly comparing against baselines such as GRU-D, IP-Net, and vanilla Trieansformer. Metrics include AUPRC and AUROC, calculated over three seeds. Notable results:

Dataset / Task	Best Baseline (Model)	SAT-Transformer (Arrow Attention)
PhysioNet 2019 (sepsis detection)	15.3 / 82.8 (GRU-D/IP-Net/Transformer)	16.7 / 83.0
MIMIC-III (in-hospital mortality)	52.8 / 86.1 (GRU-D)	53.7 / 86.4
eICU (Heart Failure)	3.16 / 78.3 (IP-Net)	3.10 / 81.0
eICU (Respiratory Failure)	3.58 / 73.5 (Transformer)	3.80 / 73.7
eICU (Kidney Failure)	3.09 / 80.8 (Transformer)	3.33 / 81.0

Arrow Attention achieves up to $\alpha_e, \beta_e > 0$710% relative gain in AUPRC over state-of-the-art approaches. The computational overhead is minimal ($\alpha_e, \beta_e > 0$87.23 ms/iter for Arrow Attention vs. 6.71 ms/iter for standard Transformer) (Kim et al., 2023). Ablation studies show that combining both exponential and periodic kernels yields optimal performance. Moreover, Arrow Attention maintains superior robustness in limited data regimes, with performance gains increasing as training data volume decreases.

6. Qualitative Analysis and Kernel Dynamics

Visualization of attention maps reveals that Arrow Attention heads frequently display a strong diagonal stripe structure, indicating systematically higher weights for temporally adjacent elements. Some heads, influenced by the periodic kernel, exhibit periodic “banding.” Analysis of learned kernel parameters demonstrates that different heads converge to distinct temporal profiles: some encode a rapid exponential decay with time-lag, others emphasize periodic motifs, and some approximate the identity (kernel ≈ 1), thus defaulting to standard data-driven attention. This diversity prevents attention collapse and enhances the representational breadth of the model. Optionally, kernel parameters can be conditioned on global sequence features (e.g., mean interval, sequence length), but this yields only minor and inconsistent benefits.

7. Relation to Broader Temporal Modeling and Limitations

Arrow Attention provides an explicit mechanism for encoding temporal priors complementary to, and more expressive than, positional encodings or masked attention. By directly learning the decay and periodicity associated with the arrow of time, it offers a flexible framework adaptable to variable temporal structures across datasets and domains. A plausible implication is improved sample efficiency and robustness in temporally structured prediction tasks, especially in low-data or data-sparse regimes. The method incurs negligible computational or parameter increase, preserving the scalability of base Transformers. Current evidence for Arrow Attention is domain-specific (EHR), and while transferable, generalization to highly non-stationary or non-monotonic sequences warrants further study (Kim et al., 2023).