Forget Attention: Importance-Aware Attention Is All You Need

Abstract: Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling. Transformers see everywhere but cannot prioritize; SSMs know what matters but cannot revisit. Existing hybrids -- Jamba (block level) and Hymba (head level) -- place the two in separate compartments, so neither informs the other during the attention computation itself. We propose SISA (SSM-Informed Softmax Attention), which adds an SSM-derived importance term directly inside the attention score and realizes the full operation as a single SDPA call on augmented query/key vectors -- no recurrent state, no custom kernel. At 152M / 5B tokens, SISA reaches LAMBADA-greedy 17.3% (vs. Transformer 13.9 and Mamba-3 15.5) and attains NIAH 100% from step 1K, 7x faster than Transformer's retrieval convergence; at 369M, Mamba-3 leads LAMBADA while SISA preserves perfect NIAH and stock-SDPA execution. SISA thus defines a third design axis for SSM-attention hybrids -- score-level fusion -- beyond the block-level and head-level paradigms that have dominated the field.

• The paper proposes score-level fusion where SSM-derived bias is directly injected into Transformer attention scores, merging global content with sequential importance.
• Its methodology uses augmented Q/K vectors to integrate decay and rotation signals, achieving faster retrieval (100% by 1K steps) and improved sample efficiency.
• Experimental results show significant gains over Transformers and Mamba variants at mid-scale, with optimal SSM bias dimensions varying non-monotonically.

SISA: Score-Level Fusion of Attention and State Space Models for Language Modeling

Introduction and Motivation

The "Forget Attention: Importance-Aware Attention Is All You Need" (2606.02332) paper interrogates the limitations of Transformer-based attention and state space models (SSMs)—key architectural families underpinning contemporary sequence models. Transformers enable every token to attend globally by content-similarity, yet lack an intrinsic notion of token importance derived from sequence structure. Mamba and related SSMs can track importance through recurrence and decay but, by their very nature, are unable to revisit queries arbitrarily in the past. Prior hybrid architectures (e.g., Jamba, Hymba) compartmentalize attention and SSM modules at the block or head level, which prevents information from one path directly modulating the score computation in the other. This paper introduces SISA (SSM-Informed Softmax Attention), which directly fuses the SSM-derived sequential importance signal into the attention score itself, defining a novel and orthogonal “score-level” hybridization.

Figure 1: Three levels of SSM-attention fusion. Block- and head-level hybrids combine outputs independently; SISA injects the SSM signal directly into the attention score as augmented Q/K.

Methodology: SISA Layer Architecture and Score-Level Fusion

SISA’s innovation is fusing a vector-valued SSM-derived bias into the Transformer’s attention score function. Conceptually, it augments the traditional content-similarity term with an SSM-based importance signal. The core attention score between query $i$ and key $j$ is redefined as:

$$s_{ij}^{\text{SISA}} = \frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d_h}} + \lambda \cdot \bar{\mathbf{C}}_i^\top \bar{\mathbf{B}}_j$$

where $\bar{\mathbf{C}}_i, \bar{\mathbf{B}}_j$ encode SSM-induced decay and data-dependent phase (rotation), while $\lambda$ is a learned scaling parameter. This addition coherently integrates global content lookup (attention) with importance tracking (SSM), computed in a single, efficient SDPA (scaled dot-product attention) kernel call after augmenting the Q/K vectors.

The SSM channels used for the bias signal adapt Mamba-3’s mathematical formalism: they include per-token decay (cumulative, data-dependent) and rotation (data-dependent RoPE) terms. Through concatenation, Q and K are expanded to include the SSM-derived quantities, and all computation remains compatible with standard FlashAttention kernels—no custom CUDA or external state needed.

Figure 2: Compute graph of a SISA layer, illustrating how attention and SSM paths merge via augmented Q/K vectors for single-call SDPA.

Experimental Evaluation and Results

SISA is compared against three baselines (Transformer, Mamba-2, Mamba-3) across multiple model scales (50M, 152M, 369M parameters) and evaluated on five benchmarks: LAMBADA (long-range context), NIAH (retrieval/retention), HellaSwag (commonsense), ARC-Easy (factual knowledge), and WinoGrande (coreference).

Key Results at 152M Parameters:

• On LAMBADA-greedy, SISA achieves 17.3% accuracy, exceeding Transformer (13.9%) by +3.4 points (+24.5% rel.) and Mamba-3 (15.5%) by +1.8 points;
• SISA maintains NIAH (Needle-in-a-Haystack) retrieval at 100% from step 1K (10% of training), seven times faster than Transformer and unattainable for Mamba;
• SISA's efficiency: surpasses Mamba-3 in throughput (+25%) due to SDPA fusion with no custom kernel requirements.

Figure 3: NIAH convergence at 152M; SISA reaches 100% retrieval from step 1K, significantly outpacing Transformer and Mamba family.

Figure 4: Scaling behavior across model sizes. SISA peaks on LAMBADA at 152M; Mamba-3 overtakes at 369M. SISA and Transformer maintain perfect NIAH.

Figure 5: LAMBADA trajectory at 152M; SISA attains Transformer’s final accuracy by step 2K and converges 24% higher.

Scaling Trends and Tradeoffs:

• The performance gain from SISA is maximal at the mid-scale (152M), while at large scale (369M), Mamba-3 outperforms on LAMBADA but SISA retains perfect retrieval (NIAH) and superior compatibility/throughput.
• The efficacy of SISA’s score-level fusion depends non-monotonically on the SSM projection dimension $d_s$; optimal $d_s$ varies by model scale, with smaller $d_s$ preferred at 152M and larger $d_s$ at 50M/369M.

Figure 6: $d_s$ ablation demonstrates non-monotonic optimal values across scales; optimal $j$0 is 64 at 50M, 16 at 152M, and (among retrains) 128 at 369M.

Analysis: Architectural Implications

SISA operationalizes the SSM as a dynamic, data-dependent importance bias rather than a memory (no extra state is carried between steps). The decay factor supplies a flexible, content-adaptive distance weighting that can privilege some tokens over others; the rotation differentiates structurally similar, but sequentially distinct, positions. This approach directly addresses the observed failures of prior hybrids, which could not modulate attention weights with sequential importance during score computation.

Pragmatically, SISA’s innovation is significant in that it merges both mechanisms in the score computation, keeping the compute path simple (single SDPA), and remains compatible with FlashAttention. Empirical evidence shows that SISA induces much faster acquisition of retrieval capability and, for some data/scale regimes, improved sample efficiency.

One limitation is observed “softmax dilution”—when many tokens are equally relevant, the SSM bias is weakened, suggesting sigmoid-based attention could further exploit the importance information, motivating the proposed SISA-2 extension.

Implications and Future Directions

The introduction of score-level (vector-valued, data-dependent) fusion as the third axis of SSM–attention hybridization provides an actionable route for research beyond mere architectural stacking or parallelization. It points to the attention score itself as the natural interface for fusing heterogeneous sequence inductive biases. This work’s open questions include:

• Scaling laws for SSM bias dimension $j$1: The optimal allocation varies non-monotonically; principled, possibly adaptive, selection schemes are needed.
• Enabling longer context: The current implementation, based on RoPE, inherits its context window limitations.
• Softmax replacement: SISA’s score-level bias could be further exploited with alternative normalization (e.g., sigmoid attention), as suggested for future SISA-2 variants.

Practically, the design leads to strong improvements in sample efficiency and hardware throughput for certain task/model regions. Future efforts could generalize or extend the bias mechanism, encode richer information in the score, or fuse other sequential inductive biases.

Conclusion

SISA advances the theory and practice of sequence modeling architectures by introducing score-level fusion—a principled and efficient hybridization of attention and SSM mechanisms. This approach delivers strong empirical gains in both retrieval convergence and long-range modeling, outperforms previous hybrids, and establishes the attention score as a potent interface for integrating complementary sequence-biases. The analysis identifies nontrivial scaling trade-offs and motivates follow-up work on score-level fusion generalizations, adaptive importance weighting, and normalization alternatives. SISA thus opens new avenues for efficient, hardware-compatible, and interpretable sequence model architectures that can simultaneously exploit global context and sequential importance tracking.