--- title: Interleaving FA/SWA Layers in Transformers url: https://www.emergentmind.com/topics/interleaving-fa-swa-layers type: topic --- # Interleaving FA/SWA Layers in Transformers Interleaving FA/SWA Layers constitutes a hybridization technique in Transformer-based large language models (LLMs) where full causal self-attention (FA) layers are systematically interleaved with sliding-window attention (SWA) layers. This approach enables pretrained FA models to efficiently adapt to long-context inference using SWA without requiring full retraining, mitigating the accuracy loss associated with naive SWA deployment. The method strategically alternates global (FA) and local (SWA) receptive fields to maintain high accuracy on deep-context tasks, while achieving substantial reductions in memory and computational cost [2512.10411]. ## 1. Motivation for Interleaving Transformer models pretrained with full causal attention possess global context propagation at every layer. Switching all layers to local sliding-window attention with window size $W$ drastically reduces the attention's receptive field, leading to severe degradation in performance for tasks requiring long-range context (e.g., question answering with inputs >16K tokens). Pure SWA delivers $\mathcal{O}(N \cdot W)$ inference, substantially improving speed and lowering memory use, but at the cost of downstream accuracy due to the lack of long-range information flow. Interleaving FA and SWA layers serves as a compromise: maintaining a subset of layers with global context re-injects distant information that otherwise would be forgotten by SWA, thereby restoring downstream task performance while still retaining most of the efficiency gains of SWA [2512.10411]. ## 2. Layer Scheduling and Algorithmic Implementation Interleaving operates by defining a periodic pattern across the $L$ self-attention layers of a Transformer. Every $n$th layer is designated as an FA layer, with the remainder as SWA layers. Let $l$ index the layers, $n$ the interleaving period, and $offset \in \{0, \dots, n-1\}$ the phase. For typical settings, $n=2$ (half FA, half SWA) and $offset=0$ or $1$, resulting in interleaved patterns such as FA at layer indices $[0,2,4,\dots]$ or $[1,3,5,\dots]$. The choice of $offset$ is model- and calibration-dependent. The algorithm is summarized as follows: ```python # Given: # L ← total Transformer layers # n ← period: every n-th layer uses FA # offset ← phase: 0 (even) or 1 (odd) # W ← SWA window size # k_sink ← number of sink tokens # prefill_or_decode ← FA-Decode flag for layer l in 0 … L−1: if (l mod n) == offset: compute_attention(FA_mask) else: if keep_first_k_tokens: compute_attention(SWA_with_sink_mask(W, k_sink)) else: compute_attention(SWA_mask(W)) ``` In practical configurations, $W \in \{2K, 4K, 8K\}$ and $k_{sink}\in\{0,10,100,1000\}$ are reported [2512.10411]. ## 3. Attention Masks and Signal Propagation Attention masking determines whether a position $i$ in the sequence can attend to another position $j$. For each block, a pre-softmax logit mask $M \in \mathbb{R}^{N \times N}$ is constructed as: - **Full causal attention (FA):** $$ M_{i,j}^{FA} = \begin{cases} 0 & \text{if } j < i \\ -\infty & \text{otherwise} \end{cases} $$ - **Sliding-window attention (SWA) of window $W$:** $$ M_{i,j}^{SWA} = \begin{cases} 0 & \text{if } i-W \leq j < i \\ -\infty & \text{otherwise} \end{cases} $$ - **SWA with $k$ “sink” tokens:** $$ M_{i,j}^{SWA\_sink} = \begin{cases} 0 & \text{if } j < k \\ 0 & \text{if } i-W \leq j < i \\ -\infty & \text{otherwise} \end{cases} $$ The output at position $i$ is then $$ A_i = \sum_j \text{softmax}\left(\frac{Q_i K_j^\top}{\sqrt{d}} + M_{i,j}\right) V_j. $$ Switching $M$ per layer interleaves access to global and local context, thus enabling recovery of critical global signals otherwise inaccessible in pure SWA [2512.10411]. ## 4. Role of Sink Tokens in Interleaving While interleaving alone does not require special tokens, empirical adaptation strategies often combine it with the “Keep First k Tokens” approach. This ensures that in every SWA block, the first $k$ input tokens are always visible (“sink” tokens), thereby stabilizing attention distributions on context “hubs” and reducing catastrophic context collapse in zero-train adaptation settings. The sink tokens’ mask is implemented as a logical OR between the standard SWA mask and a mask enabling unimpeded access to the initial $k$ positions. When SWA-aware fine-tuning is performed, sink-token preservation is rendered optional, but in inference-only adaptation, it is essential [2512.10411]. ## 5. Hyperparameter Regimes and Rationale Empirically, several hyperparameters govern the efficacy of interleaving FA/SWA layers: - **Window size $W$:** An aggressive $W=2$K window highlights auxiliary method gains; practical trade-offs favor $W=4$K or $8$K, as increasing $W$ after interleaving and FA Decode typically closes most of the performance gap. - **Interleaving period $n$:** $n=2$ (half FA/SWA) achieves the bulk of accuracy recovery while halving the quadratic cost. More aggressive sparsification (e.g., $n=4$) further reduces compute but degrades performance. - **Phase offset:** The optimal phase (even vs. odd layers using FA) varies by architectural family (e.g., Qwen3-4B favors offset=1; Qwen3-30B and Llama3.1 the opposite), so a small calibration sweep is recommended. - **Sink token count $k_{sink}$:** For inference-only, $k=10$ confers most benefit; $k=100$ yields marginal improvement, with diminishing returns and negligible runtime cost above $k=1000$. - **FA Decode:** Using SWA for prefilling and reverting to FA at decode time improves performance without architecture changes, but doubles KV memory unless recomputation per turn is enabled. A robust default is: $W=4$K, $n=2$, offset swept, $k_{sink}=10$–$100$ [2512.10411]. ## 6. Empirical Ablations and Performance Outcomes Ablations in [2512.10411] on Qwen3-4B-Thinking and -Instruct with LongMemEval_24k demonstrate: | Configuration | Think Acc. | Non-Think Acc. | Efficiency (TTFT) | |------------------------------------------------------------------|------------|----------------|------------------------| | Pure SWA (2K) | 3.2% | 11.0% | $8\times$ speedup | | Half Interleaved (no Keep-First/FA Decode, n=2) | 13–18% | – | $\approx3\times$ speedup| | Interleaving + FA Decode | 32% | 26% | – | | Interleaving + FA Decode + Keep-First $k=10$ | 59.2% | 34.8% | – | | Best Zero-Train (Int. [1,3,5,…], k=100, FA Decode) | 68.8% | 50.6% | – | | FA Baseline (All Layers FA) | 73.0% | 62.0% | Baseline (slowest) | A plausible implication is that interleaving strategy, especially combined with FA Decode and light sink-token preservation, achieves nearly 90% recovery of baseline accuracy with only half the quadratic cost. With SWA-aware fine-tuning, interleaving alone suffices to approach baseline accuracy, rendering sink-token strategies largely optional [2512.10411]. Efficiency trade-offs indicate that pure SWA achieves $8\times$ speedup, half-interleaving yields $3\times$, and increasing $W$ to 4K maintains most efficiency. ## 7. Deployment Recommendations and System Integration Integration of interleaving FA/SWA layers in existing serving stacks (e.g., Flash-Attention–2 or vLLM) is operationally straightforward. For inference-only adaptation without fine-tuning, the combination of FA Decode, Interleaving, and Keep-First $k=10$–$100$ provides a strong default. When some SWA-aware fine-tuning is feasible (e.g., LoRA), interleaving with $n=2$ and offset calibrated suffices, and Keep-First becomes unnecessary. Increasing window size ($W$) improves accuracy with little adverse impact on throughput. FA Decode can also be introduced for further gains as needed. In summary, interleaving FA/SWA layers is identified as the single most cost-effective component of Sliding Window Attention Adaptation, offering substantial memory and speed gains, and accurately restores long-context model competence with minimal modification to existing architectures [2512.10411]. Source: https://www.emergentmind.com/topics/interleaving-fa-swa-layers