NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation

Abstract: Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of \emph{which layers} should retain full attention remains unsolved. Existing methods use either fixed periodic patterns or attention-based heuristics that may not capture what matters for downstream accuracy. We propose NLL-guided layer selection, a training-free method that directly measures each layer's importance by computing the negative log-likelihood degradation on answer tokens when that layer uses sliding-window instead of full attention. On LongMemEval with Qwen3-4B, our method achieves 64.6\% accuracy using only 1/4 full-attention layers, matching the 1/2-FA periodic baseline (65.0\%) while halving the computational budget. NLL-guided selection outperforms the SWAA-reported periodic 1/4-FA baseline by 10.4 percentage points and a matched LightTransfer-style baseline by 26.4 percentage points. De-confounding analysis shows the signal is consistent with long-range attention needs rather than generic layer sensitivity. The method requires only $\sim$15 minutes of one-time calibration, advancing the efficiency-accuracy Pareto frontier for long-context LLM deployment.

• The paper presents a training-free approach that uses NLL degradation to select the most beneficial full-attention layers for hybrid attention models.
• It demonstrates that using only 25% of layers with full attention can achieve comparable accuracy to traditional methods while reducing computational cost by 50%.
• The method proves robust across diverse long-context tasks and provides a principled alternative to heuristic-based layer selection strategies.

NLL-Guided Full-Attention Layer Selection for Sliding-Window Adaptation in LLMs

Introduction

The paper "NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation" (2606.27791) addresses a fundamental challenge in deploying LLMs for long-context inference: mitigating inference-time computational costs without significantly degrading downstream accuracy. While hybrid attention architectures—where full and sliding-window attention mechanisms are interleaved among Transformer layers—offer a pragmatic solution, the optimal selection of layers for full attention remains unsolved. This work introduces a training-free, data-driven method based on negative log-likelihood (NLL) degradation to select the most critical layers for full attention, empirically demonstrating strong performance and efficiency gains in long-context settings.

Figure 1: Overview of NLL-guided full-attention layer selection for SWAA. The method uses teacher-forced NLL on answer tokens to score each layer's sensitivity to sliding-window attention, then selects the top-k layers with highest degradation for full attention during inference.

Methodology and Framework

The authors focus on adapting pretrained, full-attention LLMs to leverage hybrid attention during prefill: only a subset of layers uses full attention, with the remainder operating under a sliding-window regime. Their method (NLL-guided selection) employs a straightforward calibration process requiring no gradient updates or retraining. For each layer, the degradation in mean NLL on answer tokens is measured when swapping full attention for sliding-window attention during the prefill phase, while all answer tokens still attend to the full prompt, aligning with the full-attention decode protocol.

The selection protocol is as follows:

1. For a calibration set of long-context examples, each layer's $\Delta$-NLL is computed by toggling its attention mode from full to sliding-window, leaving all others in sliding-window mode.
2. The top-$k$ layers with the largest $\Delta$-NLL are selected to retain full attention under the imposed computational budget.
3. All layers use full attention in decoding, emulating the SWAA protocol.

This procedure is entirely training-free and typically requires around 15 minutes for calibration on modest compute resources, amortizing quickly relative to real-world inference workloads.

Empirical Results

On the LongMemEval benchmark—tasking LLMs with retention and reasoning over 24k-token prompts—NLL-guided 1/4-FA (9 full-attention layers out of 36; Qwen3-4B) yields 64.6% accuracy, essentially matching the 1/2-FA periodic baseline's 65.0% while halving the number of computationally expensive layers. Notably, this method outperforms the SWAA-reported periodic 1/4-FA baseline by 10.4 percentage points and the LightTransfer heuristic (attention-lazy ratio) by a substantial 26.4 percentage points under the same compute regime.

Per-task breakdown indicates gains across all evaluated categories, with the greatest improvements observed on single-session-user and temporal-reasoning tasks—highlighting the method's efficacy for challenging, long-range memory and logical operations.

Robustness and Analysis

A critical analytical dimension explored is whether NLL-based layer sensitivity reflects mere general importance or is specific to long-range dependencies. The authors demonstrate through correlation and overlap analyses that calibrations on long versus short prompts yield uncorrelated layer rankings (Spearman $\rho=0.306$, Jaccard=0.2), and the signal magnitude is substantially higher for long-context settings. This provides compelling evidence that the NLL signal is highly task (and thus context)-specific.

Figure 2: Layer ranking comparison between long-prompt (16k--32k tokens) and short-prompt (1.5k tokens) calibration. Low correlation (Spearman $\rho=0.306$) and minimal overlap (Jaccard=0.2) are consistent with the NLL signal being specific to long-range attention needs.

The nature of the selected layers is non-trivial: the top-$k$ set naturally spans early, middle, and late layers (rather than a periodic or stratified distribution), indicating that long-range information flow in Transformer networks is neither uniform nor easily captured by naive heuristics.

Figure 3: Per-layer NLL degradation ($\Delta$-NLL) when using SWA instead of FA. Blue bars indicate the 9 layers selected for full attention. The selected layers span early, middle, and late depths with a non-periodic pattern.

Calibration stability is also rigorously assessed. Reducing the calibration set size from 64 to 16 examples results in some drop (2.4pp) in downstream accuracy and partial divergence in selected layers, but core, high-impact layers are still reliably captured, underscoring the signal's robustness for production deployment.

Implications and Future Directions

This work advances the efficiency-accuracy Pareto frontier for long-context LLMs, enabling dramatically lower compute budgets for prefill without sacrificing accuracy on information-intensive tasks. It also exposes the limitations of periodic or attention-heuristic layer selection, advocating for explicit downstream quality metrics as the guiding principle.

From a theoretical perspective, the results underscore the heterogeneous and context-dependent roles of different Transformer layers in modeling long-range dependencies—a finding with implications for future architecture search, pruning, and efficient inference strategies.

For practical applications, the approach enables scalable LLM deployment for use cases with large and frequent inference workloads (retrieval-augmented generation, long-form QA, memory-augmented interactive agents), and can be combined with other KV cache compression or attention sparsity methods for even greater efficiency.

Open directions include exploration of per-instance or dynamic layer selection, cross-model and cross-benchmark generalization, and the development of unified frameworks for layer-budgeting that natively incorporate model, hardware, and task characteristics.

Conclusion

NLL-guided full-attention layer selection presents a principled, training-free method for hybrid attention adaptation in LLMs. By directly targeting downstream accuracy, the method achieves near-optimal results under strict computational constraints, outperforming heuristic or periodic baselines and yielding practical and theoretical insights into Transformer layer roles in long-context processing. This approach is a recommended instantiation for efficient, production-grade long-context model deployment and provides a foundation for future research in context-adaptive inference strategies.