Lighthouse Attention for Long-Context Training

• Lighthouse Attention is a hierarchical training-time method that compresses long input sequences into multi-resolution pyramids to wrap around standard dense attention.
• It employs symmetrical pooling of queries, keys, and values along with gradient-free top-K selection to efficiently reduce computation while preserving causal consistency.
• A short dense-recovery phase after extensive compressed training enables models to match or exceed dense-from-scratch baselines with significant training speedups.

Searching arXiv for the cited Lighthouse Attention paper and closely related context. Lighthouse Attention is a training-time hierarchical attention scheme for long-context causal transformer pre-training that is designed to accelerate training at extreme sequence lengths and then be removed near the end of training, returning the model to ordinary dense scaled dot-product attention (SDPA) (Peng et al., 7 May 2026). Rather than defining a sparse inference architecture that must be preserved at deployment, it wraps around standard SDPA during most of pre-training by compressing the sequence hierarchically, selecting important entries, running dense attention on the resulting compressed subsequence, and then reconstructing a dense full-length output. Its defining empirical claim is recoverability: most training can be performed with the compressed hierarchical approximation, followed by a short dense-attention recovery phase that yields a standard full-attention model matching or surpassing a dense-from-scratch baseline at the same token budget (Peng et al., 7 May 2026).

1. Problem setting and design objective

The method is introduced for the standard causal transformer bottleneck at long context. For a sequence of length $N$, one attention head computes

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$

where $M$ is the causal mask. The paper states that both time and memory scale as

$\Theta(N^2 d),$

so attention dominates training cost once sequence lengths reach $10^5$ tokens and beyond (Peng et al., 7 May 2026).

The central design target is specifically training-time sparsity, not inference-time sparsity. The distinction is methodological. For inference-only approximations, it is sufficient that a sparse mechanism behave well when inserted into a model already trained with dense attention. For training-time approximations, the stricter criterion is whether a model trained through the approximation remains a good dense-attention model afterward. Lighthouse Attention is designed around that criterion, which is why removability near the end of training is a primary requirement rather than a secondary engineering convenience (Peng et al., 7 May 2026).

The intended workflow is therefore fixed by construction: pre-train at very long context lengths using Lighthouse Attention, switch back to standard dense SDPA for a short continuation, and then deploy or evaluate the resulting model as an ordinary full-attention transformer. A plausible implication is that the method should be understood less as an alternative transformer architecture than as a pre-training acceleration wrapper for long-context regimes.

2. Core mechanism: hierarchical, symmetrical, and selection-based attention

At a high level, Lighthouse Attention builds a multi-resolution summary pyramid of the current layer’s queries, keys, and values; selects the most important summaries and tokens; runs ordinary dense attention only on the selected compressed subsequence; and then spreads the result back over the original sequence positions in a causal way (Peng et al., 7 May 2026).

The method is described as a symmetrical selection-based hierarchical attention mechanism. “Hierarchical” means that it forms multiple levels of pooled sequence representations, from token level to coarse spans. “Selection-based” means that it does not attend over all pooled entries, but instead scores them and chooses a top-$K$ subset. “Symmetrical” means that queries, keys, and values are pooled together at every level rather than compressing only keys and values while leaving queries dense (Peng et al., 7 May 2026).

That symmetry is one of the paper’s principal distinctions. Because pooled queries and pooled keys exist in the same representation space, each selected entry is a coherent $(Q,K,V)$ triple summarizing a single span. The consequence emphasized by the authors is that the dense inner attention scales as $\mathcal{O}(S^2 d)$, because both the query and key sides are compressed, rather than $\mathcal{O}(N S d)$ if queries remained dense. The paper further states that this improves parallelism because all subsequent attention is just dense SDPA on a smaller sequence (Peng et al., 7 May 2026).

Another defining property is that Lighthouse wraps around ordinary SDPA instead of replacing it with a custom sparse attention operator. The sparse logic is external to the inner attention kernel: pyramid construction, scoring, top-$K$ selection, gathering into a contiguous dense subsequence, standard FlashAttention or SDPA on that subsequence, and scatter-back reconstruction. This preserves stock dense-attention execution inside the expensive kernel (Peng et al., 7 May 2026).

The selection stage is also gradient-free. Top-$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$0 is treated as discrete; there is no straight-through estimator, no Gumbel-softmax relaxation, no auxiliary selector loss, and no learnable selector. The stated motivation is implementation and optimization simplicity, especially the avoidance of a complicated sparse backward pass tied to selection (Peng et al., 7 May 2026).

3. Algorithmic structure and causal reconstruction

A Lighthouse layer replaces full attention with four stages: pyramid construction, scoring and top-$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$1, gathered-sequence attention, and scatter-back reconstruction (Peng et al., 7 May 2026). In the single-head formulation given in Algorithm 1, the pipeline is: project $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$2 to $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$3, build the pyramid, compute scores, run chunked-bitonic top-$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$4, gather, FlashAttention, and scatter-back.

Pyramid construction

Given $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$5, Lighthouse constructs an $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$6-level pyramid with pooling factor $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$7. At level $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$8, the $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$9-th window is

$M$0

Each pyramid entry is formed by mean pooling: $M$1 Level $M$2 is the original sequence; higher levels summarize blocks of size $M$3. Because the three projections are pooled together, each triple $M$4 summarizes the same base span. The total number of pyramid entries satisfies

$M$5

so pyramid construction is linear in $M$6 (Peng et al., 7 May 2026).

Scoring and selection

Each pyramid entry receives two scalar scores, one on the query side and one on the key side. At the finest level,

$M$7

At coarser levels, Lighthouse uses max pooling from level-0 scores: $M$8 The method is defined using this norm-based parameter-free scorer, although the paper also evaluates an alternative dilated softmax-attention scorer (Peng et al., 7 May 2026).

Selection runs jointly over the concatenated QK and KQ score streams: $M$9 If an entry is selected by its KQ score, the gathered object is still the full triple $\Theta(N^2 d),$0. The coarsest level is always fully retained, which guarantees at least one contributor covering every base position (Peng et al., 7 May 2026).

Gathered dense attention

After indices $\Theta(N^2 d),$1 are chosen, the selected entries are gathered into a contiguous sequence: $\Theta(N^2 d),$2 and dense causal attention is applied: $\Theta(N^2 d),$3 The gathered length is given as

$\Theta(N^2 d),$4

The paper explains the intended interpretation as a sum: the coarsest level contributes all $\Theta(N^2 d),$5 entries, each finer level contributes at most $\Theta(N^2 d),$6 entries, and $\Theta(N^2 d),$7 is therefore much smaller than $\Theta(N^2 d),$8. For $\Theta(N^2 d),$9, the paper states that $10^5$0 (Peng et al., 7 May 2026).

The gathered subsequence is topologically sorted so that a standard causal mask can be applied over it. The authors emphasize that the hierarchy is gathered without “holes” in the sequence, because compressed queries would make missing query positions particularly problematic during training (Peng et al., 7 May 2026).

Scatter-back and causality

After attention on the compressed subsequence, outputs are written back to the original sequence. For a selected entry at level $10^5$1, position $10^5$2, the output is scattered to

$10^5$3

This shift by $10^5$4 is the causal mechanism: a summary over a span is only written starting at the last token of that span, so no earlier token receives information containing its own future (Peng et al., 7 May 2026). Outputs are summed across all selected entries whose ranges cover a base position: $10^5$5 Within a level, consecutive windows write to disjoint adjacent ranges; across levels, contributions add; and fan-in per base position is bounded by $10^5$6, independent of $10^5$7 (Peng et al., 7 May 2026).

4. Gradient flow, complexity, and systems considerations

The selector branch is explicitly non-differentiable. Gradients flow only through the main computational trunk,

$10^5$8

while the selection indices carry no gradient (Peng et al., 7 May 2026). The rationale given is practical: no sparse backward kernel coupled to selection, no learnable selector that could collapse or misalign with the attention computation, and reuse of stock FlashAttention backward on the gathered subsequence.

The paper’s per-layer complexity breakdown is summarized below (Peng et al., 7 May 2026).

Stage	Complexity
Projections $10^5$9	$K$0
Pyramid pool	$K$1
Scoring	$K$2
Top-$K$3 selection	$K$4
Gather	$K$5
Dense sub-sequence attention	$K$6
Scatter-back	$K$7

The only super-linear term in $K$8 is therefore the dense attention over the gathered subsequence. In the appendix, the paper chooses

$K$9

so that

$(Q,K,V)$0

for constant $(Q,K,V)$1. Under bounded $(Q,K,V)$2, the paper concludes that total per-layer compute is effectively linear in $(Q,K,V)$3 up to a $(Q,K,V)$4 factor, contrasting this with dense softmax attention at $(Q,K,V)$5 (Peng et al., 7 May 2026).

The method is expected to help most when context length $(Q,K,V)$6 is very large, $(Q,K,V)$7 can remain moderate, and attention dominates the training cost rather than MLP or optimizer-state costs. The paper names very long context pre-training regimes such as $(Q,K,V)$8K, $(Q,K,V)$9K, and up to $\mathcal{O}(S^2 d)$0M context as the relevant target regime (Peng et al., 7 May 2026).

From an implementation perspective, only two stages require custom kernels: chunked-bitonic Top-$\mathcal{O}(S^2 d)$1 and scatter-back (Peng et al., 7 May 2026). The chunked-bitonic selection partitions the score stream into chunks of size

$\mathcal{O}(S^2 d)$2

keeps a running top-$\mathcal{O}(S^2 d)$3 buffer with

$\mathcal{O}(S^2 d)$4

and performs in-register bitonic merges per chunk. The resulting indices are then reordered by a single torch.sort into causal order. The paper notes that this is not guaranteed to equal exact global top-$\mathcal{O}(S^2 d)$5; if many high-scoring entries cluster in one chunk, some may be replaced by lower-scoring entries from other chunks. The authors interpret this as a kind of stratified top-$\mathcal{O}(S^2 d)$6 that discourages span collapse and yields broader coverage (Peng et al., 7 May 2026).

A practical systems claim is compatibility with standard dense-attention infrastructure. Because the gathered tensor is contiguous and dense, the same FlashAttention kernel can be used, and standard ring attention or context parallelism works without sparse-specific collectives. For contexts beyond 128K, the paper uses context parallelism in which pooling, scoring, and top-$\mathcal{O}(S^2 d)$7 are shard-local, each rank selects from the tokens it owns, and the gathered subsequences participate in standard ring attention. The paper states that this scales to 1M-token pre-training across 32 Blackwell GPUs (Peng et al., 7 May 2026).

5. Two-stage training and empirical results

The training recipe has two stages. In Stage 1, most training is done with Lighthouse replacing attention in most layers. In Stage 2, training resumes from the Lighthouse checkpoint using ordinary dense SDPA with the same optimizer state and the same dataloader continuation (Peng et al., 7 May 2026). No architectural surgery beyond disabling Lighthouse selection is required.

The total budget is held fixed at 16,000 optimizer steps and about 50.3B tokens. The tested schedules are 12k Lighthouse + 4k SDPA, 11k Lighthouse + 5k SDPA, and 10k Lighthouse + 6k SDPA (Peng et al., 7 May 2026). The paper states that the recipe does not hinge on a precise schedule.

The authors’ explanation for why a short recovery suffices is empirical rather than theoretical: because Lighthouse wraps around the same underlying $\mathcal{O}(S^2 d)$8 and uses standard SDPA on a selected subsequence, the model’s representations remain compatible with dense attention. On switching to dense SDPA, training loss spikes transiently to about 1.12–1.57, then recovers within about 1k–1.5k SDPA steps, and by step 16k all tested schedules match or beat dense-from-scratch (Peng et al., 7 May 2026).

Final losses reported for the schedule comparison are:

Training recipe	Final loss
Dense SDPA baseline	0.7237
LH $\mathcal{O}(S^2 d)$9 SDPA 12k+4k	0.7102
LH $\mathcal{O}(N S d)$0 SDPA 11k+5k	0.7001
LH $\mathcal{O}(N S d)$1 SDPA 10k+6k	0.6980

These results support the paper’s central claim of recoverability: a model pre-trained mostly with Lighthouse can be turned back into an ordinary dense-attention model with a short continuation and can match or outperform a dense baseline trained from scratch under the same total token budget (Peng et al., 7 May 2026).

The reported speed gains operate at both layer and full-training scale. For single-layer latency on one B200, with $\mathcal{O}(N S d)$2, sparsity about $\mathcal{O}(N S d)$3, Lighthouse is 21× faster than cuDNN SDPA on the forward pass at $\mathcal{O}(N S d)$4K, and 17.3× faster on forward+backward (Peng et al., 7 May 2026). At full-model training scale on the 530M model with 98K context, dense SDPA baseline throughput is about 45.6k tok/s, whereas Lighthouse Stage 1 throughput across ablations is 84k–126k tok/s, roughly a 2× per-step advantage (Peng et al., 7 May 2026).

End-to-end, over the fixed 16k-step and 50.3B-token budget, the dense SDPA baseline requires 303.2 B200-hours and 37.9h, whereas Lighthouse plus recovery requires 179.6–215.7 B200-hours and 22.5h–27.0h. The paper summarizes this as a 1.40× to 1.69× wall-clock speedup while matching or improving final loss (Peng et al., 7 May 2026).

6. Experimental configuration and ablations

The experiments are described as preliminary small-scale LLM pre-training (Peng et al., 7 May 2026). The model is a 530M-parameter Llama-3-style decoder with $\mathcal{O}(N S d)$5, 30 layers, 8 heads, head dim 128, FFN 1536, and a byte-level tokenizer. Layers $\mathcal{O}(N S d)$6 remain dense SDPA, while the other 26 use Lighthouse (Peng et al., 7 May 2026).

Training uses the C4 dataset with sequence length 98,304 and global batch 32. Optimization is AdamW with lr $\mathcal{O}(N S d)$7, $\mathcal{O}(N S d)$8, $\mathcal{O}(N S d)$9, weight decay $K$0, linear warmup for 2k steps, grad clip 1, bf16, and FSDP only (Peng et al., 7 May 2026). Hardware for the 98K runs is a single NVIDIA BGX 8×B200 node, with multi-node setups for longer contexts using context parallelism (Peng et al., 7 May 2026).

The primary baseline is full dense SDPA training from scratch with matched architecture, data, and tokens. The paper also compares scorer variants, pooling factors $K$1, levels $K$2, and top-$K$3 budgets $K$4 (Peng et al., 7 May 2026).

Several ablation results are emphasized. For scorer choice, projection-norm and dilated scorer variants do not uniformly dominate one another; results are typically within about 0.01 loss, while the projection-norm scorer is about 9% cheaper in B200-hours (Peng et al., 7 May 2026). At $K$5, the paper reports $K$6: dilated 0.6881, norm 0.6946; and $K$7: dilated 0.6969, norm 0.6921 (Peng et al., 7 May 2026).

For pooling factor at $K$8, the paper reports $K$9: 0.6825, $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$00: 0.6881, and $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$01: 0.6828 (Peng et al., 7 May 2026). For number of levels at $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$02, it reports $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$03: 0.6825, $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$04: 0.6978, and $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$05: 0.6991, with deeper pyramids consistently worse in that setup (Peng et al., 7 May 2026).

For top-$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$06 budget at $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$07, dilated scorer, the values are $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$08: 0.6825, $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$09: 0.6880, $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$10: 0.6890, $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$11: 0.6951, and $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$12: 0.6831 (Peng et al., 7 May 2026). The paper highlights the surprising result that smaller $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$13 often performs better and speculates that stronger compression may regularize at this token budget, while leaving open whether that trend reverses at larger scales (Peng et al., 7 May 2026).

In a simplified Needle-in-a-Haystack evaluation at 98K training, after SDPA recovery, three of four Lighthouse configurations match or beat the dense baseline mean retrieval rate of 0.72: $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$14, dilated 0.76; $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$15, dilated 0.73; $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$16, norm 0.72; $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$17, norm 0.65 (Peng et al., 7 May 2026). This suggests that recovery is not only about training loss.

7. Interpretation, limitations, and relation to deployment

The paper’s practical interpretation is explicit: Lighthouse Attention should be regarded as a pre-training acceleration wrapper for long context rather than as a new serving architecture (Peng et al., 7 May 2026). The workflow is to keep the transformer architecture essentially standard, replace most attention layers with Lighthouse during the bulk of long-context pre-training, train faster because attention is performed on a selected compressed subsequence, disable Lighthouse near the end, continue with ordinary dense SDPA for a short recovery phase, and deploy the recovered dense model (Peng et al., 7 May 2026). At inference time after recovery, there is therefore no architectural change or extra inference-time complexity; the deployed model is a standard full-attention transformer (Peng et al., 7 May 2026).

This design also explains one of the principal limitations. Because Lighthouse symmetrically pools queries, it assumes that all queries are present in one forward pass. That is natural during training but not in autoregressive decoding, where queries arrive one at a time. The paper explicitly states that this makes the method not a direct autoregressive inference method (Peng et al., 7 May 2026). A plausible implication is that the method belongs to the class of train-time approximations whose operational value is measured by their compatibility with later dense evaluation and serving.

A second limitation is that the inner attention remains quadratic in the gathered length: $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$18 Thus the method is subquadratic in the original $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$19, but not strictly linear unless $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$20 remains bounded and moderate. The paper does not characterize regimes in which $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$21 must grow substantially with $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$22 (Peng et al., 7 May 2026).

A third limitation concerns evidence scale. The reported experiments use a 530M model and 16k steps and are repeatedly described as preliminary (Peng et al., 7 May 2026). The paper does not establish behavior at frontier model scales. It explicitly leaves open whether the favorable small-$Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$23 trend persists at larger token budgets, how well the method scales in quality to much larger models, whether adaptive per-layer or per-head $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$24 would help, whether recovery could target a serveable sparse architecture rather than dense SDPA, and how the training-time gains translate into serving systems with batching, speculative decoding, and KV-cache constraints (Peng et al., 7 May 2026).

Within those stated limits, Lighthouse Attention occupies a specific position in the long-context training landscape. Its novelty is not merely hierarchical compression or sparse scoring in isolation, but the combination of symmetrical pooling of $Q = X W_Q,\quad K = X W_K,\quad V = X W_V,\qquad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\tfrac{Q K^\top}{\sqrt{d} + M\right) V,$25, parameter-free and gradient-free hierarchical selection, dense FlashAttention on a gathered contiguous subsequence, causal scatter-back reconstruction, and a two-stage training recipe centered on recovery to full dense attention (Peng et al., 7 May 2026). The paper’s main claim is that these ingredients jointly permit faster long-context pre-training while preserving the ability to ship an ordinary dense-attention transformer.