Characterize jointly informative and nonredundant layer combinations

Determine which combinations of target-model hidden layers are jointly informative rather than geometrically redundant for multi-layer conditioning in speculative decoding.

Background

The paper studies hidden-state selection for EAGLE3-style speculative decoding, in which multiple hidden states from a target decoder are fused and supplied to a draft model. Existing approaches commonly select layers using fixed positional heuristics such as [2, N//2, N-3], even though selecting an optimal subset is combinatorial.

The unresolved issue is whether particular layer combinations provide complementary information or instead encode substantially redundant representations. The paper addresses this issue empirically using matrix entropy of concatenated hidden states as an informativeness measure and centered kernel alignment (CKA) to assess representational redundancy, but the general characterization of favorable layer combinations remains unresolved.

References

Scientifically, it remains unclear which layer combinations are jointly informative versus geometrically redundant.

— Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding  (2610.12939 - Goel et al., 9 Oct 2026) in Introduction, paragraph beginning “This creates both a practical and scientific gap.”

Drafters can thus inherit the long-context ability of their target models, which shifts the open question from how distant entries are positioned to how they are attended to. FOLD controls how much weight they receive, and richer ways of reading the memory may exploit it further.

— FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding  (2610.20392 - Li et al., 14 Oct 2026) in Section 5, Conclusion