Milder assumptions for per-token marginal influence identifiability

Determine whether a milder assumption than the stated lag-decay condition suffices to establish the paper’s model-independent bound for the per-token marginal influence, and whether the normalization used for that statistic is appropriate.

Background

The paper proves near-identifiability of the per-token marginal influence between two LLMs whose next-token logits are sufficiently similar, under assumptions of Jacobian smoothness, preceding-token dependence, and lag decay. The lag-decay assumption requires the normalized dependence of predictions on increasingly distant tokens to decrease according to an envelope that is independent of sequence length and embedding dimension.

The authors note that lag decay is partly a consequence of the chosen normalization and that empirical measurements show slow, approximately power-law decay. The unresolved issue is whether the theorem can be obtained under a weaker assumption and whether the current normalization is the correct one for this statistic.

References

Whether a milder assumption suffices for this statistic, and whether its normalisation is the right one, is an open question.

— Emergent Structure in the Marginal Attention Space of Language Models  (2610.03109 - Maiorca et al., 2 Oct 2026) in Appendix, Section “Theory: measurements,” Section “Proofs,” subsection “Assumptions”