Generalization of GSRS Attention Dynamics Beyond Video DiTs

Determine whether the attention dynamics observed by Growing Sparsity and Reduced Search (GSRS), and the growing-sparsity and reduced-search mechanisms derived from them, generalize to other architectures such as large language models.

Background

Growing Sparsity and Reduced Search (GSRS) is designed and evaluated specifically for Diffusion Transformer (DiT)-based video generation models. Its acceleration strategy relies on empirical observations that attention becomes increasingly concentrated during denoising and that blocks selected early retain substantial attention weight at later steps. The paper explicitly leaves unresolved whether these attention dynamics and the resulting mechanisms apply to architectures outside video DiTs, including LLMs.