Cause of PI-induced timing degradation under RoPE scaling

Determine whether the slowdown of periodic sounds and temporal desynchronization observed when extending SoundReactor’s context window via Position Interpolation (PI) on Rotary Positional Embeddings (RoPE) is caused by PI’s scaling of positions that lowers the effective RoPE angular frequency. Characterize the mechanism by which RoPE frequency scaling impacts timing in interleaved, frame-aligned audio–visual token sequences and establish conditions under which NTK-aware interpolation or sliding-window attention preserve synchronization.

Background

The paper investigates zero-shot context-window extension for SoundReactor using three approaches: Position Interpolation (PI), NTK-aware interpolation (NTK), and Sliding Window Attention (SWA). Empirically, PI degrades timing and slows periodic sounds (e.g., footsteps), while NTK and SWA preserve temporal alignment.

The authors conjecture that PI’s position-domain scaling alters RoPE frequencies, reducing high-frequency positional components needed for precise audio–visual timing. They hypothesize that NTK’s base rescaling maintains these components, explaining its superior synchronization compared to PI. A formal verification of this causal mechanism remains outstanding.