Block-Recurrent Dynamics in Vision Transformers (2512.19941v1)

Abstract: As Vision Transformers (ViTs) become standard vision backbones, a mechanistic account of their computational phenomenology is essential. Despite architectural cues that hint at dynamical structure, there is no settled framework that interprets Transformer depth as a well-characterized flow. In this work, we introduce the Block-Recurrent Hypothesis (BRH), arguing that trained ViTs admit a block-recurrent depth structure such that the computation of the original $L$ blocks can be accurately rewritten using only $k \ll L$ distinct blocks applied recurrently. Across diverse ViTs, between-layer representational similarity matrices suggest few contiguous phases. To determine whether these phases reflect genuinely reusable computation, we train block-recurrent surrogates of pretrained ViTs: Recurrent Approximations to Phase-structured TransfORmers (Raptor). In small-scale, we demonstrate that stochastic depth and training promote recurrent structure and subsequently correlate with our ability to accurately fit Raptor. We then provide an empirical existence proof for BRH by training a Raptor model to recover $96\%$ of DINOv2 ImageNet-1k linear probe accuracy in only 2 blocks at equivalent computational cost. Finally, we leverage our hypothesis to develop a program of Dynamical Interpretability. We find i) directional convergence into class-dependent angular basins with self-correcting trajectories under small perturbations, ii) token-specific dynamics, where cls executes sharp late reorientations while patch tokens exhibit strong late-stage coherence toward their mean direction, and iii) a collapse to low rank updates in late depth, consistent with convergence to low-dimensional attractors. Altogether, we find a compact recurrent program emerges along ViT depth, pointing to a low-complexity normative solution that enables these models to be studied through principled dynamical systems analysis.

• The paper introduces the Block-Recurrent Hypothesis, showing that Vision Transformers self-organize into a few recurrent blocks using max-cut segmentation.
• It demonstrates that RAPTOR models with k=2–3 blocks preserve high accuracy (up to 98% recovery) by leveraging intra-block similarity and stochastic depth.
• The study underscores that block recurrence enables efficient compression, dynamical system analysis, and improved model interpretability.

The Block-Recurrent Hypothesis: Depth Reuse and Simplicity in Vision Transformers

Introduction: Emergence of Block Structure in ViTs

The study presents a comprehensive exploration of the Block-Recurrent Hypothesis (BRH) for Vision Transformers (ViTs), positing that post-training, the sequential layers in ViT models self-organize into a small number of contiguous phases—blocks—within which computations are functionally similar and amenable to recurrence. This hypothesis is motivated by consistent empirical observations: layer-layer similarity matrices across ViT variants display clear block-diagonal structure, indicating contiguous regions of high representational similarity along model depth.

Figure 1: Layer-layer similarity matrices across diverse Vision Transformers reveal phase-segmented contiguous block structure indicating recurrent computational regimes.

This structural organization raises the critical question: does representational similarity correspond to functional reuse, or is it merely superficial co-variance? The BRH claims the existence of $k \ll L$ distinct blocks, each repeated along depth, that suffice to reconstruct the original ViT's entire representational flow with negligible loss. This represents an implicit simplicity bias in ViTs, trading parameter overparameterization for iterative phase-local computation.

Formalizing Block Discovery: Max-Cut Segmentation

To operationalize BRH, the paper introduces a max-cut algorithm to segment the layer-layer similarity matrix, optimizing for high intra-block similarity and minimal cross-block similarity. This algorithm robustly identifies candidate phase boundaries—locations of sharp representational transition—enabling systematic mapping from raw depth to block assignments.

Figure 2: Max-cut segmentation partitions depth into functionally distinct contiguous blocks, exposing sharp boundaries that coincide with qualitative changes in representational dynamics.

Experimental validation demonstrates that the discovered partitions are predictive of functional compressibility: recurrent surrogates with the indicated partitioning, termed RAPTOR models, are able to closely match teacher ViTs both in representational trajectory and downstream task performance. Critically, swaps of layers within blocks (intra-block) preserve functionality, while inter-block substitutions result in catastrophic accuracy degradation—strongly confirming the functional uniqueness of block recurrence.

Mechanistic Drivers: Training Dynamics and Stochastic Depth

To dissect the origins of block-recurrent structure, the authors systematically vary training procedures and architectural regularization. A major finding is the role of stochastic depth: independently dropping layers at random during training, with probability $p$, increases layer-layer representational similarity and strengthens block structure.

Figure 3: Increasing stochastic depth dropout probability enhances block-wise representational similarity and RAPTOR reconstruction fidelity, demonstrating functional compressibility is promoted by regularization.

RAPTOR models trained to fit these regularized ViTs achieve higher fidelity in reconstructing hidden trajectories, especially for mid-to-high $p$ values. The correlation between representational similarity and recurrent approximator performance underscores that stochastic depth encourages functional compressibility and recurrence. Additionally, training dynamics, as observed during overfitting and removal of skip connections, reveal that block recurrence is not an artifact of initialization or architecture alone but emerges from the interaction of learning and network design.

Constructive Verification: RAPTOR Surrogates at Scale

Scaling the methods to foundation models, RAPTOR surrogates are trained to reconstruct internal activations of DINOv2-Base ViT on ImageNet-1k using only $k=2$–$4$ recurrent blocks, as determined by max-cut segmentation. Training utilizes a hybrid procedure combining teacher forcing and autoregressive objectives to achieve stable, high-fidelity alignment.

Figure 4: Three training paradigms clarify the necessity of closed-loop autoregressive training; only the hybrid protocol yields self-consistent trajectory reconstruction and accurate block recurrence.

Strong numerical findings are reported: a $k=2$ block RAPTOR recovers $96\%$ of DINOv2's linear probe accuracy, with $k=3$ closing the gap to $98\%$, all at iso-compute. Cosine similarity between intermediate representations from RAPTOR and those from DINOv2 exceeds $0.7$ throughout model depth.

Figure 5: Cosine similarity between RAPTOR and DINOv2-Base activations remains high through depth, indicating accurate dynamic reconstruction.

Causal intervention experiments further solidify these claims: swapping layers within blocks preserves function, while inter-block swaps abolish accuracy, demonstrating the necessity of phase-local recurrence.

Figure 6: Intra-block layer substitutions maintain accuracy, whereas inter-block swaps collapse output, verifying the functional uniqueness of block recurrence.

Dynamical Systems Interpretation: Attractors, Low-Rank Collapse, and Token-Specific Dynamics

Interpreting ViT depth as the evolution of a discrete-time dynamical system, the study reveals directional convergence of token representations to class-dependent attractors in angular space. Feature norms grow with depth, but directions stabilize, supporting the existence of angular fixed points. Perturbed trajectories self-correct, evidencing basin stability and contraction toward attractors.

Figure 7: Depth-wise normalized trajectories exhibit collapse into compact basins, with progressive cosine alignment to the final representation, consistent with angular attractors.

Figure 8: Perturbed trajectories revert toward baseline, with patch-token sensitivity decaying, confirming basin stability and self-correcting recurrence.

Token groups display distinct dynamical profiles: late-stage CLS tokens undergo sharp reorientation (aggregation for readout), patch tokens accumulate coherence culminating in high collective alignment, and register tokens stabilize early. The layer-to-layer update matrices progressively collapse to low rank, especially in late phases, indicating that the depth flow contracts to a restricted subspace.

Figure 9: Depth increases drive low-rank collapse of the update matrix and sharp rise in patch-token coherence, revealing collective motion and mean-field effects; DMD fits show weak contraction and token-specific spectral features.

Empirical dynamic mode decomposition (DMD) confirms that depth-wise propagation can be linearly approximated with eigenvalues clustered just inside the unit circle, indicating predominantly angular updates with mild contraction and multiple long-memory directions for CLS token evolution.

Theoretical Perspective: Algorithmic Complexity and Simplicity Bias

A theoretical analysis shows that block-recurrent ViTs have low Levin complexity—they admit concise program representations with schedule and tied block parameter descriptions, at constant computational cost. This is in contrast to classical Kolmogorov complexity; here, the computational cost is preserved while the algorithmic description length shrinks. The existence of such compact algorithms substantiates a simplicity bias in deep learning: ViTs discover a small set of primitives reused via recurrence, with implications for understandability, interpretability, and potential future model design.

Practical and Theoretical Implications

• Interpretability and Mechanistic Analysis: The emergence of block-recurrent structure enables tractable dynamical systems analysis and suggests new approaches for mechanistic interpretability in vision models.
• Compression without Loss of Function: Functional compressibility via recurrence allows substantial simplification with little loss in accuracy, opening avenues for more efficient deployment and distillation.
• Role of Regularization: Stochastic depth and architectural choices can induce or suppress block recurrence, informing training strategies for both accuracy and interpretability.
• Low-Rank and Collective Dynamics: Later layer contraction implies that coordinated, low-dimensional updates drive late-stage inference, pointing toward new directions for extracting key features and representations.

Conclusion

This work advances the block-recurrent hypothesis for Vision Transformers, combining empirical analysis, algorithmic formalism, and large-scale constructive validation. The findings demonstrate that trained ViTs can be rewritten as a compact, recurrent composition of a handful of functionally distinct blocks, with equivalent computational cost and near-equivalent accuracy. These emergent patterns have critical theoretical and practical implications, suggesting an inherent simplicity bias in modern architectures. Future research will benefit from this framework, enabling deeper mechanistic understanding, efficient model compression, and robust interpretability for vision and potentially other transformer-empowered domains.

(2512.19941)