Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers

Abstract: Recent recursive Transformer studies have primarily reused shared parameters across computation steps to construct compact, parameter-efficient models. In this work, we leverage recursion to build effectively deeper Transformers with stronger representational capacity. However, in Vision Transformers, simply increasing recursion depth does not reliably improve performance, as existing recursive approaches do not fully utilize the intermediate representations produced throughout recursive computation. We propose Soft Mixture-of-Recursions (SoftMoR) and its Vision Transformer instantiation, Soft Recursive Vision Transformer (SR-ViT). SoftMoR learns token-wise mixture weights to softly combine outputs from all recursion steps, allowing intermediate representations to be utilized in a learnable and flexible way. Across diverse vision tasks, SR-ViT consistently improves as recursion depth increases with minimal parameter overhead. On ImageNet-1K, increasing recursion depth from 1 to 4 improves SR-ViT-S top-1 accuracy from 79.83% to 82.48% with only 1.7M additional parameters, outperforming the substantially larger DeiT-B while using approximately 27% of its parameters. These results demonstrate that SoftMoR provides a parameter-efficient path to deeper and stronger Vision Transformers through recursion.

• The paper proposes SoftMixture-of-Recursions (SoftMoR), a novel mechanism that uses learnable token-wise soft aggregation to combine outputs from multiple recursion steps in Vision Transformers.
• It demonstrates that deeper recursive computation enhances performance on tasks like image classification and segmentation while keeping parameter growth minimal.
• The study validates that a coarse-to-fine recursive organization and lightweight step transforms effectively improve representation diversity and computational efficiency.

Soft Mixture-of-Recursions: Deeper Vision Transformers via Flexible Recursive Computation

Introduction and Motivation

Scaling Transformer-based vision architectures has established itself as a principal route to enhance visual representational capacity. However, brute-force parameterization through stacking more layers incurs severe computational and memory costs. Recursive Vision Transformers (ViTs) have emerged as a parameter-efficient alternative by repeatedly applying shared blocks, but such methods have been primarily leveraged for model compression rather than systematic scaling of capacity. Crucially, empirical evidence demonstrates that simply deepening recursive computation by increasing recursion steps—a naive approach—does not guarantee consistent performance improvements beyond shallow depths. This limitation is rooted in under-utilization of rich intermediate recursive representations, as most prior works either restrict final outputs to the deepest recursion state or rely on hard token-wise routing, discarding intermediate outputs.

The paper "Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers" (2607.00774) introduces a new architectural mechanism—Soft Mixture-of-Recursions (SoftMoR)—and its ViT instantiation, Soft Recursive Vision Transformer (SR-ViT). The main innovation is the use of learnable, token-wise mixture weights over all recursion steps, enabling effective aggregation of representations traversing different recursion depths. This soft mixture mechanism, complemented by lightweight step transforms, maximizes the utility of recursion for scaling representational power without proportionally increasing parameters. The formulation is evaluated across image classification, object detection, instance segmentation, and semantic segmentation, demonstrating significant performance gains and parameter efficiency surpassing larger non-recursive baselines.

Methodology: Recursive Computation with Soft Aggregation

The SR-ViT architecture generalizes traditional ViTs by segmenting the backbone into a sequence of recursive units interleaved with non-recursive blocks. Each recursive unit comprises $K$ sequential Transformer layers with parameters shared across $T$ recursion steps. The resulting effective depth is thereby $K \times T$. All recursion-step outputs are made available for subsequent aggregation.

SoftMoR replaces one-hot (hard) recursion-step selection strategies with a softmax-weighted sum. For each token, a lightweight routing head predicts $T$ logits, which are normalized across recursion steps:

$$\alpha_n^{(t)} = \frac{\exp(Z_n^{(t)})}{\sum_{k=1}^T \exp(Z_n^{(k)})}$$

where $Z_n^{(t)}$ is the $t$-th logit for token $n$. The token's output is then the weighted sum over all recursion steps:

$$\mathbf{X}_n^{\rm out} = \sum_{t=1}^T \alpha_n^{(t)}\, \mathbf{X}_n^{(t)}$$

To enhance inter-step diversity, a learned step transform—a token-wise linear projection with residual connection—is injected between recursion steps. The overall transformer architecture (SR-ViT) organizes recursive units in a "coarse-to-fine" fashion to maximize representation progression across depth.

Figure 1: Overview of SR-ViT with SoftMoR, detailing recursive unit organization, soft mixture aggregation, step transforms, and learned spatially structured mixture maps.

Empirical Results Across Multiple Vision Tasks

Image Classification

SR-ViT demonstrates clear monotonic improvement as recursion depth increases, with minimal parameter inflation. For instance, SR-ViT-S escalates from 79.83% to 82.48% Top-1 on ImageNet-1K as recursion depth rises from $T=1$ to $T$0, while parameter count grows only by 1.7M. This accuracy ($T$1) notably outperforms the non-recursive DeiT-B by 0.68pp using only 27% of its parameters, under comparable computational budgets.

Figure 2: SoftMoR scaling on SR-ViT-S; increasing recursion depth correlates directly with improved accuracy across tasks.

Further, distillation protocols are compatible, pushing accuracy above 85% at high resolutions. Notably, naive recursion without soft mixture fails to provide consistent gains as depth increases past $T$2; only SoftMoR unlocks continued benefit from recursive computation.

Object Detection, Instance Segmentation, and Semantic Segmentation

Transfer experiments using Mask R-CNN (COCO) and UPerNet (ADE20K) underscore the generality of SoftMoR; deeper recursive backbones yield incremental improvements in both box/mask AP and mIoU, with SR-ViT variants at $T$3 matching or surpassing DeiT-B using fewer parameters (and lower FLOPs for the backbone). For example, SR-ViT-S outperforms DeiT-B on ADE20K multi-scale mIoU with one-third the parameter count.

Qualitative Analyses

Visualization of learned mixture weights reveals meaningful spatial adaptation: SoftMoR assigns recursion steps different spatial roles, often correspondingly emphasizing objects, boundaries, or backgrounds in a context-sensitive manner, with patterns becoming increasingly diverse as $T$4 increases.

Figure 3: Visualized learned token-wise mixture weights, where blue, red, and green reflect increasing recursion steps, and darker indicates greater token-step impact.

Additional visualizations on COCO and ADE20K reinforce the semantic coherence of mixture weights, demonstrating that SoftMoR indeed learns to leverage recursive depth heterogeneously depending on spatial and contextual cues.

Figure 4: COCO: Mixture weights displayed for recursion depths T=2,3,4; map colors reflect recursion-step allocation per token.

Figure 5: ADE20K: Mixture weight maps reveal that tokens in structurally important scene regions integrate distinct recursion steps.

Analysis of Recursive Unit Organization and Representation Dynamics

Ablative studies clarify that both soft mixture and step transform are required for effective utilization of increased recursion. Hard selection (as in MoR) provides marginal improvement only at shallow depth and quickly saturates or regresses as $T$5 grows; in contrast, soft mixture scales reliably.

Moreover, experiments on recursive unit granularity (single coarse unit, fine-grained units, fine-to-coarse, and coarse-to-fine) demonstrate that only "coarse-to-fine" organizations preserve representational progression across effective depth, as evidenced by CKA analysis. The layer similarity drops off more sharply for coarse-to-fine, mirroring the trend in non-recursive ViTs, while other structures exhibit redundant or abrupt transitions.

Figure 6: Token-level linear CKA heatmaps across recursive organization strategies; coarse-to-fine yields gradual, non-redundant representation evolution.

Figure 7: Mean CKA decreases fastest with depth in the coarse-to-fine setup, indicating higher representational diversity per step.

Implications and Future Directions

The methodological advance introduced by SoftMoR establishes recursion not merely as a means to compress ViTs, but as a scalable and flexible axis for augmenting network capacity. By utilizing all intermediate recursive states and integrating them through dynamically learned, spatially varying mixtures, SoftMoR circumvents the depth-saturation failure mode observed in both naive recursion and hard routing paradigms.

From a practical perspective, this positions recursive ViTs as strong backbones for compute/memory-constrained applications, since competitive accuracies are achievable at a fraction of the parameter budget (with costs predominantly shifted to increased computation, which is more easily parallelizable in most modern hardware settings).

Theoretically, the demonstrated success of token-wise, soft mixture aggregation over recursion steps opens new research directions: architectural exploration into even more granular or hierarchical mixture strategies, further integration with local and global routing policies, or extension to multimodal and temporally recursive domains.

Conclusion

Soft Mixture-of-Recursions and its ViT instantiation SR-ViT significantly advance the depth scaling efficiency of Vision Transformers. SoftMoR's token-wise aggregation of all recursion-step outputs, empowered by spatially aware mixture weights and enhanced via lightweight step transforms, enables deeply recursive architectures to consistently improve across multiple vision tasks with minimal parameter expansion. This work establishes recursion as a principal and practical scaling dimension for ViTs and sets the foundation for more sophisticated multi-depth aggregation mechanisms in transformer-based models.