RecursiveLink Modules in RecursiveVLM

• RecursiveLink Modules are compact adapters in recursive Transformer architectures, merging features across layers to refine multimodal representations.
• They employ RMSNorm and modality-specific projections with learnable residual scaling to normalize and fuse vision and text features.
• Empirical results show accuracy improvements of up to +7.2 percentage points over naive baselines, demonstrating efficiency and scalability.

RecursiveLink Modules are compact, per-selected-layer adapters designed for recursive Transformer architectures in large multimodal models (LMMs). They enable efficient parameter reuse with on-demand representation refinement, combining layer-wise normalization, modality-specific projections, and learnable residual scaling. RecursiveLink Modules are situated within the RecursiveVLM framework, connecting outputs from multiple Transformer layers and feeding fused representations recursively, thereby addressing scale and distribution mismatches present in naive recursive or parameter-sharing schemes (Xu et al., 9 Feb 2026).

1. Placement and Functional Role within RecursiveVLM

In RecursiveVLM, the backbone is a standard Vision-Language Transformer decoder with depth $L$ (e.g., 28 layers), processing concatenated vision $(V)$ and text $(T)$ token embeddings through $L$ shared Transformer layers. Instead of a single forward pass, the model undergoes $R$ recursive loops. At each recursion $t$, the initial embeddings $E^{(1)} = [V^{(1)}, T^{(1)}]$ are propagated through the Transformer stack, producing intermediate hidden states $H_1^{(t)}, \ldots, H_L^{(t)}$ per loop.

After each loop—except the final one—a RecursiveLink Module is applied at a user-selected subset of $k$ layers $$S = \{ \ell_1, \ldots, \ell_k \} \subset \{1, \ldots, L\}$$ (commonly spaced at quartiles of depth). The module normalizes the hidden states, projects and fuses them, and integrates their outputs with the original embeddings to construct the inputs $(V)$0 for the next recursion. This placement makes the RecursiveLink Module an inter-loop fusion and refinement adapter rather than an intra-layer block.

2. Mathematical Definition of RecursiveLink Modules

At recursion step $(V)$1 and each selected layer $(V)$2, the RecursiveLink Module operates on hidden states partitioned by modality: $(V)$3. The transformation consists of the following stages for each modality $(V)$4 (vision, text):

1. Normalization:

$(V)$5

2. Modality-specific projection and residual scaling:

$(V)$6

where $(V)$7 if $(V)$8 and $(V)$9 if $(T)$0; $(T)$1 is a trainable scale initialized to 0; $(T)$2 is a nonlinearity (such as GELU); $(T)$3 are up- and down-projection matrices; and “$(T)$4” denotes element-wise scaling.

3. Fusion into the next-step embeddings:

$(T)$5

The new input embedding is $(T)$6.

Through this construction, the RecursiveLink Module aggregates multi-layer features, aligns statistical scales, and projects them compatibly across recursion steps, crucially respecting modality-specific distributions.

3. Training with Monotonic Recursion Loss

RecursiveVLM employs a Monotonic Recursion Loss to supervise every recursion step. At recursion step $(T)$7, the standard cross-entropy loss is computed:

$(T)$8

With $(T)$9 output tokens, token-wise loss adjustments enforce monotonic improvement:

$L$0

where $L$1 (set to $L$2). The loss for each step is $L$3. The total loss is summed over all recursion steps: $L$4. This enforces that each recursion’s outputs are at least as good as prior ones and discourages any performance regression.

4. Empirical Performance and Benchmark Outcomes

RecursiveVLM with RecursiveLink Modules was instantiated on a Ming-Lite-Omni LMM backbone (28 Transformer layers, 3B parameters). Pretraining and supervised fine-tuning were conducted on datasets comprising 9M multimodal QA + 3M text and 8M CoT + 2.6M text samples. Recursion depth was set to $L$5, $L$6, and $L$7. Evaluation across eight multimodal benchmarks (AI2D, MM-Star, MM-Vet, MMMU, MMB, MathVista, OCRBench, HallusionBench) yielded:

• At recursion step two, accuracy gains of +3.0 percentage points (pp) over standard non-recursive Transformers.
• +7.2 pp over naive (vanilla) recursive baselines.
• Even a single recursive step consistently improved performance by +1–2 pp, indicating enhanced representations and optimization stability (Xu et al., 9 Feb 2026).

5. Implementation and Deployment Practices

RecursiveLink Modules are designed for minimal overhead:

• All connector weights are small, modality-specific MLPs; no backbone parameters are added.
• RMSNorm and connector add $L$8 parameter and compute overhead, where $L$9 is the number of selected layers.
• Initialization of residual scalers to zero ensures identity mapping at the first step, allowing safe adoption from any pretrained Vision-LLM.
• For latency reduction, $R$0 can be pruned or connector weights quantized.
• Deployment is adaptable: with $R$1 or 2, the modules deliver accuracy boosts with minimal compute, suitable for resource-constrained environments. For maximum precision, larger $R$2 (e.g., $R$3) further refines outputs, reducing the incidence of hallucinations.

A summary of deployment options and trade-offs:

Recursion Depth ($R$4)	Inference Mode	Performance
1-2	Mobile/Edge	Strong, efficient
3+	Cloud/Server	Higher, refined

6. Formal Definition and Usage Guidelines

RecursiveLink Modules (also denoted “Recursive Connector”) are adapters operating across recursion steps and selected Transformer layers, comprising:

1. Hidden state normalization via RMSNorm;
2. Modality-specific MLPs with learnable residual scalers per vision/text channel;
3. Summation/fusion into the next-step input embeddings;
4. Distinct, per-modality projections to maintain statistical independence and optimize for vision-language signal structure.

These modules address hidden state misalignment that arises in naive parameter reuse, regulate distributional scale, and facilitate robust, on-demand recursive refinement.

Guidelines for effective usage:

• Uniformly space $R$5 layers $R$6 to capture multi-scale features.
• Employ separate projection matrices for each modality.
• Initialize residual scalers to zero.
• Supervise each recursion step with monotonic penalty to prevent degradation.
• Select recursion depth $R$7 to balance device compute against target accuracy (Xu et al., 9 Feb 2026).

RecursiveLink Modules thus underpin a flexible, efficient method for recursive representation refinement in large multimodal models, with demonstrated empirical advantages and negligible parameter overhead.