---
title: Recurrent Routing & Self-Rethinking
url: https://www.emergentmind.com/topics/recurrent-routing-and-self-rethinking
type: topic
---

# Recurrent Routing & Self-Rethinking

Recurrent routing and self-rethinking are architectural strategies for neural networks aimed at enabling dynamic, input-adaptive computation—moving beyond fixed, feedforward transformation. These techniques allow a model to iteratively revisit, refine, and recombine intermediate representations by leveraging controllers that repeatedly route information through pools of computational units or experts. The result is a form of iterative, example-specific reasoning within both vision and language models, with demonstrated gains in robustness, expressivity, and computational efficiency across multiple benchmarks [1811.04380] [2501.07890].

## 1. Architectural Foundations of Recurrent Routing

In both vision (ReSet) and language (GRAPHMOE) domains, the central innovation is replacing fixed sequential block structures with modules that contain a pool of expert or computational units (CUs) and a learnable routing controller. The controller selects, at each iteration, a subset or mixture of these units for processing the current state, and the process repeats over several rounds.

In ReSet [1811.04380], ResNet’s standard residual block sequence is segmented into “ReSet blocks,” each orchestrating $k$ recurrent routing steps through a small pool of $n$ learnable CUs—each a residual block with independent parameters. The routing for each input is governed by a lightweight controller (CNN- or RNN-based), producing a distribution over the pool at each iteration. The selected units’ outputs are combined and added residually to the current feature state, allowing for state refinement across iterations.

GRAPHMOE [2501.07890] extends this paradigm to language models, embedding multi-expert MoE transformations in Transformer decoder layers. Each MoE layer is augmented with a “virtual node” implemented via a low-rank GRU. Routing is performed recurrently: at each round, the router uses the refined current state—incorporating prior expert mixtures—to select a top-$k$ subset of experts. The outputs from selected experts are aggregated and fed through the GRU, producing a residual message that updates the layer’s hidden state for the next routing round.

## 2. Mathematical Formalism and Routing Controllers

The routing controller, central in both models, computes a distribution over available computational units conditional on the current feature and, in recurrent variants, the evolving hidden/controller state.

- For ReSet, the RNN-based controller at iteration $t$ produces $[\pi_t, h_{t+1}] = C(x_t, h_t)$, where $\pi_t \in \mathbb{R}^n$ are expert-selection logits. After softmax normalization,
  $$
  y_{t,i} = \frac{\exp(\pi_{t,i})}{\sum_{j=1}^n \exp(\pi_{t,j})}
  $$
  The next state is composed as $x_{t+1} = x_t + \sum_{i=1}^n y_{t,i} F_i(x_t)$, allowing both soft mixtures and cycles.

- In GRAPHMOE, the router $R^\ell$ projects the feature to $n$ dimensions, and after applying softmax and top-$k$ sparsification, yields mixture weights $s_t^\ell \in \mathbb{R}^n$:
  $$
  \hat{s}_t^\ell = \text{Softmax}(R^\ell x_t^\ell), \qquad s_t^\ell = \text{Top-}k(\hat{s}_t^\ell)
  $$
  The expert outputs are $y_t^\ell = \sum_{i=1}^n s_t^\ell[i] E_i^\ell(x_t^\ell)$, where each $E_i^\ell$ is either a LoRA-adapted FFN (language) or a learnable CU (vision).

A recurrent controller (LSTM in ReSet, GRU in GRAPHMOE) provides the capacity for multi-step, state-aware routing, enabling the model to refine decisions with each pass.

## 3. The Self-Rethinking Mechanism

Self-rethinking refers to the capacity of the model to iteratively “revisit” its previous routing and computation, refining its intermediate representation before handing it off to the next layer or stage.

In ReSet [1811.04380], repeated routing enables cycles—the network may reuse or skip CUs multiple times within the block. The LSTM controller’s hidden state $h_t$ encodes information about past decisions, supporting dynamic adjustment and sharpening. Empirical analysis shows that early iterations in ReSet blocks are similar across inputs, while later iterations diverge semantically.

GRAPHMOE [2501.07890] formalizes self-rethinking as message passing on a pseudo-graph. Each inference round, the virtual node aggregates the expert mixture via a GRU, with the hidden state update:
\[
z_t^\ell=\sigma(W_z^\ell[h_{t-1}^\ell;y_t^\ell])
\]
\[
r_t^\ell=\sigma(W_r^\ell[h_{t-1}^\ell;y_t^\ell])
\]
\[
\tilde h_t^\ell=\tanh(W_o^\ell[r_t^\ell\odot h_{t-1}^\ell;y_t^\ell]+b_o^\ell)
\]
\[
h_t^\ell=(1-z_t^\ell)\odot h_{t-1}^\ell + z_t^\ell\odot\tilde h_t^\ell
\]
\[
g_t^\ell = W_g^\ell h_t^\ell, \quad x_{t+1}^\ell = x_t^\ell + g_t^\ell
\]
After $T$ iterations, the final refined output $y_T$ is passed up the stack. This process yields deeper, input-sensitive reasoning within each MoE layer.

## 4. Training Objectives, Regularization, and Handling Discrete Routing

Both approaches augment standard per-task losses with regularizers and techniques to stabilize or diversify routing:

- ReSet [1811.04380] applies an entropy-based regularization:
  - $-\lambda_1$ times the entropy of $E_x[y_t]$ (encouraging global diversity of paths)
  - $+\lambda_2$ times the expectation over $x$ of $H(y_t)$ (encouraging each sample’s routing to be sharp/deterministic)
  - A skip-reward term biases the controller toward skipping computation when appropriate, introducing a trade-off between accuracy and compute.
  - Methods for differentiating through discrete routing include REINFORCE (unstable), Gumbel-Softmax, and Straight-Through Gumbel-Softmax.

- GRAPHMOE [2501.07890] minimizes
  $$
  \mathcal{L}_{\mathrm{total}} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{LB}
  $$
  where $\mathcal{L}_{LB}$ is a load-balancing loss that penalizes unbalanced expert usage:
  $$
  f_i = \frac{1}{|\mathcal{M}|} \sum_{(x,y)\in\mathcal{M}} \mathbf{1}\{s_x[i]>0\}, \qquad p_i = \frac{1}{|\mathcal{M}|} \sum_{(x,y)\in\mathcal{M}} s_x[i]
  $$
  $$
  \mathcal{L}_{LB} = n\sum_{i=1}^n f_i p_i
  $$

Implementation requires backpropagation through time over the recurrent routing steps.

## 5. Empirical Findings and Model Behavior

Empirical analyses highlight multiple benefits and properties of recurrent routing and self-rethinking:

| Model    | Domain   | Routing Mechanism     | Self-Rethinking Implementation | Core Empirical Gains                                                                |
|----------|----------|----------------------|-------------------------------|-------------------------------------------------------------------------------------|
| ReSet    | Vision   | Recurrent CU routing | LSTM controller, recurrent residual CU selection | Improved accuracy and generalization on CIFAR-10.1; up to 40% CU skipping with small accuracy cost; class/sample adaptive routing [1811.04380] |
| GRAPHMOE | Language | Recurrent MoE routing| Virtual node (GRU) aggregator | +2.1–2.3 points avg. gain across 8 commonsense benchmarks vs. prior LoRA+MoE; SOTA with MixLoRA/MixDoRA; >20% improved expert utilization uniformity [2501.07890] | 

- In ReSet, RNN-based controllers outperform CNN-based: they reliably replicate the original ResNet order, support nontrivial input-dependent routing, and can skip or reuse CUs flexibly.
- In GRAPHMOE, accuracy increases with self-rethinking rounds up to a point (2–3 passes), then declines (“overthinking”).
- Routing diversity: In both models, visualization of controller distributions demonstrates that early iterations remain generic, while later rounds show class- or sample-specific routing.

A plausible implication is that recurrent routing enables a trade-off between depth (more reasoning rounds) and computational cost, adjustable even at inference time.

## 6. Related Methodologies and Integrations

GRAPHMOE [2501.07890] explicitly integrates LoRA-based parameter-efficient finetuning, broadening applicability to large frozen pretrained models. The MoE expert weights are composed of frozen base weights plus learned low-rank adapter matrices:
\[
\tilde W = W + B A
\]
This enables scalable deployment and extension of the self-rethinking mechanism to large language model settings.

ReSet [1811.04380] demonstrates compatibility with established deep architectures (ResNet), requiring only local modifications to residual stage structure.

Both approaches make the routing controller and expert pool sizes tunable architectural hyperparameters, and allow for additional structure such as a skip/identity unit to control computational budget.

## 7. Significance and Prospects

Recurrent routing and self-rethinking represent a significant shift toward adaptive, iterative processing in neural architectures. By enabling models to “rethink” and refine intermediate representations via input-dependent multi-step computation within a layer, these approaches confer benefits in robustness, accuracy, and efficiency, as well as improving expert utilization in mixture models.

Open directions include investigation of scalability to deeper, more structured pseudo-graphs, refinement of routing regularization strategies, improved handling of overthinking, and further integration with other architectural innovations. The empirical evidence across both vision and language domains confirms the impact and versatility of recurrent routing and self-rethinking as design principles [1811.04380][2501.07890].

Source: https://www.emergentmind.com/topics/recurrent-routing-and-self-rethinking