Patchwise Expert Routing in Vision

• Patchwise expert routing is a Mixture-of-Experts mechanism that selectively routes computation at the patch level to focus on semantically relevant image regions.
• Architectural realizations vary across frameworks, featuring per-patch gating in robot learning, token-choice in diffusion transformers, and prototype-based routing in CNNs.
• Empirical and theoretical studies demonstrate that adaptive expert allocation via patchwise routing improves computational efficiency and accuracy in complex vision tasks.

Patchwise expert routing is a class of Mixture-of-Experts (MoE) mechanisms in which routing decisions are made at the level of image patches or patch-like tokens rather than at the level of whole samples. Across contemporary vision systems, the method appears in several distinct forms: per-patch token-choice routing in robot learning, expert-choice routing in diffusion transformers, prioritized patch selection in convolutional MoE, and prototype-based token assignment in guided diffusion MoE. Despite these differences, the common objective is spatially adaptive computation: experts are activated for task-relevant, semantically salient, or class-discriminative regions while irrelevant or redundant regions receive reduced expert processing (Wang et al., 6 Oct 2025, Sun et al., 2024, Chowdhury et al., 2023, Wei et al., 28 Oct 2025).

1. Conceptual basis and problem setting

Patchwise expert routing is motivated by the observation that visual information is not uniformly informative across space. In the theoretical pMoE formulation, each example is decomposed into $n$ disjoint patches, and only one patch contains the class-discriminative pattern while the other $n-1$ patches are class-irrelevant noise patterns. Under that setting, routing only a small number $l \ll n$ of patches to each expert yields both computational and statistical advantages (Chowdhury et al., 2023). This suggests that patchwise routing is especially well matched to tasks in which the decisive signal is localized.

In transformer-based vision models, the same intuition is expressed in token form. EC-DIT begins from a $256\times256$ image, splits it into non-overlapping latent patch tokens after a frozen VAE, and replaces every even-numbered feed-forward layer with an MoE block. The routing objective is not merely sparsity; it is heterogeneous compute allocation aligned with varying text-image complexities, so that high-detail or text-relevant patches can receive disproportionately more expert capacity than flat background regions (Sun et al., 2024). ProMoE makes a related but more explicit claim about visual tokens: compared with language tokens, they exhibit spatial redundancy and functional heterogeneity, which hinders expert specialization in vision MoE unless routing guidance is strengthened (Wei et al., 28 Oct 2025).

In robot learning, VER frames the issue differently. The motivating problem is that individual pretrained vision foundation models excel only in specific domains, while naïve distillation into a single policy representation can produce inflexible task-specific feature selection. VER therefore distills multiple VFMs into a vision expert library and then fine-tunes only a lightweight routing network to dynamically select task-relevant experts for downstream robot tasks. Patchwise Expert Routing with Curriculum Top-K Annealing is the mechanism used to make that selection spatially adaptive and dynamically sparse (Wang et al., 6 Oct 2025).

2. Architectural realizations

The architectural placement of patchwise routing differs substantially across frameworks, even though all of them operate over image patches or patch tokens.

In VER, the backbone is a ViT whose last $N$ transformer layers are converted into an MoE block, called the Vision Expert Library. Each MoE layer contains $L$ expert MLPs $\{E^n_l\}_{l=1\ldots L}$ and one lightweight Robot Router $R^n_{\text{robot}}$. During robot-policy fine-tuning, the backbone and experts are frozen; only the router, which is approximately $0.4\%$ of total parameters, and the policy head are trained. The routing module itself is a per-patch two-layer MLP, Linear→GELU→Linear, that takes the input token for patch $p$ at layer $n-1$0 and outputs $n-1$1 routing logits (Wang et al., 6 Oct 2025).

In EC-DIT, patchwise routing is inserted into the feed-forward path of a diffusion transformer. The base DiT block remains LayerNorm → Self-Attention → (add skip) → [Cross-Attention](https://www.emergentmind.com/topics/cross-attention) → (add skip) → Feed-Forward → (add skip), but every second block’s final FFN is replaced by an MoE block. Routing is computed from the sum of the self-attention output and the cross-attention output, so the dispatch decision incorporates both image and text context (Sun et al., 2024).

In pMoE for CNNs, the architecture is not transformer-based. Instead, there are $n-1$2 parallel two-layer CNN experts, each with its own gating kernel $n-1$3. For expert $n-1$4, the router scores each patch $n-1$5 by $n-1$6, selects the top-$n-1$7 patches, and forwards only those to the expert. The baseline single-expert CNN corresponds to the dense case $n-1$8, $n-1$9 (Chowdhury et al., 2023).

In ProMoE, routing is explicitly decomposed into two sequential stages. Conditional routing first partitions incoming patch tokens into unconditional and conditional subsets according to classifier-free guidance status. Prototypical routing then assigns only the conditional subset to experts by comparing token embeddings with a bank of learnable prototypes, while shared experts contribute across tokens (Wei et al., 28 Oct 2025).

Framework	Patchwise routing rule	Distinctive property
VER	Per-patch top-$l \ll n$0 token-choice gating	Curriculum Top-K Annealing
EC-DIT	Top-$l \ll n$1 patches selected per expert	Perfect balance without auxiliary load-balance loss
pMoE	Top-$l \ll n$2 patches selected per expert by gating kernel	Sample-complexity analysis for CNNs
ProMoE	Hard conditional split, then prototype-based top-$l \ll n$3 routing	Explicit routing guidance via prototypes and contrastive loss

3. Gating functions and expert aggregation

The defining mathematical object in patchwise expert routing is the patch–expert assignment map, but the form of that map varies across systems.

In VER, let $l \ll n$4 be the embedding of patch $l \ll n$5 entering an MoE layer. The router computes raw logits

$l \ll n$6

with optional exploration noise $l \ll n$7, and softmax scores

$l \ll n$8

Curriculum Top-K Annealing restricts the active expert set through

$l \ll n$9

and a binary dispatch mask $256\times256$0 that keeps only the top-$256\times256$1 logits. The final routing coefficient is

$256\times256$2

Each selected expert processes the same patch in parallel,

$256\times256$3

and the routed output is the weighted sum

$256\times256$4

Because experts outside the top-$256\times256$5 have $256\times256$6, the mechanism is sparse at the patch level; because the router is per-patch, the active expert set can vary across spatial locations within the same image (Wang et al., 6 Oct 2025).

EC-DIT uses a different perspective. Instead of each token choosing experts, each expert chooses tokens. With router input

$256\times256$7

the model computes logits

$256\times256$8

then affinities

$256\times256$9

Each expert has capacity

$N$0

and keeps its top-$N$1 patches. The sparse gating tensor is

$N$2

Aggregation is then

$N$3

This routing rule is “patchwise” in the sense that dispatch is still defined over patch tokens, but control is exercised by expert capacity rather than token-local top-$N$4 selection (Sun et al., 2024).

ProMoE introduces a two-step formulation. Conditional routing is deterministic:

$N$5

For the conditional subset $N$6, prototypes $N$7 define routing logits by cosine similarity,

$N$8

with identity activation $N$9. A sparse top-$L$0 vector $L$1 is formed by keeping only the largest entries of $L$2, and the conditional-token output is

$L$3

plus shared expert output. The full token output combines shared, unconditional, and conditional-expert paths (Wei et al., 28 Oct 2025).

The pMoE CNN formulation is simpler but structurally analogous. For expert $L$4, $L$5 denotes the top-$L$6 patches by gating score, and the overall model output is

$L$7

The common pattern across all four formulations is sparse patch dispatch followed by expert-local processing and weighted recombination (Chowdhury et al., 2023).

4. Objectives, regularization, and training regimes

Patchwise expert routing does not imply a single optimization protocol. The role of auxiliary losses, balancing constraints, and parameter freezing differs sharply by application.

In VER’s robot-policy fine-tuning, the only loss is the downstream policy loss $L$8, such as imitation or reinforcement loss, applied after the policy head. PER does not use the mutual-information or load-balancing regularizers that appeared in pretraining. Exploration and balanced expert usage are instead induced by the router noise $L$9 and the wide-to-narrow annealing of $\{E^n_l\}_{l=1\ldots L}$0. In practice, $\{E^n_l\}_{l=1\ldots L}$1 is kept fixed, so the annealing schedule alone acts as an implicit regularizer: large $\{E^n_l\}_{l=1\ldots L}$2 early promotes exploration of many expert combinations, while small $\{E^n_l\}_{l=1\ldots L}$3 late yields sparse, stable routing (Wang et al., 6 Oct 2025).

EC-DIT is trained end-to-end with the same $\{E^n_l\}_{l=1\ldots L}$4 diffusion objective as dense DiT,

$\{E^n_l\}_{l=1\ldots L}$5

No routing-specific loss is added. The reason is structural rather than heuristic: expert-choice routing gives each expert exactly $\{E^n_l\}_{l=1\ldots L}$6 tokens, so perfect balance is guaranteed by construction. The capacity factor $\{E^n_l\}_{l=1\ldots L}$7 then becomes the explicit knob controlling the trade-off between increased compute per expert and the risk that some tokens are dropped completely (Sun et al., 2024).

ProMoE takes the opposite view and adds explicit routing supervision. Its Routing Contrastive Loss defines, for each active expert, a positive centroid $\{E^n_l\}_{l=1\ldots L}$8 from the tokens routed to that expert and uses an InfoNCE-style objective

$\{E^n_l\}_{l=1\ldots L}$9

With $R^n_{\text{robot}}$0, this pulls each prototype toward its own routed-token centroid and pushes it away from the centroids of other experts. The production model does not rely on manual labels, but the paper reports that both explicit routing supervision and implicit $R^n_{\text{robot}}$1-means clustering in preliminary experiments produced large gains over a naive MoE, motivating the semantic guidance strategy (Wei et al., 28 Oct 2025).

In pMoE, both separate-training and joint-training regimes are analyzed. Separate-training first learns the router weights on a router-only objective, fixes them, and then trains the experts by SGD on logistic loss

$R^n_{\text{robot}}$2

Joint-training instead learns router and expert weights simultaneously, with $R^n_{\text{robot}}$3 given by a softmax over router scores. The distinction is important because the strongest theoretical guarantees are presented separately for the fixed-router and jointly trained settings (Chowdhury et al., 2023).

5. Empirical and theoretical evidence

The empirical record shows that patchwise routing changes not only efficiency but also the spatial distribution of representation capacity.

For VER, Patchwise Expert Routing with Curriculum Top-K Annealing outperforms both Framewise/Layerwise Teacher Routing and PER without annealing, especially on tasks requiring fine spatial focus such as pen-in-cup and relocate. Feature-norm visualizations show that PER+CTA suppresses large-norm “outlier” activations in background patches and concentrates high norms on task-relevant regions such as the target pen shape or the bin mouth. Patchwise mutual-information analysis further shows reduced information in irrelevant patches while preserving it in relevant ones. Across 17 diverse robotic tasks, policies using PER+CTA achieve higher success rates with fewer spurious expert activations and less variance across random seeds, and ablations on $R^n_{\text{robot}}$4 show a trade-off between computational cost and final success rate without retraining the backbone (Wang et al., 6 Oct 2025).

EC-DIT reports complementary evidence in diffusion transformers. Against a token-choice MoE baseline with the same activated parameters, expert-choice routing converges faster in training FID and CLIP Score and matches a 16-expert token-choice model with only 8 experts. Increasing the number of experts from $R^n_{\text{robot}}$5 at fixed dense width yields consistent gains in GenEval/DSG alignment and FID without increasing per-token FLOPs. Routing visualizations show that later layers can route up to 44% of experts onto a single high-detail patch, while flat background patches may receive zero expert activations. The largest models achieve a GenEval score of 71.68%, and inference overhead is measured at 20–28% over the dense baseline, somewhat above the theoretical FLOP increase because of sparse-kernel and communication costs (Sun et al., 2024).

The most explicit theoretical support comes from pMoE. For separate-training, the required number of training samples satisfies

$R^n_{\text{robot}}$6

whereas the corresponding single-expert CNN requires

$R^n_{\text{robot}}$7

so that

$R^n_{\text{robot}}$8

For hidden units, the ratio is

$R^n_{\text{robot}}$9

The joint-training result gives a reduction factor in sample complexity of $0.4\%$0 compared to a single CNN, under the stated routing-lock-in assumption. Empirically, on synthetic MNIST-based data, pMoE reaches 95% test accuracy with approximately 60% of the samples required by the CNN baseline, and on CIFAR-10, CIFAR-Vehicles, and CelebA it is reported to be less prone to overfit spurious correlations such as the hair-color cue in CelebA (Chowdhury et al., 2023).

ProMoE evaluates routing quality more directly. Intra-expert coherence is visualized with t-SNE, where assigned tokens form tight, well-separated clusters. Inter-expert diversity is quantified by principal-subspace similarity derived from SVD of expert FFN weights, and ProMoE with Routing Contrastive Loss achieves lower subspace similarity than baselines. These routing changes translate into improved FID and IS on ImageNet; at the same activated parameters, the reported gain includes up to approximately 30% FID reduction relative to prior MoE DiTs (Wei et al., 28 Oct 2025).

6. Design trade-offs, misconceptions, and broader significance

A recurring misconception is that patchwise expert routing is a single routing algorithm. The literature instead contains multiple incompatible routing paradigms. VER uses per-patch token-choice gating with curriculum-scheduled sparsification; EC-DIT uses expert-choice routing with fixed per-expert capacity; ProMoE uses a hard functional split followed by prototype-based top-$0.4\%$1 dispatch; pMoE uses prioritized top-$0.4\%$2 patch selection via expert-specific kernels. The shared object is the patch token, not the routing rule itself (Wang et al., 6 Oct 2025, Sun et al., 2024, Wei et al., 28 Oct 2025, Chowdhury et al., 2023).

A second misconception is that sparse vision MoE must rely on explicit load-balancing losses. EC-DIT demonstrates that expert-choice can enforce perfect balance structurally, so no auxiliary load-balance loss is needed. VER shows a different possibility: during robot-policy training, mutual-information and load-balancing regularizers can be omitted if exploration is induced by router noise and by wide-to-narrow annealing of the allowed expert set. ProMoE, by contrast, argues that explicit routing guidance is crucial for vision MoE, but its added loss is not a load-balancing term; it is a semantic coherence-and-diversity term implemented through Routing Contrastive Loss (Sun et al., 2024, Wang et al., 6 Oct 2025, Wei et al., 28 Oct 2025).

A third misconception is that every patch should always be assigned substantial expert computation. The evidence points in the opposite direction. In EC-DIT, background patches may receive zero expert activations. In VER, successful routing suppresses large-norm outliers in background regions and preserves concentration on task-critical regions. In pMoE theory, the benefit arises precisely because class-irrelevant patches are filtered away from experts that should focus on discriminative patterns (Sun et al., 2024, Wang et al., 6 Oct 2025, Chowdhury et al., 2023).

The principal design trade-offs exposed by these works are sparsity versus stability, compute versus coverage, and semantic flexibility versus explicit guidance. In VER, $0.4\%$3 controls the trade-off between the number of experts per token and final success rate; in EC-DIT, the capacity factor $0.4\%$4 controls the trade-off between increased compute per expert and token dropping; in ProMoE, the addition of prototype guidance and contrastive routing supervision is intended to improve expert specialization. A plausible implication is that patchwise expert routing is becoming a general abstraction for visual conditional computation across robotics, diffusion modeling, and supervised classification, but the optimal routing mechanism remains application-specific.