LTDR: Long-Tailed Distribution-aware Router

• LTDR is a routing mechanism that explicitly adapts expert activation to address long-tailed data distributions across modalities.
• It relaxes load-balancing constraints for vision tokens, ensuring rare but critical elements receive tailored expert attention.
• Empirical results show LTDR improves accuracy and reduces errors in both vision-language and vision-only tasks with minimal efficiency loss.

A Long-Tailed Distribution-aware Router (LTDR) is a routing mechanism designed for mixture-of-experts (MoE) architectures to explicitly address long-tailed or imbalanced data distributions—a ubiquitous characteristic in vision, language, and multimodal model training. LTDR methods adapt both the routing logic and the expert activation strategy to the specific statistical properties of head and tail components within the data, yielding improved generalization, especially for rare (tail) cases.

1. Core Principles and Motivation

LTDR is motivated by the observation that standard MoE and routing frameworks assume either uniform input distributions or attempt to load-balance expert usage indiscriminately, often under-serving rare but semantically critical elements—commonly termed tail tokens or tail classes. In large vision-LLMs (LVLMs), vision tokens empirically exhibit long-tailed occurrence frequencies, in contrast to language tokens which are typically more uniform. This statistical discrepancy can negatively affect both the specialization and utility of MoE experts, as well as downstream performance on rare or under-represented categories.

The LTDR concept incorporates explicit distribution-awareness into the routing mechanism. It enables dynamic, input-type-sensitive routing and, in some variants, intentionally increases redundancy or expert exposure for tail cases, compensating for their statistical under-representation and inherently higher learning difficulty (Cai et al., 2 Jul 2025).

2. Modality- and Distribution-Aware Routing

A defining feature of LTDR frameworks is the adoption of routing policies explicitly tailored to the empirical input distribution. For example, in LVLMs, the router is adapted as follows (Cai et al., 2 Jul 2025):

• Language tokens: Standard load balancing loss is retained, enforcing uniform expert utilization suited to the (approximately) uniform distribution of language tokens.
• Vision tokens: The load balancing constraint is relaxed or disabled, allowing routing probabilities to more accurately reflect the intrinsic long-tailed distribution. This increases the variance of token-to-expert assignments for vision tokens, so important but rare visual elements (“tail tokens”) are not forced into the same routing regime as frequent “head” tokens.

Mathematically, for token $x$ and expert $i$,

$$\mathcal{P}(x)_i = \frac{\exp(f(x)_i)}{\sum_{j=1}^K \exp(f(x)_j)}$$

is the router probability; the load-balancing loss only applies to language tokens.

3. Enhanced Expert Activation for Vision Tail Tokens

LTDR further addresses the issue of insufficient specialization for vision tail tokens by adaptively oversampling their exposure to experts—without duplicating tokens, but instead disabling the usual top-$k$ expert restriction for those tokens. This is operationalized as follows (Cai et al., 2 Jul 2025):

• Compute Routing Probability Variance (RPV) per vision token:

$\mathrm{RPV}(v_i) = \mathrm{Var}(\mathcal{P}(v_i))$

• If $\mathrm{RPV}(v_i)$ is higher than the mean among all vision tokens, label $v_i$ as a “tail” token.
• For these, increase the number of activated experts from $k$ (baseline) to $a$ (often $a=K$, all experts):

$$\mathrm{MoE}(v_i) = \sum_{j=1}^{a} \mathcal{P}(v_i)_j \cdot \mathcal{E}(v_i)_j$$

This selective expert oversampling ensures robust representation and learning for vision tokens that are both rare and information-rich.

Notably, only approximately 13% of tokens (those identified as tails via high RPV) are routed to an expanded set of experts, maintaining computational efficiency.

4. Relationship to Contemporary Architectures and Methods

LTDR extends beyond standard MoE routing, which typically load-balances tokens across experts regardless of distribution, and contrasts with approaches such as cluster-based, instruction, or task-based routing. Empirical benchmarking on vision-language and vision-only tasks consistently shows LTDR surpassing these alternatives, due to its ability to tailor expert specialization and coverage to the underlying data statistics (Cai et al., 2 Jul 2025).

Some recent frameworks for long-tailed recognition (e.g., RIDE (Wang et al., 2020), DQRoute (Wei et al., 27 Aug 2025)) incorporate similar principles:

• RIDE uses dynamic expert activation (via a lightweight learned routing module) and a diversity loss scaled by class frequency, reducing both bias and variance in model predictions.
• DQRoute integrates class difficulty estimation and decentralized expert routing, assigning higher weight to intrinsically hard (often tail) classes, and leveraging mixture-of-experts fusion without a single centralized router.

LTDR shares with these approaches the central insight: explicit modeling of distributional characteristics—frequency, difficulty, and uncertainty—markedly increases generalization for tail cases.

5. Empirical Benchmarks and Measured Impact

LTDR has been validated across a wide range of vision-language and vision-only benchmarks:

Domain	System	Avg. Accuracy Gain vs. Baselines	Efficiency Impact
Vision-Language	LTDR	+0.4–1.2% (across backbones)	Negligible (<13% tokens rerouted)
Vision-only	LTDR	~+0.9%	Minimal

• LTDR achieves consistently higher scores than prior MoE-LLaVA and GMoE frameworks on datasets such as GQA, ScienceQA, TextVQA, PACS, and DomainNet.
• Ablation studies confirm that both distribution-aware routing and tail token oversampling are necessary for observed improvements.
• In object hallucination benchmarks (e.g., POPE), LTDR reduces hallucination rates and improves semantic grounding relative to prior MoE methods.

A plausible implication is that LTDR's approach—explicit, data distribution-sensitive expert allocation—generalizes across modalities and scales robustly with model size.

6. Downstream Implications and Research Directions

LTDR methods demonstrate the necessity of adapting MoE and router logic to the true, often highly skewed, statistics of modern datasets. This has several implications:

• Uniform load balancing may be fundamentally suboptimal for modalities or domains with long-tailed or otherwise heterogeneous token/class distributions.
• Adaptive, distribution-aware routing mechanisms (as in LTDR) are required for optimal use of overparameterized models, particularly for rare class or instance specialization.
• Future research may investigate further integration of confidence, difficulty, and density signals for even more granular control of routing, as well as broadening these methods to unsupervised or few-shot regimes (e.g., generalized category discovery under long tail (Zhao et al., 14 Jun 2025)).

The LTDR paradigm highlights a critical, currently indispensable, dimension in mixture-of-experts modeling: the explicit encoding and handling of real-world data distribution properties at the routing and expert-allocation level, with measurable effects on both efficiency and fairness.