---
title: Multimodal Interaction-Aware Router
url: https://www.emergentmind.com/topics/multimodal-interaction-aware-router
type: topic
---

# Multimodal Interaction-Aware Router

Searching arXiv for the primary paper and closely related router works to ground the article in current literature.
A multimodal interaction-aware router is a routing mechanism that conditions computation, information flow, or model selection on the interaction structure between modalities rather than on a fixed compression rule, a static expert assignment, or a modality-agnostic dispatch policy. In recent work, the term spans several layers of multimodal systems: budgeted evidence allocation in long-video understanding, expert specialization under missing modalities, soft expert separation in multimodal MoE language models, modality-native delivery across agent boundaries, channel-level information routing before fusion, and pre-answer model selection across heterogeneous multimodal model pools. Across these settings, the common objective is to spend limited compute, bandwidth, or token budget where multimodal interactions matter most for the downstream prediction or generation [2605.05848] [2606.30355] [2605.11301].

## 1. Conceptual scope and problem setting

The primary motivation for multimodal interaction-aware routing is that multimodal evidence is rarely uniform. In long-video understanding, visual evidence is described as “sparse and unevenly distributed over time,” so uniform compression wastes budget on redundant frames while indiscriminate high resolution exceeds model limits. VideoRouter addresses this by turning long-video token compression into budgeted evidence allocation conditioned on the user’s query [2605.05848]. In incomplete multimodal learning, the central problem is different: some modalities are missing at inference, and representations from incomplete inputs deviate from their full-modality counterparts. MARS therefore routes experts according to the residual $r = z_{\text{full}} - z_{\text{miss}}$, treating missingness-induced representational change itself as the routing signal [2606.30355].

A second motivation is modality imbalance. MoIR argues that modality dominance in VLMs cannot be resolved solely by attention reallocation, because attention can change where the model focuses but cannot create information that is absent. Its router instead modifies information availability by identifying less informative channels and routing complementary content from a stronger modality before fusion [2604.16264]. SMAR addresses a related but distinct problem in multimodal MoE LLMs: naive multimodal adaptation can erode language capability because experts mix modality-specific signals in ways that damage pretrained text competence. Its solution is not hard partitioning, but a soft constraint on the divergence between text and image routing distributions [2506.06406].

A third motivation is systems-level heterogeneity. Some routers select among different models rather than among tokens or experts. Router-Suggest dynamically selects between text-only auto-completion models and VLMs for visually grounded dialog completion; ECVL-ROUTER dispatches multimodal queries between small edge VLMs and large cloud VLMs according to a scenario-specific quality–speed–energy objective; LatentRouter estimates each candidate MLLM’s counterfactual utility before the answer is generated [2601.05851] [2510.27256] [2605.11301]. In multi-agent systems, MMA2A routes voice, image, and text parts in their native modalities across agent boundaries whenever the destination agent declares support for them, rather than collapsing all content to text [2604.12213].

These works collectively distinguish interaction-aware routing from several neighboring ideas. It is not identical to ordinary MoE gating, because the routed object may be a frame budget, a channel slice, an inter-agent message part, or an entire model call rather than a token-to-expert assignment. It is not identical to attention steering, because some methods explicitly alter token content or modality delivery before the main reasoning stage. It is also not reducible to query difficulty estimation: LatentRouter explicitly states that effective MLLM routing must match multimodal requirements to model capabilities, and not merely estimate how hard a query is [2605.11301].

## 2. Core mathematical patterns

Despite their diversity, recent multimodal interaction-aware routers follow a small number of recurring mathematical patterns. One pattern is constrained evidence allocation. VideoRouter defines a visual-token budget
$$
B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,
$$
with $\epsilon \approx 100$, and seeks to maximize task performance under the constraint $\sum_t k_t \le B_{\mathrm{img}}$, where $k_t = N / s_t^2$ depends on the frame-specific compression scale $s_t$. The routing decision first selects a policy variable $p \in \{\text{Global}, \text{Fragment}\}$ and then maps frame relevance to two-tier scales $s_1 = 1$ and $s_0 = 4$ [2605.05848].

A second pattern is privileged routing with deployable imitation. MARS computes
$$
z_{\text{full}} = f_{\text{full}}(x), \qquad z_{\text{miss}} = f_{\text{miss}}(x_{\text{miss}}), \qquad r = z_{\text{full}} - z_{\text{miss}},
$$
and uses a residual router to assign experts during training, while a feature router learns to imitate that routing using only incomplete inputs. The total loss combines task loss, imitation loss, load–importance balancing, and discrepancy-aware noise regularization:
$$
L_{\text{total}} = L_{\text{task}} + \lambda_{\text{imit}} L_{\text{imit}} + \lambda_{\text{LB}} L_{\text{LB}} + \lambda_{\text{noise}} L_{\text{noise}}.
$$
This formulation makes the train-time signal explicitly interaction-aware while preserving test-time deployability [2606.30355].

A third pattern is routing-distribution regularization. SMAR summarizes expert usage by modality through modality routing distributions and penalizes symmetric KL divergence only when it falls outside a tolerance band:
$$
L_{\text{SMAR}} = [d_{\min} - d_{\text{sym-KL}}]_+ + [d_{\text{sym-KL}} - d_{\max}]_+.
$$
This mechanism increases separation if text and image route too similarly and decreases separation if they route too differently, thereby encouraging specialization without rigid expert partitioning [2506.06406].

A fourth pattern is information-level channel routing. MoIR computes per-channel informativeness scores from the singular vectors and singular values of modality token matrices, selects the bottom $k'$ fraction of channels for each modality, and replaces weak channels by gated cross-modal mixtures:
$$
\bar{F}^A_{:,:,d} = (1 - \alpha_d^A)\,F^A_{:,:,d} + \alpha_d^A\,F^B_{:,:,d}.
$$
The key distinction is that the router operates on information density rather than on attention weights alone [2604.16264].

A fifth pattern is utility-based model selection. LatentRouter defines per-model utility as
$$
u_i(x) = y_i(x) - \lambda c_i,
$$
learns a shared per-model score $s_\theta(x,m_i)$ that approximates that utility, and selects
$$
\pi_\theta(x) = \arg\max_{i \in \Omega(x)} s_\theta(x,m_i),
$$
where $\Omega(x)$ masks unavailable models. ECVL-ROUTER uses a related thresholded policy, routing to the edge model when the predicted probability that the SVLM is competent exceeds $\tau$, and evaluating routing with APSP, CA, AIL, and
$$
\mathrm{RCS}(R) = \alpha\,\mathrm{APSP}(R) + \beta\,\mathrm{CA}(R) - \gamma\,\mathrm{AIL}(R).
$$
Both formulations treat routing as a constrained utility problem rather than as fixed dispatch [2510.27256] [2605.11301].

A sixth pattern is temporal interaction decomposition. Time-MoE quantifies temporal multimodal interactions through redundancy, uniqueness, and synergy over time lag, then uses those signals to guide expert routing. High redundancy encourages shared regular experts, high uniqueness encourages diversified routing, and high synergy pushes tokens toward synergy experts that implement cross-modal attention followed by FFN [2509.25678]. This suggests that interaction-aware routing can be grounded not only in static cross-modal similarity but also in lag-dependent structure.

## 3. Representative architectural forms

The architectural realization of a multimodal interaction-aware router depends on what is being routed: token budget, expert traffic, information channels, message parts, or full model invocations.

| System | Routed object | Interaction signal |
|---|---|---|
| VideoRouter | frame resolution and temporal sampling | early query–video interactions |
| MARS | sparse expert assignments | residual $z_{\text{full}} - z_{\text{miss}}$ |
| SMAR | expert-routing distributions | symmetric KL between text and image MRDs |
| MoIR | weak channel slices | SVD-based channel informativeness |
| MMA2A | native vs transcoded message parts | modality support in Agent Cards |
| LatentRouter | model selection | latent communication between routing capsules and model capability tokens |

VideoRouter uses a dual-router design on InternVL3. A Semantic Router predicts whether the query requires broad temporal coverage or concentrated detail, and an Image Router scores frame relevance from early multimodal fusion states extracted by a shadow stack consisting of the first four decoder layers. The routed sequence $Z_{\text{final}}$ then replaces image placeholders before the main LLM cross-attention and decoding stages [2605.05848].

MARS is structurally a sparse MoE with two routers. The residual router operates only during training and routes according to deviation patterns caused by missing modalities; the feature router is a lightweight MLP over $z_{\text{miss}}$ and imitates residual-router behavior for deployment. Expert outputs are aggregated by top-$K$ routing probabilities, and discrepancy-aware noise is injected into residual-router logits to reduce the train–test router gap [2606.30355].

SMAR does not alter the underlying Mixtral 8×7B MoE architecture. Instead, it adds modality-aware bias vectors $b_v$ and $b_t$ to router logits and regularizes the resulting modality routing distributions layer by layer. Its significance lies in being a routing intervention without architectural changes or extra forward passes [2506.06406].

MoIR sits between modality-specific encoders and the LLM decoder. It reshapes token tensors, computes per-modality SVDs, selects less informative channels, applies per-channel gates initialized to $0.5$, and emits enriched token streams $\bar{F}^v$ and $\bar{F}^t$ for downstream decoding [2604.16264].

At the systems layer, MMA2A’s Modality-Aware Router is a lightweight proxy that intercepts `tasks/send` and `tasks/sendSubscribe`, inspects Agent Card declarations such as `inputModes` and `outputModes`, and either forwards `FilePart` objects natively or transcodes them to text. The decision is binary per message part because destination assignment is fixed by the orchestrator [2604.12213]. Router-Suggest is another systems-layer design, but its decision is model choice rather than modality preservation: a lightweight neural classifier over 768D EmbeddingGemma-300m prefix embeddings selects among candidate MAC models per typed prefix [2601.05851].

LatentRouter represents the query with learned routing capsules and each candidate MLLM with a model capability token. Through two layers of latent communication, model tokens read capsule states, compare themselves pool-aware through masked self-attention with pairwise comparison bias, and send decision-aware feedback back to the capsules. A distributional outcome head predicts model-specific counterfactual quality, and a bounded capsule correction refines close decisions [2605.11301].

## 4. Supervision, labels, and optimization regimes

Training regimes for multimodal interaction-aware routers vary substantially because the routed decision is often not identical to the benchmark answer. VideoRouter explicitly trains routers “to allocate evidence—not to answer benchmarks directly.” It introduces Video-FLR-200K, with 200,664 QA pairs and binary frame-necessity labels, and Video-QTR-10K, with 10,000 video-query pairs labeled by dominant allocation policy. Its optimization is staged: Image Router training with binary cross-entropy, Semantic Router training with cross-entropy, and joint fine-tuning with
$$
L = L_{\text{LLM}} + \lambda_s L_{\text{sem}} + \lambda_v L_{\text{img}},
$$
where $\lambda_s = \lambda_v = 0.01$ [2605.05848].

MARS uses paired full-modality and masked-modality views of the same sample. The routing teacher is privileged because it requires complete modalities, while the student feature router is deployable because it sees only incomplete inputs. Missingness patterns are sampled uniformly in early epochs and later prioritized using router disagreement. This training protocol makes the router explicitly aware of interaction changes induced by absent evidence [2606.30355].

Router-Suggest constructs router labels by exhaustive model comparison on each training prefix: every candidate model generates a completion, partial-F1 is computed against the ground-truth completion, and the best model becomes the gold label. The router itself sees only text-prefix embeddings, while multimodal relevance enters implicitly through those performance-derived labels. The training objective combines ordinary cross-entropy with a latency-weighted expected cost term [2601.05851].

ECVL-ROUTER uses a different supervision source. Its Response Score Dataset contains approximately 22k image–text instances scored by an LLM-as-a-Judge on a 1–10 scale, and the label for “edge competency” is defined relative to a Minimal Expectation Score. Threshold $\tau$ is then selected by validation grid search to maximize the scenario-specific routing comprehensive score [2510.27256].

Time-MoE requires an additional estimation stage for temporal interaction dynamics. It trains lag-conditioned discriminators to estimate redundancy, uniqueness, and synergy across lags, and uses Sinkhorn-normalized alignment tensors to approximate the marginal-matching distributions required by the decomposition. Those estimated temporal interaction signals then supervise routing behavior through auxiliary losses on redundancy consistency, uniqueness diversity, and synergy promotion [2509.25678].

SMAR, by contrast, requires no routing labels at all. Its supervisory signal is purely regularization-based: it computes batch-level modality routing distributions in each MoE layer during multimodal fine-tuning and adds the KL band-penalty to the task loss [2506.06406]. This contrast illustrates two broad optimization regimes within the literature: routers trained from explicit routing supervision, and routers trained from differentiable structural constraints on expert usage.

## 5. Empirical behavior and system trade-offs

Representative results show that interaction-aware routing is typically evaluated not only by downstream accuracy but also by token count, latency, memory, cost, or model-utilization balance.

| System | Representative result | Reported trade-off |
|---|---|---|
| VideoRouter | up to a 67.9% token reduction; 7,748 tokens, TTFT $\approx 4.7$ s, peak memory 23.9 GB vs dense 16,384 tokens, TTFT $\approx 8.4$ s, peak memory 36.5 GB | lower prefill cost and memory under query-aware compression |
| MARS | CASIA-SURF average ACER 2.43 vs DMRNet 3.58; CREMA-D average 65.52% vs 61.35% | robustness to missing modalities with moderate overhead |
| SMAR | 86.6% language retention with only 2.5% pure-text | specialization without severe language degradation |
| MMA2A | 52% task completion accuracy vs 32% for text-bottleneck baseline | accuracy gain with a $1.8\times$ latency cost |
| Router-Suggest | 2.3×–10× speedup over the best-performing VLM | lower latency with near-best PR-F1 under router configurations |
| MoIR | VizWiz accuracy 28.22 $\rightarrow$ 32.47; unchanged prediction rate under image noise 62.13% $\rightarrow$ 29.63% | better modality balance and greater sensitivity to visual evidence |
| LatentRouter | 0.792 / 0.786 on MMR-Bench and 0.815 / 0.812 on VL-RouterBench | strongest non-oracle routing in performance-only and performance-cost settings |

The long-video case is especially explicit about the efficiency–accuracy relation. Under a controlled InternVL3-8B comparison with 64 frames, VideoRouter used 7,748 tokens and achieved 66.5 on Video-MME Overall, 61.9 on LongVideoBench, and 72.1 on MLVU, compared with 16,384 tokens and 66.3 / 58.8 / 71.4 for the dense baseline and 7,936 tokens with 65.8 / 59.7 / 71.0 for CLIP Relevance Pooling. The reported interpretation is that query-adaptive policy plus frame-level relevance outperforms relevance-only compression under similar or smaller budgets [2605.05848].

In incomplete multimodal learning, the evidence favors deviation-aware specialization. MARS reports that on CASIA-SURF, average ACER drops to 2.43 from 3.58 for DMRNet, and on CREMA-D the average rises to 65.52% from 61.35% for DMRNet and 57.66% for MMANet. On MCubeS segmentation, MARS reaches 0.4773 mIoU versus 0.4683 for DMRNet, while remaining substantially lighter than some competing MoE-style baselines in parameter count or FLOPs [2606.30355].

Several papers show that interaction-aware routing is not only an accuracy mechanism but also a language-retention or systems-control mechanism. SMAR reports 86.6% average retention across C-EVAL, MMLU, GSM8K, ARC-Challenge, BBH, and MBPP while using only 2.5% pure-text, outperforming both the no-aux-loss and load-balancing-only baselines [2506.06406]. Router-Suggest reports Router-4-L at 0.351 s on MMDD with PR-F1 = 0.240, approximately 5× faster than MiniCPM-V at similar completion quality, and Router-2-L at 0.170 s on MMDD, reported as roughly a 10× speedup versus the best-performing VLM [2601.05851].

At the agent and deployment levels, MMA2A and ECVL-ROUTER show that routing design alters overall system behavior. MMA2A achieves 52% TCA versus 32% for the text-bottleneck baseline on CrossModal-CS, with the largest gains on product defect reports and visual troubleshooting, while paying an approximately $1.8\times$ latency cost [2604.12213]. ECVL-ROUTER reports, for the 38B/1B pair, CA = 0.824, APSP = 0.506 versus 0.549 for All-at-Large, and AIL = 4.53 s versus 7.44 s for All-at-Large; this is summarized as routing over 80% of queries to the small model with less than 10% drop in problem-solving probability [2510.27256].

Finally, routing gains are not uniform across problem types. LatentRouter reports that gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and its ablations show that removing latent communication causes the largest performance drop. Time-MoE reports that redundancy is especially important on PAMAP2, synergy is especially important on MIMIC-IV IHM and Opportunity, and uniqueness is critical across many tasks, indicating that the most useful routing signal is domain-dependent [2605.11301] [2509.25678].

## 6. Limitations, misconceptions, and open directions

A recurring limitation is that routing quality depends on the informativeness of the interaction signal itself. VideoRouter notes that subtle evidence such as tiny objects or brief occlusions may be undervalued by early-layer signals, and that misleading or under-specified queries can cause incorrect policy choice. Under extreme budgets, even prioritized sampling among critical frames may still discard necessary context [2605.05848]. In MoIR, the corresponding risk is misestimating channel informativeness from batch-level SVD; the paper states that these scores can be noisy with small batches or highly diverse content, and that large exchange ratios can overmix modalities [2604.16264].

A second limitation is train–test mismatch. MARS depends on complete modalities during training to compute the privileged residual, so the method is not directly applicable when training data are themselves incomplete without additional assumptions. Its feature router may also fail to imitate residual routing in unseen or extreme missingness regimes, which discrepancy-aware noise only partially mitigates [2606.30355]. LatentRouter exhibits an analogous dependence on calibration quality: model capability tokens must be built from representative calibration statistics, and domain shift or poorly calibrated newly added models can degrade routing until the pool is re-calibrated [2605.11301].

A third limitation is that routing alone is not always sufficient. MMA2A provides a direct ablation: when the final decision agent is replaced by keyword matching, modality-native routing and text-bottleneck routing both achieve 36%, eliminating the accuracy gap. Only with LLM-backed reasoning does the 52% versus 32% gap appear. This establishes a two-layer requirement: protocol-level routing must be paired with a downstream reasoning mechanism capable of exploiting the preserved evidence [2604.12213]. MoIR makes a parallel conceptual point from the opposite direction: attention steering can be insufficient if the underlying modality tokens are information-poor, so routing may need to alter information content rather than only importance weights [2604.16264].

A fourth limitation is sensitivity to routing hyperparameters and granularity. SMAR depends on the KL band $[d_{\min}, d_{\max}]$ and on $\beta$; small bands can cause routing collapse, while overly aggressive separation can harm multimodal performance [2506.06406]. Time-MoE notes that overly large lag windows can degrade the multi-scale RUS estimator, and same-lag alignment is used in practice for efficiency even though cross-lag extensions would be more expressive [2509.25678]. Router-Suggest notes that its embedding-based classifier may be brittle under domain shift and does not directly ingest image features at inference; visual grounding affects routing only through performance-derived training labels [2601.05851].

These limitations frame several open directions already stated in the literature. VideoRouter suggests streaming or chunked routing and extension to audio or other modalities; MoIR suggests uncertainty-aware or layerwise routing and extension to more modalities; LatentRouter emphasizes pool changes and cold-start model insertion; ECVL-ROUTER proposes privacy-first or scenario-aware policies over edge and cloud pools; Time-MoE points toward richer temporal interaction estimators and broader modality sets [2605.05848] [2604.16264] [2605.11301] [2510.27256] [2509.25678]. A plausible implication is that “multimodal interaction-aware router” is not a single architecture class but a unifying systems principle: routing should be driven by how modalities interact under the current query, task, or deployment constraint, not by a fixed global allocation rule.

Source: https://www.emergentmind.com/topics/multimodal-interaction-aware-router