Multimodal Interaction-Aware Router

• Multimodal interaction-aware routers allocate computational resources by conditioning decisions on dynamic interplay between modalities rather than fixed, uniform rules.
• They employ strategies such as constrained evidence allocation, residual routing, and channel-level information routing to optimize task performance under resource constraints.
• These routers improve efficiency, accuracy, and robustness by aligning modality-specific signals with system-level goals while addressing challenges like missing data and modality imbalance.

Searching arXiv for the primary paper and closely related router works to ground the article in current literature. A multimodal interaction-aware router is a routing mechanism that conditions computation, information flow, or model selection on the interaction structure between modalities rather than on a fixed compression rule, a static expert assignment, or a modality-agnostic dispatch policy. In recent work, the term spans several layers of multimodal systems: budgeted evidence allocation in long-video understanding, expert specialization under missing modalities, soft expert separation in multimodal MoE LLMs, modality-native delivery across agent boundaries, channel-level information routing before fusion, and pre-answer model selection across heterogeneous multimodal model pools. Across these settings, the common objective is to spend limited compute, bandwidth, or token budget where multimodal interactions matter most for the downstream prediction or generation (Lin et al., 7 May 2026, Baek et al., 29 Jun 2026, Cheng et al., 11 May 2026).

1. Conceptual scope and problem setting

The primary motivation for multimodal interaction-aware routing is that multimodal evidence is rarely uniform. In long-video understanding, visual evidence is described as “sparse and unevenly distributed over time,” so uniform compression wastes budget on redundant frames while indiscriminate high resolution exceeds model limits. VideoRouter addresses this by turning long-video token compression into budgeted evidence allocation conditioned on the user’s query (Lin et al., 7 May 2026). In incomplete multimodal learning, the central problem is different: some modalities are missing at inference, and representations from incomplete inputs deviate from their full-modality counterparts. MARS therefore routes experts according to the residual $r = z_{\text{full}} - z_{\text{miss}}$, treating missingness-induced representational change itself as the routing signal (Baek et al., 29 Jun 2026).

A second motivation is modality imbalance. MoIR argues that modality dominance in VLMs cannot be resolved solely by attention reallocation, because attention can change where the model focuses but cannot create information that is absent. Its router instead modifies information availability by identifying less informative channels and routing complementary content from a stronger modality before fusion (Kim et al., 17 Apr 2026). SMAR addresses a related but distinct problem in multimodal MoE LLMs: naive multimodal adaptation can erode language capability because experts mix modality-specific signals in ways that damage pretrained text competence. Its solution is not hard partitioning, but a soft constraint on the divergence between text and image routing distributions (Xia et al., 6 Jun 2025).

A third motivation is systems-level heterogeneity. Some routers select among different models rather than among tokens or experts. Router-Suggest dynamically selects between text-only auto-completion models and VLMs for visually grounded dialog completion; ECVL-ROUTER dispatches multimodal queries between small edge VLMs and large cloud VLMs according to a scenario-specific quality–speed–energy objective; LatentRouter estimates each candidate MLLM’s counterfactual utility before the answer is generated (Mishra et al., 9 Jan 2026, Tang et al., 31 Oct 2025, Cheng et al., 11 May 2026). In multi-agent systems, MMA2A routes voice, image, and text parts in their native modalities across agent boundaries whenever the destination agent declares support for them, rather than collapsing all content to text (Srinivasan, 14 Apr 2026).

These works collectively distinguish interaction-aware routing from several neighboring ideas. It is not identical to ordinary MoE gating, because the routed object may be a frame budget, a channel slice, an inter-agent message part, or an entire model call rather than a token-to-expert assignment. It is not identical to attention steering, because some methods explicitly alter token content or modality delivery before the main reasoning stage. It is also not reducible to query difficulty estimation: LatentRouter explicitly states that effective MLLM routing must match multimodal requirements to model capabilities, and not merely estimate how hard a query is (Cheng et al., 11 May 2026).

2. Core mathematical patterns

Despite their diversity, recent multimodal interaction-aware routers follow a small number of recurring mathematical patterns. One pattern is constrained evidence allocation. VideoRouter defines a visual-token budget

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$

with $\epsilon \approx 100$, and seeks to maximize task performance under the constraint $\sum_t k_t \le B_{\mathrm{img}}$, where $k_t = N / s_t^2$ depends on the frame-specific compression scale $s_t$. The routing decision first selects a policy variable $p \in \{\text{Global}, \text{Fragment}\}$ and then maps frame relevance to two-tier scales $s_1 = 1$ and $s_0 = 4$ (Lin et al., 7 May 2026).

A second pattern is privileged routing with deployable imitation. MARS computes

$$z_{\text{full}} = f_{\text{full}}(x), \qquad z_{\text{miss}} = f_{\text{miss}}(x_{\text{miss}}), \qquad r = z_{\text{full}} - z_{\text{miss}},$$

and uses a residual router to assign experts during training, while a feature router learns to imitate that routing using only incomplete inputs. The total loss combines task loss, imitation loss, load–importance balancing, and discrepancy-aware noise regularization:

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$0

This formulation makes the train-time signal explicitly interaction-aware while preserving test-time deployability (Baek et al., 29 Jun 2026).

A third pattern is routing-distribution regularization. SMAR summarizes expert usage by modality through modality routing distributions and penalizes symmetric KL divergence only when it falls outside a tolerance band:

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$1

This mechanism increases separation if text and image route too similarly and decreases separation if they route too differently, thereby encouraging specialization without rigid expert partitioning (Xia et al., 6 Jun 2025).

A fourth pattern is information-level channel routing. MoIR computes per-channel informativeness scores from the singular vectors and singular values of modality token matrices, selects the bottom $$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$2 fraction of channels for each modality, and replaces weak channels by gated cross-modal mixtures:

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$3

The key distinction is that the router operates on information density rather than on attention weights alone (Kim et al., 17 Apr 2026).

A fifth pattern is utility-based model selection. LatentRouter defines per-model utility as

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$4

learns a shared per-model score $$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$5 that approximates that utility, and selects

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$6

where $$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$7 masks unavailable models. ECVL-ROUTER uses a related thresholded policy, routing to the edge model when the predicted probability that the SVLM is competent exceeds $$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$8, and evaluating routing with APSP, CA, AIL, and

$$B_{\mathrm{img}} = L_{\max} - L_{\text{text}} - L_{\text{gen}} - \epsilon,$$9

Both formulations treat routing as a constrained utility problem rather than as fixed dispatch (Tang et al., 31 Oct 2025, Cheng et al., 11 May 2026).

A sixth pattern is temporal interaction decomposition. Time-MoE quantifies temporal multimodal interactions through redundancy, uniqueness, and synergy over time lag, then uses those signals to guide expert routing. High redundancy encourages shared regular experts, high uniqueness encourages diversified routing, and high synergy pushes tokens toward synergy experts that implement cross-modal attention followed by FFN (Han et al., 30 Sep 2025). This suggests that interaction-aware routing can be grounded not only in static cross-modal similarity but also in lag-dependent structure.

3. Representative architectural forms

The architectural realization of a multimodal interaction-aware router depends on what is being routed: token budget, expert traffic, information channels, message parts, or full model invocations.

System	Routed object	Interaction signal
VideoRouter	frame resolution and temporal sampling	early query–video interactions
MARS	sparse expert assignments	residual $\epsilon \approx 100$0
SMAR	expert-routing distributions	symmetric KL between text and image MRDs
MoIR	weak channel slices	SVD-based channel informativeness
MMA2A	native vs transcoded message parts	modality support in Agent Cards
LatentRouter	model selection	latent communication between routing capsules and model capability tokens

VideoRouter uses a dual-router design on InternVL3. A Semantic Router predicts whether the query requires broad temporal coverage or concentrated detail, and an Image Router scores frame relevance from early multimodal fusion states extracted by a shadow stack consisting of the first four decoder layers. The routed sequence $\epsilon \approx 100$1 then replaces image placeholders before the main LLM cross-attention and decoding stages (Lin et al., 7 May 2026).

MARS is structurally a sparse MoE with two routers. The residual router operates only during training and routes according to deviation patterns caused by missing modalities; the feature router is a lightweight MLP over $\epsilon \approx 100$2 and imitates residual-router behavior for deployment. Expert outputs are aggregated by top-$\epsilon \approx 100$3 routing probabilities, and discrepancy-aware noise is injected into residual-router logits to reduce the train–test router gap (Baek et al., 29 Jun 2026).

SMAR does not alter the underlying Mixtral 8×7B MoE architecture. Instead, it adds modality-aware bias vectors $\epsilon \approx 100$4 and $\epsilon \approx 100$5 to router logits and regularizes the resulting modality routing distributions layer by layer. Its significance lies in being a routing intervention without architectural changes or extra forward passes (Xia et al., 6 Jun 2025).

MoIR sits between modality-specific encoders and the LLM decoder. It reshapes token tensors, computes per-modality SVDs, selects less informative channels, applies per-channel gates initialized to $\epsilon \approx 100$6, and emits enriched token streams $\epsilon \approx 100$7 and $\epsilon \approx 100$8 for downstream decoding (Kim et al., 17 Apr 2026).

At the systems layer, MMA2A’s Modality-Aware Router is a lightweight proxy that intercepts tasks/send and tasks/sendSubscribe, inspects Agent Card declarations such as inputModes and outputModes, and either forwards FilePart objects natively or transcodes them to text. The decision is binary per message part because destination assignment is fixed by the orchestrator (Srinivasan, 14 Apr 2026). Router-Suggest is another systems-layer design, but its decision is model choice rather than modality preservation: a lightweight neural classifier over 768D EmbeddingGemma-300m prefix embeddings selects among candidate MAC models per typed prefix (Mishra et al., 9 Jan 2026).

LatentRouter represents the query with learned routing capsules and each candidate MLLM with a model capability token. Through two layers of latent communication, model tokens read capsule states, compare themselves pool-aware through masked self-attention with pairwise comparison bias, and send decision-aware feedback back to the capsules. A distributional outcome head predicts model-specific counterfactual quality, and a bounded capsule correction refines close decisions (Cheng et al., 11 May 2026).

4. Supervision, labels, and optimization regimes

Training regimes for multimodal interaction-aware routers vary substantially because the routed decision is often not identical to the benchmark answer. VideoRouter explicitly trains routers “to allocate evidence—not to answer benchmarks directly.” It introduces Video-FLR-200K, with 200,664 QA pairs and binary frame-necessity labels, and Video-QTR-10K, with 10,000 video-query pairs labeled by dominant allocation policy. Its optimization is staged: Image Router training with binary cross-entropy, Semantic Router training with cross-entropy, and joint fine-tuning with

$\epsilon \approx 100$9

where $\sum_t k_t \le B_{\mathrm{img}}$0 (Lin et al., 7 May 2026).

MARS uses paired full-modality and masked-modality views of the same sample. The routing teacher is privileged because it requires complete modalities, while the student feature router is deployable because it sees only incomplete inputs. Missingness patterns are sampled uniformly in early epochs and later prioritized using router disagreement. This training protocol makes the router explicitly aware of interaction changes induced by absent evidence (Baek et al., 29 Jun 2026).

Router-Suggest constructs router labels by exhaustive model comparison on each training prefix: every candidate model generates a completion, partial-F1 is computed against the ground-truth completion, and the best model becomes the gold label. The router itself sees only text-prefix embeddings, while multimodal relevance enters implicitly through those performance-derived labels. The training objective combines ordinary cross-entropy with a latency-weighted expected cost term (Mishra et al., 9 Jan 2026).

ECVL-ROUTER uses a different supervision source. Its Response Score Dataset contains approximately 22k image–text instances scored by an LLM-as-a-Judge on a 1–10 scale, and the label for “edge competency” is defined relative to a Minimal Expectation Score. Threshold $\sum_t k_t \le B_{\mathrm{img}}$1 is then selected by validation grid search to maximize the scenario-specific routing comprehensive score (Tang et al., 31 Oct 2025).

Time-MoE requires an additional estimation stage for temporal interaction dynamics. It trains lag-conditioned discriminators to estimate redundancy, uniqueness, and synergy across lags, and uses Sinkhorn-normalized alignment tensors to approximate the marginal-matching distributions required by the decomposition. Those estimated temporal interaction signals then supervise routing behavior through auxiliary losses on redundancy consistency, uniqueness diversity, and synergy promotion (Han et al., 30 Sep 2025).

SMAR, by contrast, requires no routing labels at all. Its supervisory signal is purely regularization-based: it computes batch-level modality routing distributions in each MoE layer during multimodal fine-tuning and adds the KL band-penalty to the task loss (Xia et al., 6 Jun 2025). This contrast illustrates two broad optimization regimes within the literature: routers trained from explicit routing supervision, and routers trained from differentiable structural constraints on expert usage.

5. Empirical behavior and system trade-offs

Representative results show that interaction-aware routing is typically evaluated not only by downstream accuracy but also by token count, latency, memory, cost, or model-utilization balance.

System	Representative result	Reported trade-off
VideoRouter	up to a 67.9% token reduction; 7,748 tokens, TTFT $\sum_t k_t \le B_{\mathrm{img}}$2 s, peak memory 23.9 GB vs dense 16,384 tokens, TTFT $\sum_t k_t \le B_{\mathrm{img}}$3 s, peak memory 36.5 GB	lower prefill cost and memory under query-aware compression
MARS	CASIA-SURF average ACER 2.43 vs DMRNet 3.58; CREMA-D average 65.52% vs 61.35%	robustness to missing modalities with moderate overhead
SMAR	86.6% language retention with only 2.5% pure-text	specialization without severe language degradation
MMA2A	52% task completion accuracy vs 32% for text-bottleneck baseline	accuracy gain with a $\sum_t k_t \le B_{\mathrm{img}}$4 latency cost
Router-Suggest	2.3×–10× speedup over the best-performing VLM	lower latency with near-best PR-F1 under router configurations
MoIR	VizWiz accuracy 28.22 $\sum_t k_t \le B_{\mathrm{img}}$5 32.47; unchanged prediction rate under image noise 62.13% $\sum_t k_t \le B_{\mathrm{img}}$6 29.63%	better modality balance and greater sensitivity to visual evidence
LatentRouter	0.792 / 0.786 on MMR-Bench and 0.815 / 0.812 on VL-RouterBench	strongest non-oracle routing in performance-only and performance-cost settings

The long-video case is especially explicit about the efficiency–accuracy relation. Under a controlled InternVL3-8B comparison with 64 frames, VideoRouter used 7,748 tokens and achieved 66.5 on Video-MME Overall, 61.9 on LongVideoBench, and 72.1 on MLVU, compared with 16,384 tokens and 66.3 / 58.8 / 71.4 for the dense baseline and 7,936 tokens with 65.8 / 59.7 / 71.0 for CLIP Relevance Pooling. The reported interpretation is that query-adaptive policy plus frame-level relevance outperforms relevance-only compression under similar or smaller budgets (Lin et al., 7 May 2026).

In incomplete multimodal learning, the evidence favors deviation-aware specialization. MARS reports that on CASIA-SURF, average ACER drops to 2.43 from 3.58 for DMRNet, and on CREMA-D the average rises to 65.52% from 61.35% for DMRNet and 57.66% for MMANet. On MCubeS segmentation, MARS reaches 0.4773 mIoU versus 0.4683 for DMRNet, while remaining substantially lighter than some competing MoE-style baselines in parameter count or FLOPs (Baek et al., 29 Jun 2026).

Several papers show that interaction-aware routing is not only an accuracy mechanism but also a language-retention or systems-control mechanism. SMAR reports 86.6% average retention across C-EVAL, MMLU, GSM8K, ARC-Challenge, BBH, and MBPP while using only 2.5% pure-text, outperforming both the no-aux-loss and load-balancing-only baselines (Xia et al., 6 Jun 2025). Router-Suggest reports Router-4-L at 0.351 s on MMDD with PR-F1 = 0.240, approximately 5× faster than MiniCPM-V at similar completion quality, and Router-2-L at 0.170 s on MMDD, reported as roughly a 10× speedup versus the best-performing VLM (Mishra et al., 9 Jan 2026).

At the agent and deployment levels, MMA2A and ECVL-ROUTER show that routing design alters overall system behavior. MMA2A achieves 52% TCA versus 32% for the text-bottleneck baseline on CrossModal-CS, with the largest gains on product defect reports and visual troubleshooting, while paying an approximately $\sum_t k_t \le B_{\mathrm{img}}$7 latency cost (Srinivasan, 14 Apr 2026). ECVL-ROUTER reports, for the 38B/1B pair, CA = 0.824, APSP = 0.506 versus 0.549 for All-at-Large, and AIL = 4.53 s versus 7.44 s for All-at-Large; this is summarized as routing over 80% of queries to the small model with less than 10% drop in problem-solving probability (Tang et al., 31 Oct 2025).

Finally, routing gains are not uniform across problem types. LatentRouter reports that gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and its ablations show that removing latent communication causes the largest performance drop. Time-MoE reports that redundancy is especially important on PAMAP2, synergy is especially important on MIMIC-IV IHM and Opportunity, and uniqueness is critical across many tasks, indicating that the most useful routing signal is domain-dependent (Cheng et al., 11 May 2026, Han et al., 30 Sep 2025).

6. Limitations, misconceptions, and open directions

A recurring limitation is that routing quality depends on the informativeness of the interaction signal itself. VideoRouter notes that subtle evidence such as tiny objects or brief occlusions may be undervalued by early-layer signals, and that misleading or under-specified queries can cause incorrect policy choice. Under extreme budgets, even prioritized sampling among critical frames may still discard necessary context (Lin et al., 7 May 2026). In MoIR, the corresponding risk is misestimating channel informativeness from batch-level SVD; the paper states that these scores can be noisy with small batches or highly diverse content, and that large exchange ratios can overmix modalities (Kim et al., 17 Apr 2026).

A second limitation is train–test mismatch. MARS depends on complete modalities during training to compute the privileged residual, so the method is not directly applicable when training data are themselves incomplete without additional assumptions. Its feature router may also fail to imitate residual routing in unseen or extreme missingness regimes, which discrepancy-aware noise only partially mitigates (Baek et al., 29 Jun 2026). LatentRouter exhibits an analogous dependence on calibration quality: model capability tokens must be built from representative calibration statistics, and domain shift or poorly calibrated newly added models can degrade routing until the pool is re-calibrated (Cheng et al., 11 May 2026).

A third limitation is that routing alone is not always sufficient. MMA2A provides a direct ablation: when the final decision agent is replaced by keyword matching, modality-native routing and text-bottleneck routing both achieve 36%, eliminating the accuracy gap. Only with LLM-backed reasoning does the 52% versus 32% gap appear. This establishes a two-layer requirement: protocol-level routing must be paired with a downstream reasoning mechanism capable of exploiting the preserved evidence (Srinivasan, 14 Apr 2026). MoIR makes a parallel conceptual point from the opposite direction: attention steering can be insufficient if the underlying modality tokens are information-poor, so routing may need to alter information content rather than only importance weights (Kim et al., 17 Apr 2026).

A fourth limitation is sensitivity to routing hyperparameters and granularity. SMAR depends on the KL band $\sum_t k_t \le B_{\mathrm{img}}$8 and on $\sum_t k_t \le B_{\mathrm{img}}$9; small bands can cause routing collapse, while overly aggressive separation can harm multimodal performance (Xia et al., 6 Jun 2025). Time-MoE notes that overly large lag windows can degrade the multi-scale RUS estimator, and same-lag alignment is used in practice for efficiency even though cross-lag extensions would be more expressive (Han et al., 30 Sep 2025). Router-Suggest notes that its embedding-based classifier may be brittle under domain shift and does not directly ingest image features at inference; visual grounding affects routing only through performance-derived training labels (Mishra et al., 9 Jan 2026).

These limitations frame several open directions already stated in the literature. VideoRouter suggests streaming or chunked routing and extension to audio or other modalities; MoIR suggests uncertainty-aware or layerwise routing and extension to more modalities; LatentRouter emphasizes pool changes and cold-start model insertion; ECVL-ROUTER proposes privacy-first or scenario-aware policies over edge and cloud pools; Time-MoE points toward richer temporal interaction estimators and broader modality sets (Lin et al., 7 May 2026, Kim et al., 17 Apr 2026, Cheng et al., 11 May 2026, Tang et al., 31 Oct 2025, Han et al., 30 Sep 2025). A plausible implication is that “multimodal interaction-aware router” is not a single architecture class but a unifying systems principle: routing should be driven by how modalities interact under the current query, task, or deployment constraint, not by a fixed global allocation rule.