---
title: 'MMR-Bench: Adaptive Routing in MLLMs'
url: https://www.emergentmind.com/topics/mmr-bench
type: topic
---

# MMR-Bench: Adaptive Routing in MLLMs

MMR-Bench is a comprehensive benchmark for adaptive routing among multimodal large language models (MLLMs). It targets the core issue in practical MLLM deployment—no single model is uniformly optimal across the heterogeneous spectrum of vision–language tasks and computational efficiency constraints. MMR-Bench presents a controlled, offline framework for quantifying and comparing routing strategies, enabling precise cost–accuracy trade-offs using pre-tabulated model outputs and costs over a broad suite of multimodal tasks. Its design incorporates explicit modality-aware fusion, standardized budget constraints, and representative policies from trivial bounds to parametric decision rules, defining a foundation for scalable, efficient, and adaptive model selection in real-world scenarios [2601.17814].

## 1. Motivation and Problem Scope

Vision–language workloads require diverse operational capabilities, such as lightweight OCR, complex diagram reasoning, or general visual question answering (VQA). The MLLM ecosystem is highly heterogeneous in architecture, alignment, and inference cost: compact open-weight models are efficient for routine perception, while large commercial APIs dominate complex reasoning but incur high latency or financial cost. Traditional approaches that deploy a single 'jack-of-all-trades' MLLM cannot simultaneously optimize for both utility and efficiency—forcing a trade-off between over-provisioned compute on simple tasks and loss of accuracy on difficult ones. Query-level routing addresses this by allocating model resources per-instance, but in the multimodal domain faces new challenges: joint representation of text-image signals, nonuniform model cost profiles, and lack of standardized benchmarks for budget-sensitive routing [2601.17814].

## 2. Dataset Composition and Task Suite

MMR-Bench is constructed as an offline table of $\approx 11,000$ multimodal instances $x_i=(x^{\mathrm{text}}_i, x^{\mathrm{img}}_i)$, paired with:

- Modality availability vectors $\mathbf{m}_i\in\{0,1\}^2$ (presence of text/image),
- Model utility vectors $\mathbf{u}_i=(u_{i,1},...,u_{i,K})\in[0,1]^K$ (normalized accuracy or task-specific score of each of $K$ candidate models),
- Model cost vectors $\mathbf{c}_i=(c_{i,1},...,c_{i,K})\in\mathbb{R}_+^K$ (normalized inference cost based on provider pricing, e.g., OpenRouter \$ per 1M output tokens).

Tasks cover three primary scenarios:

1. **Document-centric OCR and understanding**: Includes benchmarks such as OCRBench and SEED-Bench V2 Plus; queries range from simple text extraction to layout-conditioned generation.
2. **General VQA and grounding**: MMStar, RealWorldQA and related benchmarks; queries span natural images, charts, and scene analysis with both open-ended and multiple-choice formats.
3. **Multimodal math and diagram reasoning**: MathVista, MathVerse, MathVision; queries require compositional reasoning over diagrams, equations, and text.

This variety ensures that model utility and cost profiles vary nontrivially across instances, making routing meaningful and non-degenerate [2601.17814].

## 3. Modality Fusion and Routing Feature Space

Effective multimodal routing demands representations that encode both visual and textual complexity. MMR-Bench supplies:

- **Frozen CLIP-based text/image embeddings** for every instance.
- Routable feature spaces: $\phi_{\mathrm{text}}(x)$ (text-only), $\phi_{\mathrm{img}}(x)$ (image-only), and fused multimodal $\phi_{\mathrm{mm}}(x)$.

Fusion is facilitated via a confidence estimator—using prototype similarity scores and embedding norms per modality—followed by softmax weighting, inclusion of the elementwise product, and feature difference:

$$
z_i = w_{\mathrm{txt}}\,x_{\mathrm{txt}} + w_{\mathrm{img}}\,x_{\mathrm{img}} + \alpha(x_{\mathrm{txt}} \odot x_{\mathrm{img}}) + \beta|x_{\mathrm{txt}} - x_{\mathrm{img}}|
$$

This design enables routers to exploit multimodal cues for optimal policy learning [2601.17814].

## 4. Candidate Model Pool and Cost Normalization

The routing space is defined over a diverse pool of ten models:

| Type                                | Example Models         | Cost Range (\$/1M tokens) |
|-------------------------------------|------------------------|---------------------------|
| Commercial API                     | GPT-5-0807, Claude 3.7, Gemini 2.5 | \$10–\$3                |
| Open-weight VL backbone             | InternVL3 78B, Qwen2.5-VL (3B, 7B, 72B), Gemma3 4B | \$0.03–\$0.14           |

Each model's raw cost per 1M output tokens is normalized: $c_{i,j} = \frac{\text{Cost}_{\text{raw}}(j)}{\max_{k}\text{Cost}_{\text{raw}}(k)}$. This provides uniform budgeting across models and queries, supporting direct metric computation of utility–cost trade-offs [2601.17814].

## 5. Routing Policies and Formalism

Routing policies in MMR-Bench range from trivial to complex, including:

| Policy             | Description                                      |
|--------------------|--------------------------------------------------|
| RandomRouter       | Uniform selection over candidate models           |
| OracleRouter       | Selects model with true maximal utility given cost|
| KNNRouter          | Nearest-neighbor selection in feature space      |
| KMeansRouter       | Cluster-based assignment                         |
| LinearRouter       | Linear regression on utility/cost                |
| LinearMFRouter     | Low-rank matrix-factorization variant            |
| MLPRouter          | Shallow multi-layer perceptron                   |
| MLPMFRouter        | Matrix-factorization style shallow MLP           |
| CrossModalRouter   | Cross-modal attention-based selection            |

Parametric routers predict $\{\hat u_{i,j}, \hat c_{i,j}\}$ from input features $z_i$ and select:

$$
j^\star(i;\lambda) = \arg\min_j\, [1 - \hat u_{i,j} + \lambda\,\hat c_{i,j}]
$$

where $\lambda \ge 0$ trades off cost-sensitivity, enabling sweeping along the Pareto frontier $p(c)$ [2601.17814].

## 6. Evaluation Metrics and Empirical Findings

Performance metrics are defined offline by aggregating model outputs:

- **Normalized AUC**:
  $$
  \mathrm{nAUC} = \frac{1}{c_{\max}-c_{\min}} \int_{c_{\min}}^{c_{\max}} p(c) \, \mathrm{d}c
  $$
- **Peak Score**:
  $$
  P_s = \max_c p(c)
  $$
- **Quality-Neutral Cost (QNC)**:
  $$
  \mathrm{QNC} = \frac{1}{c_{\mathrm{best}}} \min\,\{c : p(c) \geq p_{\mathrm{best}}\}
  $$

where $p_{\mathrm{best}}$ and $c_{\mathrm{best}}$ are the accuracy and cost of the strongest single model. QNC $<1$ indicates the router matches best-model accuracy at lower cost.

Core empirical outcomes:

- **Multimodal routing outperforms unimodal**: At matched cost, multimodal policies achieve higher accuracy than text- or image-only policies, especially for image-governed queries.
- **Efficient Pareto gains**: Adaptive routing exceeds the best single model’s accuracy at approximately 33% of its cost under tight budget constraints.
- **Generalization**: Routing policies trained on subsets generalize zero-shot within scenario (e.g., OCR, VQA, math) and even to text-only datasets (GSM8K, MMLU, ARC) by masking the image channel, showing modality-agnostic difficulty modeling [2601.17814].

## 7. Significance, Limitations, and Future Directions

MMR-Bench isolates multimodal model selection as an explicit research task, providing standardized, reproducible cost–accuracy evaluations. It highlights the necessity of multimodal routing for real-world deployments, where unimodal signals systematically misallocate compute. Limitations include reliance on frozen features, fixed candidate pools, and the offline evaluation nature (no 'live' API calls). A plausible implication is that future work should incorporate dynamic state, uncertainty modeling, or retraining on live feedback. The zero-shot generalization of routing policies suggests that difficulty signals learned in multimodal domains transfer robustly, motivating further work on hybrid and cross-modal architectures, dynamic budget management, and multi-agent systems for vision–language model orchestration [2601.17814].

---

MMR-Bench defines a reference framework for adaptive and budget-aware multimodal model routing, enabling empirical benchmarking of policies, illuminating cost–utility trade-offs, and charting pathways for scalable MLLM system design and deployment.

Source: https://www.emergentmind.com/topics/mmr-bench