Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference

Abstract: Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks. Using BudgetDoc, we train DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B) that predicts ordinal model performance across budget levels, achieving a 0.753 weighted F1. When dynamically allocating reasoning budgets across five frontier models and three datasets, DRB matches or improves F1 scores compared to always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost. Finally, preliminary evaluations demonstrate DRB's potential to generalize to cross-model selection.

• The paper introduces BudgetDoc and DRB, a roughly 1-billion-parameter multimodal estimator that predicts seven performance classes across five frontier models, five reasoning budgets, and 9,000 labeled document-task outcomes.
• DRB achieves 0.753 weighted F1 and 0.932 recall for near-perfect outputs, enabling budget optimization that reduces costs by 5.2%–99% in 13 of 15 configurations while matching or improving F1 in 9.
• The results show that external pre-flight estimation complements model self-assessment by limiting over-thinking, although latency overhead, weak intermediate-class predictions, and limited cross-provider transfer remain important constraints.

Motivation: the over-thinking penalty and the limits of self-assessment

Modern frontier LLMs expose user-controllable reasoning budgets at inference time, but allocating maximum reasoning compute uniformly across inputs is frequently counterproductive. The paper documents a systematic over-thinking penalty: for many document tasks, additional test-time compute degrades accuracy or merely inflates cost. Gemini 2.5 Flash on RVL-CDIP loses 6% F1 as the budget rises from 0 to 2048 tokens, and GPT-5.2 drops from 0.68 to 0.58 F1 on CheckboxQA moving from low to high effort. Provider-side mitigations—Gemini's dynamic auto-thinking and GPT-5's reasoning_effort tiers—are fundamentally in-flight and self-assessed: the model calibrates its own reasoning based on its own perception of difficulty. The authors argue this is insufficient, citing evidence that 32% of nominally cheaper model pairs cost more in practice due to thinking-token overrun (with up to 9.7× per-query cost variance for identical prompts), and that models' self-predictions of token consumption correlate only weakly with realized usage (Kendall $\tau \leq 0.39$). This motivates an external, pre-flight estimator that predicts, per sample, how well a given model will perform at each budget level before any expensive API call is issued.

The central scientific question is whether a lightweight multimodal model—small enough that its own cost is negligible—can learn this performance surface for much larger frontier LLMs.

The BudgetDoc benchmark

BudgetDoc is the first multimodal benchmark designed specifically for supervised LLM performance estimation. It spans three document-centric tasks chosen to cover distinct difficulty drivers:

• RVL-CDIP (multi-class classification), made harder by concatenating three PDFs from different categories into multi-page documents requiring page-level prediction;
• TAT-QA/TAT-DQA (arithmetic reasoning over semi-structured financial tables);
• CheckboxQA (form-checkbox state extraction from scanned images).

A core set of 360 base samples (120 per task) is exhaustively evaluated across five frontier models (three Gemini variants; GPT-5.1 and GPT-5.2) and five budget levels $b \in \{0, 512, 1024, 1536, 2048\}$ (mapped to effort tiers for the GPT family), yielding a dense combinatorial grid of exactly 9,000 labeled pairs. Per-sample F1 scores are discretized into seven ordinal performance classes, from Class 0 ($f_1 < 0.5$, total failure) to Class 6 ($f_1 \geq 0.99$, perfect output). Prompts are paraphrased to prevent syntactic overfitting, and splits are enforced strictly at the document level: 240 documents for training (6,000 pairs), 20 for validation (500 pairs), and 100 held-out test documents whose complete 2,500-pair grid provides leak-free ground truth for out-of-sample evaluation.

Two caveats bear on generalization claims. The benchmark covers only two provider families and only document-centric tasks where visual layout drives difficulty; and the budget grid is coarse (five levels), so the learned surface is discretized rather than continuous.

DRB architecture and estimation quality

The Document-Reasoning Balancer (DRB) is a ~1B-parameter multimodal estimator comprising four components fused by late concatenation into a 2048-unit MLP head trained with cross-entropy over the seven classes:

• a SigLIP-2-Large-patch16-512 vision encoder (428M parameters) producing page embeddings;
• a 12-layer fusion transformer (hidden dimension 1152) performing cross-page attention and mean pooling;
• a Qwen3-0.6B prompt encoder with a linear projection aligning text and vision dimensions;
• learned embeddings conditioning on target model name and reasoning budget.

On the held-out BudgetDoc test set ($n = 2500$), DRB achieves a weighted F1 of 0.753. The most consequential result is Class 6 recall of 0.932 (F1 = 0.860): DRB reliably detects "easy" samples that succeed at any budget, which is precisely the signal needed to assign zero or minimal thinking tokens without sacrificing accuracy. Estimation is hardest at intermediate classes (F1 of 0.46–0.74 for Classes 1–5), reflecting the stochasticity of partial reasoning successes; the authors note this is acceptable because the downstream savings mechanism depends primarily on correct Class 6 identification, not fine-grained middle-class ranking. Class 1 recall is notably low (0.361), though distinguishing near-total from total failures has little practical consequence since both map to low-cost assignments.

Application I: per-sample budget optimization

DRB's estimates drive budget selection via hierarchical scanning: budgets are queried lowest-to-highest, scanning terminates early upon predicting Class 6, and the highest predicted class wins with lower cost as tie-breaker. Against the default baseline (auto-budget for Gemini, high/xhigh effort for GPT), evaluated on the 100 unseen test documents:

Result	Detail
Cost reduction	13 of 15 configurations reduce cost, ranging from 5.2% to 99%
Accuracy	9 of 15 configurations match or improve F1
Largest F1 gains	gemini-2.5-flash on CDIP (+5.7%), TAT-DQA (+5.2%), CheckboxQA (+5.1%)
Extreme GPT savings	gpt-5.2: −99.0% cost on CDIP, −98.5% on CheckboxQA, −98.2% on TAT-DQA

The GPT results are the strongest demonstration of practical value: extended-thinking GPT models are disproportionately expensive at maximum budget (gpt-5.2 costs $0.096/sample on CDIP and $0.318/sample on CheckboxQA, 10–60× above Gemini), yet most samples do not require that compute. DRB selects minimal budgets and captures nearly all available savings with F1 changes within ±3.2%. Adjusted costs explicitly charge the estimator's own infrastructure overhead ($0.00015/s on a GCP T4), so these figures are net of DRB's compute. Latency effects are mixed: GPT configurations see large latency reductions (up to −85%), while some Gemini configurations incur +20–147% latency overhead from the scanning step—an important operational trade-off the paper reports transparently.

TAT-DQA is consistently the hardest target, with F1 losses up to −5.7% under DRB-guided selection; the performance surface over budgets for hybrid numerical reasoning is irregular, making confident identification of the optimal operating point difficult even for an oracle-quality estimator. Even here, cost savings of 40–98% persist because avoiding expensive configurations requires less precision than ranking them optimally. An appendix comparison against all four GPT effort tiers confirms DRB's F1 nearly matches every tier, with TAT-DQA the sole consistent exception.

Application II: model selection (exploratory)

As a secondary case study, the authors probe whether DRB's learned utility surface transfers to selecting among models—a task it was not designed or trained for. Against an oracle best-single-model baseline, results degrade monotonically with selection scope:

Scope	F1 loss vs. oracle	Adjusted cost savings
Gemini family	−2.5% to −9.9%	9.6–75.9%
GPT family	−9.3% to −10.1%	98.7–99.4%
Cross-family	−6.4% to −11.2%	59.6–95.2%

The pattern is a stark quality-cost trade-off: GPT-family selection sacrifices 9–10% F1 while capturing nearly all cost savings, because DRB systematically routes inputs to low-effort configurations. Cross-family selection compounds calibration mismatch, confirming the paper's own conclusion that dedicated estimators trained with explicit multi-provider supervision are required for routing. These results should be read as preliminary; the authors explicitly disclaim that DRB constitutes a routing system.

Analysis and positioning relative to in-flight controls

Three analytical points stand out. First, the savings mechanism is robust to estimation error: mispredicting adjacent middle classes rarely produces catastrophically wrong budgets, so 0.75 weighted F1 suffices provided Class 6 detection remains reliable. Second, DRB is architecturally complementary to in-flight controls rather than redundant: auto-thinking calibrates how much the model reasons, but cannot override internally perceived reasoning needs—the paper cites cases where dynamic auto-thinking still produced over 60,000 thinking tokens while a comparable model needed 25, yielding realized costs 35% higher despite lower listed prices. A pre-flight estimator imposes a hard cap at API-call time that self-assessment cannot. Third, the value of estimation scales with the gap between baseline and optimal cost: it is most valuable for high-cost models on complex documents, and least valuable for cheap models on short documents where scanning latency can dominate.

Hierarchical scanning exhibits efficient search dynamics, testing on average 1.0–3.9 of 5 budget levels per sample, with a 100% early-stop rate for all Gemini models on CDIP—keeping estimator overhead modest relative to downstream savings.

Limitations and open questions

The paper concedes three principal limitations. Adapting to new providers or architectures requires collecting a fresh calibration sample of configuration-outcome pairs, so the estimator does not transfer zero-shot to unseen model families. Evaluation is restricted to document-centric tasks; extension to non-visual domains such as code generation or open-ended QA remains untested. Finally, hierarchical scanning introduces latency overhead that makes the framework favorable for token-heavy, high-cost inference but unsuitable for ultra-low-latency pipelines. Open questions left by the work include whether explicit multi-provider supervision can close the cross-family F1 gap observed in model selection, and whether finer-grained (continuous-budget) supervision would improve the weak intermediate-class estimates on irregular surfaces like TAT-DQA.

Conclusion

This paper answers affirmatively the question of whether a lightweight multimodal model can estimate frontier LLM reasoning performance on document tasks. DRB, a ~1B-parameter SigLIP-2/Qwen3-based estimator, attains 0.753 weighted F1 on the BudgetDoc benchmark with 0.932 recall on the perfect-performance class, and its estimates translate into up to 99% cost reductions across five frontier models while maintaining or improving F1 in 9 of 15 configurations. The over-thinking penalty is shown to be systematic across modern model families, and external pre-flight estimation is positioned as a necessary complement to in-flight self-assessed controls. Model selection emerges as a promising but immature secondary application, with intra-family transfer encouraging and cross-family transfer motivating dedicated future work.