Confidence-Aware Tool Orchestration for Robust Video Understanding

Abstract: Video reasoning LLMs implicitly assume that every input frame is equally reliable. This leads to what we term the Blind Trust Problem: under realistic perturbations such as motion blur, glare, or occlusion, frontier video reasoning models can suffer 15-30%p accuracy drops on real-world embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods.

• The paper introduces Robust-TO, which integrates per-frame reliability estimation and confidence-calibrated evidence aggregation to address the Blind Trust Problem in Video-LLMs.
• It employs a three-stage pipeline—quality profiling, confidence-guided tool routing, and reliability-tiered synthesis—to adaptively handle corrupted video frames.
• Experimental results show significant improvements in accuracy and latency reduction, with Robust-TO outperforming baselines under realistic video perturbations.

Confidence-Aware Tool Orchestration for Robust Video Understanding

Introduction and Problem Setting

This work addresses a key failure mode in current Video LLMs (Video-LLMs): the blind trust assumption that all sampled video frames are equally reliable for downstream inference. In real-world video data, frames are often corrupted by perturbations such as motion blur, glare, or partial occlusions. The consequence of this assumption is a silent but significant degradation in video reasoning accuracy, particularly under distribution shift and real-world corruptions, while model confidence estimates remain unaffected. The paper formalizes this issue as the Blind Trust Problem, highlighting severe accuracy drops (15–30%) on standard embodied video benchmarks subjected to realistic perturbations.

Robust-TO: Framework Overview

The proposed solution, Robust-TO, is an agentic video understanding architecture that explicitly incorporates frame-level reliability estimation and confidence-calibrated evidence aggregation throughout its reasoning pipeline. The framework couples a unified evidence interface—where all perceptual tool results are packaged with calibrated reliability scores—with a three-stage pipeline:

1. Frame Selection via Quality Profiling: Parametric-free signal-level analysis produces a disturbance profile for each frame, scoring it for blur, brightness deviation, and occlusion. Frames are prioritized based on a multiplicative score of reliability and query relevance, allowing for adaptive exclusion of unreliable, albeit query-relevant, distractors.
2. Confidence-Guided Tool Routing: Natural language queries are decomposed into atomic sub-queries. Each sub-query is assigned to the optimal perception tool based on the dominant corruption signature in its source frames, ensuring that the evidence acquisition process is robustly matched to degradation type.
3. Reliability-Aware Reasoning and Synthesis: Collected evidence tuples—each containing a tool result, calibrated confidence, source frames, and per-frame disturbance metrics—are synthesized in a reliability-aware manner. Evidence is grouped into high, medium, and low reliability tiers. Only consistent medium- and, if necessary, low-reliability signals may supplement but not override high-reliability evidence, and residual epistemic uncertainty is explicitly surfaced when applicable.

The host VLM is trained end-to-end using Group Relative Policy Optimization (GRPO) with a reward signal jointly optimizing answer correctness, evidence reliability, tool-use efficiency, and the minimality of the sub-query decomposition.

Technical Contributions

Signal-Level Frame Profiling

The paper eschews trainable quality assessment in favor of lightweight, signal-based disturbance estimation. By summarizing each frame via the mean of Laplacian-based blur score, normalized mean luminance deviation, and Sobel-magnitude-based occlusion estimation, the framework provides fine-grained corruption-awareness with negligible computational overhead. This enables real-time adaptation to sample-specific and type-specific corruptions without additional supervision.

Adaptive Tool Routing and Unified Confidence Interface

Robust-TO’s routing policy learns a non-trivial, data-driven mapping from the joint space of sub-query semantics and dominant disturbance mode to tool selection. The confidence contract for tools ($c_j$) multiplies intrinsic certainty (e.g., detection score, OCR char-level confidence, tracker IoU, etc.) with input reliability aggregated conservatively (worst-K mean) over source frames. This design prevents overconfident reliance on degraded evidence, a systemic weakness in previous agentic approaches that rely on tool self-assessment decoupled from input quality.

Reliability-Tiered Reasoning

Reasoning is tiered: conclusions are synthesized primarly from high-reliability evidence; lower tiers contribute only if strictly consistent and necessary. The answer interface explicitly reports both the reasoning process and residual uncertainties. This explicit, interpretable evidence attribution is particularly salient for safety-critical or forensic audit use-cases.

GRPO-based End-to-End Training with Confidence-Cost Reward

The GRPO reward combines:

• Correctness: Final answer accuracy.
• Confidence-cost trade-off: Encourages high-confidence evidence and penalizes costly tool calls on degraded inputs.
• Sub-query efficiency: Penalizes both over- and under-decomposition, using a frozen estimator to suggest the minimal sufficient sub-query count.
• Format: Adherence to structured output.

This interplay guides the model toward parsimonious, robust strategies that favor reliable, cost-efficient, and faithful evidence collection and integration.

Experimental Results

Robust-TO yields clear advantages in both clean and corrupted regimes. Key performance highlights include:

• On Urban Video-Bench and VSI-Bench, Robust-TO with Qwen3-VL-7B backbone achieves 56.4% average accuracy on clean data, outperforming Gemini-2.5-Pro (46.2%) and Video-R1 (48.5%).
• Under realistic corruptions (motion blur, glare, occlusion, Gaussian noise, low-light), Robust-TO maintains 54.3% mean accuracy and exhibits the smallest performance drop (2.1% from clean to corrupted) in all benchmarks and backbone settings.
• Adaptive key-frame selection reduces the average processed frames by 35%, lowers inference and training latency, and further boosts accuracy over strong uniform-sampling baselines (+2.0% to +2.1%).
• Ablation studies confirm the impact of each design choice: removal of confidence-cost reward drops accuracy by 2.3 points, and ablations of the reliability interface or worst-K aggregation show up to 7.6 points drop.
• The confidence-guided routing policy achieves substantial task-dependent gains over fixed or query-only routing, particularly when type-specific tools can compensate for particular degradation types (e.g., captioning under blur, action recognition under occlusion).

Practical and Theoretical Implications

By integrating per-frame reliability scores and confidence-aware tool orchestration, Robust-TO sets a new standard in video understanding architectures for embodied, safety-critical, and open-world domains. Practically, the model’s evidence attribution and reporting mechanisms significantly increase the interpretability and trustworthiness of video reasoning systems. This is particularly critical in forensic analysis, automated surveillance, and AV incident investigation, where silent failures or unjustified confidence can have catastrophic implications.

Theoretically, this work demonstrates that sample-wise and type-wise disturbance modeling provides a robust inductive bias for video question answering under out-of-distribution and non-i.i.d. conditions. The results suggest that even with high-capacity multimodal backbones, explicit quality modeling and reliability-based reasoning can outperform naively scaled architectures. Moreover, the unified (result, confidence) abstraction enables plug-and-play extension to new tools and corruption types with minimal retraining.

Future Directions

Limitations noted in the study include a focus on three disturbance types (blur, brightness, occlusion) and the use of a frozen, inference-based estimator for sub-query decomposition complexity. Future work may extend the disturbance vocabulary (e.g., handling adversarial perturbations, semantic occlusion, or audio-visual desynchrony), refine decomposition strategies, and investigate adversarial robustness of the disturbance estimator itself. The plug-and-play interface design readily enables incorporation of new perceptual primitives and evaluation in additional multimodal and adversarial settings.

Conclusion

The Blind Trust Problem is a critical bottleneck for the deployment of Video-LLMs in realistic, safety-sensitive domains. Robust-TO addresses this by agentizing video reasoning with explicit, dynamic, and interpretable reliability estimation at both the evidence acquisition and reasoning levels. It produces state-of-the-art robustness and maintains high accuracy under challenging degradations at low computational and latency cost. The approach is validated through strong empirical gains, rigorous ablation, and qualitative case studies. Its modular interface and signal-driven reliability mechanism form a compelling foundation for next-generation robust embodied video understanding systems (2606.26904).