Detect in Any Scene: An Agentic Framework for Object Detection with Experience-Aware Reasoning

Abstract: Object detection in real-world scenarios remains challenging due to diverse image degradations and heterogeneous object distributions, which significantly hinder the generalization of existing detectors. Conventional approaches, including scene-specific representation learning and end-to-end pipeline design, are inherently limited by their reliance on predefined conditions and lack adaptability to dynamic environments. In this paper, we propose DetAS, an agentic detection framework that formulates object detection as a dynamic decision process. Instead of relying on static pipelines, DetAS leverages a Multimodal LLM (MLLM) as a central agent to adaptively compose detection workflows by selecting from a toolbox of restoration modules and specialized detectors. Specifically, DetAS consists of two key components: Self-Adaptive Image Restoration, which dynamically determines whether and how to enhance images for downstream detection, and Multi-Expertise Detection, which integrates multiple domain-specialized detectors and resolves their predictions through instance-level reasoning. To further improve decision quality under fine-grained conditions, we introduce Self-Evolving Experience Harvesting and extend the framework to DetAS-X, which accumulates node-level decision experience from a small set of annotated data and enables experience-aware reasoning during inference. This mechanism allows the system to progressively refine its decision policy and adapt to diverse real-world scenarios. Extensive experiments on six challenging benchmarks demonstrate that DetAS-X significantly outperforms existing MLLM-based detectors, achieving an average improvement of 28.36% in F1 score, with up to 37.01% gain on DarkFace. These results demonstrate the promise of agentic detection and establish a solid foundation for its application in complex and dynamic environments.

• The paper introduces DetAS-X, an agentic object detection framework that dynamically orchestrates adaptive image restoration, multi-expertise detector fusion, and experience harvesting.
• Its modular architecture, comprising SAIR, MED, and SEEH, enables context-sensitive decision-making, leading to statistically significant F1 improvements—including a 37.01% gain on DarkFace.
• The work demonstrates practical scalability in real-world scenarios, while highlighting future extensions for domain-specific adaptations and lifelong learning.

Agentic Object Detection via Experience-Aware Reasoning: A Review of DetAS-X

Motivation and Context

Robust object detection in the wild remains elusive due to heterogeneous degradations (e.g., haze, rain, underwater distortions, low light) and nonstationary object distributions. Existing paradigms, notably scene-specific representation learning and static end-to-end pipelines, suffer from brittle generalization, as they presume fixed environmental priors and adaptation is either coarse or heavily manual. Recent advances in multimodal LLMs (MLLMs) and agentic reasoning suggest an opportunity to upgrade visual systems by elevating detection from rigid functional composition to adaptive, input- and context-sensitive decision processes.

DetAS and DetAS-X: Architecture and Agentic Pipeline

The proposed DetAS framework casts object detection as a dynamic decision process, realized through a multimodal agent that orchestrates restoration and detection components tailored to the current scene. The architecture is organized into three major modules: Self-Adaptive Image Restoration (SAIR), Multi-Expertise Detection (MED), and Self-Evolving Experience Harvesting (SEEH).

Figure 1: Overview of the DetAS-X framework showing SAIR, MED, and SEEH modules composing adaptive detection through LLM-enabled decision-making and experience harvesting.

SAIR leverages an LLM-based perception module to infer scene-specific degradations and selects an optimal sequence of restoration tools (dehazing, deraining, denoising, brightness enhancement, SR, etc.). A detection-centric criterion, rather than classical image quality, governs whether restoration is applied, ensuring only transformations that benefit downstream detection are utilized.

MED addresses categorical and environmental heterogeneity via a detector pool. For each image, the agent considers both visual context and object category requirements to select $K$ appropriate detectors from distinct specializations (general-purpose, dense/small-object, autonomous-driving, drone-view, underwater, face, etc.). Detector outputs are post-processed by an instance grouping mechanism that combines spatial and visual similarity metrics, and final decisions are refined at the instance level using LLMs for efficient allocation of token context and precise reasoning.

SEEH extends the system to DetAS-X by enabling node-level experience acquisition and retrieval. Upon deployment, a small annotated corpus is used to empirically evaluate diverse decision configurations at restorer, super-resolution, and detector choice nodes. Scene profiles—incorporating degradation type, visibility, object scale, density, and illumination—are used to index accumulated experience, which is subsequently integrated into inference-time reasoning through top-$K$ nearest neighbor retrieval. This augments LLM decision-making beyond fixed rule sets, imbuing the agent with tailored, context-aware priors and facilitating progressive domain adaptation as new data arrive.

Empirical Results and Analysis

Experimental validation on six complex benchmarks (HazyDet, MARIS, DarkFace, B-Night, B-Rainy from BDD100K, and standard COCO) provides a comprehensive evaluation across fog, underwater, low light, and driving scenarios. DetAS-X delivers statistically significant improvements across all tasks relative to baseline MLLM detectors (Ovis2.5, Qwen3.5, MiniCPM-V, GLM-4.1V, InternVL3.5, GPT-4.1, Gemini-2.5-Flash, and Qwen3-VL).

Notably, DetAS-X exceeds the best MLLM baseline (Qwen3-VL-8B) by an average F1 improvement of 28.36%, peaking at a 37.01% gain on DarkFace, a challenging low-light face detection benchmark. The ablation study isolates the marginal gain attributable to SEEH (experience-based adaptation), which yields consistent improvements over the rule-driven DetAS system, further demonstrating the advantage of experience-centric node-level guidance.

Qualitative visualization substantiates these quantitative improvements. DetAS-X generates perceptually faithful restored images, leverages multi-detector consensus for robust proposal generation, and applies LLM-based reasoning for fine-grained label and spatial selection. Across a broad set of real-world distortions, DetAS-X consistently resolves ambiguous detections and maintains high recall and precision, outperforming strong MLLM baselines that typically underperform under severe degradations.

Figure 2: Qualitative visualization of the DetAS-X pipeline demonstrating adaptive restoration and multi-detector fusion across varied degradation scenarios.

Figure 3: Qualitative comparison of detection results between DetAS-X and baseline MLLMs, illustrating improved accuracy and localization in challenging real-world cases.

Limitations and Implications

The authors note two salient limitations. First, DetAS-X remains bounded by the intrinsic detection accuracy of current MLLMs, particularly in fixed-scene conditions where specialized conventional detectors may excel. Second, the suite of restoration tools and detectors is calibrated for common degradations and categories, excluding long-tail or domain-specific tasks such as medical or infrared imaging. Despite these bounds, the agentic, modular design is natively extensible, enabling integration of additional tools, detectors, and experience modalities as new domains and MLLM capabilities emerge.

Practically, DetAS-X provides a robust baseline for scalable, real-world deployment of object recognition agents, particularly in nonstationary or weakly curated domains. Theoretically, it supports the efficacy of agentic perception: separating high-level decision-making from low-level vision, guided dynamically by both instantaneous input and accumulated experience, is a fruitful direction for generalizable AI.

Future Directions

Advancements in MLLM grounding, open-vocabulary understanding, and lifelong learning mechanisms will compound the efficacy of agentic detection architectures. As language-vision agents acquire richer semantic priors and better compositional reasoning, frameworks such as DetAS-X will inherit improved capabilities for dynamic reconfiguration, specialized adaptation, and out-of-distribution robustness. Key areas for future development include (1) learning-based end-to-end experience assimilation beyond empirical configuration search, (2) closed-loop online experience harvesting with semi-supervised feedback, and (3) expanding detector pools with emergent domain-specialized models.

Conclusion

DetAS-X presents an agentic, experience-aware decision framework for robust object detection under arbitrary real-world degradations. By orchestrating adaptive restoration, multi-expertise detector fusion, and context-sensitive experience utilization through an MLLM agent, the system systematically addresses key obstacles in open-world detection. Empirical results provide strong evidence of its effectiveness, and the modular, extensible design signals a compelling trajectory for future agentic perception systems (2605.31174).