LLM-Guided Agentic Object Detection

• LLM-Guided Agentic Object Detection is a framework for fully label-free, zero-shot detection that dynamically generates scene-specific candidate object names using an LLM.
• The system utilizes a two-stage pipeline where a multimodal LLM infers object names and an open-vocabulary detector localizes objects, benchmarked with CAAP and SNAP metrics.
• LAOD bridges gaps in open-world perception by autonomously generating semantic hypotheses and enabling dynamic prompt re-querying for adaptive and precise detection.

LLM-Guided Agentic Object Detection (LAOD) is a framework for open-world visual understanding in which object detection is no longer conditioned on a fixed ontology or a manually supplied prompt, but instead on scene-specific object names generated at inference time by a LLM. In the formulation introduced in "LLM-Guided Agentic Object Detection for Open-World Understanding," LAOD addresses a specific gap between Open-World Object Detection (OWOD), which can detect unknowns but lacks semantic labels for them, and Open-Vocabulary Object Detection (OVOD), which can localize arbitrary named categories but depends on user prompts. LAOD therefore targets fully label-free, zero-shot detection by having a multimodal LLM generate candidate object names from the image itself and forwarding those names to an open-vocabulary detector for localization; the paper also introduces the evaluation metrics Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP), and reports results on LVIS, COCO, and COCO-OOD (Mumcu et al., 14 Jul 2025).

1. Conceptual scope and problem setting

The central premise of LAOD is that the detector's target vocabulary should be determined dynamically from the scene rather than predetermined during training or specified externally at inference. This departs from conventional object detection, which uses fixed category sets, and from prompt-driven OVOD, in which the semantic search space is supplied by the operator. In LAOD, the system itself determines what to detect by querying an LLM for scene-relevant object names, then uses those names as the conditioning vocabulary for an open-vocabulary detector (Mumcu et al., 14 Jul 2025).

Within the taxonomy articulated by the paper, OWOD and OVOD solve different but incomplete versions of open-world perception. OWOD improves flexibility relative to closed-set detectors, yet unknown instances remain semantically unnamed. OVOD allows arbitrary text-specified categories, but its dependence on user prompts limits autonomy. LAOD is positioned as an overview of autonomy and open-vocabulary localization: semantic categories are inferred from the image and then localized without retraining. A common misconception is that this is merely OVOD with an automatically written prompt. The distinction drawn by the paper is stronger: LAOD makes category proposal itself an agentic step, so the system's goals can be revised dynamically through prompt re-querying, label merging, and downstream feedback rather than being fixed ex ante (Mumcu et al., 14 Jul 2025).

A plausible implication is that LAOD reframes object detection as a two-part inference problem: first infer a scene-contingent semantic hypothesis set, then perform localization under that hypothesis set. This is also why the paper separates localization and naming in evaluation rather than treating them as a single AP-like score.

2. Two-stage inference pipeline

At inference time, LAOD processes an image $X$ in two stages: LLM-guided label generation and open-vocabulary object detection. The label-generation stage uses a multimodal LLM $L$ to produce a scene-specific vocabulary:

$C_g = L(X, p),$

where $p$ is a natural-language prompt. The default prompt is:

$$C_m = C_u \,\cup\, \bigl\{g\in C_g\mid \max_{u\in C_u}\,\mathrm{sim}(\mathrm{Emb}(g),\mathrm{Emb}(u)) < \tau \bigr\}.$$2

The generated label set $C_g=\{g_1,\dots,g_n\}$ is then parsed, deduplicated, and optionally merged with any user-provided labels $C_u$ through CLIP-based similarity filtering. When user labels are present, the merged vocabulary is

$$C_m = C_u \,\cup\, \bigl\{g\in C_g\mid \max_{u\in C_u}\,\mathrm{sim}(\mathrm{Emb}(g),\mathrm{Emb}(u)) < \tau \bigr\}.$$

The detector stage then applies an open-vocabulary detector $O$ to the image and the final label set:

$D = O(X, C_m),$

yielding detections $D=\{(b_i,\ell_i,s_i)\}$ consisting of bounding boxes $L$0, predicted class names $L$1, and confidence scores $L$2 (Mumcu et al., 14 Jul 2025).

The operational details are important. The paper specifies that the LLM output may be comma-separated or line-separated; parsing lowercases, strips punctuation, and removes duplicates. If user labels are provided, optional CLIP-based filtering suppresses near-duplicates to those labels. The implementation uses CLIP text embeddings and a similarity threshold $L$3, with ViT-B/32 as the CLIP model in the reported experiments. In the concrete implementation described in the paper, the multimodal LLM is Gemma-3 Small and the open-vocabulary detector is YOLO-World X (Mumcu et al., 14 Jul 2025).

Prompt design is treated as a first-class control variable. The default prompt encourages enumeration of primary scene entities, but rephrased prompts can request "main objects," "all visible components," or task-specific subsets such as "List currently moving vehicles in this scene." In streaming or multi-view settings, the agent can re-query the LLM with modified prompts based on previous detection results or external signals. The paper describes this as dynamic goal adaptation. A plausible implication is that LAOD does not merely inject semantics into detection; it creates a controllable loop in which semantics, task specification, and visual search can be renegotiated during inference (Mumcu et al., 14 Jul 2025).

3. Evaluation methodology: disentangling localization from naming

Because LAOD generates its own label set, standard class-aware AP cannot by itself distinguish whether failures arise from missed localization, semantically inappropriate naming, or both. The paper therefore introduces two metrics designed to separate these factors: Class-Agnostic Average Precision (CAAP) for localization and Semantic Naming Average Precision (SNAP) for naming (Mumcu et al., 14 Jul 2025).

CAAP ignores class labels and measures how well predicted boxes cover any object. Let predicted boxes be $L$4 and ground-truth boxes be $L$5. At IoU threshold $L$6, true positives, false positives, and false negatives are defined using IoU-only matching. Precision and recall at score threshold $L$7 are:

$L$8

$L$9

The single-threshold CAAP is the area under the precision-recall curve,

$C_g = L(X, p),$0

and [email protected]:.95 averages over $C_g = L(X, p),$1 (Mumcu et al., 14 Jul 2025).

SNAP evaluates naming quality independently of localization. Let the predicted label set be $C_g = L(X, p),$2 and the ground-truth label set be $C_g = L(X, p),$3. Semantic matching is defined using CLIP text embeddings and cosine similarity:

$C_g = L(X, p),$4

At semantic threshold $C_g = L(X, p),$5, a predicted label matches one ground-truth label if $C_g = L(X, p),$6. Precision and recall are computed analogously, and

$C_g = L(X, p),$7

[email protected]:.9 averages over $C_g = L(X, p),$8 (Mumcu et al., 14 Jul 2025).

This metric split directly addresses a recurring ambiguity in open-world detection. A system may localize an object correctly while naming it with a semantically adjacent but non-identical expression, or may produce semantically plausible names without correspondingly accurate boxes. CAAP and SNAP are intended to make these failure modes separately observable rather than collapsing them into a single score. The paper nonetheless also evaluates against standard unknown-object metrics to preserve comparability with prior work.

4. Experimental configuration and reported results

The reported experimental setup uses LVIS-Minival, COCO-Val, and COCO-OOD. LVIS-Minival is described as a long-tail benchmark with approximately 1,200 rare categories; COCO-Val contains 80 common classes; COCO-OOD contains 504 images with 1,655 unknown object instances. The implementation uses Gemma-3 Small for multimodal label generation, YOLO-World X as the open-vocabulary detector, the default prompt "List the objects that you see in this image.", and CLIP ViT-B/32 for embedding and filtering. CAAP uses IoU thresholds from 0.50 to 0.95 in steps of 0.05, and SNAP uses semantic thresholds from 0.60 to 0.90 in steps of 0.10 (Mumcu et al., 14 Jul 2025).

The core CAAP/SNAP results are organized by low-, mid-, and high-threshold regimes. The paper defines LO/MID/HI for CAAP as IoU ranges $C_g = L(X, p),$9, $p$0, and $p$1, with analogous semantic ranges for SNAP.

Dataset	Localization	Naming
LVIS-Minival	CAAP$p$2=0.15	SNAP$p$3=0.41
COCO-Val	CAAP$p$4=0.25	SNAP$p$5=0.54
COCO-OOD	CAAP$p$6=0.61	—

On COCO-Val, the paper reports CAAP$p$7=0.25 and SNAP$p$8=0.54. On COCO-OOD, it reports CAAP$p$9=0.61 despite the system never seeing those classes in training. The full table also includes LVIS-Minival values of CAAP$C_g=\{g_1,\dots,g_n\}$0=0.06, CAAP$C_g=\{g_1,\dots,g_n\}$1=0.13, SNAP$C_g=\{g_1,\dots,g_n\}$2=0.10, and SNAP$C_g=\{g_1,\dots,g_n\}$3=0.38; COCO-Val values of CAAP$C_g=\{g_1,\dots,g_n\}$4=0.08, CAAP$C_g=\{g_1,\dots,g_n\}$5=0.22, SNAP$C_g=\{g_1,\dots,g_n\}$6=0.19, and SNAP$C_g=\{g_1,\dots,g_n\}$7=0.52; and COCO-OOD values of CAAP$C_g=\{g_1,\dots,g_n\}$8=0.26 and CAAP$C_g=\{g_1,\dots,g_n\}$9=0.56 (Mumcu et al., 14 Jul 2025).

For compatibility with prior unknown-object detection literature, the paper also reports standard unknown metrics at IoU $C_u$0 against Grounding DINO, OW-DETR, VOS, UnSniffer, and UN-DETR. LAOD achieves $C_u$1, $C_u$2, $C_u$3, and $C_u$4, exceeding the listed baselines in U-AP, U-F1, and U-REC while maintaining balanced U-PRE. For comparison, UN-DETR is reported at $C_u$5 and Grounding DINO at $C_u$6 (Mumcu et al., 14 Jul 2025).

The paper's ablation summary isolates the contribution of the agentic LLM stage. Removing the LLM stage, described as using a fixed, broad prompt, drops CAAP$C_u$7 by approximately 30%. Using a smaller distilled LLM trades off approximately 5 points of SNAP for a $C_u$8 speedup. No formal $C_u$9-values are reported, but the paper states that standard 95%-CI over validation splits indicate that all reported improvements exceed random variations (Mumcu et al., 14 Jul 2025).

Qualitative examples are used to illustrate the kinds of scene-conditioned behavior the framework permits. On COCO-OOD, the paper highlights detections such as "broom," "glasses," and "faucet," including objects beyond annotated ground truth. In an anomaly-detection example on ComplexVAD, LAOD labels a "cardboard box" anomaly whereas Detectron2 misses it. In a prompt-conditioned granularity example, the prompt "List moving vehicles…" returns only the bicycle in motion, in contrast to broader detection under the default prompt. These examples reinforce that the framework is intended not only to expand category coverage but also to make the semantic scope of detection queryable and revisable (Mumcu et al., 14 Jul 2025).

5. Position within the broader agentic detection literature

LAOD belongs to a broader 2025-2026 trend in which LLMs and multimodal LLMs intervene in object detection not merely as text encoders but as planners, prompt constructors, critics, memory systems, or training-time semantic adapters. Closely related work includes Real-LOD, which addresses language-based object detection by using an LLM-controlled workflow with planning, tool use, and reflection to refine object descriptions cyclically. Real-LOD operates on detected objects and VLM-generated raw expressions, uses a discrete state/action design with actions such as Rewrite, Crop, Ext_Crop, and Highlight, runs for up to $$C_m = C_u \,\cup\, \bigl\{g\in C_g\mid \max_{u\in C_u}\,\mathrm{sim}(\mathrm{Emb}(g),\mathrm{Emb}(u)) < \tau \bigr\}.$$0 cycles, and constructs a dataset with about 1.18M image-expression pairs from 188k images; the associated model is reported to surpass existing LOD methods by around 50% on standard benchmarks (Chen et al., 30 Mar 2025).

A distinct branch of agentic detection is represented by DetAS and DetAS-X, which formulate object detection as a dynamic decision process rather than a fixed pipeline. In that framework, a Multimodal LLM profiles the scene, selects restoration modules such as RIDCP, MPRNet, SwinIR, and LLFlow, applies ESRGAN super-resolution, chooses top-$$C_m = C_u \,\cup\, \bigl\{g\in C_g\mid \max_{u\in C_u}\,\mathrm{sim}(\mathrm{Emb}(g),\mathrm{Emb}(u)) < \tau \bigr\}.$$1 specialized detectors from a detector pool, and resolves candidate predictions through instance-level reasoning; DetAS-X augments this with Self-Evolving Experience Harvesting and reports an average improvement of 28.36% in F1 score, with up to 37.01% gain on DarkFace (Zhang et al., 29 May 2026). Relative to LAOD, this relocates the agentic intervention from semantic vocabulary generation to workflow composition over restoration and expert-detector modules.

A third line uses LLMs or MLLMs as semantic guidance layers rather than autonomous vocabulary generators. The edge-cloud framework "Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detection" instruction-fine-tunes an MLLM to output structured scene descriptions in JSON, maps the resulting semantics into parameter adjustments for an edge detector, and routes between edge-only and cloud-enhanced inference according to confidence. It reports latency reductions of over 79% and computational-cost reductions of 70% in low-light and highly occluded scenes while maintaining accuracy (Hu et al., 24 Sep 2025). Here the LLM does not decide what categories exist in the scene; it modulates how an existing detector should operate.

YOLOA occupies yet another point in the design space: an LLM Adapter refines object and affordance predictions during training by generating class priors, box offsets, and affordance gates, but the adapter is disabled at inference. The reported model reaches 52.8 / 73.1 mAP on ADG-Det / IIT-Heat with real-time performance up to 89.77 FPS, and the lightweight variant reaches up to 846.24 FPS (Ji et al., 3 Dec 2025). This suggests that "LLM-guided detection" can also mean training-time distillation of semantic priors rather than online agentic reasoning.

System-level autonomy is illustrated by "Agentic UAVs," which embeds YOLOv11 into a five-layer Perception-Reasoning-Action-Integration-Learning stack. In simulated search-and-rescue scenarios, the framework combines tool-calling, policy-graph generation, and action execution, and reports higher detection confidence (0.79 vs. 0.72), improved person detection rates (91% vs. 75%), and action recommendation rates (92% vs. 4.5%) (Koubaa et al., 14 Sep 2025). A plausible implication is that LAOD is best understood as one member of a larger family of agentic perception systems in which the LLM may determine categories, refine captions, choose detectors, allocate compute, or trigger embodied actions.

6. Limitations, misconceptions, and open directions

The primary limitations reported for LAOD are computational and semantic rather than purely detection-theoretic. The paper notes that the compute overhead of large LLMs may hinder real-time deployment and proposes evaluation of distilled or task-specific LLMs as future work. It also identifies persistent ambiguity in object granularity and suggests reinforcement learning to refine prompt wording based on downstream detection feedback. Finally, it argues that there is a benchmarking gap: new datasets are needed to explicitly measure agentic, prompt-driven detection capabilities rather than only static detection accuracy (Mumcu et al., 14 Jul 2025).

These limitations clarify several common misconceptions. One is that autonomous label generation removes prompt engineering; in fact, prompt wording remains a governing variable for object granularity and task focus. Another is that open-vocabulary localization alone solves open-world understanding; LAOD's introduction of SNAP indicates that semantic naming remains a distinct evaluation target even when localization is adequate. A third is that agentic detection necessarily requires online, high-latency language reasoning at every step. Related systems show several alternative strategies: training-time semantic refinement in Real-LOD, experience-aware detector orchestration in DetAS-X, semantic parameter modulation in edge-cloud detection, and training-only LLM adapters in YOLOA (Chen et al., 30 Mar 2025, Zhang et al., 29 May 2026, Hu et al., 24 Sep 2025, Ji et al., 3 Dec 2025).

A plausible implication is that future work will differentiate along at least three axes. The first is where the agent acts: vocabulary generation, restoration selection, detector selection, parameter adaptation, or embodied action planning. The second is when the agent acts: offline dataset construction, training-time supervision, online inference, or closed-loop deployment. The third is how the agent is supervised: by benchmark metrics such as CAAP and SNAP, by validation-set experience memories, or by downstream task feedback. Within that landscape, LAOD's distinctive contribution is to define object detection as an autonomous inference process in which the semantic target set is generated from the scene itself and evaluated with separate localization and naming metrics (Mumcu et al., 14 Jul 2025).