GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine

Abstract: Reasoning segmentation requires localizing targets based on complex, implicit queries. Current end-to-end models typically entangle perception and deduction into an opaque black box, severely limiting interpretability and scalability. To address this, we propose GEAR-Seg (Grounded Explainable Agent for Reasoning Segmentation), an explicitly decoupled agent that shifts the paradigm by translating visual pixels into dense, attribute-rich text. By decoupling class-agnostic segmentation, semantic description, and LLM deduction, GEAR-Seg transforms implicit reasoning into an explicit, trackable logic chain. As a zero-shot inference framework, it achieves highly competitive performance across diverse reasoning and fine-grained referring segmentation benchmarks. Furthermore, GEAR-Seg inherently functions as a highly scalable data engine. Utilizing this engine, we construct GEAR-131K, a massive benchmark (over 38k images, 656k QA-mask pairs) introducing a multifaceted taxonomy tailored for complex real-world manipulation-oriented reasoning. Finally, distillation experiments demonstrate that lightweight models supervised exclusively by our automated pipeline closely match the upper-bound performance of costly human-annotated baselines.

• The paper demonstrates how explicit decoupling of vision and language modules enables zero-shot reasoning segmentation with enhanced interpretability.
• It introduces a modular architecture combining SAM, dense semantic description, and logic-driven reasoning to generate scalable, high-quality pseudo-labels.
• Empirical results show substantial gains in gIoU and mAP metrics, validating the approach for manipulation-centric and long-tail segmentation tasks.

GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Scalable Data Generation

Introduction and Motivation

The task of reasoning segmentation (RSeg) extends beyond closed-set and open-vocabulary segmentation by demanding explicit localization of target regions in images based on complex, often implicit linguistic commands. Most state-of-the-art approaches intermix perception and deduction via end-to-end Vision-LLMs (VLMs) or direct visual token injection into LLMs, yielding high performance at the expense of interpretability, modularity, and scalability. These limitations are particularly profound when addressing long-tail, attribute-intensive domains and high-level visual reasoning required for robotics and manipulation-centric applications.

This paper introduces GEAR-Seg, a fully decoupled, grounded, and explainable agent paradigm for reasoning segmentation (2607.00544). The paradigm shift centers on explicit modularization: instead of tightly entangling visual and textual modalities, GEAR-Seg implements independent, composable modules for perception, dense pixel-to-text attribute grounding, and logic-driven segmentation via LLMs. This decoupling fundamentally changes the capability boundary, interpretability, and scalability of reasoning segmentation systems.

Figure 1: GEAR-Seg enables both zero-shot inference and scalable data engine operation, explicitly translating pixels into dense text for open-ended reasoning segmentation, dense referring tasks, and fine-grained attribute grounding.

GEAR-Seg Architecture and Operational Modes

GEAR-Seg leverages explicit decomposition into three main modules:

1. Class-Agnostic Segmentation (SAM 2): All prominent and subtle candidate regions are segmented using the latest Segment Anything 2 (SAM 2) in "Everything Mode," filtered via Mask-NMS.
2. Dense Semantic Description (DAM): For each mask, DAM generates context-rich, fine-grained attribute descriptions leveraging localized and global visual context, ensuring interpretable region-level representations.
3. Logic-Driven Reasoning (LLM): The LLM module, independent of the previous vision stages, is tasked with deductive logic—matching queries to dense textual mask representations to resolve target(s) via explicit, trackable logic chains.

This design provides inherent support for dual operation: (i) zero-shot reasoning segmentation through logic deduction, and (ii) dense, attribute-centric referring segmentation suitable for manipulation and long-tail domains.

Figure 2: GEAR-Seg framework: explicit decoupling into class-agnostic perception (SAM), dense description (DAM), and logic-driven reasoning (LLM), supporting inference and data engine modes.

GEAR-131K: An Automated Complex Reasoning Segmentation Benchmark

GEAR-Seg's modularity and explicit pixel-to-text logic facilitate direct scaling into a data engine. The pipeline autonomously generates annotated data pairs by fusing global scene context, mask-level descriptions, and LLM reasoning chains.

The GEAR-131K benchmark is constructed through closed-loop automation, spanning 39k high-resolution images, 131k filtered QA-mask pairs, and a linguistic expansion protocol that generates over 656k diverse, contextually rich, query+mask+explanation triplets. Compared to prior benchmarks, GEAR-131K introduces an explicit taxonomy relevant for manipulation–segmentation, with task divisions including commonsense, functional, manipulation-related, part-based, and attribute-based scenarios.

Figure 3: The GEAR-Seg data generation engine, integrating multi-granular visual context and logic-driven QA-mask synthesis.

Figure 4: Statistics for GEAR-131K: image/dataset composition, scenario proportion, word cloud for semantic diversity, and categorical comparison vs. existing datasets.

Empirical Results and Analysis

Zero-Shot Inference on Reasoning Segmentation

GEAR-Seg achieves superior zero-shot performance. On ReasonSeg, it yields a gIoU of 57.5—surpassing fully fine-tuned LISA-13B by 1.3 points—while also leading all competitors on the critical size-normalized ncIoU (54.0 vs. 52.8). On LLM-Seg40k, it offers a 6.7-point improvement over fine-tuned LLM-Seg-7B (52.2 vs. 45.5), substantiating the effectiveness of the decoupled, attribute-driven architecture.

Long-Tail and Attribute-Dense Generalization

Testing on specialized agri-domain benchmarks (StrawDI_Db1, Mega_Blueberry, Mega_Peach), GEAR-Seg attains a dominant lead in mAP$_{50:95}$, e.g., 13.2 points higher than the closest alternative on StrawDI_Db1, and demonstrates robust zero-shot auto-labeling capabilities, uncovering semantic categories absent from annotation.

Moreover, for fine-grained state classification tasks (e.g., fruit maturity grading), the system achieves 83.2% zero-shot accuracy, relying on dense text-derived attributes even under severe occlusion.

Figure 5: Qualitative results: (a) complex multi-step reasoning segmentation, (b) zero-shot auto-labeling in agricultural domains, (c) precise maturity grading under occlusion.

Knowledge Distillation and Scalable Supervision

The data-centric paradigm allows direct supervision of lightweight architectures (e.g., YOLOv8s, LISA-13B) using only automated GEAR-131K pseudo-labels. Student models trained exclusively using GEAR-131K attain 92.4% of the upper-bound (human-annotated) gIoU performance on ReasonSeg when matched for data size, and surpass the baseline when scaled in volume. Critically, the inclusion of logic chain supervision leads to a 10.6-point gIoU gain over mask-only training, clearly indicating effective transfer of deductive reasoning—not just mask fitting.

In instance segmentation, YOLOv8s trained with GEAR-Seg pseudo-labels recovers up to 93.2% of manual annotation upper-bound performance in mAP$_{50:95}$ on Mega_Blueberry, dramatically outperforming other generative pipelines.

Modularity, Error Analysis, and Amortized Efficiency

Plug-and-play evaluations show that dense, attribute-rich DAM descriptions are indispensable—replacing them with VLM captions collapses ReasonSeg gIoU by over 22 points. Cognitive module substitutions (e.g., local Qwen-8B/30B vs. API-based Gemini-3.1-Pro) demonstrate high flexibility and scalability. In edge scenarios with explicit referring tasks, lightweight LLMs suffice thanks to DAM's high-fidelity representations.

Cascading error analysis reveals that attribute hallucination in DAM and LLM decision errors dominate failure modes, with SAM's perceptual foundation remaining consistently robust. While single-pass inference is computationally heavier than black-box approaches, amortized cost for multi-query and offline dataset generation is highly efficient due to reusability of intermediate text representations.

Figure 6: Evaluation of agent accuracy and typical error propagation, highlighting cascading errors in LLM-based reasoning chains and DAM attribute hallucination.

Dataset Visualizations and Linguistic Diversity

GEAR-131K exhibits significant linguistic variability and scenario complexity through protocol-driven linguistic expansion. Representative examples show strong syntactic and semantic diversity, providing high-instruction coverage.

Figure 7: Top: Illustration of linguistic expansion producing five distinct query rewrites per scenario. Bottom: Additional visualizations of GEAR-131K's contextual and reasoning richness.

The pipeline is effective for auto-label extraction in open-world, long-tail settings, consistently identifying entities overlooked in manual annotation, thus enhancing downstream learning and robustness.

Figure 8: Zero-shot auto-label extraction: GEAR-Seg autonomously discovers and tags both frequent and long-tail entities across unseen agricultural datasets.

Theoretical and Practical Implications

The explicit modularization of GEAR-Seg establishes critical advances in transparency, extensibility, and functional reliability for reasoning segmentation. By decoupling interpretation, attribute grounding, and logic deduction, the framework is uniquely positioned for rapid domain transfer and low-cost, high-quality annotation generation—a capability highly relevant for embodied AI, robotics, and real-world manipulation where explicit reasoning and robust supervision are mandatory.

The data-centric distillation pipeline demonstrates that high-level reasoning capabilities can be effectively transferred to lightweight, task-specific models using only scalable, automated data, paving the way for efficient deployment under tight resource constraints. Additionally, the design enables seamless adaptation via plug-and-play LLM/Cognition module upgrades and granular error analysis, offering robust avenues for future extensibility.

Conclusion

GEAR-Seg redefines the reasoning segmentation landscape by explicitly decoupling perception, dense text attribute grounding, and abstract logic reasoning. This results in state-of-the-art zero-shot performance, unified support for open-ended and long-tail tasks, and the capacity to operate as an automated data engine—illustrated by the release of GEAR-131K. The paradigm demonstrates highly effective data-centric distillation, translating advanced cognitive capabilities to deployable lightweight models. Ongoing challenges, including DAM attribute hallucination and perception error propagation, are intrinsic to agentic architectures and represent key directions for further work. Ultimately, GEAR-Seg and GEAR-131K establish concrete foundations for interpretable, scalable, and application-ready reasoning segmentation.