RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs

Abstract: Earth Observation (EO) systems are essentially designed to support domain experts who often express their requirements through vague natural language rather than precise, machine-friendly instructions. Depending on the specific application scenario, these vague queries can demand vastly different levels of visual precision. Consequently, a practical EO AI system must bridge the gap between ambiguous human queries and the appropriate multi-granularity visual analysis tasks, ranging from holistic image interpretation to fine-grained pixel-wise predictions. While Multi-modal LLMs (MLLMs) demonstrate strong semantic understanding, their text-based output format is inherently ill-suited for dense, precision-critical spatial predictions. Existing agentic frameworks address this limitation by delegating tasks to external tools, but indiscriminate tool invocation is computationally inefficient and underutilizes the MLLM's native capabilities. To this end, we propose RemoteAgent, an agentic framework that strategically respects the intrinsic capability boundaries of MLLMs. To empower this framework to understand real user intents, we construct VagueEO, a human-centric instruction dataset pairing EO tasks with simulated vague natural-language queries. By leveraging VagueEO for reinforcement fine-tuning, we align an MLLM into a robust cognitive core that directly resolves image- and sparse region-level tasks. Consequently, RemoteAgent processes suitable tasks internally while intelligently orchestrating specialized tools via the Model Context Protocol exclusively for dense predictions. Extensive experiments demonstrate that RemoteAgent achieves robust intent recognition capabilities while delivering highly competitive performance across diverse EO tasks.

• The paper introduces RemoteAgent, which utilizes RL-based alignment to effectively translate vague human queries into precise EO task execution, achieving 95% intent recognition accuracy.
• It employs Group Relative Policy Optimization to optimize routing between intrinsic semantic reasoning and extrinsic dense predictions, outperforming existing MLLM and tool-based methods.
• The framework offers efficient computation with a 100x inference speedup and robust performance across benchmarks, underscoring its potential for accessible real-world EO applications.

RL-based Agentic MLLMs for Bridging Ambiguous Human Intents in Earth Observation

Motivation and Problem Formulation

The usability gap between domain experts and Earth Observation (EO) AI systems arises from the disparity between ambiguous, free-form queries and rigid, machine-centric task requirements. Experts typically articulate operational needs through imprecise language, which must be robustly grounded and executed across tasks ranging from holistic image-level inference to precision pixel-wise dense predictions. Existing MLLMs are limited by their text-centric architectures, which constrain them in dense spatial output tasks; tool-augmented agents, while offering external specialization, tend to overutilize tool chains, incurring computational inefficiency and underexploiting native MLLM capabilities.

Figure 1: The gap between vague user queries and rigid EO system requirements, highlighting the failure modes of current MLLMs and tool-based agents and RemoteAgent's bridging strategy.

VagueEO Benchmark Construction

To address real-world ambiguity, the VagueEO benchmark is constructed with simulated user prompts that mirror authentic, non-expert queries and map these to precise structural annotations in EO tasks. VagueEO operationalizes diversified free-form instruction templates synthesized with LLM-driven pipelines, covering a spectrum of EO domains. Each query is deterministically paired with ground-truth labels, spanning image-level, bounding box, and pixel-wise mask supervision for multi-granularity spatial reasoning.

Figure 2: Ten representative EO tasks in the VagueEO benchmark, each pairing vague human-centric queries with standardized labels.

RemoteAgent: RL-Aligned Agentic Framework Design

RemoteAgent delineates EO task space into two orthogonal subsets: intrinsic (semantic understanding, sparse localization) and extrinsic (dense predictions). It deploys an RL-based alignment protocol, specifically Group Relative Policy Optimization (GRPO), to condition the MLLM (Qwen2.5-VL-7B-Instruct) on sparse reasoning tasks. Dense prediction tasks invoke external specialized toolkits—interfaced through the Model Context Protocol (MCP)—only when intent recognition conclusively demands.

Figure 3: Architecture overview of RemoteAgent, showing RL alignment for intrinsic tasks and dynamic routing to expert tools for dense outputs.

GRPO surpasses SFT by directly incentivizing functional correctness at the sequence level, with KL regularization to constrain policy drift. A unified multimodal reward interface dispatches to branches by answer format (coordinate, numerical, textual), enabling training on heterogeneous task outputs without bespoke loss functions or explicit task labels.

Experimental Evaluation and Empirical Results

Intent Recognition

RemoteAgent records 95.0% mean accuracy in intent recognition, decisively outperforming RL-based RemoteReasoner and SFT-based baselines (GeoChat, Falcon <8%), demonstrating strong generalization and routing flexibility attributable to RL-alignment rather than overfitting to rigid templates.

Figure 4: Intent recognition accuracy across EO tasks, showing RemoteAgent’s dominance over existing baselines.

Intrinsic Task Performance

For scene classification, RemoteAgent achieves 91.34% accuracy on the AID benchmark, eclipsing general-purpose MLLMs (Qwen2.5-VL: 63.07%) and remaining competitive versus specialist models (e.g., FUSE-RSVLM). In visual grounding and geospatial region reasoning (DIOR-RSVG, EarthReason), RemoteAgent attains IoU 48.3 and [email protected] 57.81%, respectively, consolidating its capacity for precise sparse spatial inference.

Extrinsic Task Execution

Dense tasks routed to external tools yield expert-level performance without text-centric bottlenecks. Object detection benchmarks (DIOR, DIOR-R) report AP50 77.80/73.80, nearly matching SkySense (SOTA). Semantic segmentation and referring expression segmentation achieve 93.54 mF1 (Potsdam) and 71.08 mIoU (RRSIS-D), outperforming prior segmentation architectures and MLLM-based pixel-grounding models. Building damage assessment (xBD) exhibits F1_overall 77.16, validating robust system-wide precision in complex bi-temporal change detection.

Efficiency and Ablation Studies

RemoteAgent achieves 100x inference speedup compared to multi-step agentic frameworks (e.g., Earth-Agent), with a total execution time of 1.18 seconds, due to its direct intent-recognition-driven tool invocation. Ablation confirms RL-based training preserves routing and cognitive flexibility, with SFT triggering catastrophic forgetting and degrading segmentation performance by 18.94% mIoU.

Figure 5: Qualitative results demonstrating interpretation of vague queries and dynamic routing to specialized tools with precise execution.

Practical and Theoretical Implications

RemoteAgent demonstrates that RL-based alignment is superior for intent mapping and multi-granularity task execution, circumventing both catastrophic forgetting and semantic rigidity induced by SFT. The agentic routing paradigm allows EO systems to maximally leverage central MLLM cognitive reasoning for macroscopic and sparse localization tasks, while invoking specialist tools for precision-critical dense outputs. The separation of semantic reasoning and spatial execution not only increases computational efficiency but also enhances real-world accessibility for non-technical users.

From a theoretical perspective, this approach validates RL fine-tuning for dynamic workflow orchestration in agentic MLLMs, and generalizes the principle of functional correctness reward alignment for heterogeneous multimodal outputs. Remaining challenges include scaling instruction datasets (VagueEO), automating dynamic tool integration, and mitigating compounding errors from tool chains.

Speculation on Future Developments

Continued scaling of RL-aligned agentic frameworks is projected to support open-ended tool discovery and integration workflows. Autonomous workflow evolution, error correction mechanisms, and dynamic memory management will be critical for robustness in practical deployments. The paradigm established by RemoteAgent will inform the next generation of agentic MLLMs across domains demanding multi-granular cognition and high accessibility, including environmental monitoring, urban planning, and policy support.

Conclusion

RemoteAgent establishes an RL-aligned agentic architecture that robustly bridges vague human intent and EO task execution, strategically leveraging intrinsic MLLM strengths for sparse reasoning while orchestrating precision-dense tasks via external toolkits. Its exceptional data efficiency, accuracy, and latency profile define a formal accessible paradigm for EO AI, with broad implications for agentic MLLM design in other multimodal domains (2604.07765).