RingMo-Agent: Unified RS Vision-Language Model

• The paper introduces RingMo-Agent as a remote sensing vision-language model that unifies multi-modal inputs from optical, SAR, and IR sensors for comprehensive perception and reasoning.
• It employs modality-adaptive embedding and token-conditioned decoding to handle tasks ranging from object detection and captioning to long-horizon spatial planning.
• Training on the large-scale RS-VL3M dataset with a two-stage protocol yields significant improvements in metrics like relation reasoning F1 and trajectory scheduling performance.

Searching arXiv for the specified paper to ground the article in the current record. RingMo-Agent is a remote sensing vision-LLM designed as a unified, standalone framework for multi-platform and multi-modal perception and reasoning. It operates on optical, SAR, and infrared imagery collected from satellite and UAV platforms, and it is trained to follow textual instructions across both conventional visual perception tasks and more demanding analytical tasks. The system is distinguished by three coupled design choices: a large-scale instruction-oriented dataset, RS-VL3M, with over 3 million image-text pairs; modality-adaptive representation learning via separate embedding layers for heterogeneous sensing modalities; and token-conditioned decoding mechanisms that support both ordinary language generation and long-horizon spatial outputs such as waypoint sequences (Hu et al., 28 Jul 2025).

1. Problem setting and task scope

RingMo-Agent is motivated by two limitations identified in prior remote sensing vision-LLMs. First, existing RS-VLMs primarily train on relatively homogeneous sources, especially optical imagery from a single platform. Second, they emphasize basic perception tasks such as classification and captioning, while remaining weak on heterogeneous modalities and on long-horizon spatial reasoning. The paper frames this as a mismatch between benchmark-oriented RS-VLM development and real-world remote sensing applications, where sensor characteristics, imaging geometry, and task requirements vary substantially across platforms and modalities (Hu et al., 28 Jul 2025).

Within that formulation, RingMo-Agent is intended to consume multi-platform, multi-modal inputs and perform both perception and reasoning tasks from user textual instructions. The reported task coverage includes classification, detection, captioning, and VQA on the perception side, together with relation reasoning, instruction decomposition, action decision, and task scheduling on the reasoning side. The paper further emphasizes long-horizon spatial tasks such as multi-step navigation and action planning, for which standard autoregressive coordinate emission is described as insufficient.

This task scope places RingMo-Agent at the intersection of remote sensing foundation models, instruction-tuned VLMs, and embodied spatial reasoning. A plausible implication is that the model is not merely extending optical RS captioning to additional modalities, but reorganizing heterogeneous remote sensing problems into a single prompt-conditioned interface. The paper nevertheless does not claim native support for segmentation, polygon grounding, counting, or change detection, and no task formats for these are provided (Hu et al., 28 Jul 2025).

2. RS-VL3M dataset and instruction schema

RS-VL3M is the training and instruction-tuning corpus that underpins RingMo-Agent. It comprises more than 3,000,000 image-text pairs across optical, SAR, and IR modalities and across satellite and UAV platforms. The dataset spans eight task families: task scheduling, action decision, instruction decomposition, relation reasoning, object detection, image captioning, scene classification, and VQA. It also includes a shared multi-modal dialogue subset in which the same SAR or IR images are reused for detection, captioning, and classification in multi-turn evaluation (Hu et al., 28 Jul 2025).

The paper gives selected dataset statistics and resolutions for the constituent sources:

Task family	Dataset examples	Samples / resolution
Task scheduling	CityNav	32,637; variable resolution $[617, 4001]$
Action decision	SkyAgent-Plan3k	3,000; $512 \times 512$
Instruction decomposition	ReCon1M-DEC	27,819; variable resolution $[135, 1000]$
Relation reasoning	FIT-RS; ReCon1M-REL	97,843; 125,000
Object detection	DIOR, DOTA, NWPU VHR-10, SARDet-100k, IR-DET	23,463; 108,047; 800; 116,598; 56,353
Captioning	UCM, Sydney, RSICD, NWPU, SAR-CAP, IR-CAP	10,500; 3,065; 54,605; 157,500; 582,990; 281,765
Scene classification	AID, NWPU-RESISC45, WHU-RS19, UCMerced, SAR-CLA, IR-CLA	10,000; 31,500; 1,005; 2,100; 116,598; 56,353
VQA	RSVQA-HR; RSVQA-LR	1,066,316; 77,232

The instruction schema is highly structured. Coordinates are normalized to $[0, 999]$ across tasks, including bounding boxes, regions, and reference positions. A modality label token, $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$, must be specified by users to indicate input modality. The reasoning-oriented tasks use explicit tags such as <|navigation|>, <|decision|>, <|decomposition|>, <|reasoning|>, <|pose|>, <|ref|>, <|det|>, <|rel|>, and <trajectory>. This design makes the output space partly symbolic and partly free-form: labels, captions, and VQA answers are generated as text, whereas boxes, regions, and trajectories are emitted through constrained task-specific formats (Hu et al., 28 Jul 2025).

The paper does not report explicit radiometric calibration, SAR speckle filtering, or IR normalization procedures beyond resolution handling and coordinate normalization. Images are tiled to multiples of 384 for the encoder, and a $384 \times 384$ global thumbnail is generated. CityNav includes seen and unseen splits; other public datasets follow their original splits. Licensing terms, public release URLs, and explicit availability for RS-VL3M are not stated. This suggests that the paper prioritizes system unification and cross-task coverage over documentation of dataset governance.

3. Architecture and modality-adaptive representation learning

The architecture is organized into four principal components: modality-aware visual embedding, a frozen SigLIP encoder, a frozen MLP adapter, and a LoRA-tuned DeepSeekMoE LLM, together with a trainable trajectory decoder for long-horizon spatial outputs. The visual backbone is SigLIP-SO400M-384, kept frozen during training. Input images are resized to the smallest multiples $m \times 384$ and $n \times 384$ covering the original height and width, tiled into $384 \times 384$ patches, and accompanied by a $384 \times 384$ global thumbnail. The constraint $512 \times 512$0 limits the number of local tiles for compute tractability (Hu et al., 28 Jul 2025).

The central mechanism for heterogeneous modality handling is the use of separate embedding layers for optical, IR, and SAR imagery ahead of the frozen encoder. For modality $512 \times 512$1 with input $512 \times 512$2, the modality-specific embedding $512 \times 512$3 produces

$512 \times 512$4

and the frozen visual encoder $512 \times 512$5 maps this representation to visual tokens

$512 \times 512$6

The multi-modal token sets are written as $512 \times 512$7 and $512 \times 512$8. The stated rationale is that separate $512 \times 512$9 layers avoid forcing heterogeneous remote sensing physics—reflectance, backscatter, and thermal emissivity—through shared filters, thereby reducing cross-modal interference (Hu et al., 28 Jul 2025).

The MLP projector $[135, 1000]$0 maps visual tokens into the LLM’s semantic token space, yielding

$[135, 1000]$1

The adapter performs $[135, 1000]$2 token shuffling, reducing $[135, 1000]$3 tokens to $[135, 1000]$4 per tile, and inserts special positional indicators: one special token appended at the end of each row in the global thumbnail tile, one appended to the end of the last column of local tiles, and one inserted between global and local tiles. The resulting visual token sequence $[135, 1000]$5 is concatenated directly with text tokens and fed into the LLM; the paper does not report a separate cross-attention module beyond the encoder-adapter pipeline.

The LLM is DeepSeekMoE in the activated $[135, 1000]$6B-parameter version, updated only through LoRA with rank $[135, 1000]$7 and scaling $[135, 1000]$8. DeepSeek latent key-value compression is used to reduce KV-cache memory. Generation is expressed as

$[135, 1000]$9

where $[0, 999]$0 is the LoRA-tuned LLM and $[0, 999]$1 is the textual prompt (Hu et al., 28 Jul 2025).

4. Decoding mechanisms and task conditioning

RingMo-Agent uses two distinct output mechanisms. Standard textual outputs such as labels, captions, and short VQA answers are produced through autoregressive token generation with the instruction-following language-model objective

$[0, 999]$2

Geometric outputs for HBB detection and region specification are also represented textually, but as numeric token sequences inside task-specific tags such as <|det|> ... </|det|>, with coordinates normalized to $[0, 999]$3. The object detection prompt reported in the paper is “Detect all objects shown in the remote sensing image and describe using HBBs,” and the response format lists object names together with normalized $[0, 999]$4 coordinates (Hu et al., 28 Jul 2025).

Long-horizon spatial outputs are treated differently. For task scheduling, a special token <trajectory> conditions extraction of the final-layer hidden state, and a trainable GRU-based decoder predicts a sequence of 6D waypoint-plus-attitude entries $[0, 999]$5. Let $[0, 999]$6 denote the final transformer-layer embedding of the <trajectory> token. The decoder is defined by

$[0, 999]$7

$[0, 999]$8

$[0, 999]$9

$[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$0

where $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$1 is normalized to $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$2 by $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$3. The sequence $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$4 terminates when a distance threshold $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$5 is met or when a dataset-dependent maximum step $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$6 is reached. The decoder is thus autonomous after initialization and does not depend on token-by-token textual generation (Hu et al., 28 Jul 2025).

The prompt vocabulary is explicitly task-conditioned. For relation reasoning, the prompt begins with <|reasoning|> and combines <|ref|>, <|det|>, and <|rel|> tags to ask for the relationship between detected objects together with their categories. Instruction decomposition uses <|decomposition|> and region tags to elicit multi-step outputs including counts, per-class HBBs, relations, and a final context summary. Task scheduling uses <|navigation|>, <|ref|>, <|pos|>, and <|pose|> to specify landmarks, start pose, and target description; the response is a sequence of 6D poses. Action decision uses <|decision|> and start/end poses to request a detailed flight plan grounded in visible scene cues.

A common misconception would be to treat RingMo-Agent as a purely text-generative VLM. The reported design is more hybrid: text generation handles semantic outputs, numeric text emission handles box geometry, and hidden-state trajectory decoding handles long-horizon spatial sequences. The paper explicitly contrasts this with general VLMs that rely on text-only coordinate generation and reports that such approaches underperform on trajectories (Hu et al., 28 Jul 2025).

5. Training protocol, objectives, and ablation evidence

Training follows a two-stage procedure. Stage 1 performs vision-language generation pre-adaptation to the remote sensing domain across optical, SAR, and IR data. LoRA is applied only to the LLM. The optimizer is AdamW with linear warmup up to $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$7 followed by cosine decay, warmup learning rate $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$8, minimum learning rate $[label] \in \{\text{opt}, \text{sar}, \text{ir}\}$9, and weight decay $384 \times 384$0. This stage uses more than 500,000 samples for 10 epochs at fixed image size 384 on $384 \times 384$1 NVIDIA A100 80GB GPUs. The data source consists of public RS classification, captioning, and detection datasets; GPT-4 is used to augment captions under constraints that preserve original labels, RemoteCLIP filters poor augmentations, and attribute-matching verification checks number, shape, and category consistency (Hu et al., 28 Jul 2025).

Stage 2 performs instruction tuning on RS-VL3M for task compliance and output formatting. It again uses AdamW, now with linear warmup from $384 \times 384$2 to $384 \times 384$3 and cosine decay to 0, with weight decay $384 \times 384$4. Training runs for 10 epochs, retains original image resolutions, and uses the combined 3M+ samples across tasks and modalities. Throughout the pipeline, the SigLIP encoder and the MLP adapter are frozen; LoRA updates only the LLM, and the trajectory decoder is trained for scheduling (Hu et al., 28 Jul 2025).

The total loss combines text generation and trajectory regression:

$384 \times 384$5

with $384 \times 384$6 and

$384 \times 384$7

over the predicted trajectory sequence with temporal matching to ground-truth waypoints. The paper states that RingMo-Agent does not use CLIP/InfoNCE-style contrastive alignment as part of the described training pipeline; SigLIP is frozen, and the adapter plus LLM are trained generatively. Detection-specific or segmentation-specific losses are likewise not described, since detection supervision is implemented through instruction-conditioned text generation of HBB coordinates (Hu et al., 28 Jul 2025).

The ablation evidence focuses on three design choices. First, the two-stage training scheme improves both reasoning and perception relative to one-stage instruction tuning. Relation reasoning F1 rises from 72.82 to 75.34 on FIT-RS and from 87.11 to 90.23 on ReCon1M-REL; instruction decomposition increases from 16.72 to 24.20 in mAP@50 and from 20.32 to 32.85 in F1. Second, modality-specific embeddings materially improve heterogeneous-modality performance: SARDet-100k mAP@50 increases from 48.10 to 53.84, IR-DET from 51.23 to 67.36, SAR-CAP BLEU-4 from 46.32 to 55.93, and IR-CAP BLEU-4 from 51.08 to 56.84. Third, the trajectory decoder improves CityNav scheduling, with validation-seen SR rising from 2.65 to 5.89, OSR from 14.23 to 21.39, and SPL from 2.16 to 5.24. The paper interprets these results as evidence that remote-sensing-specific pre-adaptation, modality isolation, and hidden-state trajectory decoding contribute distinct benefits (Hu et al., 28 Jul 2025).

6. Empirical performance, positioning, and limitations

The reported evaluation spans scheduling, action decision, relation reasoning, instruction decomposition, captioning, VQA, classification, and detection. On CityNav task scheduling, RingMo-Agent fine-tuned on the task reports validation-seen NE 132.0, SR 5.89, OSR 21.39, SPL 5.24; validation-unseen NE 156.0, SR 4.91, OSR 17.22, SPL 4.33; and test-unseen NE 149.6, SR 4.74, OSR 18.94, SPL 4.17. The paper states that it outperforms Seq2Seq and CMA and approaches map-augmented MGP without external maps. On SkyAgent-Plan3k action decision, BLEU-1/SPICE scores are 37.33/31.68 for Shanghai, 37.76/31.85 for Shenzhen, 42.79/28.45 for Campus, and 42.78/37.68 for Residence. Relation reasoning reaches F1 75.34 on FIT-RS, compared with SkySenseGPT at 74.33, and 90.23 on ReCon1M-REL, where Shikra zero-shot is reported as $384 \times 384$8 (Hu et al., 28 Jul 2025).

For instruction decomposition on ReCon1M-DEC, the model achieves 24.20 mAP@50 and 32.85 F1, compared with MiniGPT-v2 fine-tuned at 11.50/15.19 and DeepSeek-VL2 fine-tuned at 19.80/10.32. In captioning, the optical UCM dataset yields BLEU-4 77.63, METEOR 51.79, ROUGE-L 85.51, and CIDEr 373.68, while NWPU-Captions zero-shot yields METEOR 18.81, ROUGE-L 34.11, and CIDEr 48.39. For SAR-CAP, BLEU-1/2/3/4 are 55.93/44.49/33.57/23.94 with METEOR 25.06 and ROUGE-L 51.12; for IR-CAP, BLEU-1/2/3/4 are 56.84/40.45/29.17/21.50 with METEOR 26.15 and ROUGE-L 43.13 (Hu et al., 28 Jul 2025).

In VQA, the model reports on RSVQA-LR fine-tuned presence 93.10, comparison 87.50, and average 90.30, and on RSVQA-HR zero-shot presence 75.24, comparison 83.92, and average 79.58. In classification, optical fine-tuning yields 94.72 on NWPU-RESISC45 and 91.67 on AID, while zero-shot optical performance reaches 96.81 on WHU-RS19 and 88.00 on UCMerced-LandUse. Fine-tuned SAR-CLA and IR-CLA accuracies are 92.67 and 99.45. In detection, SARDet-100k reports per-class mAP@50 of Ship 74.43, Aircraft 52.93, Bridge 49.90, Harbor 62.30, Car 63.37, and Tank 20.10, with overall mAP@50 53.84; IR-DET reports coarse Ship 42.52, Vehicle 63.83, and overall mAP@50 59.88 (Hu et al., 28 Jul 2025).

These results position RingMo-Agent as a unified RS foundation model rather than a specialist model optimized for a single task. The paper explicitly contrasts it with RS-VLMs that are optical-only or otherwise limited in modality support, and with general VLMs that do not include dedicated mechanisms for long-horizon spatial reasoning. It also situates the model within the RingMo family by noting that it builds on the remote sensing foundation model ecosystem represented by RingMoGPT, while extending it to multi-modal embeddings and embodied reasoning tasks.

The stated limitations are equally specific. The SigLIP encoder and adapter remain frozen. Advanced pixel-level tasks such as segmentation and change detection are not reported. No explicit SAR speckle modeling or denoising is introduced, and robustness to extreme noise is not quantified. Dataset licensing, release details, preprocessing specifics beyond tiling and normalization, and inference throughput are not provided. Modalities not covered include multispectral or hyperspectral imagery, LiDAR, and SAR polarimetry. The authors note future directions in deeper logical reasoning for damage assessment and disaster response decision-making, and a plausible implication is that memory-augmented decoders, map priors, or tool-use modules would be natural extensions for the long-horizon reasoning regime described in the paper (Hu et al., 28 Jul 2025).