Geo-CoT380k: Geospatial Chain-of-Thought Dataset

• Geo-CoT380k is a remote-sensing corpus providing structured, verifiable geospatial reasoning traces for tasks like VQA, image captioning, and object detection.
• It employs a GPT-4V-driven pipeline that retrofits rationales using auxiliary ground-truth data, ensuring the reasoning process is tightly coupled to image evidence.
• The dataset underpins a Planning–Grounding–Synthesize framework that improves model performance by explicitly structuring the analytical process behind remote-sensing queries.

Geo-CoT380k is a remote-sensing supervision corpus introduced as “the first large-scale dataset of structured Geo-CoT rationales” for vision-LLMs (VLMs), with 384,591 examples spanning visual question answering, image captioning, scene classification, visual grounding, object counting, and object detection (Liu et al., 26 Sep 2025). It is designed to teach a specific reasoning protocol—Perceptually-Grounded Geospatial Chain-of-Thought (Geo-CoT)—rather than only image-to-answer mapping. In the formulation presented with RSThinker, Geo-CoT380k supplies structured supervision for Planning–Grounding–Synthesize reasoning, so that a model outputs not only a final answer but also a perceptually grounded analytical trace whose claims are intended to remain verifiable against the image.

1. Conceptual basis and problem setting

Geo-CoT380k is motivated by a critique of prevailing remote-sensing VLM training. Existing systems are described as typically learning a direct mapping from an image and a prompt to a final textual answer. That regime may yield strong benchmark scores, but it obscures the intermediate analytical process, omits the explicit evidence-gathering steps performed by human remote-sensing analysts, and makes model outputs difficult to verify (Liu et al., 26 Sep 2025).

The associated reasoning paradigm, Geo-CoT, is defined as Perceptually-Grounded Geospatial Chain-of-Thought. In this framework, reasoning is not treated as generic explanatory text appended to an answer. Instead, it is formalized as a multi-step analytical process in which the model decomposes a task, gathers visual evidence step by step, and synthesizes a conclusion only after grounding its claims in spatially identifiable perceptual cues. The paper contrasts this with prior approaches that may rely on abstract textual explanations, parametric world knowledge, or evidence references that are not localizable in the image.

Perceptual grounding is defined strictly. Reasoning steps are expected to refer to concrete objects, regions, coordinates, directions, relative positions, counts, or other image-verifiable spatial evidence. The motivating task types include scanning subregions for counting, resolving relational descriptions such as “the vehicle on the right of the vehicle on the lower left,” and identifying structures through shape-based cues such as “the building shaped like the number 8.” A recurrent misconception is therefore that Geo-CoT380k is merely a collection of chain-of-thought-style responses. The paper instead presents it as supervision for a specific geospatial reasoning protocol whose outputs are intended to maintain a “verifiable link” to corresponding visual evidence.

The dataset is also framed as a response to four shortcomings in remote-sensing VLMs: end-to-end training that bypasses explicit analytical steps, omission of systematic evidence gathering for large scenes, unverifiable outputs not tied to localizable evidence, and chain-of-thought-like text that may be semantically plausible without being faithfully grounded. Geo-CoT380k is introduced to supervise the missing process itself, not only the terminal answer.

2. Construction pipeline and supervision design

Geo-CoT380k is constructed through a scalable rationale-generation pipeline centered on GPT-4V, but the generation is constrained by ground-truth information rather than left open-ended (Liu et al., 26 Sep 2025). Instead of asking a strong VLM to invent free-form reasoning from scratch, the pipeline provides the image, the question, the correct answer, task-specific auxiliary information, and hand-crafted in-context exemplars demonstrating the desired reasoning style. Depending on the task, the auxiliary information may include object coordinates, counts, referring expressions, captions, normalized boxes, spatial attributes, and related metadata.

The paper characterizes this process as “retrofitting” a rationale: GPT-4V is asked to produce a step-by-step path that logically connects known supporting evidence to an already-known correct conclusion. The design claim is that the pipeline “ensures faithfulness by design” because the generator is constrained to explain a correct answer from known evidence. During construction, auxiliary information is used “to scaffold faithful reasoning,” but this scaffolding is removed from the released annotations.

Prompting follows a “two-tier” protocol. A shared base prompt standardizes the output format and the Planning–Grounding–Synthesize architecture. This base prompt defines the generator as a remote-sensing expert and instructs it to construct a detailed remote-sensing CoT given image, auxiliary information, question, and correct answer. It explicitly requires that the chain of thought infer the answer rather than state it abruptly, include thinking, deduction, and reflection, and be naturally segmented with sections separated by \n\n\n. Task-specific prompts then supply curated in-context exemplars spanning diverse scene types, difficulty levels, and edge cases.

The generation process is automated, but the prompt templates, canonical format, and exemplars are human designed. The paper does not report a large manual rationale-writing workforce, inter-annotator agreement, explicit human-validation percentages, or numerical post-hoc filtering thresholds. Quality control is instead largely prompt-based and schema-based: the rationales are grounded with correct-answer supervision and auxiliary metadata, and the exemplars are described as “high-quality,” “curated,” and required to follow the canonical format for reliable parsing and consistent reasoning. A plausible implication is that malformed outputs would need to be filtered for parsing, but no quantitative filtering statistics are reported.

3. Scale, task coverage, and source benchmarks

Geo-CoT380k contains 384,591 structured rationales drawn from multiple public remote-sensing benchmarks (Liu et al., 26 Sep 2025). The composition mixes holistic tasks such as image captioning and scene classification with fine-grained localization tasks such as visual grounding, counting, and detection. This mixture is important to the paper’s thesis that faithful reasoning should function as a general analytical style rather than a capability restricted to one benchmark family.

Task family	Source benchmarks	Samples
Visual question answering	VRSBench-train-VQA	85,813
Image captioning	VRSBench-train-cap; FIT-RS-cap	20,264; 65,197
Scene classification	NWPU-RESISC45-train; AID-train	31,500; 10,000
Visual grounding	DIOR-RSVG-train; VRSBench-train-VG	34,744; 35,967
Object counting	DOTAv2-train; HRRSD-train	25,769; 24,784
Object detection	DOTAv2-train; HRRSD-train	25,769; 24,784

The imagery comes from diverse publicly available remote-sensing datasets. The paper emphasizes “large-scale imagery from sources like DOTAv2” and notes that DOTAv2 imagery was tiled into $800\times 800$ patches. Some appendix examples involve GoogleEarth-sourced imagery, but these derive from the underlying public datasets rather than from a separately collected Geo-CoT380k imaging campaign.

Geo-CoT380k does not provide a standalone geographic-coverage map or a country-by-country distribution. The dataset is geographically diverse insofar as it aggregates multiple public benchmarks, but that diversity is not quantified. Data partitioning is inherited from the underlying datasets: Geo-CoT380k is built from training splits, whereas evaluation uses the corresponding held-out validation or test benchmarks. Examples given in the paper include training on NWPU-RESISC45-train and AID-train while evaluating on NWPU-RESISC45-test, AID-test, WHU-RS19, SIRI-WHU, and UCMerced for scene classification, and training on DOTAv2-train and HRRSD-train while evaluating on DOTAv2-val and HRRSD for counting and detection.

4. Annotation format and perceptual grounding

Each Geo-CoT380k example consists of a multimodal input and a structured textual output (Liu et al., 26 Sep 2025). The core input is an image $I$ and a question or task instruction $Q$. During construction, hidden auxiliary information and the correct answer may also be available. The supervised target used in training is a structured response in the exact format > ... <answer> ... </answer>.

Inside <think>, the model is expected to generate a geospatial reasoning trace that follows Planning–Grounding–Synthesize. These rationales are natural-language chains divided into several segments. They are not released as formal programs or JSON graphs; rather, they are structured prose constrained at the document level and task-specific at the sentence level. A counting rationale may identify likely target regions, scan systematically by region or direction, enumerate visible targets, check for missed or duplicated instances, reflect on decision criteria, and only then provide the count. A grounding rationale may explain scene layout, identify anchor objects, narrow the candidate region, verify spatial relations, and then output the box. A scene-classification rationale may begin from overall layout and salient objects before reasoning toward the scene label.

The defining property of “perceptually grounded” rationales is the use of observable image evidence rather than generic commonsense. The paper’s examples include references such as “top-right quadrant,” “bottom-left curved road,” “parallel to the overpass,” “on the right side of the apron,” “near the center,” and “two dense rows of parked vehicles aligned parallel to the curb.” The rationales also use perceptual signatures including color, shape, shadow, alignment, size, and relative placement. Their “verifiable” character derives from the possibility of checking these claims against the image and the final output: enumerated groups can be audited against counts, described regions can be compared with predicted boxes, and scene labels can be assessed against listed evidence.

This suggests that Geo-CoT380k operationalizes faithfulness not through hidden latent states or formal proof objects, but through externally inspectable spatial prose whose claims are intended to remain image-auditable.

5. Function in two-stage alignment and model training

Geo-CoT380k serves two distinct roles in the paper’s two-stage alignment strategy for RSThinker (Liu et al., 26 Sep 2025). In stage I, supervised fine-tuning (SFT) uses the rationale-bearing annotations to teach the model “the entire methodical workflow encoded in each structured output $o_i$.” The SFT objective is written as

$\mathcal{L}_{\text{SFT}(\theta) = - \sum_{t=1}^{|o_i|} \log p(o_{i, t} \mid o_{i, <t}, I, Q; \theta),$

where $I$ is the image, $Q$ is the question or prompt, $o_i$ is the target output sequence, and $o_{i,t}$ is the token at position $t$. The crucial point is that $I$0 contains not only the answer but the entire Geo-CoT trace plus answer. Supervision therefore targets the reasoning structure itself.

In stage II, Group Relative Policy Optimization (GRPO) no longer uses rationale-bearing targets. Instead, it draws from “the original, rationale-free instances from Geo-CoT380k,” supplemented with additional datasets. The model samples multiple outputs, each including its own generated reasoning trace, and these outputs are scored by task-specific outcome rewards. The paper presents SFT as instilling the “cognitive structure” and GRPO as refining “factual correctness.”

The GRPO formulation includes a clipped PPO-like surrogate together with a KL penalty to a reference policy initialized from the SFT checkpoint. The paper notes that this KL term is critical in ablation, because without it the reasoning format can collapse. Reward definitions are task-specific: VQA and scene classification use $I$1 for correct, partially correct, and other outputs; visual grounding uses IoU; object detection uses mAP@0.5; image captioning uses a weighted combination of BLEU-4, METEOR, CIDEr, and ROUGE-L; and object counting uses a normalized counting-error penalty based on mean absolute error, although the printed formula contains typographical corruption.

RSThinker is the model through which the value of Geo-CoT380k is demonstrated. It is initialized from GLM-4.1V-9B-Base. Its visual backbone is Aimv2-Huge, selected because it can handle variable image resolutions and aspect ratios, and its language decoder uses 3D-RoPE for spatial awareness. Training is reported on 8 NVIDIA A100 GPUs. The SFT stage runs for 3 epochs with batch size 32 using AdamW at learning rate $I$2, followed by a GRPO stage of 970 steps. The output format remains explicitly dual: a final answer plus its analytical trace.

6. Empirical evidence and benchmark behavior

The cleanest evidence for Geo-CoT380k’s effect comes from the ablation comparing answer-only SFT with SFT using Geo-CoT rationales (Liu et al., 26 Sep 2025). On the suite summarized in Table 6, the base GLM-4.1V-9B-Base achieves visual-grounding mIoU 56.26, counting MAE 10.81, detection [email protected] 3.56, image-caption BLEU-4 10.99, scene-classification accuracy 69.78, and VQA accuracy 8.16. Ordinary SFT without rationales improves these to 81.80 mIoU, 3.272 MAE, 49.36 [email protected], 31.14 BLEU-4, 93.33% scene classification, and 63.57% VQA. SFT with Geo-CoT rationales improves them further to 87.70 mIoU, 2.932 MAE, 74.03 [email protected], 33.31 BLEU-4, 96.67% scene classification, and 74.20% VQA. The full model, combining SFT with Geo-CoT and GRPO, reaches 89.02 mIoU, 2.728 MAE, 77.06 [email protected], 33.96 BLEU-4, 96.89% scene-classification accuracy, and 77.24% VQA accuracy.

The main benchmark tables reinforce the same pattern. On visual grounding, RSThinker reaches 90.4 @0.5, 77.2 @0.75, and 80.79 mIoU on VRSBench-VG, and 93.1 @0.5, 90.2 @0.75, and 89.02 mIoU on DIOR-RSVG. On object counting, it achieves 43.93% accuracy and 2.728 MAE on DOTAv2-val, and 85.26% accuracy and 0.242 MAE on HRRSD. On object detection, the appendix reports 77.06 [email protected] on DOTA and 97.99 [email protected] on HRRSD. On scene classification, it reaches 96.89 on RESISC45 and 98.17 on AID. On VRSBench-VQA, the reported category scores are 82.84 for category, 92.36 for existence, 68.97 for position, 56.67 for quantity, 73.24 for scene, 64.33 for color, and 92.87 for image questions. Captioning gains are also reported, including 55.69 BLEU-4 / 32.29 METEOR / 73.55 CIDEr on RSITMD and 85.12 / 58.88 / 94.81 on NWPU-Captions.

Qualitative examples are used to support the claim that the gains reflect verifiable reasoning rather than only higher answer accuracy. The paper describes a counting example in which the model decomposes a total into grounded subgroups such as “three on one side” and “two on the opposite.” Appendix examples show object detection proceeding from coarse enumeration of aircraft regions to refined boxes; visual grounding resolving “the vehicle on the right of the vehicle on the lower left in the image” by first grounding the lower-left vehicle and then selecting the one to its right; and shape-based grounding for “the building shaped like the number 8.” The appendix exemplars further show typical rationale style: systematic regional scanning, object enumeration with perceptual descriptors, cross-checking for omissions, and a delayed final answer only after evidence review.

The authors interpret the strongest improvements on grounding, detection, and counting as evidence that explicit process supervision matters most when tasks depend on local evidence gathering and spatial constraint satisfaction. This suggests that the main contribution of Geo-CoT380k is not merely stylistic explanation, but a task-generalized bias toward decompositional, image-auditable analysis.

7. Limitations, release, and research significance

The paper states an important limitation explicitly: although the rationales are anchored to ground-truth data, they “may inherit stylistic biases from the generative process itself” (Liu et al., 26 Sep 2025). Because GPT-4V produces the rationales, Geo-CoT380k likely reflects its language patterns, reasoning style, and possible blind spots. Since auxiliary information is available during rationale construction, some traces may be more coherent than what a model could derive from image evidence alone, and some may function as post hoc rationalizations of known answers rather than records of genuinely discovered reasoning.

Other limitations follow directly from what is not reported. The paper does not present dedicated step-level faithfulness metrics, human audits of rationale correctness, inter-annotator agreement, or quantified filtering thresholds. The case for faithfulness is therefore indirect: it rests on design choices, the constraint of known answers and auxiliary metadata, and empirical gains, especially on localization-heavy tasks. The dataset also inherits the sampling biases, class distributions, and scene-type imbalances of the public benchmarks from which it is assembled. Geographic diversity is asserted through aggregation rather than measured directly.

In terms of availability, both Geo-CoT380k and RSThinker are stated to be publicly released upon publication. The title page lists a GitHub repository at https://github.com/minglangL/RSThinker and a Hugging Face page at https://huggingface.co/minglanga/RSThinker. One practical detail for downstream users is that the auxiliary information used during dataset construction is removed from the released annotations. The provided text does not specify a license.

Within the paper’s framing, Geo-CoT380k is the data substrate that makes perceptually grounded, verifiable reasoning operational for remote-sensing VLMs. Geo-CoT denotes the protocol; Geo-CoT380k instantiates that protocol at scale across heterogeneous remote-sensing tasks. Its reported significance lies in shifting supervision from opaque end-to-end answer prediction toward explicit Planning–Grounding–Synthesize traces, with the empirical claim that such process supervision improves both performance and inspectability. Even with the limitations of synthetic rationale generation, the dataset is presented as the first large-scale attempt to impose a verifiable geospatial chain-of-thought regime on remote-sensing vision-LLMs.