Geo-DLC: Detailed Localized Captioning

• Geo-DLC is a fine-grained remote sensing captioning task that generates detailed, natural-language descriptions for localized image regions using global context.
• It employs a scale-adaptive focal strategy and textual localization to bridge object detection with open-ended geospatial reasoning.
• The integrated DE-Dataset and DE-Benchmark support applications in environmental monitoring, urban planning, and disaster management through attribute-rich annotations.

Geo-DLC, short for Geographical Detailed Localized Captioning, is an object-level, fine-grained image captioning task for remote sensing imagery introduced by the DescribeEarth work. It generalizes the “Describe Anything / DLC” paradigm from natural images to nadir-view, large-scale Earth observation scenes by taking a remote sensing image $I$ and a user-specified region of interest $R$ and generating a detailed textual description $T$ focused on that region while exploiting the global context of the full image. In the same work, Geo-DLC is presented not as an isolated benchmark but as an integrated research program comprising a task formulation, a large-scale dataset (DE-Dataset), an evaluation suite (DE-Benchmark), and a dedicated multimodal LLM (DescribeEarth) (Li et al., 30 Sep 2025).

1. Terminological scope and disambiguation

Within current arXiv usage, the explicit and formal use of Geo-DLC is the remote-sensing task introduced by DescribeEarth: object-level, fine-grained description of localized regions in remote sensing imagery (Li et al., 30 Sep 2025). In that formulation, the atomic unit is no longer the whole scene but a selected object or geographic ROI, and the output is an open-ended, context-aware paragraph rather than a short global caption.

Geo-DLC is distinct from conventional remote sensing image captioning, which typically assigns a single scene-level description to an image, and from vision-language grounding, which generally solves the inverse problem of locating a described region. It also differs from scene graph generation: scene graphs encode structured triplets, whereas Geo-DLC produces natural-language descriptions that can still express object attributes, relations, and environmental context.

The string “Geo-DLC” is not uniform across the broader literature. In cross-view geo-localization, GeoDTR is a retrieval model centered on disentangled geometric layout correspondence and counterfactual learning rather than caption generation (Zhang et al., 2022). In 3D point cloud learning, GeoCD is a geodesic-aware Chamfer-distance variant and is described as being “loosely referred to as ‘Geo-DLC’ in conversation,” but it is not a captioning framework (Alonso et al., 30 Jun 2025). In geometric diagram understanding, GeoPerceive and GeoDPO are presented as a geometry-focused capability package for VLMs; the supplied discussion characterizes that package as what “would correspond” to Geo-DLC in that setting, but again this is not the formal remote-sensing task (Yu et al., 26 Feb 2026). A further informal extension appears in discussion of diamond-like carbon membrane coatings, where “Geo-DLC” is used to frame large-scale environmental membrane applications, although the paper itself studies DLC nanostructure coatings on PES nanofiltration membranes rather than a named Geo-DLC framework (Kiamehr et al., 2024).

This terminological dispersion suggests that, for encyclopedia purposes, Geo-DLC is most precisely reserved for Geographical Detailed Localized Captioning in remote sensing, while other occurrences are better treated as unrelated or informal homonyms.

2. Task formulation and conceptual shift

Geo-DLC is defined over a remote sensing image $I$ and a localized region $R$, given by an oriented bounding box, point, mask, or equivalent cue. The output is a detailed description $T$:

$T = \text{Model}(I, R).$

The paper also frames the problem as conditional autoregressive generation:

$$p(T \mid I, R; \theta) = \prod_{t=1}^{L} p(y_t \mid y_{<t}, I, R; \theta).$$

The description is expected to cover three classes of content: intrinsic object features such as appearance, structure, materials, and scale; spatial features such as orientation and geometric layout; and contextual features such as neighboring objects, land cover, and surrounding human activity (Li et al., 30 Sep 2025).

This formulation responds to a limitation in classical remote sensing captioning. Image-level captions are too coarse when a single scene contains many distinct objects such as ships, storage tanks, bridges, vehicles, or runways. Geo-DLC instead makes the user-selected object the unit of description. The result is a shift from generic summarization toward localized semantic interpretation, enabling queries of the form “Describe this specific storage tank and its surroundings” or “Explain the condition of this building compared to neighboring buildings.”

The formulation also departs from template-based object description. In the paper’s comparison, template-driven systems generate predefined attribute phrases for a small set of categories, whereas Geo-DLC requires open-ended, paragraph-style language tied to a precise region and supported by scene context. A plausible implication is that Geo-DLC is intended not merely as a captioning task but as a bridge between localized visual recognition and downstream LLM-based geospatial reasoning.

3. DE-Dataset

DE-Dataset is the corpus introduced to support Geo-DLC. It is built from two oriented object detection datasets, DOTA and DIOR, and after tiling and filtering comprises 18,417 remote sensing images, 261,806 object instances, and 25 object classes, with an average of ~14 instances per image and an average description length of 119.83 words per instance (Li et al., 30 Sep 2025).

A critical preprocessing step is the tiling of large DOTA images. Images with resolution greater than $1024 \times 1024$ are divided into non-overlapping $1024 \times 1024$ crops together with boundary tiles so that objects crossing tile edges are preserved. Oriented bounding boxes are remapped into sub-image coordinates, and sub-images with only incomplete objects or no objects are pruned. The resulting instances retain oriented-box localization, which is especially important for classes whose orientation is semantically salient, including runways, bridges, ships, and similar infrastructure.

Annotation uses an MLLM-assisted pipeline followed by human quality control. The annotator model is Qwen2.5-VL-32B, which receives the sub-image, the localized crop, and the category name. Prompts require coverage of intrinsic features, spatial features, and contextual features, and they explicitly forbid speculative or uncertain language and hallucinations. Remote sensing experts manually review a diverse subset across categories and complexities; low-quality or incorrect captions are corrected, regenerated, or discarded.

DE-Dataset differs from earlier remote sensing caption datasets such as UCM-Captions, RSICD, Sydney-Captions, RSITMD, and NWPU-Caption in four respects stated in the source material: object-level granularity, scale, open-ended paragraph-style richness, and the inclusion of attributes together with contextual relations. It also differs from EVAttrs-95K, which provides object-level templates with attribute slots but uses templated and rigid language. The paper does not specify explicit train/validation/test splits for DE-Dataset itself; instead, DE-Benchmark is constructed from the validation portions of DIOR and DOTA plus separate out-of-distribution instances.

4. DE-Benchmark and evaluation methodology

DE-Benchmark is an LLM-assisted, attribute-based QA evaluation protocol designed specifically for Geo-DLC. Its central premise is that conventional captioning metrics such as BLEU, METEOR, ROUGE, CIDEr, and SPICE are not well suited to detailed localized captioning, because they penalize correct additional detail and do not reliably measure factual correctness at object level (Li et al., 30 Sep 2025).

Benchmark construction begins with validation-set instances from DIOR and DOTA, which are described using the same pipeline as DE-Dataset. For each category, representative instances are manually selected and then stratified by difficulty into Simple and Complex cases. Complex cases are defined as small, densely packed, occluded, or environmentally complicated instances. In addition, 10 out-of-distribution categories are drawn from xView.

Human experts then read each reference description and extract attribute–value pairs such as color, shape, orientation, nearby objects, land cover, and layout patterns. For each attribute they author a question and a rubric whose options include contradiction, omission, vague mention, implicit support, explicit correct and detailed mention, and correct extension via reasonable inference. These questions are organized into four top-level groups: Appearance, Surrounding, Usage, and Language. Five additional general language-quality questions explicitly rate grammaticality, logical structure, factual consistency, reasoning, and conciseness.

Evaluation is performed with GPT-4.1 as judge. For each instance, the judge receives the model-generated description and the attribute-based QA set, selects rubric options question by question, and returns scores by category together with an aggregated percentage. The result is a fine-grained assessment of attribute coverage, factual correctness, omissions or hallucinations, contextual adequacy, and linguistic quality. This design makes DE-Benchmark structurally closer to a localized semantic auditing protocol than to classical n-gram-based caption evaluation.

5. DescribeEarth architecture and training

DescribeEarth is the remote-sensing-oriented MLLM introduced for Geo-DLC. It is built on Qwen2.5-VL-3B-Instruct and combines three design choices: a scale-adaptive focal strategy, a domain-guided fusion module (DFM) using RemoteCLIP (ViT-B), and textualized oriented bounding boxes instead of mask injection into the vision encoder (Li et al., 30 Sep 2025).

The model uses a dual-path visual pipeline. A global path processes the full image, typically resized to $R$0, while a focal path processes a crop centered on or aligned with the target instance. The scale-adaptive crop depends on the instance scale $R$1 for the minimum enclosing axis-aligned box of the ROI. If $R$2, the focal crop is the bounding-box region itself; if $R$3, the model uses a $R$4 crop centered on the box centroid; if $R$5, it uses a $R$6 crop centered on the centroid and then resizes to the focal resolution. This design addresses the extreme scale variation of remote sensing objects.

Domain guidance enters through RemoteCLIP features extracted from the focal crop. These are projected into the Qwen vision-feature space and fused with global and focal features through hierarchical gated cross-attention in DFM. The module first enriches global and focal tokens using RemoteCLIP as key–value features and then refines focal features using the enriched global representation. The paper reports that DFM > pure gated cross attention > concatenation > Q-Former fusion, especially on complex and OOD cases.

Localization handling is deliberately textual. Instead of modifying the RGB input with masks or extra channels, the model keeps the vision encoder frozen and inserts the OBB coordinates into the prompt. This preserves the input statistics seen during pretraining and makes localization explicit in language form. The approach also supports user interactions such as clicks or rough boxes that are converted to OBBs using segmentation methods like SAM.

Training is intentionally lightweight. The Qwen2.5-VL vision encoder is frozen, RemoteCLIP is frozen apart from a learned projection, and the LLM, vision–language connector, and fusion module are fine-tuned. Training uses 2× A800 GPUs, batch size 4 with gradient accumulation every 4 steps, 1 epoch, global resolution $R$7, focal resolution $R$8, and a pure captioning cross-entropy loss. The paper reports that freezing the vision encoder yields better overall generalization than training it, with only a very small loss on complex cases.

6. Empirical findings, limitations, and significance

On DE-Benchmark, DescribeEarth consistently improves over the fine-tuned Qwen2.5-VL-3B baseline and over DAM, and it exceeds GPT-4o on in-distribution simple and complex cases. The reported aggregated scores are 74.19% on Simple, 70.24% on Complex, and 68.78% on OOD. For comparison, fine-tuned Qwen2.5-VL-3B obtains 68.07%, 63.76%, and 63.75%, while GPT-4o obtains 70.24%, 65.51%, and 68.78%, matching DescribeEarth on OOD but not surpassing it in-distribution (Li et al., 30 Sep 2025).

Category-level results show especially strong gains on remote-sensing-specific infrastructure. Examples reported in the paper include Harbor at 86.34% for DescribeEarth versus 73.45 for GPT-4o and 60.82 for DAM, Facility at 88.37% versus 72.67 and 41.86, and Locomotive at 72.22% versus 63.33 and 21.67. DAM is reported to struggle particularly on oriented infrastructure such as Tower crane, where DescribeEarth scores 64.58% and DAM 0.0%.

Question-type analysis clarifies the model’s strengths and trade-offs. On Surrounding, DescribeEarth reaches 54.41%, compared with 43.99% for GPT-4o and 6.46% for DAM, which aligns with the claim that the architecture is particularly effective at contextual environmental description. On Language, DescribeEarth scores 98.77%, slightly above GPT-4o at 98.14%. On Usage, however, GPT-4o leads with 44.64% versus 35.49% for DescribeEarth, suggesting stronger general world knowledge for inference-oriented questions. On Appearance, fine-tuned Qwen2.5-VL-3B scores 53.77%, above DescribeEarth’s 42.26%, indicating that DescribeEarth’s improvements come partly from better balancing appearance with contextual and structural description rather than maximizing one subdimension.

The paper also reports several stable ablation findings: feature-level RemoteCLIP guidance is better than no guidance or textual category hints; scale-adaptive cropping is better than fixed-size focal cropping; higher global resolution improves complex and OOD performance by up to ~2.3%; and freezing the vision encoder improves simple and OOD performance relative to making it trainable.

The framework has direct application value in environmental monitoring, urban planning, disaster management, agriculture, and resource monitoring. The authors explicitly state that integrating DescribeEarth with detection pipelines enables automatic large-scale semantic annotation and supports searchable geospatial knowledge graphs or text indices. This suggests a role for Geo-DLC as an intermediate layer between object detection and language-based geospatial querying.

The limitations are equally explicit. The current system is single-modal RGB and single-time, with no multi-temporal, SAR, LiDAR, or hyperspectral input. Category coverage is inherited from DOTA and DIOR, so the dataset is biased toward man-made structures and vehicles. Automatic annotation, even with human verification, may retain subtle MLLM-generated errors or biases. DE-Benchmark depends on GPT-4.1 as judge, and the judge may itself introduce biases. Future directions proposed by the authors include multi-temporal Geo-DLC, multi-modal extensions, richer geospatial integration with coordinates and GIS layers, and scaling to larger MLLMs and interactive systems.

Taken together, these elements define Geo-DLC as a localized, object-centric remote-sensing captioning regime whose main novelty lies not only in producing longer descriptions, but in coupling localization, attribute-rich annotation, QA-based evaluation, and scale-aware multimodal modeling into a single research framework (Li et al., 30 Sep 2025).