HQRS-IT-210K: Multimodal Robotics & RS Data

• HQRS-IT-210K is a dual-purpose dataset providing interleaved image-text(-action) pairs for both embodied robotics and remote sensing applications.
• It uses automated pipelines with robust object detection and LLM-based instruction parsing to achieve over 99% accuracy in critical tasks.
• The dataset enhances model generalization, enabling state-of-the-art performance in VLA command generation and detailed remote sensing captioning.

HQRS-IT-210K refers to two distinct, large-scale datasets in contemporary machine learning research: (1) the Open Interleaved X-Embodiment Dataset for vision-language-action (VLA) robot manipulation, and (2) the HQRS-IT-210K remote sensing image-text corpus for vision-language foundation models. Both datasets are designed to bridge critical gaps in their respective domains—multi-modal embodied robotics and high-fidelity remote sensing captioning—by constructing extensive, high-quality image-text or image-text-action pairs using systematic, pipeline-driven methodologies.

1. Vision-Language-Action Embodied Robotics: The Open Interleaved X-Embodiment Dataset

The HQRS-IT-210K Open Interleaved X-Embodiment Dataset targets the training of generalist Vision-Language-Action models by providing 210,000 real-world robot demonstration episodes. Each episode’s language instruction is automatically augmented through interleaving detected object images at key points in the instruction string. This interleaving enables VLA models to parse robot instructions as mixed text/image token streams and generate continuous, physically grounded action sequences. The dataset is constructed via automated relabeling of text-only episodes from the Open X-Embodiment project, encompassing eleven contextual corpora such as RT-1 (fractal_dataset), Berkeley Autolab UR5, Stanford Hydra, and others (Fan et al., 4 May 2025).

1.1 Automatic Conversion Pipeline

The conversion process consists of three stages:

• Instruction Parsing: Key object phrases are extracted from the instruction via an LLM (Qwen2.5), which outperforms rule-based NLP in handling compositional and long instructions.
• Open-Vocabulary Object Detection: Each key phrase is mapped to its visual referent in the episode’s RGB frames using OWLv2 object detection, achieving >99% accuracy on common objects.
• Data-Quality Verification: Challenging or ambiguous cases are re-classified by Qwen2.5-VL and, if necessary, refined by keypoint segmentation using SAM, raising corner-case segmentation accuracy from <50% to >95%.

Mathematically, the original segmented instruction sequence $\ell_1, ..., \ell_n$ is replaced with the interleaved sequence

$$\mathcal{I} = (\ell_1, \mathbf{I}_1, \ell_2, \mathbf{I}_2, ..., \ell_n, \mathbf{I}_n)$$

with $\ell_i$ signifying text and $\mathbf{I}_i$ the corresponding robot-captured object crop.

1.2 Dataset Statistics and Format

• Episodes: 210,000
• Total frames: 13 million (~62 fps)
• Unique object categories: ~3,500 (long-tail, with top 100 covering 35% of episodes and “tail” ≥1,000 covering 65%)
• Task types: Pick-and-place, stacking, tool use, open/close actions, etc.
• Action sequences: 2–25 primitives per episode (mean ≈ 8), with (RGB, proprioception)→(Δx,Δq,gripper command) tuples.
• Annotation Format: Each episode is a JSON file listing frames, proprioceptive states, an interleaved instruction array (text content and object crop image paths), and the discrete action sequence. For model training, images are converted to ViT patch embeddings, and text segments are tokenized with the corresponding Vision-LLM tokenizer.

Parameter	Value/Description	Notes
Episodes	210,000	Real robot demonstrations
Object categories	≈3,500	Long-tail distribution
Frames	13,000,000	≈62 fps; multi-view RGB streams

1.3 Quality Assurance and Coverage

OWLv2 achieves >99% accuracy in bounding-box detection. The two-stage Qwen2.5-VL + SAM pipeline boosts segmentation accuracy for rare categories from under 50% to over 95%. Instruction coverage ensures that all training samples use robot-captured crops; neither web images nor user-provided sketches are included as training data, although models built on HQRS-IT-210K generalize at >70% success to such zero-shot evaluation cases.

1.4 Limitations and Use Cases

Limitations of HQRS-IT-210K include its exclusive use of robot-view images for instruction interleaving, modest annotation granularity (bounding box crops, no full 3D pose or segmentation), and 2–3x increased computation per training sample owing to longer mixed-modal token streams. The dataset supports pretraining VLA models for multimodal command following, zero-shot generalization evaluation, and benchmarking downstream robotic manipulation tasks in both simulation and real-world settings (Fan et al., 4 May 2025).

2. Remote Sensing Image-Text: HQRS-IT-210K Dataset

HQRS-IT-210K also denotes a distinct, large-scale remote sensing image-captioning dataset used for advancing vision-language foundation models on satellite and UAV imagery (He et al., 22 Jul 2025). This corpus comprises 210,556 unique remote-sensing images annotated with approximately 1,263,336 text captions (averaging six per image, with a mean caption length of 35.6 words—significantly longer and richer than previous RS corpora).

2.1 Dataset Sources

The image assemblage spans 23 public benchmark datasets, covering multispectral and RGB satellite imagery (e.g., fMoW, AID, RESISC45, EuroSAT), UAV collections (VisDrone, Stanford Drone, CARPK), semantic segmentation sets (iSAID, LoveDA), and benchmark detection datasets (HRRSD, DOTA, HRSC).

2.2 Caption Generation Pipeline: MpGI

The Multi-Perspective Generation and Integration (MpGI) pipeline consists of:

• Stage 1 — Detailed Multi-Perspective Generation:
  • Rule-MLLM Relay: Starting from concise class labels, ChatGPT-4V generates more detailed, class-specific captions, followed by manual verification.
  • Detection & Segmentation (DET-10): Annotations are converted to natural language via Algorithm A2D, producing relational object count/location statements.
  • Instruction-Guided MLLM Generation: Kosmos-2 and LLaVA-1.6 are prompted with annotation-derived templates, yielding detailed, 220-word multi-view descriptions per image.
• Stage 2 — LLM-Based Integration:
  • A LLaMA-3-8B-Instruct LLM integrates triplets of Stage 1 captions using two prompt styles: a single summary sentence with rich adjectives, and five alternative detailed variants from which one is randomly selected. Empirically, randomly sampling between styles with $\alpha=0.5$ maximizes downstream performance.

Each image is thus assigned six distinct captions, supporting discriminative (HQRS-CLIP) and generative (RS-CoCa) model training.

Component	Details	Notes
Images	210,556	Deduplicated
Captions	1,263,336 (≈6 per image)	Long-form
Avg. caption length	35.6 words	Detailed

2.3 Quality Assessment and Metrics

• Retrieval: HQRS-CLIP ViT-B-32 boosts RSITMD zero-shot mean recall from 36.68 (GeoRSCLIP) to 40.15 (HQRS-CLIP B-32).
• Fine-tuned retrieval: R@1 (image→text) improves from 27.9 to 35.2.
• Captioning: RS-CoCa sets state-of-the-art METEOR (0.425), ROUGE-L, and SPICE (0.555) scores on RSICD benchmarks.
• Caption Distribution: Ensures uniform coverage between 20–70 words (Figure 1, (He et al., 22 Jul 2025)), reducing exposure to label length bias.

Qualitative reviews by domain experts affirm that RS-CoCa model captions are “on par with or surpassing manual annotations.” Extensive manual review eliminates hallucinations and extraneous content.

2.4 Fine-Tuning and Benchmarking

• Discriminative Model: CLIP (ViT-B-32, ViT-L-14). Trained on one caption per image (210,000 pairs), AdamW optimizer, InfoNCE loss. HQRS-CLIP B-32 achieves superior retrieval while using only 4.2% of GeoRSCLIP’s data.
• Generative Model: CoCa (ViT-L-14). Trained on all 1.26M captions. Outperforms SOTA on RS captioning metrics across RSICD, UCM datasets.
• Benchmarks: RSITMD, RSICD, UCM (retrieval/classification), LongRET3-test (long-text retrieval), AIR-SLT (semantic localization).

2.5 Release and Practical Usage

The HQRS-IT-210K dataset, including images and captions, is distributed under a permissive research license. Code, models, and dataset are available at https://github.com/YiguoHe/HQRS-210K-and-HQRS-CLIP. Preprocessing uses image tiling, P-hash and URL deduplication, bounding box extraction from segmentation, and caption cleaning. Captions are structured as JSON with fields: image_id, caption, prompt_style, source_stage. Loading and training procedures are provided as PyTorch pseudocode in the original publication (He et al., 22 Jul 2025).

3. Comparative Role and Impact

Both HQRS-IT-210K datasets drive state-of-the-art advances in their respective domains:

• The embodied robotics dataset enables model-agnostic training of VLA models that understand richly interleaved instructions, yielding 2–3x greater out-of-domain generalization than text-only baselines and facilitating robust response to heterogeneous real-world tasks (Fan et al., 4 May 2025).
• The remote sensing dataset establishes a new regime for VLFMs, showing that dense, multi-perspective long-form captions empower zero-/few-shot retrieval, classification, and captioning models to reach or surpass prior SOTA using much less data (He et al., 22 Jul 2025).

A plausible implication is that multi-modal and multi-perspective data synthesis methodologies—whether applied to robotic trajectories or to dense RS imagery—substantially enhance model generalization and reasoning across modalities.

4. Limitations

• Embodied Robotics: The dataset includes only robot-captured object crops for training instructions. Web images and sketches do not appear in the training corpus. Annotation is restricted to bounding-box crops; there is no full scene/3D segmentation. Longer token sequences increase computational cost by 2–3x per sample (Fan et al., 4 May 2025).
• Remote Sensing: Caption fusion is based on probabilistic or randomly-sampled LLM outputs, not algorithmic argmax. Final labels are constrained by annotation and model output quality and verified manually, but label noise from automatic generation remains possible (He et al., 22 Jul 2025).
• General: Both datasets are designed for research and pretraining in multimodal settings and are released under licensing terms consistent with their constituent sources.

5. Applications and Prospects

• Embodied Robotics: Pretraining VLA models to handle interleaved multimodal querying and to robustly generate continuous robot trajectories in both simulation and real-world settings. Tracking model zero-shot generalization to novel objects and modalities (e.g., instructing robots with sketches or unseen photographs) (Fan et al., 4 May 2025).
• Remote Sensing: Training, fine-tuning, and evaluating both discriminative and generative VLFMs, including CLIP and CoCa variants; benchmarking zero-/few-shot retrieval, scene classification, semantic localization, and natural language captioning in diverse RS scenarios (He et al., 22 Jul 2025).

The systematic pipeline design and demonstration of robust scaling suggest substantial potential for adapting similar large-scale interleaved datasets to new embodied or visual–linguistic domains.