RoboDesign1M: Multimodal Robot Design Dataset

• RoboDesign1M is a multimodal dataset comprising 1M unique image–text pairs extracted from robotics publications.
• Its five-stage curation pipeline combines automated extraction with human validation to ensure high-quality, diverse annotations across 1,000 robotics subfields.
• The dataset underpins benchmark tasks in visual QA, text–image retrieval, and text-to-design generation, advancing AI-driven robot design automation.

RoboDesign1M is a large-scale, multimodal dataset designed for robot design understanding, consisting of approximately 1 million image–text pairs extracted from scientific literature in the robotics domain. Encompassing a broad spectrum of robotics engineering subfields, RoboDesign1M enables research in design automation, retrieval, generative models, and AI-driven design assistance by providing high-quality annotated figures, textual descriptions, and visual question–answering dialogues. Its rigorous five-stage data curation pipeline combines automatic extraction techniques with targeted human validation to ensure relevance, diversity, and annotation quality, establishing the dataset as a benchmark for multimodal design tasks (Le et al., 9 Mar 2025).

1. Scope and Coverage

RoboDesign1M targets the full breadth of robotics engineering by sourcing image–text data from over 1,000 distinct robotics-related topics as indexed by OpenAlex. These topics span major domains—including mobile robotics, aerial robotics, industrial manipulators, humanoid systems, medical robots, soft robotics, underwater platforms, space robotics, swarm and legged systems, rehabilitation devices, and reconfigurable morphologies. The dataset's topical diversity enables both domain-specific and cross-domain research in robot design.

Robotics subfield representation in RoboDesign1M is summarized in the following table:

Subfield	Sample Count	Percentage
Mobile Robotics	210,000	21 %
Aerial Robotics	160,000	16 %
Industrial Manipulators	120,000	12 %
Medical Robotics	100,000	10 %
Humanoid Robotics	90,000	9 %
Soft Robotics	80,000	8 %
Underwater Robotics	70,000	7 %
Swarm Robotics	60,000	6 %

This distribution reflects both comprehensive coverage of central robotics areas and inclusion of more specialized domains such as soft and rehabilitation robotics.

2. Data Collection and Curation Pipeline

The construction of RoboDesign1M employs a five-stage semi-automated pipeline which synthesizes large-scale automatic data extraction with stringent targeted human annotation and post-processing. The pipeline stages are:

1. Document Retrieval: Utilizing a curated set of ∼1,000 robotics-related keywords, documents are retrieved from Google, OpenAlex, and IEEE Xplore. This yields over 1 million full PDF publications for downstream extraction.
2. Figure & Text Extraction: Application of PDFFigures2.0 extracts figures, captions, and full texts, resulting in approximately 5 million raw (image, caption, document_text) tuples.
3. Relevance Filtering: Human experts manually annotate 32,000 images for robot-design relevance, which informs training of an ensemble image classifier (OpenCLIP ImageNet-Sketch-tuned and EfficientNetV2), achieving >95% test accuracy. The classifier filters relevant samples, resulting in about 1.4 million robot-design image–text pairs.
4. Deduplication: OpenCLIP embeddings and approximate kNN search (via Faiss), supplemented by caption text similarity comparison, remove nearly identical visual and textual duplicates, yielding a final set of 1 million unique pairs.
5. Instruction-Following Data Construction: For each sample, the LLaMa 3.3 70B LLM is prompted to extract reference paragraphs and generate 1–3 visual QA pairs in instruction-tuning format. Human validation includes reference cross-checks to minimize spurious associations.

The final dataset contains approximately 1.3 million visual QA dialogues in addition to the 1 million core image–text pairs.

3. Modalities, Annotation Schemes, and Metadata

Each RoboDesign1M sample is natively multimodal, integrating imagery and multiple layers of text annotations, as well as rich metadata fields:

• Image Modality: Composition includes 2D CAD renderings, technical drawings, photographs of working prototypes, and schematic illustrations of mechanisms.
• Text Modalities:
  • Caption: The original figure caption as published.
  • Reference Text: Manually validated paragraphs referencing the figure, extracted via LLM prompting.
  • QA Annotations: One or more (user question, assistant answer) dialogue pairs in visual instruction format.
• Metadata: Each entry includes document identifier (DOI/arXiv ID), figure ID, publication venue and year, associated OpenAlex topics, author list, and extracted image/figure keywords.

Caption statistics indicate an average length of 15.14 words, and the lemmatized vocabulary comprises 113,000 unique tokens (excluding stop words). Image keyword diversity surpasses 12,000 unique technical terms.

4. Dataset Statistics

RoboDesign1M's quantitative profile is as follows:

Modality / Annotation	Count	Relative Scale
Raw image–text pairs	5,000,000	5×
Robot-relevant pairs	1,400,000	1.4×
Final unique pairs	1,000,000	1×
Visual QA dialogues	1,300,000	1.3×

Topic coverage is anchored by 1,000 OpenAlex robotics sub-domains, ensuring both topic breadth and vocabulary diversity. This heterogeneity supports generalist and specialization-oriented downstream model training.

5. Benchmark Tasks and Baseline Performance

RoboDesign1M defines and supports three flagship multimodal benchmark tasks for robotic design understanding:

1. Visual Question Answering (VQA)
   • Task: Given an image $I$ and question $Q$, predict answer $A$.
   • Formulation: $\hat{A} = \arg\max_A P(A \mid I, Q)$
   • Metrics: BLEU-N, METEOR, and L3Score (average log-likelihood of $\hat{A}$ under GPT-4).
   • Baselines:

   | Model | BLEU ↑ | METEOR ↑ | L3Score ↑ | |-------------------|--------|----------|-----------| | GPT-4o (no FT) | 0.025 | 0.208 | 0.296 | | LlaVa-1.5 (FT) | 0.008 | 0.098 | 0.181 | | Qwen2-VL (FT) | 0.013 | 0.123 | 0.311 |

2. Text–Image Retrieval

   • Task: Retrieve the correct image $I$ given caption $C$ from $N$ candidates.
   • Metric: Recall@k. Results for BLIP-2 (finetuned):

   | Training Set | R@1 ↑ | R@5 ↑ | R@10 ↑ | |--------------------|-------|--------|--------| | Text2CAD | 5.36 | 10.10 | 12.98 | | Ghezelbash | 4.78 | 9.86 | 13.20 | | RoboDesign1M (Ours)| 12.9 | 27.46 | 36.28 |

3. Text-to-Design Image Generation

   • Task: Generate novel design image $\tilde{I}$ from text $T$.
   • Metric: Fréchet Inception Distance (FID).
   • Baselines:

   | Model | Finetuned? | FID ↓ | |----------------------------|------------|--------| | Stable Diffusion XL (orig.)| No | 45.83 | | Stable Diffusion XL | Yes | 39.42 |

Performance gains suggest improvements when training on the domain-specific corpus of RoboDesign1M versus prior datasets.

6. Usage Guidelines and Licensing

The dataset is released with the following provisions:

• Data Splits: A 95/5 train/test split on unique (image, caption) pairs, ensuring no overlap of figures or documents between partitions.

• Licensing: All source documents are public-domain or open-access; RoboDesign1M itself is distributed under the CC-BY 4.0 license.

• Best Practices:

  • Use the provided relevance classifier code for dataset extension or further filtering.
  • Perform deduplication with approximate kNN and text matching for custom corpora.
  • Employ both captions and validated reference text for instruction-tuned model building to reduce hallucinations.
  • Conduct cross-dataset evaluations for transferability assessment.
  • Apply early stopping on held-out validation due to potential visual regularity in robotic designs.

A plausible implication is that, by adhering to these guidelines and leveraging RoboDesign1M's scale and diversity, model developers can advance the state of automated design understanding and generative design tasks in robotics.

7. Significance and Research Impact

RoboDesign1M establishes a new benchmark for multimodal design applications in robotics by providing a high-density, rigorously filtered corpus that supports disparate research needs: VQA, cross-modal retrieval, and text-guided generative modeling. Its diversity across 1,000 subfields, integration of multiple annotation types, and substantial QA corpus facilitate robust evaluation and instruction-tuning relevant to both foundation models and specialized robotic applications (Le et al., 9 Mar 2025).

By enabling domain-specific fine-tuning and foundation model development, RoboDesign1M addresses the persistent barrier of dataset sparsity in robot design research. This suggests enhanced potential for AI-driven design automation, retrieval, and comprehensive analysis in modern robotics workflows.