DeepPatent2: Patent Drawing Benchmark

• DeepPatent2 is a comprehensive dataset consisting of over 2.7 million segmented patent drawings, enriched with hierarchical annotations and textual metadata.
• It leverages advanced methodologies such as a DistilBERT+BiLSTM-CRF pipeline, transformer-based segmentation, and refined OCR to ensure high accuracy in entity detection.
• The dataset facilitates diverse technical applications including patent image retrieval, captioning, cross-modal analysis, and even 3D reconstruction from multi-view drawings.

DeepPatent2 is a large-scale benchmark dataset for technical drawing understanding, especially patent image retrieval, captioning, and cross-modal analysis. Developed as an order-of-magnitude extension over its predecessor DeepPatent, DeepPatent2 incorporates more than 2.7 million segmented patent drawings with hierarchical class annotations, textual metadata, and multi-view image organization, making it the de-facto standard dataset for computational analysis of patent imagery and vision-language tasks in the technical domain (Ajayi et al., 2023, Kavimandan et al., 16 Jun 2025, Lo et al., 2024).

1. Dataset Construction and Scale

DeepPatent2 was assembled by collecting all US design patents (XML full text and associated TIFF figures) published between 2007 and 2020 via USPTO bulk downloads. The preprocessing pipeline consists of:

• Extraction of XML captions and in-text references.
• Semantic labeling using a DistilBERT + BiLSTM-CRF entity recognizer to tag OBJECT and VIEWPOINT entities in patent captions, achieving F₁ scores of 0.927 (OBJECT), 0.992 (VIEW), overall F₁ = 0.960 (tested on 300 hand-labeled captions).
• Figure segmentation using MedT (a transformer network fine-tuned on 500 annotated patent panels), producing 97% segmentation accuracy at a runtime ≈1/35 of heuristic approaches.
• OCR of text labels within figures via AWS Rekognition, followed by post-processing to fuse multi-token detections (raising label F₁ score from 0.807 to 0.968).
• Matching of image panels to figure labels and XML captions using nearest-centroid matching and integer label keys, reaching 97% panel-to-label assignment accuracy.

After alignment and filtering, the finalized corpus contains 2,785,762 segmented figures (“sub-images”) with paired semantic metadata. The collection captures 366,275 patents, approximately 132,890 unique object names, 22,394 distinct viewpoint labels, and an estimated compressed size of 314 GB (Ajayi et al., 2023).

2. Annotation Schema and Metadata

Every DeepPatent2 image is annotated at multiple levels:

• Class Taxonomy: Each image is classified by the Locarno International Classification: 32 main classes, subdivided into hundreds of subclasses (407 fine-grained classes used in retrieval experiments). Each image receives a triple: (main-class ID, subclass ID, patent ID) (Kavimandan et al., 16 Jun 2025, Lo et al., 2024).
• Object and Viewpoint: Automatic NLP pipelines extract both object labels (e.g., “circuit board,” “seating unit”) and viewpoint descriptions (e.g., “front, left perspective”).
• Additional Metadata: Provided per image are fields such as patent_ID, unique object_name, caption, figure/subfigure file names, as well as bounding box coordinates (physical and normalized) for subfigure panels and associated labels.
• Semantic Enrichment: For advanced retrieval and cross-modal experiments, the dataset supports enriched caption schemes—e.g., via GPT-4V and BLIP-2 captioners, with further alias and synonym generation using GPT-4 and prompt-based templates, yielding 20+ highly descriptive English texts per image (Lo et al., 2024).

Image entries are stored per year in consolidated JSON files containing all visual and textual fields. The approximate vocabulary size for captions is 25,000 tokens, with an average caption length of 12 words (Ajayi et al., 2023).

3. Label Distribution and Statistical Properties

DeepPatent2 is explicitly long-tailed. The Locarno subclass distribution shows the top 40% of classes (163/407) account for ≈85% of processed images, while the bottom 60% (244/407) cover just ≈15%, mirroring real-world design innovation and patenting trends (Lo et al., 2024). For example, “Optical apparatus” (class 16) and “Furnishing” (class 6) are the largest main classes, each accounting for approximately 12% and 11% of images, respectively (Ajayi et al., 2023).

Absolute per-class, per-subclass frequencies are not tabulated in all studies; frequency histograms are typically presented only in retrieval-specific works.

4. Benchmarking Protocols and Experimental Splits

DeepPatent2 is employed across vision, language, and multimodal benchmarks, notably retrieval and captioning:

• Default Splits: The distributed data does not enforce fixed splits; it is common practice to implement random train/validation/test splits at the figure level (e.g., 80/10/10) for captioning or vision-language tasks (Ajayi et al., 2023). For retrieval tasks, some studies restrict to patents from a specific year (e.g., only 2007) or apply temporal splits for simulating prior-art search (training: pre-2020; test: 2020 patents) (Kavimandan et al., 16 Jun 2025, Lo et al., 2024).
• Experimental Batching: Common training practice samples mini-batches of 64 patents, with two images per patent selected to form strongly positive “same-patent” pairs for contrastive learning (Kavimandan et al., 16 Jun 2025).
• Retrieval Evaluation: Test queries are typically constructed by selecting images from “query” years and retrieving similar images from a reference pool of earlier years, enforcing temporal eligibility (i.e., only prior granted patents are valid for retrieval). The standard protocol removes all images from the same patent from the retrieval pool except for the designated query image(s) (Lo et al., 2024, Kavimandan et al., 16 Jun 2025).
• Preprocessing: All images are resized to 224×224 pixels; training augmentations include random horizontal flip, rotation (±10°), Gaussian noise, random crop, random erasing, and GridMask. No color conversion is specified; original drawings are generally monochrome (Ajayi et al., 2023, Kavimandan et al., 16 Jun 2025, Lo et al., 2024).

5. Objectives, Loss Functions, and Metrics

Retrieval and captioning models using DeepPatent2 typically follow either single-positive or hierarchical contrastive regimes:

• Instance Contrastive (InfoNCE):

$$\mathcal{L}_\text{clip} = -\log\frac{\exp(\mathbf{t}_+^\top\mathbf{v}/\tau)} {\sum_{i=1}^B\exp(\mathbf{t}_i^\top\mathbf{v}/\tau)}$$

where $\tau$ is the temperature parameter.

• Hierarchical Multi-Positive Contrastive Loss: Assigns graded positive scores depending on semantic similarity based on Locarno taxonomy:

$$h_{ij} = \begin{cases} s_p = 1 & \text{if same patent ID}\ s_s = 0.35 & \text{if same subclass only}\ s_m = 0.20 & \text{if same main class only}\ 0 & \text{otherwise} \end{cases}$$

Each anchor’s loss is weighted by normalized relevance $H_i = \sum_j h_{ij}$. The multi-positive contrastive loss:

$$L_i = -\sum_{j=1}^K (h_{ij}/H_i)\cdot\log\left[\frac{\exp(\text{sim}(z_i, \tilde{z}_j)/\tau)}{\sum_{l=1}^K \exp(\text{sim}(z_i, \tilde{z}_l)/\tau)}\right]$$

(Kavimandan et al., 16 Jun 2025).

• Category-level Coarse Losses: For head/tail class balance, models introduce category-level InfoNCE losses and combine all losses with learned variance weighting (Lo et al., 2024).

Retrieval metrics include mean Average Precision (mAP), Recall@K, and MRR@K, calculated only over temporally eligible (prior-art) databases:

$$\mathrm{mAP} = \frac{1}{|Q|}\sum_{q\in Q}AP(q),\quad \mathrm{R}@K = \frac{1}{|Q|}\sum_{q\in Q}\mathbf{1}[\exists\,\text{relevant in top }K],\quad \mathrm{MRR}@K = \frac{1}{|Q|}\sum_{q\in Q}1/\min(r_q,K+1)$$

For image captioning, metrics include accuracy (joint match of object and viewpoint), METEOR, NIST, TER, and ROUGE scores (Ajayi et al., 2023).

6. Applications and Impact Across Domains

DeepPatent2 has become foundational for multiple technical drawing benchmarks and downstream tasks:

• Image Retrieval: Enables benchmarking of learning-to-retrieve architectures (contrastive, triplet, distribution-aware, language-informed) in both single-class and hierarchical relevance regimes (Kavimandan et al., 16 Jun 2025, Lo et al., 2024).
• Vision-to-Language: Supports vision-LLMs and captioners, demonstrating increased caption accuracy ($\sim$65%) with large-scale, semantically rich technical drawings (Ajayi et al., 2023).
• 3D Reconstruction: The multi-view structure (averaging 13.8 views per object) supports volumetric and mesh-based learning for reconstructing 3D geometry from sketches, surpassing datasets with limited canonical views.
• Cross-modal Retrieval: These data facilitate retrieval by text, image, or hybrid queries, leveraging both object and viewpoint semantics and LLM-generated captions (Lo et al., 2024).
• Design Generation and Document Summarization: Serves as training ground for generative models and as a large, well-annotated set for figure selection and scientific document summarization.
• Segmentation Research: Compound-figure segmentation masks (including 7.5% flagged as challenging/mis-segmented) serve as a realistic testbed for figure parsing and contour detection.

The corpus’ scale, annotation depth, and open availability have led to its adoption as a gold-standard benchmark in patent drawing retrieval research and technical drawing understanding.

7. Licensing, Access, and Reproducibility

DeepPatent2 is described under Ajayi et al., “DeepPatent2: A large-scale benchmarking corpus for technical drawing understanding,” published in Scientific Data (10:772, 2023). The article is under CC BY 4.0; users must cite Ajayi et al. (2023) and verify any further usage restrictions on the data repository or journal site (Ajayi et al., 2023, Kavimandan et al., 16 Jun 2025, Lo et al., 2024). Data and code (segmentation pipeline, semantic extraction tools) are accessible via Harvard Dataverse (DOI 10.7910/DVN/UG4SBD) and corresponding GitHub repositories.

Consistent protocols for train/validation/test division and preprocessing are documented in various benchmarking publications. However, certain works implement study-specific splits (e.g., temporal splits for retrieval), and absolute per-class counts or temporal breakdowns may vary by study.

---

References:

• Ajayi K. et al., “DeepPatent2: A large-scale benchmarking corpus for technical drawing understanding,” Scientific Data 10:772 (2023) (Ajayi et al., 2023)
• Lo et al., "LLM Informed Patent Image Retrieval" (Lo et al., 2024)
• "Hierarchical Multi-Positive Contrastive Learning for Patent Image Retrieval" (Kavimandan et al., 16 Jun 2025)