Pap2Pat-EvalGold Dataset

• Pap2Pat-EvalGold is a gold-standard evaluation dataset that benchmarks automated patent drafting by aligning technical details with statutory requirements.
• It adopts a multi-dimensional annotation protocol using the Chain-of-Legal-Thought paradigm to ensure precise technical mapping and legal compliance.
• Empirical evaluations show strong correlations with expert assessments, underlining its pivotal role in advancing legal AI and patent evaluation research.

Pap2Pat-EvalGold is a gold-standard evaluation dataset designed specifically for rigorous assessment of patent description generation systems under both technical and statutory constraints. Developed as part of the Pat-DEVAL framework, it enables quantitative and legal-level benchmarking of automated patent drafting, addressing deficiencies in prior metrics that fail to capture long-form coherence, enablement, and legal compliance. Pap2Pat-EvalGold adopts a multi-dimensional annotation protocol reflecting statutory requirements and technical fidelity, leveraging the Chain-of-Legal-Thought (CoLT) paradigm for comprehensive, legally-constrained evaluation using LLM-as-a-judge methodology (Yoo et al., 1 Jan 2026).

1. Dataset Definition and Scope

Pap2Pat-EvalGold consists of candidate patent descriptions generated by LLMs, paired with their corresponding source technology documents (typically academic papers), and systematically annotated across four evaluation dimensions. Each example includes:

• Source Document ($R$): An original technology disclosure (e.g., peer-reviewed paper).
• Generated Patent Description ($D_{gen}$): Output of an automated drafting system.
• Dimension Scores ($\{s_i\}$): Likert-style ratings for Technical Content Fidelity (TCF), Data Provenance (DP), Sectional Coverage (SC), and Legal-Professional Compliance (LPC).
• Rationales ($\{\rho_i\}$): Brief textual justifications for each rating authored by patent experts.

The gold labels are assigned through patent professional review, ensuring the dataset reflects jurisprudential standards (such as 35 U.S.C. § 112(a)) essential for downstream patent validity and enforceability.

2. Annotation Methodology and Statutory Constraints

Annotation in Pap2Pat-EvalGold is governed by the CoLT evaluation protocol. Each example is analyzed via a sequential, single-pass, layer-by-layer reasoning trace:

1. Technical Mapping Layer: Verifies element-level correspondence between $R$ and $D_{gen}$ regarding mechanisms, architectural modules, algorithms, and reference numerals. Detects factual hallucination and omission.
2. Statutory Compliance Layer: Examines enablement (reproducibility without undue experimentation) and possession as stipulated by 35 U.S.C. § 112(a).
3. Formal Consistency Layer: Assesses structural coverage (mandatory patent sections) and professional drafting conventions.

Crucially, annotations are forbidden from emitting numeric scores until substantiating each dimension with a documented legal rationale. The system prompt provided to evaluators embeds the full text of statutory constraints, functioning as a hard constraint on all assessment phases.

3. Scoring Functions and Evaluation Metrics

Pap2Pat-EvalGold operationalizes its annotation protocol into quantifiable evaluation metrics:

• For each example, the CoLT function

$$\left( \{s_i\}, \{\rho_i\} \right) = f_{CoLT}(P, L, R, D_{gen})$$

yields dimension-wise scores after complete reasoning passes.

• The aggregate Pat-DEVAL score for a document:

$S_{doc} = \frac{1}{4}\sum_{i=1}^4 s_i$

• Empirical validation employs Pearson correlation coefficients between automated metric outputs (Pat-DEVAL and baselines) and human expert scores on Pap2Pat-EvalGold ($N=146$):

$$\rho_{TCF}=0.68,\quad \rho_{DP}=0.72,\quad \rho_{SC}=0.64,\quad \rho_{LPC}=0.73,\quad \rho_{Avg}=0.69$$

These metrics quantitatively establish coherence and statutory validity correlating with professional audit.

Table: Metric Correlations on Pap2Pat-EvalGold ($N=146$)

Metric	TCF	DP	SC	LPC	Avg
BLEU	0.12	0.08	0.05	0.03	0.07
BERTScore	0.31	0.27	0.18	0.15	0.23
Standard CoT	0.45	0.41	0.38	0.32	0.39
G-Eval	0.58	0.54	0.51	0.45	0.52
Pat-DEVAL (CoLT)	0.68	0.72	0.64	0.73	0.69

Removal of CoLT logic sharply decreases compliance scores, highlighting the necessity of statutory guidance for legal fidelity in evaluation protocols.

4. Role in LLM-as-a-Judge and CoLT Evaluation Paradigm

Pap2Pat-EvalGold directly operationalizes the LLM-as-a-judge paradigm by requiring the evaluator model $M$ to simulate a Person Having Ordinary Skill in the Art (PHOSITA). This boosts alignment with professional patent standards and reduces abstract or generic errors common in surface-level LLM metrics.

The CoLT pipeline mandates:

• Explicit role prompts (“You are a Senior Patent Examiner (PHOSITA)”).
• Embedded statutory instructions.
• Single-pass, multi-layered reasoning before score emission.

Pap2Pat-EvalGold thus enables robust, legally-constrained LLM evaluation, providing a model for chain-of-thought based legal audit far beyond traditional similarity metrics or generic QA datasets.

5. Comparative Context and Related Evaluation Resources

Pap2Pat-EvalGold distinguishes itself from prior gold sets by targeting patent description, not merely claims or abstracts, and by encoding both technical and statutory compliance dimensions. Previous resources (e.g., BLEU- or BERTScore-based grading, or datasets from general legal text LLMs such as Legal$\Delta$ (Dai et al., 17 Aug 2025)) lack granular, section-specific and law-specific labeling.

In comparative studies, Pat-DEVAL demonstrates superior correlation and legal fidelity, especially in Legal-Professional Compliance ($\rho=0.73$), against both standard CoT and transformer-based evaluators, underpinning the dataset’s value for rigorous patent AI deployment.

6. Empirical Impact, Limitations, and Prospective Applications

Pap2Pat-EvalGold has established itself as a methodological benchmark for automated patent description systems, driving the development and external validation of legally-grounded LLM frameworks. Its empirical findings inform best practices in prompt design, statutory constraint injection, and scorer calibration.

The main limitation remains its domain specificity: it is tailored to US patent law (notably 35 U.S.C. § 112(a)), and its framework may require adaptation for international statutes or cross-jurisdiction patent analysis. A plausible implication is that future datasets may need to embrace multi-jurisdiction legal standards to scale evaluation beyond the US context.

For practical deployment, Pap2Pat-EvalGold supports not only benchmarking but also reinforcement learning and supervised fine-tuning protocols by offering annotated rationales and granular flaws. This suggests it will continue to play a central role in legal AI research concerned with both technical and statutory validity of generated patent documents.