PILOT-Bench: USPTO PTAB Appeals Benchmark

• PILOT-Bench is a large-scale benchmark framework that aligns PTAB decisions with USPTO patent data and formalizes legal reasoning into IRAC-structured classification tasks.
• It organizes evaluations into three tasks—Issue Type, Board Authorities, and Subdecision—demonstrating LLM performance on structured legal analyses.
• The benchmark reveals significant performance gaps between closed- and open-source models, offering actionable insights for improving legal NLP systems.

PILOT-Bench is a large-scale benchmark framework for evaluating legal reasoning in the patent domain, aligned with ex parte appeal decisions from the United States Patent and Trademark Office’s Patent Trial and Appeal Board (USPTO PTAB). PILOT-Bench formalizes three IRAC-structured classification tasks—Issue Type, Board Authorities, and Subdecision—at the case level, systematically aligning PTAB decisions with USPTO patent data. It establishes, for the first time, a PTAB-centric foundation for benchmarking the structured legal reasoning capabilities of LLMs in the patent context, with open access to all data and resources (Jang et al., 8 Jan 2026).

1. Dataset Construction and Alignment

1.1 Data Sources and Size

PILOT-Bench integrates three key sources:

• PTAB Metadata: 170,000+ records from the USPTO PTAB API v2.
• PTAB Decisions: ∼25,000 decisions spanning 2009–2024, obtained via PDF-to-OCR processing.
• USPTO Patent Full Text: XML data (2006–2024) including patent titles, specifications, and claims.

Following quality-control (e.g., OCR cover-page stability, exclusion of pre-2006 or ambiguously aligned applications), PILOT-Bench comprises 18,738 PTAB cases, with 15,482 cases successfully mapped to USPTO patent records.

1.2 Case-Level Alignment

Alignment is achieved by matching the PTAB appellantApplicationNumberText to the USPTO applicationNumber extracted from the patent XML. In cases with multiple publications per application, the publication closest to the PTAB decision date is selected. Exclusions include pre-2006 applications and ambiguous or duplicate case threads.

1.3 Summary Statistics

From Statement of the Case and Analysis (pre–Opinion-Split):

• Mean per-decision length: 1,864 words (±1,144)
• Statement of the Case: 433 words (±276)
• Analysis: 1,435 words (±1,065)

After the Opinion Split (retaining only appellant_arguments and examiner_findings):

• Average per-case input: 1,409 words (±936)
• Each case links on average to approximately 2.05 patents (one base, one prior).

2. IRAC-Aligned Classification Tasks

PILOT-Bench operationalizes three stages of the IRAC methodology—Issue, Rule, Conclusion—as explicit classification tasks, fully mapped to PTAB decision content.

2.1 Issue Type (IRAC–Issue)

Definition: Multi-label classification to identify the statutory grounds contested (based on 35 U.S.C.), given appellant_arguments and examiner_findings.

Labels (5 total):

• 101: subject-matter eligibility
• 102: novelty
• 103: non-obviousness
• 112: written description/enablement/definiteness
• Others: any additional statutory grounds (e.g., priority, new matter)

Labels are regularized into this five-label structure, with multi-label instances for cases citing multiple statutory sections.

2.2 Board Authorities (IRAC–Rule)

Definition: Multi-label classification of procedural authorities (predominantly under 37 C.F.R. with rare 35 U.S.C. provisions) cited by the Board.

Labels (9 total):

• 37 C.F.R. § 41.50, including sub-provisions (a)–(f)
• § 1.131 (swear-behind affidavits)
• § 1.132 (traversing affidavits)
• 35 U.S.C. § 251, § 161 (reissue/plant cases)
• Others (all other citations)

Labels are annotated from PTAB boardRulings, collapsed for infrequent provisions.

2.3 Subdecision (IRAC–Conclusion)

Definition: Single-label multiclass prediction of the Board’s final determination.

Fine-grained labels (23) include: “Affirmed”, “Reversed with New Ground of Rejection”, “Vacated”, and others. For evaluation, these are deterministically consolidated to the following coarse classes:

1. Affirmed
2. Affirmed with New Ground of Rejection
3. Affirmed-in-Part
4. Affirmed-in-Part with New Ground
5. Reversed (±New Ground)
6. Vacated/Remanded
7. Others

3. Evaluation Protocol

3.1 Data Split Strategy

There are no explicit training/validation/test splits. All 15,482 cases are evaluated in a zero-shot regime under a unified evaluation protocol.

3.2 Input Variations

Three alternative input structures probe sensitivity to claim and argument role structure:

1. Split (Base): Separate “Appellant’s Arguments” and “Examiner’s Findings.”
2. Merge: Role-neutral concatenation.
3. Split + Claim: Split base with full claim text appended.

3.3 Evaluation Metrics

Metrics per task:

• Issue Type & Board Authorities (multi-label): Exact Match, Macro-F1, Micro-F1
• Subdecision (multi-class): Accuracy, Macro-F1, Weighted-F1

Key formulas:

$$\mathrm{Precision} = \frac{TP}{TP + FP},\quad \mathrm{Recall} = \frac{TP}{TP + FN}$$

$$\mathrm{Micro\!-\!F1} = \frac{2\sum_c TP_c}{2\sum_c TP_c + \sum_c FP_c + \sum_c FN_c}$$

$$\mathrm{Macro\!-\!F1} = \frac1C\sum_{c=1}^C \frac{2\,TP_c}{2\,TP_c + FP_c + FN_c}$$

4. Model Families and Benchmark Results

4.1 Model Families

Benchmarked LLMs include:

Closed-source (commercial): Claude-Sonnet-4, Gemini-2.5-pro, GPT-4o, GPT-o3, Solar-pro2 Open-source: LLaMA-3.1-8B, Mistral-7B, Qwen-8B, T5-2B (Gemma)

4.2 Performance Summary

Model performance for the central classification tasks (Split Base input):

Model	Issue Type Micro-F1	Board Authorities Micro-F1	Subdecision Accuracy
Claude-Sonnet-4	0.790	0.65 (≈ mean)	0.59
Gemini-2.5-pro	0.792	0.69	0.59
GPT-4o	0.786	0.68	0.58
GPT-o3	0.797	0.66	0.57
Solar-pro2	0.771	0.62	0.56
Qwen-8B	0.774	0.20	0.47
LLaMA-3.1-8B	0.579	≪0.30	0.48
Mistral-7B	0.608	≪0.30	0.07
T5-2B	0.447	≪0.30	0.03

A substantial gap is observed between commercial and open-source models, especially for Board Authorities and Subdecision tasks. For Issue Type, the best open-source (Qwen-8B) approaches high closed-source performance, but for Board Authorities, open-source models do not exceed Micro-F1 = 0.20.

Table: Issue Type (Split Base) Results

Model	Exact Match	Macro-F1	Micro-F1
Claude-Sonnet-4	0.587	0.546	0.790
Gemini-2.5-pro	0.587	0.663	0.792
GPT-4o	0.575	0.652	0.786
GPT-o3	0.596	0.664	0.797
Solar-pro2	0.558	0.524	0.771
LLaMA-3.1-8B	0.183	0.105	0.579
Mistral-7B	0.341	0.211	0.608
Qwen-8B	0.556	0.525	0.774
T5-2B	0.077	0.385	0.447

5. Error Analysis and Evaluation Insights

5.1 Error Tendencies

• All models exhibit low Macro-F1 on long-tailed labels (notably rare subdecision outcomes).
• Open-source models frequently emit out-of-schema or free-form answers (e.g., bare numerals “102(e)”, hallucinated citations) not consistent with the label schema.
• T5-2B, in multi-label settings, tends to over-predict all possible labels, resulting in inflated recall but low precision.

5.2 Input Variation Effects

• Split (Base) input is most consistent.
• Merge input sometimes gives smoother predictions for Claude-Sonnet-4 and GPT-o3 but erases helpful role assignment signals.
• Split + Claim input generally impairs performance, especially for Board Authorities: claim text increases input length threefold and introduces semantic noise rather than relevant content.

5.3 Recommendations for Dataset and Model Alignment

Recommended directions:

• Selective claim-augmentation: Append only claim passages that are directly relevant to the contested statutory ground (e.g., claim 1 when §103 is at issue).
• Instruction tuning with schema-enforcing auxiliary losses to curtail out-of-schema or invalid responses.
• Oversampling or label rebalancing to compensate for rare subdecision types.
• Extension to generation-based IRAC application tasks to enable multi-step, non-classification reasoning.

A plausible implication is that structured legal datasets with role-aware input segmentation and robust label schemas are necessary for strong LLM performance on complex legal reasoning in the patent domain.

6. Context and Significance

PILOT-Bench is the first benchmark to provide large-scale, IRAC-aligned, case-level evaluation directly targeting LLM legal reasoning in the context of PTAB ex parte appeals. By integrating PTAB and USPTO data and structuring reasoning per the IRAC paradigm, it enables detailed assessment of statutory interpretation, procedural rule mapping, and conclusion synthesis at real-world case scale. Results expose limitations in both open- and closed-source model alignment, especially regarding rare or nuanced legal outcomes. The benchmark provides a reproducible substrate for legal NLP researchers aiming to improve LLM performance on patent casework and for the design of new reasoning-focused legal datasets and model architectures (Jang et al., 8 Jan 2026).