LaborBench: AI Benchmark for UI Law
- LaborBench is a benchmark that defines and tests AI systems on statutory and regulatory extraction tasks in U.S. unemployment insurance law.
- The dataset is constructed by transforming detailed DOL tables into over 3,700 machine-readable QA pairs with specialized Boolean supplements.
- Empirical evaluations show that retrieval-augmented generation systems, especially STARA, can significantly improve accuracy despite challenges in retrieval and reasoning.
Searching arXiv for LaborBench and related benchmark papers. LaborBench is a question–answer benchmark for evaluating AI systems on statutory and regulatory extraction, comparison, and multi-jurisdictional survey tasks in U.S. unemployment insurance (UI) law. It was introduced from the U.S. Department of Labor’s annual “Comparison of State Unemployment Insurance Laws” tables and later used to benchmark retrieval-augmented generation (RAG), the Statutory Research Assistant (STARA), and commercial AI statutory survey tools. In this setting, the benchmark operationalizes a technically demanding legal-research problem: fifty different codes, scattered provisions, cross-references, defined terms, exceptions, and temporal qualifiers, all within a domain whose annual DOL reference spans roughly 200 pages and over 101 dimensions of state variation (Hariri et al., 26 Aug 2025, Afane et al., 7 Feb 2026).
1. Origin in unemployment-insurance statutory comparison
LaborBench emerged from work by Hariri and Ho on AI for statutory simplification. Their starting point was the DOL’s “Comparison of State Unemployment Insurance Laws” (CSUIL), a 200-page collection of 86 tables spread across eight UI subfields. The source process is itself unusually intensive: a six-lawyer DOL team spends six months combing through state statutes and regulations to produce a table-based survey covering all 53 jurisdictions and roughly 3,700 questions answered. In the later benchmarking paper, the same domain is described as involving state-by-state variations in UI law across over 101 distinct dimensions, with substantial complexity arising from dispersed provisions, cross-references, exceptions, and time-sensitive qualifiers (Hariri et al., 26 Aug 2025, Afane et al., 7 Feb 2026).
The benchmark’s motivation is tied to legal AI use cases in code simplification and statutory survey generation. The earlier paper frames UI law as a particularly acute test bed because statutory and regulatory heterogeneity is high, yet the task remains structured enough to support answer validation. A plausible implication is that LaborBench occupies a middle ground between doctrinal QA and full legal memo generation: it is narrow enough to admit typed answers, but realistic enough to expose failures in retrieval, statutory interpretation, and citation grounding (Hariri et al., 26 Aug 2025).
2. Dataset construction, scope, and ground truth
The original LaborBench transformation pipeline converts raw CSUIL tables into machine-usable QA entries. For each table and column, the papers report extraction of prelude text as context, derivation of a natural-language question template, and instantiation of that template across jurisdictions. The resulting resource contains 3,700+ QA pairs typed by bool, int, or str, with footnotes stored separately as metadata. In addition, a Boolean supplement was derived from 20 multi-jurisdiction tables with categorical or string entries, producing 1,272 binary questions and a 4,972-row × 21-column Boolean QA table (Hariri et al., 26 Aug 2025).
The later legal-RAG evaluation paper describes a specific LaborBench evaluation slice consisting of 40 question types and 1,647 Boolean classification items. Each question asks, for each state, whether a particular statutory feature exists. Reported examples of question types include self-employment assistance, SNAP overissuance deductions, alternative base period, and part-time search rules. This suggests that the published benchmark family contains both a broader QA conversion of the DOL materials and a narrower Boolean subset used for controlled model comparisons (Afane et al., 7 Feb 2026).
Ground truth was created through a multi-month manual enumeration by U.S. Department of Labor attorneys. The attorneys systematically reviewed every state UI code, extracted relevant provisions, and tabulated True/False answers, with the output double-checked by senior DOL staff. The later paper characterizes this as authoritative but not flawless, which becomes central to subsequent error analysis and benchmark reinterpretation (Afane et al., 7 Feb 2026).
3. StateCodes, retrieval architecture, and evaluation protocol
LaborBench is tightly coupled to StateCodes, a legal corpus introduced to support end-to-end RAG research. StateCodes has a reported size of 8.7 GB and contains 1.8 million statutory code sections totaling 488 million words and 2.2 million regulatory code sections totaling 268 million words. Statutes were scraped via Justia and regulations sourced from Fastcase, with content organized by state, then titles or chapters, then sections, each retaining URL, title path, and body text (Hariri et al., 26 Aug 2025).
The retrieval stack reported in the earlier paper uses 1,000-token chunking with 200-token overlap via tiktoken and cl100k_base. Dense retrieval is implemented with DPR-style embeddings, including E5\textsubscript{large}, OpenAI text-embedding-3-small, Google text-embedding-004, and Gemini embeddings, indexed with MIPS; sparse retrieval uses Okapi BM25. At inference time, the system retrieves up to chunks, deduplicates sections, and supplies the full section texts, titles, and URLs to the LLM prompt. Generation is schema-constrained: the model is prompted to reason and then emit JSON with typed fields and a citation URL (Hariri et al., 26 Aug 2025).
The evaluation methodology differs slightly across the two LaborBench papers. For the broad RAG study, the Boolean supplement is scored with precision, recall, , and accuracy, alongside citation recall@5 and citation accuracy. For the later comparative study of STARA and commercial tools, the core metric is Boolean accuracy over 1,647 questions, defined as
This metric choice emphasizes exact survey correctness rather than partially crediting high-recall retrieval pipelines (Hariri et al., 26 Aug 2025, Afane et al., 7 Feb 2026).
4. Empirical performance of RAG, STARA, and commercial systems
The initial LaborBench study found that retrieval materially improves performance relative to no-retrieval baselines. Averaged across five generators and five retrievers, the no-retrieval baseline achieved accuracy .536 ± .012, precision .452 ± .014, recall .515 ± .018, and .481 ± .013. Under RAG, the best retrieval-plus-LLM configurations achieved accuracy .656, precision .567, recall .806, and .665, with the best reported configuration—Gemini retriever plus Deepseek V3—reaching approximately . State-by-state, baseline ranged from .316 in Louisiana to .590 in Mississippi, while RAG ranged from .520 in Virginia to .810 in Arkansas; the reported delta ranged from +.05 in California to +.30 in Idaho (Hariri et al., 26 Aug 2025).
The later paper benchmarks four systems on LaborBench’s 1,647 Boolean questions and reports a sharper comparative picture:
| System | Accuracy | Correct count |
|---|---|---|
| Standard RAG | 66% | |
| STARA | 83% | |
| Westlaw AI | 58% | 0 |
| Lexis+ AI | 64% | 1 |
These results place STARA substantially above the earlier standard RAG baseline, while both commercial platforms underperform not only STARA but also standard RAG on this benchmark. The papers therefore do not support the view that marketed AI statutory survey capabilities reliably solve multi-jurisdictional legal research in this domain (Afane et al., 7 Feb 2026).
The earlier paper also measures citation behavior. In a sample of 30 instances, retriever recall@5 was .93, citation accuracy was .73, and conditional citation accuracy given a correct section in the top-5 was .77. This matters because LaborBench is not merely a label-prediction task: the benchmark is designed around legal propositions that require traceable statutory support, and citation failures can persist even when answer labels are correct (Hariri et al., 26 Aug 2025).
5. Retrieval failures, reasoning errors, and revision of the “ground truth”
The error analysis in the STARA paper divides failures into two categories: retrieval failures and reasoning errors. Retrieval failures occur when the relevant provision is not surfaced. A reported STARA example is Utah part-time work search eligibility, which appears only in administrative regulations and not in the statutory text captured by STARA’s RegEx filters, producing a false negative. A reported Lexis+ AI example is Alabama’s self-employment assistance statute, which the system failed to retrieve despite standard “self-employment assistance” terminology (Afane et al., 7 Feb 2026).
Reasoning errors occur when a provision is retrieved but then misinterpreted or misclassified. The paper’s STARA example is Michigan: Mich. Comp. Laws § 421.11 authorizes information sharing with the U.S. Department of Agriculture but does not authorize deducting SNAP overissuances, yet STARA labeled the item True. Westlaw AI’s Alaska example cites AS § 23.20.401, concerning child support, and AS § 23.20.085, concerning interstate UI reciprocity, as though they authorized SNAP offsets. Lexis+ AI is reported to have confused eligibility thresholds requiring wages in two quarters with actual multi-quarter benefit calculation formulas (Afane et al., 7 Feb 2026).
The most consequential finding is that many apparent model errors were actually omissions in the DOL enumeration. Of STARA’s 181 apparent false positives, manual review found that 135, or 75%, were legitimate state provisions missing from the DOL ground truth. Using the paper’s notation, with initial STARA correct count 2, omitted-but-valid items 3, and total 4, the corrected score is
5
that is, 6 (Afane et al., 7 Feb 2026).
This revision has methodological significance beyond the single system comparison. The omitted provisions reportedly included overpayment waiver grounds, self-employment assistance, alternative base periods, part-time search exceptions, and retirement exclusions. The papers therefore complicate a common benchmarking assumption: expert-produced legal reference tables are highly valuable, but they are not guaranteed to be exhaustive. In LaborBench, transparent statutory citations enabled a system not only to answer questions but also to expose defects in the benchmark labels themselves (Afane et al., 7 Feb 2026).
6. Design principles and position within the benchmark landscape
The later LaborBench paper distills five design principles for multi-jurisdictional legal RAG. Precise question specification and context requires prompts to define temporal scope, treatment of historical or expired provisions, and handling of exceptions; the paper gives “in effect as of January 1, 2025” as the relevant type of temporal qualifier. Domain expertise in statutory interpretation is needed to distinguish substantive doctrinal differences from harmless drafting variation. Strategic corpus selection and retrieval scope requires either restricting retrieval to the relevant UI code titles or explicitly including regulations and interpretations if the benchmark does so. Transparent retrieval and citation practices require exact statutory sections rather than opaque prose justifications. Recognition of non-statutory authorities requires benchmarks and systems to document whether they include regulations, agency policies, or statutes only (Afane et al., 7 Feb 2026).
These principles align closely with the limitations already surfaced in the earlier RAG study. That paper recommends always integrating retrieval, using approximately 1,000-token chunks with overlap, combining dense and sparse retrieval, adopting JSON schemas for structured outputs, eliciting citations, validating citations manually or through additional IR checks, considering hierarchical retrieval from sections to paragraphs to clauses, and keeping a human in the loop for high-stakes decisions (Hariri et al., 26 Aug 2025).
Within the broader benchmark ecosystem, LaborBench is distinct from UpBench, which evaluates agentic systems on real jobs drawn from the global Upwork marketplace; LABBench2, which measures AI systems performing biology research across nearly 1,900 tasks; and LabourLawBench, which covers Chinese labor-law tasks such as statute recitation, compensation computation, NER, and case analysis (Yi et al., 15 Nov 2025, Laurent et al., 4 Feb 2026, Lan et al., 15 Jan 2026). This situates LaborBench as a jurisdiction-specific legal-research benchmark focused on statutory survey fidelity, retrieval grounding, and cross-state doctrinal comparison rather than on general labor-market work, experimental biology, or Chinese labor adjudication.
LaborBench’s principal significance lies in making fifty-state statutory survey generation empirically measurable. The benchmark shows that RAG materially improves over no-retrieval baselines, that system architecture matters more than marketing claims, and that benchmark maintenance in law must account for omissions in even expert-produced reference materials. In that sense, LaborBench functions not only as an evaluation set for legal AI, but also as a stress test for the epistemic infrastructure of legal benchmarking itself (Hariri et al., 26 Aug 2025, Afane et al., 7 Feb 2026).