InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

Abstract: Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.

• The paper introduces InsufficiencyBench, a 202-item benchmark spanning six legal domains and 24 US jurisdictions to test whether models identify outcome-determinative missing elements before giving advice.
• Evaluations of 10 frontier models found no score above F2 = 0.46, with median recall of 0.44; GPT-5.2 led at F2 = 0.455 and 0.666 recall, while models often answered substantively without acknowledging gaps.
• The findings show that legal safety depends more on calibrated gap detection than on explanation quality, with procedural posture and party-status requirements especially overlooked and over-hedging creating false alarms.

Motivation: premature legal closure

Existing legal LLM benchmarks—LegalBench, CaseHOLD, LexGLUE, LawBench, and hallucination-focused evaluations such as LegalBench-RAG—share a strong assumption: that the query arrives fully specified. Real client intake does not. A user asking "Can my employer enforce this non-compete?" may omit the jurisdiction, which alone determines whether the agreement is void (California, Bus. & Prof. Code §16600), enforceable subject to reformation (Texas), or conditioned on compensation thresholds (Illinois). Vincent et al. name the resulting failure mode premature legal closure: producing a substantive legal answer before legally material inputs are known. The failure is distinct from hallucination—a model may state the law correctly for a silently assumed jurisdiction while giving advice that is wrong for the user's actual one. It is also distinct from abstention failure: the target behavior is not refusal but recognition of what is missing and targeted clarification.

The paper's central reframing is from answer correctness to answerability: whether the model recognizes when no final answer can yet be safely produced. The legal domain makes this measurable in a way general ambiguity benchmarks cannot, because materiality is fixed by statute and doctrine rather than annotator preference.

Benchmark design

InsufficiencyBench comprises 202 items—58 fully specified base queries and 144 deficient variants—spanning six US legal domains (tort, commercial, criminal, employment, real property, civil procedure) across 24 jurisdictions. Two attorneys with over 20 years of combined experience authored and annotated every item at the sentence level: each material element receives a sub-tag, source sentence, canonical category, required-flag, and an expert rationale for why it matters.

Deficient variants are constructed by removing sentences that supply required elements, yielding ground-truth missing-element sets ($M_v$) without requiring gold answers. Variants carry 1–6 missing elements (mean 1.9); the corpus contains 541 annotated material elements, 96% flagged as required.

Taxonomy of insufficiency

The benchmark formalizes three structural failure modes and eight canonical categories:

• Switch: the missing element selects between materially different frameworks (jurisdiction, controlling text, facts of harm).
• Gating: the missing element determines whether a framework attaches at all (parties and status, procedural posture, timing, user goal).
• Fatal prerequisite: the missing element is dispositive within an otherwise applicable framework (consideration).

This structure tracks practitioner intake logic—identify the framework, determine whether it attaches, check prerequisites. The authors concede the eight categories are not exhaustive, that timing and consideration have context-dependent roles, and that mixed modes occur (e.g., FMLA employer headcount functions as both gate and prerequisite).

Evaluation protocol

Three metrics are computed per item via an LLM judge (fixed to GPT-5):

1. Element-identification F2, weighting recall over precision because a silently presumed missing element is more dangerous than excessive caution.
2. Explanation accuracy (ExplAcc): whether identified gaps are accompanied by legally correct rationales, judged against withheld attorney explanations.
3. Safety rate: whether responses avoid fabricating conclusions dependent on missing elements, computed over the full ground-truth set since fabrication on unacknowledged gaps is still harmful.

Ten frontier models spanning six providers were evaluated under a deliberately minimal system prompt ("You are a legal assistant. Please answer the query."), mimicking real deployment.

Results

No model handles insufficiency well. No model exceeds F2 = 0.46; median recall is 0.44, meaning typical models miss more than half of legally material elements. GPT-5.2 leads (F2 = 0.455, recall = 0.666); DeepSeek-V4-Pro trails (F2 = 0.278, recall = 0.321).

Model	F2	Recall	Hedge	ExplAcc	Safety	Over-flag
GPT-5.2	0.455	0.666	0.868	0.763	0.918	0.724
Claude Opus 4.7	0.399	0.495	0.660	0.766	0.863	0.534
Mistral Large 3	0.383	0.442	0.556	0.627	0.756	0.276
DeepSeek-V4-Pro	0.279	0.321	0.361	0.694	0.698	0.224

Silence, not inaccuracy, drives the shortfall. Decomposing aggregate F2 into hedge rate and conditional F2 reveals that when models do hedge, identification quality is markedly higher (e.g., DeepSeek-V4-Pro reaches F2 = 0.524 given hedging). The primary failure is that models hedge rarely: DeepSeek-V4-Pro answers 63.9% of deficient queries with substantive advice while acknowledging no gap.

Hedging behavior is miscalibrated and largely query-independent. The same disposition produces both over-hedging and silence: GPT-5.2 flags spurious gaps on 72.4% of complete base queries while Claude Opus 4.7 does so on 53.4%. Conversely, the best-calibrated-looking models earn that appearance only by rarely flagging anything. Performance on this benchmark tracks a single behavioral axis rather than scale, provider, or reasoning regime.

Structural gaps are systematically missed. Procedural posture shows mean recall of only 0.09 (zero for three models; max 0.231), and parties-and-status elements average 0.258 recall. Controlling text (0.635) and facts of harm (0.437) are caught far more often because their absence leaves a visible trace in the narrative. Detecting procedural posture requires external legal knowledge—for instance, that Title VII relief requires a timely EEOC charge, or that FMLA coverage depends on workforce geography—that the query itself never signals.

Once a gap is flagged, reasoning quality is uniform. ExplAcc compresses into a narrow 0.63–0.77 band despite identification recall varying by more than a factor of two. The bottleneck is upstream of legal reasoning: the decision to flag, not the rationale that follows.

Robustness to judge choice. Re-evaluation with two alternative judges (Claude-Haiku-4.5, GLM-5) confirms no model exceeds F2 = 0.46 or recall = 0.67 under any judge; GPT-5 is in fact the most lenient, making the reported numbers charitable to the evaluated models. Per-item inter-judge correlation is moderate (r = 0.54–0.76), but disagreement shifts scores down, not up.

Limitations and open questions

The authors identify several constraints plainly. The dataset is small (202 items) and confined to six domains within contentious US common-law matters. Judge agreement was moderate and not validated against human scoring, though the judge role is extractive and headline findings are judge-invariant. Evaluation is single-turn; premature closure may also manifest across multi-turn clarification dialogues, which the benchmark does not test. Per-category results should be read directionally given uneven category support. An open question the paper leaves explicit is whether training interventions can shift models off the single hedging-disposition axis without inducing indiscriminate over-flagging—the calibration tension between GPT-5.2's high hedge rate and 72.4% over-flag rate suggests current models lack the discriminative signal needed for both.

Conclusion

InsufficiencyBench isolates an intake-stage capability absent from prior legal evaluation: recognizing that a query omits outcome-determinative facts, naming them, and declining to conclude under silent presumptions. Across ten frontier models, the capability is weak, structured around a fixed hedging disposition rather than genuine sufficiency assessment, and weakest precisely on structurally hidden prerequisites such as procedural posture. Safe legal assistants require mechanisms for deciding when clarification must precede reasoning—an optimization target this benchmark makes concrete and measurable.