Vietnamese Legal Benchmark (VLegal-Bench)

• Vietnamese Legal Benchmark (VLegal-Bench) is a cognitively grounded framework designed to assess Vietnamese legal reasoning through statutory interpretation, document hierarchy, and amendment tracking.
• It organizes 22 tasks into five cognitive levels—from basic recall to ethics and fairness—mirroring real-world legal-assistant workflows in a civil-law context.
• The benchmark compiles 10,450 expert-validated samples, offering detailed evaluation metrics and insights on both closed and open LLM performances.

Vietnamese Legal Benchmark (VLegal-Bench) is a cognitively grounded benchmark for evaluating LLMs on Vietnamese legal reasoning in a civil-law setting. It is introduced as the first comprehensive benchmark designed to assess Vietnamese legal tasks systematically, with an explicit focus on codified statutory reasoning, hierarchical legal structure, and legal validity under amendment and replacement. The benchmark comprises 10,450 expert-validated samples across 22 tasks, organized by a legal adaptation of Bloom’s taxonomy into five cognitive levels ranging from recall to ethics and fairness, and is designed to mirror practical Vietnamese legal-assistant workflows rather than isolated academic subtasks (Dong et al., 16 Dec 2025).

1. Jurisdictional motivation and benchmark scope

VLegal-Bench is motivated by a structural mismatch between prevailing legal NLP benchmarks and the characteristics of Vietnamese law. The benchmark paper frames Vietnamese law as statute-centered, hierarchically organized through units such as Article, Clause, and Point, and frequently revised through amendments, replacements, and repeal. These properties make evaluation based on common-law assumptions, especially case-centered reasoning and precedent interpretation, insufficient for Vietnamese legal AI (Dong et al., 16 Dec 2025).

The benchmark is explicitly aligned with the Vietnamese civil-law system. Its task design emphasizes codified statutory interpretation, legal hierarchy, document relationships, legal validity periods, and conflict checking between provisions. In that framing, VLegal-Bench is not merely a collection of Vietnamese legal datasets, but a standardized evaluation framework intended to test whether an LLM can act as a Vietnamese legal assistant across recognition, structuring, inference, generation, and ethical judgment (Dong et al., 16 Dec 2025).

The benchmark is also presented as potentially extensible beyond Vietnam. The paper states that its civil-law-oriented design could be adapted to other codified legal systems, including France, Germany, Japan, South Korea, and Thailand. This suggests that VLegal-Bench is intended not only as a local benchmark, but as a template for civil-law legal evaluation where statute hierarchy and amendment tracking are central.

2. Cognitive organization and task inventory

The defining structural feature of VLegal-Bench is its legal adaptation of Bloom’s taxonomy. The benchmark is divided into five cognitive levels: Level 1 – Recognition / Recall, Level 2 – Understanding / Structuring, Level 3 – Reasoning / Inference, Level 4 – Interpretation / Generation, and Level 5 – Ethics, Fairness / Bias (Dong et al., 16 Dec 2025).

At Level 1, the benchmark tests basic legal factual knowledge and schema awareness. The five tasks are Legal Entity Recognition, Legal Topic Classification, Legal Concept Recall, Article Recall, and Legal Schema Recall. These tasks target recognition of legal entities, topic routing, definitional recall, citation of the correct article, and recognition of hierarchical and temporal relations such as amendment, replacement, and repeal. The reported test counts for these tasks are 750, 700, 300, 1000, and 800 respectively, for a Level-1 total of 3,550 instances (Dong et al., 16 Dec 2025).

At Level 2, VLegal-Bench moves from recall to internal legal structure. The tasks are Relation Extraction, Legal Element Recognition, Legal Graph Structuring, Judgment Verification, and User Intent Understanding. These target extraction of subject-object-content relations, identification of hypothesis-disposition-sanction structure, conversion of legal text into triplets over Articles, Clauses, and Points, verification of whether judicial reasoning matches the facts and legal content of a judgment, and inference of the real intent behind a user’s legal query. Their test counts are 253, 300, 296, 600, and 1,359 respectively, totaling 2,808 Level-2 instances (Dong et al., 16 Dec 2025).

At Level 3, the benchmark concentrates on core legal inference. The five tasks are Article / Clause Prediction, Legal Court Decision Prediction, Multi-Article Reasoning, Conflict / Consistency Detection, and Penalty / Remedy Estimation. These measure whether a model can infer applicable provisions from underspecified queries, predict a court decision from case facts, reason across multiple provisions, detect contradiction or overlap between clauses or interpretations, and estimate sanctions or remedies. Their test counts are 600, 600, 292, 161, and 358, for 2,011 Level-3 instances (Dong et al., 16 Dec 2025).

At Level 4, the benchmark evaluates productive legal language generation. The three tasks are Legal Document Summarization, Judicial Reasoning Generation using IRAC, and Objective Legal Opinion Generation. Their test counts are 384, 299, and 498 respectively, totaling 1,181 instances (Dong et al., 16 Dec 2025).

At Level 5, VLegal-Bench incorporates explicit trustworthiness tests. The tasks are Bias Detection, Privacy / Data Protection, Ethical Consistency Assessment, and Unfair Contract Detection, with 250, 216, 200, and 234 test instances respectively, totaling 900 instances. The benchmark therefore treats fairness, privacy, and ethical alignment as first-class evaluation targets rather than auxiliary audits (Dong et al., 16 Dec 2025).

Across all five levels, VLegal-Bench uses four output formats: multi-label / multiple-choice classification, binary classification, extraction, and generation. This heterogeneity is central to its design: the benchmark is intended to evaluate both narrow label prediction and open-ended legal production in one framework.

3. Source collection, database construction, and annotation process

VLegal-Bench is grounded in a large legal source collection of 55,000 centrally issued and currently effective legal documents. The underlying material is gathered from official government and state-agency websites, published court judgments, legal news and contracts, and citizen question-answer pairs from private datasets provided by Vietnamese law firms. The benchmark paper states that the database records metadata such as promulgation date, effective date, and validity period, which is essential for tasks involving amendment, replacement, and conflict (Dong et al., 16 Dec 2025).

The preprocessing pipeline includes HTML parsing, OCR, text cleaning, deduplication, tokenization, topic classification, and citation extraction. From that pipeline, the authors build two databases. The Knowledge Graph Database stores legal schema and relations among documents, Articles, Clauses, Points, and cross-document references. The Legal Corpus Database is a key-value corpus whose keys are document identifiers and hierarchical legal units, and whose values include text, promulgation date, effective date, validity period, and real citizen legal questions (Dong et al., 16 Dec 2025).

Task creation is expert-supervised. A senior legal expert specifies task-specific topics, selects authoritative sources, and prepares raw material for annotation. Sample quotas are then divided between two junior legal experts, who independently create realistic legal scenarios and answers. The resulting questions are designed to reflect actual legal-assistant workflows and are derived from statutory texts, real judgments, citizen legal questions, and contractual or policy scenarios (Dong et al., 16 Dec 2025).

The annotation team consists of 3 senior legal experts and 8 junior legal experts, recruited through two Vietnamese law firms and one university law faculty. Senior experts are licensed Vietnamese lawyers with at least five years of practice, domain specialization, and prior teaching or training experience. Junior experts are licensed lawyers or final-year law graduates who passed the bar, have one to three years of practical experience, and are native Vietnamese speakers. All annotators underwent a two-day training program covering the benchmark’s cognitive framework, hands-on work with a pilot set of 50 samples per task, and calibration on edge cases and labeling conventions. Before entering the main annotation phase, annotators had to achieve at least 85% agreement with gold pilot labels (Dong et al., 16 Dec 2025).

The cross-validation mechanism is unusually explicit. Every batch of 100 samples is blind cross-verified: Junior A answers Junior B’s questions, Junior B answers Junior A’s questions, and neither sees the original answer. The benchmark reports Initial agreement: $92.39\%$, computed as $\frac{9656}{10450}$, with $\kappa = 0.89$. Of the 794 disagreement cases, 683 were resolved through structured discussion between the junior annotators and 111 were escalated to a supervising senior expert for final adjudication. The most disagreement-prone tasks were Task 3.4 Conflict / Consistency Detection with 31 cases, Task 3.3 Multi-Article Reasoning with 27, and Task 5.4 Unfair Contract Detection with 22, indicating that the benchmark’s most difficult tasks were also the most difficult for human annotators (Dong et al., 16 Dec 2025).

Operationally, the annotation campaign required about 1,400 person-hours over 14 weeks, with fatigue controls including maximum four-hour sessions, mandatory breaks, and weekly task rotation. The paper repeatedly emphasizes that every sample is grounded in authoritative legal sources.

4. Evaluation protocol, prompting settings, and model coverage

VLegal-Bench evaluates models under zero-shot and few-shot prompting. In few-shot mode, one task-specific demonstration is prepended from a separate development set, and the paper explicitly states that these demonstrations are not included in the reported test set. For both zero-shot and few-shot evaluation, the benchmark supports direct answer prompting and Chain-of-Thought (CoT) prompting. It also defines a ReAct-style agent setting in which the model can use search tools over the Legal Corpus Database and the Knowledge Graph Database, although the main quantitative tables discussed in the paper focus on the standard model-comparison setting (Dong et al., 16 Dec 2025).

To improve reproducibility, the benchmark uses temperature = 0. If an input exceeds a model’s context limit, the benchmark applies middle truncation, motivated by the observation that the beginning and the end of legal texts often both contain important information (Dong et al., 16 Dec 2025).

The paper states that VLegal-Bench is evaluated on 24 LLMs, although the main zero-shot result tables enumerate 23 models. The model pool includes closed multilingual systems such as GPT-4o, GPT-4o-mini, Claude 4.5 Haiku / Sonnet, and Gemini 2.5 Flash / Pro; open multilingual models including Qwen 2.5 / Qwen 2.5 Instruct at multiple scales, Llama 2 Chat, Llama 3.1 Instruct, InternLM 3 8B Instruct, and Gemma 2 Instruct; and Vietnamese-focused or legal-adapted models such as SeaLLMs v3 Chat, BloomVN 8B Chat, VLSP2025-LegalSML 4B, and CMC-AI-Legal-32B (Dong et al., 16 Dec 2025).

Metrics are task-specific. The benchmark uses Accuracy for most recognition and reasoning tasks, macro-F1 for User Intent Understanding, Binary F1 for Conflict / Consistency Detection, and ROUGE-L for Legal Graph Structuring and the generation tasks. For Task 3.4, the paper reports binary F1 as separate $\mathrm{F1}_{\text{Yes}}$ and $\mathrm{F1}_{\text{No}}$ values. The paper does not provide a benchmark-wide aggregate score formula, so performance is reported task by task rather than via a single composite index (Dong et al., 16 Dec 2025).

5. Empirical profile and benchmark difficulty

The central empirical finding of VLegal-Bench is that performance degrades sharply with cognitive complexity. The benchmark paper contrasts Task 1.4 Article Recall, whose best score is 87.91%, with Task 3.1 Article / Clause Prediction, whose best score is 43.83%. This gap is used to show that recalling a provision when directly prompted is much easier than inferring which provision applies to a new legal query (Dong et al., 16 Dec 2025).

Even some ostensibly lower-level tasks are difficult. Task 1.5 Legal Schema Recall, which concerns amendment chains, legal basis relations, and document hierarchy, remains below 28% for all models. This is one of the paper’s strongest demonstrations that Vietnamese legal reasoning depends on structural statutory competence rather than surface article recall alone (Dong et al., 16 Dec 2025).

The benchmark’s hardest task is Task 3.4 Conflict / Consistency Detection. The paper describes it as a “catastrophe”: of the 23 models shown in the main results tables, 16 achieve 0.00 Y-F1, meaning they fail entirely on positive conflict cases while still obtaining moderate No-class F1 around 39–46. Only a small subset show meaningful positive-case capability, including CMC-AI-Legal-32B with 86.41 Y-F1, Llama-3.1-8B-Instruct with 37.66 Y-F1, and GPT-4o with 27.21 Y-F1. The benchmark thus identifies a strong default-to-no-conflict bias in many general-purpose models (Dong et al., 16 Dec 2025).

The results also support a broad claim that legal specialization matters more than raw model scale on several difficult tasks. The paper reports that qwen3-4b-legal-pretrain achieves the best performance on Task 3.1 Article / Clause Prediction at 43.83% and on Task 4.1 Summarization at 0.4361 ROUGE-L, despite being much smaller than frontier general models. CMC-AI-Legal-32B is best on Task 3.2 Court Decision Prediction at 90.67%, Task 3.3 Multi-Article Reasoning at 76.71%, Task 3.4 Conflict Detection at 86.41 Y-F1, and Task 5.4 Unfair Contract Detection at 73.50% (Dong et al., 16 Dec 2025).

Closed proprietary models retain strengths on some tasks. GPT-4o leads Task 3.5 Penalty / Remedy Estimation at 67.97% and Task 4.3 Objective Legal Opinion Generation at 0.4975 ROUGE-L. Claude Sonnet is strongest on Task 1.3 Legal Concept Recall at 83.00%, Task 2.2 Legal Element Recognition at 75.33%, Task 2.3 Legal Graph Structuring at 0.808 ROUGE-L, and Task 2.4 Judgment Verification at 87.81%. The pattern is therefore not one of simple open-versus-closed dominance, but of task-specific specialization (Dong et al., 16 Dec 2025).

The benchmark also reveals that some reasoning tasks are comparatively tractable. Task 3.2 Court Decision Prediction reaches 82–90% for top models, whereas Task 3.1 Article / Clause Prediction and Task 3.4 Conflict / Consistency Detection remain much weaker. On the ethical side, performance is uneven: Task 5.3 Ethical Consistency Assessment is relatively easy for many models, often above 85%, while Task 5.1 Bias Detection is substantially harder, with results roughly in the 15–58% range (Dong et al., 16 Dec 2025).

Generation tasks are not solved. Stronger models achieve roughly 0.30–0.50 ROUGE-L on Level-4 tasks, with Legal Opinion Generation easier than Summarization and IRAC-style Judicial Reasoning Generation. The paper treats this as evidence that productive legal writing remains a mid-level challenge rather than a solved subproblem.

6. Place within the Vietnamese legal benchmark ecosystem

VLegal-Bench occupies a unifying position relative to earlier Vietnamese legal resources, which largely addressed narrower tasks. Earlier Vietnamese legal QA and retrieval work centered on article retrieval or answer extraction under low-resource conditions. ALQAC 2023, as described by the NeCo system paper, focused on legal article retrieval and mixed-format question answering with only 100 training and 100 public-test samples in the competition release, and emphasized hybrid lexical-semantic retrieval and data enrichment rather than broad legal reasoning evaluation (Nguyen et al., 2023). The civil-law retrieval QA system in “Improving Vietnamese Legal Question–Answering System based on Automatic Data Enrichment” treated Vietnamese legal QA primarily as article-level retrieval over 117,557 civil-law articles, with weak labeling from article titles used to improve a BM25-plus-BERT pipeline (Vuong et al., 2023). More recently, VLQA introduced a large statutory retrieval-and-QA benchmark with 3,129 expert-annotated triplets, 59,636 legal articles, and 27 legal domains, but its focus remained on statutory retrieval and long-form answer generation rather than the broader cognitive spectrum covered by VLegal-Bench (Nguyen et al., 26 Jul 2025).

Other Vietnamese legal resources isolated individual reasoning dimensions. ViLegalNLI introduced a sentence-pair legal inference benchmark of 42,012 premise-hypothesis pairs derived from 168 active statutory documents across 27 legal sub-domains, focusing on binary entailment versus non-entailment (Duong et al., 30 Apr 2026). VLSP 2023 LTER framed statute-grounded legal textual entailment as a shared task with 76 training and 139 test examples, explicitly stressing negation and logical consistency rather than large-scale coverage (Tran et al., 2024). These resources are directly comparable to Level-3 reasoning tasks in VLegal-Bench, but VLegal-Bench generalizes beyond NLI into retrieval, generation, and ethics.

Specialized domain benchmarks further illustrate the fragmentation that VLegal-Bench attempts to overcome. ViHERMES is a domain-specific benchmark for multihop QA over Vietnamese healthcare regulations, with a 1,560-sample test split evenly distributed across hop counts 1–5 and a graph-aware retrieval system for amendment tracing and cross-document procedural synthesis (Nguyen et al., 7 Feb 2026). VLSP 2025 MLQA-TSR extends Vietnamese legal evaluation into the multimodal setting through traffic-sign regulation, using article-level retrieval and image-grounded legal QA over a legal database of 402 articles from two official legal documents (Luu et al., 23 Oct 2025). These benchmarks are narrower in domain but deeper in specialized reasoning modalities.

Methodologically, VLegal-Bench also sits alongside infrastructure papers on legal retrieval and evaluation. Synthetic-data-driven legal retrieval with passage-level supervision was explored in “Improving Vietnamese Legal Document Retrieval using Synthetic Data,” which released the TVPL benchmark and a 507,152-pair synthetic legal query dataset for retrieval pretraining (Tien et al., 2024). Two-stage legal retrieval with semi-hard negative mining over a 261,446-segment corpus was studied in “Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining,” which introduced the Exist@m metric and showed strong gains from candidate-pool-aligned negatives (Le et al., 19 Jul 2025). VLegal-Bench does not replace these task-specific contributions; rather, it provides a common umbrella under which such retrieval, reasoning, and generation capabilities can be evaluated together.

7. Limitations, interpretive cautions, and broader significance

The benchmark paper makes clear that VLegal-Bench is difficult, unevenly so, and not reducible to a single leaderboard number. Some tasks are much more ambiguous than others, and the tasks with the highest annotator disagreement are also among the hardest for models. The benchmark therefore exposes not only model weakness, but also genuine legal interpretive complexity (Dong et al., 16 Dec 2025).

There are also reporting limits. The paper provides exact test counts per task and notes the use of a separate development set for few-shot demonstrations, but full numerical train/dev/test partition details are not fully disclosed in the excerpted benchmark description. In addition, the benchmark uses ROUGE-L for generation tasks even though the paper acknowledges, in its discussion of related work, that overlap-based metrics correlate imperfectly with legal factuality. This suggests that Level-4 generation results should be interpreted as partial indicators rather than definitive measures of legal correctness.

At a broader level, VLegal-Bench formalizes a specific view of legal AI evaluation in Vietnam. It treats legal competence as layered: a model must recall provisions, understand document structure, infer applicable law, generate legally usable text, and remain fair and privacy-aware under deployment constraints. The inclusion of Level-5 tasks is itself a methodological statement that legal AI cannot be evaluated solely on factual correctness or retrieval success (Dong et al., 16 Dec 2025).

The benchmark’s larger significance lies in this integration. Earlier Vietnamese legal resources demonstrated that retrieval, entailment, multihop regulatory reasoning, multimodal legal QA, and long-form statutory answering are all independently challenging (Nguyen et al., 26 Jul 2025, Duong et al., 30 Apr 2026, Nguyen et al., 7 Feb 2026, Luu et al., 23 Oct 2025). VLegal-Bench brings these concerns into a single cognitively structured framework and shows that current models remain far from robust Vietnamese legal reasoning, especially on legal schema understanding, sparse-query article prediction, and statutory conflict detection. A plausible implication is that future progress in Vietnamese legal AI will depend less on raw model scale than on legal-domain adaptation, stronger grounding in authoritative sources, and evaluation protocols that make structural legal reasoning visible rather than treating it as a hidden byproduct of answer fluency.