---
title: Vietnamese Legal Benchmark (VLegal-Bench)
url: https://www.emergentmind.com/topics/vietnamese-legal-benchmark-vlegal-bench
type: topic
---

# Vietnamese Legal Benchmark (VLegal-Bench)

Vietnamese Legal Benchmark (VLegal-Bench) is a cognitively grounded benchmark for evaluating large language models on Vietnamese legal reasoning in a civil-law setting. It is introduced as the first comprehensive benchmark designed to assess Vietnamese legal tasks systematically, with an explicit focus on codified statutory reasoning, hierarchical legal structure, and legal validity under amendment and replacement. The benchmark comprises 10,450 expert-validated samples across 22 tasks, organized by a legal adaptation of Bloom’s taxonomy into five cognitive levels ranging from recall to ethics and fairness, and is designed to mirror practical Vietnamese legal-assistant workflows rather than isolated academic subtasks [2512.14554].

## 1. Jurisdictional motivation and benchmark scope

VLegal-Bench is motivated by a structural mismatch between prevailing legal NLP benchmarks and the characteristics of Vietnamese law. The benchmark paper frames Vietnamese law as statute-centered, hierarchically organized through units such as Article, Clause, and Point, and frequently revised through amendments, replacements, and repeal. These properties make evaluation based on common-law assumptions, especially case-centered reasoning and precedent interpretation, insufficient for Vietnamese legal AI [2512.14554].

The benchmark is explicitly aligned with the Vietnamese civil-law system. Its task design emphasizes codified statutory interpretation, legal hierarchy, document relationships, legal validity periods, and conflict checking between provisions. In that framing, VLegal-Bench is not merely a collection of Vietnamese legal datasets, but a standardized evaluation framework intended to test whether an LLM can act as a Vietnamese legal assistant across recognition, structuring, inference, generation, and ethical judgment [2512.14554].

The benchmark is also presented as potentially extensible beyond Vietnam. The paper states that its civil-law-oriented design could be adapted to other codified legal systems, including France, Germany, Japan, South Korea, and Thailand. This suggests that VLegal-Bench is intended not only as a local benchmark, but as a template for civil-law legal evaluation where statute hierarchy and amendment tracking are central.

## 2. Cognitive organization and task inventory

The defining structural feature of VLegal-Bench is its legal adaptation of Bloom’s taxonomy. The benchmark is divided into five cognitive levels: **Level 1 – Recognition / Recall**, **Level 2 – Understanding / Structuring**, **Level 3 – Reasoning / Inference**, **Level 4 – Interpretation / Generation**, and **Level 5 – Ethics, Fairness / Bias** [2512.14554].

At **Level 1**, the benchmark tests basic legal factual knowledge and schema awareness. The five tasks are **Legal Entity Recognition**, **Legal Topic Classification**, **Legal Concept Recall**, **Article Recall**, and **Legal Schema Recall**. These tasks target recognition of legal entities, topic routing, definitional recall, citation of the correct article, and recognition of hierarchical and temporal relations such as amendment, replacement, and repeal. The reported test counts for these tasks are 750, 700, 300, 1000, and 800 respectively, for a Level-1 total of 3,550 instances [2512.14554].

At **Level 2**, VLegal-Bench moves from recall to internal legal structure. The tasks are **Relation Extraction**, **Legal Element Recognition**, **Legal Graph Structuring**, **Judgment Verification**, and **User Intent Understanding**. These target extraction of subject-object-content relations, identification of hypothesis-disposition-sanction structure, conversion of legal text into triplets over Articles, Clauses, and Points, verification of whether judicial reasoning matches the facts and legal content of a judgment, and inference of the real intent behind a user’s legal query. Their test counts are 253, 300, 296, 600, and 1,359 respectively, totaling 2,808 Level-2 instances [2512.14554].

At **Level 3**, the benchmark concentrates on core legal inference. The five tasks are **Article / Clause Prediction**, **Legal Court Decision Prediction**, **Multi-Article Reasoning**, **Conflict / Consistency Detection**, and **Penalty / Remedy Estimation**. These measure whether a model can infer applicable provisions from underspecified queries, predict a court decision from case facts, reason across multiple provisions, detect contradiction or overlap between clauses or interpretations, and estimate sanctions or remedies. Their test counts are 600, 600, 292, 161, and 358, for 2,011 Level-3 instances [2512.14554].

At **Level 4**, the benchmark evaluates productive legal language generation. The three tasks are **Legal Document Summarization**, **Judicial Reasoning Generation** using **IRAC**, and **Objective Legal Opinion Generation**. Their test counts are 384, 299, and 498 respectively, totaling 1,181 instances [2512.14554].

At **Level 5**, VLegal-Bench incorporates explicit trustworthiness tests. The tasks are **Bias Detection**, **Privacy / Data Protection**, **Ethical Consistency Assessment**, and **Unfair Contract Detection**, with 250, 216, 200, and 234 test instances respectively, totaling 900 instances. The benchmark therefore treats fairness, privacy, and ethical alignment as first-class evaluation targets rather than auxiliary audits [2512.14554].

Across all five levels, VLegal-Bench uses four output formats: **multi-label / multiple-choice classification**, **binary classification**, **extraction**, and **generation**. This heterogeneity is central to its design: the benchmark is intended to evaluate both narrow label prediction and open-ended legal production in one framework.

## 3. Source collection, database construction, and annotation process

VLegal-Bench is grounded in a large legal source collection of **55,000 centrally issued and currently effective legal documents**. The underlying material is gathered from official government and state-agency websites, published court judgments, legal news and contracts, and citizen question-answer pairs from private datasets provided by Vietnamese law firms. The benchmark paper states that the database records metadata such as promulgation date, effective date, and validity period, which is essential for tasks involving amendment, replacement, and conflict [2512.14554].

The preprocessing pipeline includes **HTML parsing**, **OCR**, **text cleaning**, **deduplication**, **tokenization**, **topic classification**, and **citation extraction**. From that pipeline, the authors build two databases. The **Knowledge Graph Database** stores legal schema and relations among documents, Articles, Clauses, Points, and cross-document references. The **Legal Corpus Database** is a key-value corpus whose keys are document identifiers and hierarchical legal units, and whose values include text, promulgation date, effective date, validity period, and real citizen legal questions [2512.14554].

Task creation is expert-supervised. A senior legal expert specifies task-specific topics, selects authoritative sources, and prepares raw material for annotation. Sample quotas are then divided between two junior legal experts, who independently create realistic legal scenarios and answers. The resulting questions are designed to reflect actual legal-assistant workflows and are derived from statutory texts, real judgments, citizen legal questions, and contractual or policy scenarios [2512.14554].

The annotation team consists of **3 senior legal experts** and **8 junior legal experts**, recruited through two Vietnamese law firms and one university law faculty. Senior experts are licensed Vietnamese lawyers with at least five years of practice, domain specialization, and prior teaching or training experience. Junior experts are licensed lawyers or final-year law graduates who passed the bar, have one to three years of practical experience, and are native Vietnamese speakers. All annotators underwent a **two-day training program** covering the benchmark’s cognitive framework, hands-on work with a pilot set of **50 samples per task**, and calibration on edge cases and labeling conventions. Before entering the main annotation phase, annotators had to achieve **at least 85% agreement** with gold pilot labels [2512.14554].

The cross-validation mechanism is unusually explicit. Every batch of **100 samples** is blind cross-verified: Junior A answers Junior B’s questions, Junior B answers Junior A’s questions, and neither sees the original answer. The benchmark reports **Initial agreement: \(92.39\%\)**, computed as \(\frac{9656}{10450}\), with **\(\kappa = 0.89\)**. Of the **794** disagreement cases, **683** were resolved through structured discussion between the junior annotators and **111** were escalated to a supervising senior expert for final adjudication. The most disagreement-prone tasks were **Task 3.4 Conflict / Consistency Detection** with 31 cases, **Task 3.3 Multi-Article Reasoning** with 27, and **Task 5.4 Unfair Contract Detection** with 22, indicating that the benchmark’s most difficult tasks were also the most difficult for human annotators [2512.14554].

Operationally, the annotation campaign required about **1,400 person-hours** over **14 weeks**, with fatigue controls including maximum four-hour sessions, mandatory breaks, and weekly task rotation. The paper repeatedly emphasizes that every sample is grounded in authoritative legal sources.

## 4. Evaluation protocol, prompting settings, and model coverage

VLegal-Bench evaluates models under **zero-shot** and **few-shot** prompting. In few-shot mode, one task-specific demonstration is prepended from a separate development set, and the paper explicitly states that these demonstrations are not included in the reported test set. For both zero-shot and few-shot evaluation, the benchmark supports **direct answer prompting** and **Chain-of-Thought (CoT)** prompting. It also defines a **ReAct-style agent setting** in which the model can use search tools over the Legal Corpus Database and the Knowledge Graph Database, although the main quantitative tables discussed in the paper focus on the standard model-comparison setting [2512.14554].

To improve reproducibility, the benchmark uses **temperature = 0**. If an input exceeds a model’s context limit, the benchmark applies **middle truncation**, motivated by the observation that the beginning and the end of legal texts often both contain important information [2512.14554].

The paper states that VLegal-Bench is evaluated on **24 LLMs**, although the main zero-shot result tables enumerate **23 models**. The model pool includes closed multilingual systems such as **GPT-4o**, **GPT-4o-mini**, **Claude 4.5 Haiku / Sonnet**, and **Gemini 2.5 Flash / Pro**; open multilingual models including **Qwen 2.5 / Qwen 2.5 Instruct** at multiple scales, **Llama 2 Chat**, **Llama 3.1 Instruct**, **InternLM 3 8B Instruct**, and **Gemma 2 Instruct**; and Vietnamese-focused or legal-adapted models such as **SeaLLMs v3 Chat**, **BloomVN 8B Chat**, **VLSP2025-LegalSML 4B**, and **CMC-AI-Legal-32B** [2512.14554].

Metrics are task-specific. The benchmark uses **Accuracy** for most recognition and reasoning tasks, **macro-F1** for **User Intent Understanding**, **Binary F1** for **Conflict / Consistency Detection**, and **ROUGE-L** for **Legal Graph Structuring** and the generation tasks. For Task 3.4, the paper reports binary F1 as separate **\(\mathrm{F1}_{\text{Yes}}\)** and **\(\mathrm{F1}_{\text{No}}\)** values. The paper does not provide a benchmark-wide aggregate score formula, so performance is reported task by task rather than via a single composite index [2512.14554].

## 5. Empirical profile and benchmark difficulty

The central empirical finding of VLegal-Bench is that performance degrades sharply with cognitive complexity. The benchmark paper contrasts **Task 1.4 Article Recall**, whose best score is **87.91%**, with **Task 3.1 Article / Clause Prediction**, whose best score is **43.83%**. This gap is used to show that recalling a provision when directly prompted is much easier than inferring which provision applies to a new legal query [2512.14554].

Even some ostensibly lower-level tasks are difficult. **Task 1.5 Legal Schema Recall**, which concerns amendment chains, legal basis relations, and document hierarchy, remains below **28%** for all models. This is one of the paper’s strongest demonstrations that Vietnamese legal reasoning depends on structural statutory competence rather than surface article recall alone [2512.14554].

The benchmark’s hardest task is **Task 3.4 Conflict / Consistency Detection**. The paper describes it as a “catastrophe”: of the 23 models shown in the main results tables, **16 achieve 0.00 Y-F1**, meaning they fail entirely on positive conflict cases while still obtaining moderate No-class F1 around 39–46. Only a small subset show meaningful positive-case capability, including **CMC-AI-Legal-32B** with **86.41 Y-F1**, **Llama-3.1-8B-Instruct** with **37.66 Y-F1**, and **GPT-4o** with **27.21 Y-F1**. The benchmark thus identifies a strong default-to-no-conflict bias in many general-purpose models [2512.14554].

The results also support a broad claim that legal specialization matters more than raw model scale on several difficult tasks. The paper reports that **qwen3-4b-legal-pretrain** achieves the best performance on **Task 3.1 Article / Clause Prediction** at **43.83%** and on **Task 4.1 Summarization** at **0.4361 ROUGE-L**, despite being much smaller than frontier general models. **CMC-AI-Legal-32B** is best on **Task 3.2 Court Decision Prediction** at **90.67%**, **Task 3.3 Multi-Article Reasoning** at **76.71%**, **Task 3.4 Conflict Detection** at **86.41 Y-F1**, and **Task 5.4 Unfair Contract Detection** at **73.50%** [2512.14554].

Closed proprietary models retain strengths on some tasks. **GPT-4o** leads **Task 3.5 Penalty / Remedy Estimation** at **67.97%** and **Task 4.3 Objective Legal Opinion Generation** at **0.4975 ROUGE-L**. **Claude Sonnet** is strongest on **Task 1.3 Legal Concept Recall** at **83.00%**, **Task 2.2 Legal Element Recognition** at **75.33%**, **Task 2.3 Legal Graph Structuring** at **0.808 ROUGE-L**, and **Task 2.4 Judgment Verification** at **87.81%**. The pattern is therefore not one of simple open-versus-closed dominance, but of task-specific specialization [2512.14554].

The benchmark also reveals that some reasoning tasks are comparatively tractable. **Task 3.2 Court Decision Prediction** reaches **82–90%** for top models, whereas **Task 3.1 Article / Clause Prediction** and **Task 3.4 Conflict / Consistency Detection** remain much weaker. On the ethical side, performance is uneven: **Task 5.3 Ethical Consistency Assessment** is relatively easy for many models, often above **85%**, while **Task 5.1 Bias Detection** is substantially harder, with results roughly in the **15–58%** range [2512.14554].

Generation tasks are not solved. Stronger models achieve roughly **0.30–0.50 ROUGE-L** on Level-4 tasks, with **Legal Opinion Generation** easier than **Summarization** and **IRAC-style Judicial Reasoning Generation**. The paper treats this as evidence that productive legal writing remains a mid-level challenge rather than a solved subproblem.

## 6. Place within the Vietnamese legal benchmark ecosystem

VLegal-Bench occupies a unifying position relative to earlier Vietnamese legal resources, which largely addressed narrower tasks. Earlier Vietnamese legal QA and retrieval work centered on article retrieval or answer extraction under low-resource conditions. **ALQAC 2023**, as described by the NeCo system paper, focused on legal article retrieval and mixed-format question answering with only **100** training and **100** public-test samples in the competition release, and emphasized hybrid lexical-semantic retrieval and data enrichment rather than broad legal reasoning evaluation [2309.05500]. The civil-law retrieval QA system in “Improving Vietnamese Legal Question–Answering System based on Automatic Data Enrichment” treated Vietnamese legal QA primarily as article-level retrieval over **117,557** civil-law articles, with weak labeling from article titles used to improve a BM25-plus-BERT pipeline [2306.04841]. More recently, **VLQA** introduced a large statutory retrieval-and-QA benchmark with **3,129** expert-annotated triplets, **59,636** legal articles, and **27** legal domains, but its focus remained on statutory retrieval and long-form answer generation rather than the broader cognitive spectrum covered by VLegal-Bench [2507.19995].

Other Vietnamese legal resources isolated individual reasoning dimensions. **ViLegalNLI** introduced a sentence-pair legal inference benchmark of **42,012** premise-hypothesis pairs derived from **168** active statutory documents across **27** legal sub-domains, focusing on binary entailment versus non-entailment [2605.00116]. **VLSP 2023 LTER** framed statute-grounded legal textual entailment as a shared task with **76** training and **139** test examples, explicitly stressing negation and logical consistency rather than large-scale coverage [2403.03435]. These resources are directly comparable to Level-3 reasoning tasks in VLegal-Bench, but VLegal-Bench generalizes beyond NLI into retrieval, generation, and ethics.

Specialized domain benchmarks further illustrate the fragmentation that VLegal-Bench attempts to overcome. **ViHERMES** is a domain-specific benchmark for multihop QA over Vietnamese healthcare regulations, with a **1,560**-sample test split evenly distributed across hop counts 1–5 and a graph-aware retrieval system for amendment tracing and cross-document procedural synthesis [2602.07361]. **VLSP 2025 MLQA-TSR** extends Vietnamese legal evaluation into the multimodal setting through traffic-sign regulation, using article-level retrieval and image-grounded legal QA over a legal database of **402** articles from two official legal documents [2510.20381]. These benchmarks are narrower in domain but deeper in specialized reasoning modalities.

Methodologically, VLegal-Bench also sits alongside infrastructure papers on legal retrieval and evaluation. Synthetic-data-driven legal retrieval with passage-level supervision was explored in “Improving Vietnamese Legal Document Retrieval using Synthetic Data,” which released the **TVPL** benchmark and a **507,152**-pair synthetic legal query dataset for retrieval pretraining [2412.00657]. Two-stage legal retrieval with semi-hard negative mining over a **261,446**-segment corpus was studied in “Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining,” which introduced the **Exist@m** metric and showed strong gains from candidate-pool-aligned negatives [2507.14619]. VLegal-Bench does not replace these task-specific contributions; rather, it provides a common umbrella under which such retrieval, reasoning, and generation capabilities can be evaluated together.

## 7. Limitations, interpretive cautions, and broader significance

The benchmark paper makes clear that VLegal-Bench is difficult, unevenly so, and not reducible to a single leaderboard number. Some tasks are much more ambiguous than others, and the tasks with the highest annotator disagreement are also among the hardest for models. The benchmark therefore exposes not only model weakness, but also genuine legal interpretive complexity [2512.14554].

There are also reporting limits. The paper provides exact test counts per task and notes the use of a separate development set for few-shot demonstrations, but full numerical train/dev/test partition details are not fully disclosed in the excerpted benchmark description. In addition, the benchmark uses **ROUGE-L** for generation tasks even though the paper acknowledges, in its discussion of related work, that overlap-based metrics correlate imperfectly with legal factuality. This suggests that Level-4 generation results should be interpreted as partial indicators rather than definitive measures of legal correctness.

At a broader level, VLegal-Bench formalizes a specific view of legal AI evaluation in Vietnam. It treats legal competence as layered: a model must recall provisions, understand document structure, infer applicable law, generate legally usable text, and remain fair and privacy-aware under deployment constraints. The inclusion of Level-5 tasks is itself a methodological statement that legal AI cannot be evaluated solely on factual correctness or retrieval success [2512.14554].

The benchmark’s larger significance lies in this integration. Earlier Vietnamese legal resources demonstrated that retrieval, entailment, multihop regulatory reasoning, multimodal legal QA, and long-form statutory answering are all independently challenging [2507.19995], [2605.00116], [2602.07361], [2510.20381]. VLegal-Bench brings these concerns into a single cognitively structured framework and shows that current models remain far from robust Vietnamese legal reasoning, especially on legal schema understanding, sparse-query article prediction, and statutory conflict detection. A plausible implication is that future progress in Vietnamese legal AI will depend less on raw model scale than on legal-domain adaptation, stronger grounding in authoritative sources, and evaluation protocols that make structural legal reasoning visible rather than treating it as a hidden byproduct of answer fluency.

Source: https://www.emergentmind.com/topics/vietnamese-legal-benchmark-vlegal-bench