CS-TaxoBench: A Multi-Domain LLM Benchmark

• CS-TaxoBench is a benchmark framework offering large-scale, curated datasets and standardized schemas for evaluating both scholarly taxonomy generation and Chinese tax practice using LLMs.
• It employs a detailed methodology including PDF conversion, hierarchical tree extraction, and unified JSON output formats, assessed via metrics such as BLEU, BERTScore, and EM accuracy.
• The framework supports both academic NLP research and compliance-centric applications, rigorously testing structural and semantic capabilities while highlighting LLM limitations in quantitative tasks.

CS-TaxoBench is a multi-faceted benchmark framework referenced in several domains, notably scholarly taxonomy generation and real-world tax practice evaluation with LLMs. It provides large-scale, systematically curated datasets, standardized input/output schemas, and rigorous evaluation protocols for assessing both structural and semantic capabilities of automated systems in high-precision tasks. The benchmark enforces comprehensive, fine-grained evaluation using multiple automatic and expert-driven metrics, and has influenced methodological standards for both academic NLP research and compliance-critical industrial applications.

1. Origin and Purpose

CS-TaxoBench was originally introduced as a benchmark suite for evaluating automated scholarly taxonomy generation in the context of survey paper organization and, independently, as a structured evaluation protocol in the domain of real-world tax practice, particularly within the Chinese legal and fiscal context. In the taxonomy setting, it is derived from open-access survey articles and is targeted at quantifying the fidelity of LLM-generated taxonomies when compared to human expert-constructed structures (Lahiri et al., 20 Oct 2025). In tax compliance, it forms the backbone of a process-oriented, hierarchical evaluation of LLMs’ abilities to handle tasks ranging from regulatory recitation through risk prevention to strategy planning, as encountered in professional Chinese tax scenarios (Hu et al., 10 Apr 2026).

2. Benchmark Construction and Dataset Organization

Scholarly Taxonomy Generation

• Data Sources: CS-TaxoBench comprises 460 taxonomies extracted from ACM Computing Surveys (2020–2024), filtered to open-access and arXiv-indexed papers. Further, it includes 80 test taxonomies from major AI conference surveys.
• Taxonomy Extraction Pipeline: PDF conversion to text (using Docling), hierarchical outline parsing, pruning of non-taxonomic sections (based on section heading keywords), and reference metadata integration via Semantic Scholar.
• Instance Format: Each JSON record encapsulates a topic (survey title), tree-structured outline (section heading hierarchy), and a references list (paper metadata). The split is 400 training, 60 test, 80 held-out test cases.
• Annotation Quality: Manual TreeLib annotation and extraction-quality evaluation demonstrate high extraction fidelity: precision 83.9%, recall 94.4%, F₁ 88.8%.

Chinese Tax Practice Evaluation

• Multi-Task Structure: 14 datasets (≈7,300 instances) organized according to Bloom’s taxonomy: Knowledge Memorization (tax law recitation), Understanding (summarization, classification, regulatory Q&A), and Application (computation, risk, planning).
• End-to-End Scenarios: Three real-world KA scenarios—tax risk prevention, inspection analysis, and strategy planning—incorporating mixed-format (textual + numeric) data.
• Input/Output Protocol: Every task produces output as fixed-schema JSON, enabling uniform parsing and comparison.
• Data Integration: The benchmark fuses practical, exam-style, and official forum/government Q&A data, ensuring coverage of both declarative and procedural facets.

3. Structural and Semantic Representation

Taxonomy Trees

• Structure: Each taxonomy is encoded as an undirected, rooted tree T; nodes are topical phrases (section headings), and parent–child edges represent inclusion (survey topic → subtopic → subsubtopic).
• JSON Encoding: Taxonomies are stored as recursive lists with “label” and “children” fields, preserving explicit hierarchy without special tokens.
• Coverage: Trees have root (topic), first-level subtopics, and up to third-level granularity.

Tax Practice Tasks

• Schema: Task outputs utilize unified, domain-specific JSON schemas with strongly typed fields (e.g., “Risks Encountered,” “Penalty Outcome,” “Core Rationale”) for each scenario.
• Hybrid Content: Fields may be free-text, categorical, or strictly numeric (e.g., tax values), enabling joint semantic and quantitative evaluation.

4. Evaluation Methodology and Metrics

Taxonomy Metrics

• Structural Alignment: Average Degree Score (Δ)—normalized branching factor—assesses bushiness congruence between candidate and gold taxonomies; Δ ≈ 1 implies structural parity.
• Level-Order Comparison: BLEU-2, ROUGE-L, and BERTScore compute n-gram and embedding overlap over the level-order traversal of trees.
• Semantic Coherence:
  • Node Soft Recall (NSR): soft cardinality-based semantic set similarity using Sentence-BERT cosine similarity.
  • Node Entity Recall (NER): noun-phrase chunk overlap using FLAIR.
• Human and LLM Judging: Both expert raters (Likert 1–5) and GPT-4.1 as LLM-judge provide aggregated quality ratings; agreement (Krippendorff’s α, Spearman’s ρ) validates automatic metrics (Lahiri et al., 20 Oct 2025).

Chinese Tax Practice Metrics

• Structured Parsing and Field Matching: A four-stage pipeline cleans outputs, aligns predicted–gold fields, and isolates text vs. numeric fields.
• Accuracy, Precision, Recall, F₁: Classical metrics for classification-style subtasks.
• EM Accuracy: Stringent exact-match over structured (JSON) output fields.
• BERTScore/BARTScore: Semantic similarity scoring for text fields.
• Custom Aggregates: Composite metrics (e.g., InspectScore, PlanScore) blend EM and BERTScore to rate hybrid outputs.
• Cognitive-Level Reporting: Results are bucketed by Bloom’s Taxonomy level (Memorization, Understanding, Application).

5. Experimental Findings

Taxonomy Generation

• TaxoAlign Model Performance: Closest-to-human Δ (≈1.67), highest BERTScore (0.8501), NSR (1.3244), and positive human/LLM-judge validation (2.42/5). Baselines (AutoSurvey, STORM, topic/keyphrase methods) are significantly outperformed both structurally and semantically.
• Correlation: Human annotator scores correlate with LLM-judge (ρ ≈ 0.527), indicating valid automatic evaluation.
• Extraction Fidelity: High precision and recall in manual annotation support data reliability.

Tax Practice

• Model Ranking: Closed-source large-parameter LLMs (ERNIE-3.5, Grok3, GPT-4o, ChatGPT) consistently outperform open-source or domain-fine-tuned Chinese models (Qwen2.5, GLM4, YaYi2).
• Cognitive Task Difficulty: Knowledge Application (computational, hybrid reasoning) is most challenging (top scores ≈ 0.47), while basic Knowledge Memorization and Understanding are higher (up to ≈ 0.64).
• One-shot Prompts: Marginal gains for many models, but some degrade with prompt length.
• Low Numerical Robustness: Tasks involving tax-liability or strategic planning score < 0.20 (EM), underscoring current LLM limitations for quantitative compliance.

6. Extensibility and Implications

The structured, schema-driven evaluation paradigm of CS-TaxoBench is architected for extensibility across domains:

• Law: Benchmarking hybrid reasoning (verdict rationale) plus numeric penalty extraction.
• Finance: Audit report generation plus discrepancy computation.
• Medicine: Discharge summaries (semantic) and vitals/lab value extraction (quantitative).

This modular, hybrid approach enables rigorous, automatic evaluation of both qualitative reasoning and exact quantitative outputs, addressing reliability mandates for professional and regulated domains (Hu et al., 10 Apr 2026).

7. Significance and Limitations

CS-TaxoBench establishes an open, reproducible standard for benchmarking complex, real-world LLM tasks at the intersection of semantic structure and numeric precision. These tasks are critical in compliance, research synthesis, and expert communication domains. Identified limitations include task-specific input/output simplifications, domain restriction to select languages and professional domains, and reliance on strict input–output schemas. The continuous update of datasets, schemas, and metrics is required to persist relevance amid regulatory and domain evolution.

All data, code, and evaluation scripts are publicly available to support transparency and ongoing benchmark improvement (Lahiri et al., 20 Oct 2025, Hu et al., 10 Apr 2026).