TaxCalcBench: AI Tax Calculation Benchmark

• TaxCalcBench is a domain-specific benchmark designed to evaluate AI models' precision in computing US personal tax returns.
• It employs structured tax scenarios with rigorous metrics, including strict and lenient accuracy evaluations of individual tax form lines.
• Experimental results highlight recurring arithmetic and tax-code interpretation errors, underscoring the need for hybrid AI-deterministic systems.

TaxCalcBench is a domain-specific benchmark designed to evaluate frontier AI models on the precise task of calculating US personal income tax returns. It targets the “calculation” subtask of tax filing, requiring models to transform structured taxpayer data into accurate federal tax returns conforming to official forms. By simulating real-world tax engine conditions and assessing both line-item and aggregate accuracy, TaxCalcBench probes both the arithmetic competence and tax-code understanding of LLMs. Experimental results indicate a significant gap between current model reliability and practical deployment standards, revealing systematic errors and the necessity for additional computational infrastructure.

1. Benchmark Structure and Task Definition

TaxCalcBench consists of 51 test cases representing typical federal tax scenarios for Tax Year 2024. Each case is constructed as an input/output pair: the input is a JSON object encoding user tax data (W-2 incomes, filing status, relevant credits, and other required inputs), and the output is a mockup of a completed tax return, generally following a simplified Modernized e-File (MeF) XML structure or a prescribed text format that closely mirrors key elements of the official IRS Form 1040.

The central requirement is that models perform the substantive calculations necessary to complete each line of the return given all inputs—excluding steps related to data transcription or error correction. The dataset is intentionally limited to federal-only, straightforward return types (e.g., Single, Married Filing Jointly, Head of Household) with commonly encountered credits, deductions, and income sources. Prompts provided to models explicitly specify the expected output structure, requesting calculated line values alongside supporting explanations or calculation steps for auditability.

2. Evaluation Methodologies and Metrics

TaxCalcBench employs multiple rigorous evaluation criteria, assessing both the accuracy of entire returns and the correctness of individual tax form lines:

• Correct Returns (strict): Requires exact correspondence between the model’s output and the reference line values for every relevant field. Any error, even of a single dollar, causes a failure.
• Correct Returns (lenient): Allows a $5 tolerance per line; this is not valid for actual tax compliance but facilitates analysis of near-correct computations.
• Line-Level Accuracy: Calculates the percentage of individual return lines that match exactly or within the lenient window.
• Pass@k and pass^k Metrics: For each test case, models are evaluated over multiple independent runs; pass^k (as used in τ-bench) measures the proportion of cases that achieve a correct result at least once within k model attempts, capturing reliability under sampling.

To mimic practical usage constraints, models are tested under five “thinking budgets” (token limits allocated to the reasoning phase): lobotomized (minimal), low, medium, high, and ultrathink (maximal), with each configuration receiving four evaluation runs per test case.

3. Error Analysis and Model Shortcomings

Empirical analysis reveals critical, recurring error modes among participating models:

Error Type	Frequency / Impact	Underlying Cause
Tax Table Misuse	15–20% of cases	Models use formulaic brackets vs. lookup
Arithmetic/Calculation Error	Cascading, multiple lines affected	Fault propagation, imprecise reasoning
Eligibility Misclassification	Common in complex credit situations	Inadequate translation of legal criteria
Output Format Misalignment	Sporadic, e.g., line swaps on forms	Schema adherence failure

A principal issue is failure to use the IRS Tax Table when required (incomes below $100,000), with LLMs frequently reverting to marginal rate calculations, causing systematic dollar-level errors. Additional pitfalls include propagating errors from a single line to multiple downstream fields and mistakes in eligibility determination for complex credits (e.g., Child Tax Credit, Earned Income Tax Credit), reflecting difficulties in encoding legal intricacies as reasoning chains.

4. Role of Infrastructure and Hybrid Approaches

TaxCalcBench highlights the inadequacy of pure neural approaches for authoritative tax calculation. Deterministic, hand-crafted engines—essential in existing tax preparation software—demonstrate superior consistency. The need arises for an orchestration layer that synergizes LLM flexibility with the rigor of deterministic algorithms. Two key infrastructural components are identified:

• Enhanced Prompting/Scaffolding: Structured prompting, potentially incorporating relevant tax form segments and tax table data, offers a path to steering LLM outputs more reliably.
• API Adaptation: Mechanisms such as dynamic thinking budgets and context-aware activation of extended reasoning are needed to tune model computation per task demands.

The possibility of multi-stage or interactive processing is suggested, where the model is “walked through” discrete computational substeps (e.g., isolating tax table lookups before calculating the final return).

5. Mathematical Formulations and Formulaic Representation

Although TaxCalcBench is not a symbolic mathematics benchmark, its structure inherently depends on arithmetic aggregation and formulaic expressions. For instance, the computation for Form 1040 Line 1a can be represented as:

line_1a = sum(w2.box_1 for w2 in w2s) – sum(sch_c.temporary_statutory_employee for sch_c in schedules_c) – schedule_1.nonqualified_deferred_compensation

Evaluation criteria leverage LaTeX-style expressions to formalize correctness:

Strict criterion: $$\text{Correct Return} = \{\, \forall\, i,\ |\text{ModelValue}_i - \text{ExpectedValue}_i| = 0 \,\}$$ Lenient: $$\text{Lenient Match} = \{\, \forall\, i,\ |\text{ModelValue}_i - \text{ExpectedValue}_i| \leq 5 \,\}$$

This blending of natural language scaffolding and code-style or mathematical formulae is emblematic of the challenges in applying LLMs to technical calculation tasks.

6. Implications for AI-Augmented Tax Calculation

Findings from TaxCalcBench delineate the boundary between the current capabilities of LLMs and the requirements of domains with zero-tolerance for computational error. AI is not yet ready to independently file taxes, as even state-of-the-art models achieve less than one-third accuracy on the strict metric for the 51-case benchmark. Calculations of tax owed, credits, and eligibility for reliefs remain error-prone in the absence of explicit symbolic augmentation or hard-coded domain logic.

A plausible implication is that near-term advances will require hybrid systems, in which the LLM component handles document interpretation and complex deductions/logic, while all arithmetic and rule-based operations are delegated to deterministic code or purpose-built engines. This division of labor mirrors the architecture suggested by the Calc-X/Calcformers framework, where arithmetic reasoning is offloaded to a symbolic system to ensure reliability (Kadlčík et al., 2023).

7. Prospects for Extension and Future Research

TaxCalcBench is positioned as an evolving benchmark, with several key directions for future development:

• Broader Coverage: Expansion to encompass more complex tax scenarios, additional forms (e.g., state returns), and the full IRS MeF XML format.
• Iterative Yearly Updates: Annual refreshes to reflect changing tax law and accommodate the progress of AI model capabilities.
• Benchmarking “Scaffolded” AI: Experimentation with systems that explicitly blend prompting, staged calculations, and modular integrations with tax engines.
• Comparative Studies: Systematic analysis of model performance under varying “thinking budgets” and temperature/randomness controls, probing the trade-offs between creativity and reliability in computational output.

The progression of TaxCalcBench and parallel efforts is likely to recalibrate the standard for LLM-based automation in domains demanding both semantic understanding and precision computation. The integration of robust symbolic reasoning infrastructure remains central to advancing practical and auditable AI for tax and related regulatory tasks.