CarbonPDF-QA: Carbon Footprint QA Benchmark

• CarbonPDF-QA is an open-source benchmark for carbon footprint QA over messy PDF sustainability reports, requiring document retrieval, evidence localization, and numerical reasoning.
• The dataset comprises 1,735 PDF reports with over 18,000 questions, challenging systems with noisy extractions and mixed text-table layouts.
• The benchmark leverages a retrieval-augmented method using TF-IDF, critic models, and executable Python for reliable arithmetic and ranking over distributed evidence.

Searching arXiv for CarbonPDF-QA and closely related PDF-QA work. Using arXiv search for "CarbonPDF-QA CF-RAG carbon footprint QA PDF". CarbonPDF-QA is an open-source benchmark for carbon-footprint question answering over product sustainability reports distributed in PDF format, introduced together with the CarbonPDF method in the paper "CF-RAG: A Dataset and Method for Carbon Footprint QA Using Retrieval-Augmented Generation" (Zhao et al., 5 Aug 2025). It is designed for a setting in which useful evidence is dispersed across text, tables, and chart-derived textual content, while PDF parsing introduces noise, structural ambiguity, inconsistent reading order, and spurious extracted text. Within that formulation, CarbonPDF-QA is not treated as ordinary reading comprehension or conventional table QA, but as open-domain document QA over realistic PDF-extracted content, often requiring document retrieval, localization of relevant fragments, and arithmetic over multiple values.

1. Problem formulation and scope

The benchmark targets product sustainability reports that are important for regulatory compliance, consumers, and life-cycle assessment, but are difficult to process automatically because PDF files do not encode tables as tables. Instead, the document stores text and lines at coordinates, not rows and columns, so table structure must be reconstructed heuristically from layout, and the reconstruction is error-prone. The paper further states that mixed text/table layouts are common, evidence for one question may be scattered across sections, pages, or paragraphs, extracted reading order may differ from visual order, and PDFs can contain hidden or spurious text and numbers that are not visually obvious but are extracted programmatically (Zhao et al., 5 Aug 2025).

These design conditions distinguish CarbonPDF-QA from benchmarks that assume cleanly formatted context. The paper explicitly positions it against systems and datasets such as SQuAD, HybridQA, TabFact, and TAT-QA, arguing that those settings generally assume structured, clean input, whereas CarbonPDF-QA is built around inconsistent, loosely connected, and noisy PDF extractions from real reports. This suggests that the benchmark is intended to probe robustness to document-format pathology as much as raw question-answering ability.

2. Corpus composition and dataset structure

CarbonPDF-QA is derived from product carbon footprint reports collected from four companies: HP, Dell, Acer, and Lenovo. The corpus contains 1,735 PDF reports, with an average file size of about 3,759 characters, 539 words, and 1.76 pages per file. The dataset includes question-answer pairs and human-annotated answers, with document-level train/test separation (Zhao et al., 5 Aug 2025).

Each example is tied to a reference document and contains a question, the corresponding answer, the source PDF report as context or reference, and, in the dataset construction pipeline, a generated Python program that computes the answer. The paper reports approximately 18,000 questions across the corpus, with exact split counts of 14,382 training questions and 3,744 test questions. It also notes that each product document contributes, on average, at least 10 questions.

Split or type	Train	Test
Total questions	14,382	3,744
Word Match	7,105	1,841
Max/Min	1,863	486
Top 3/5	1,242	324
Calculation	4,172	1,093

The dataset structure is closely tied to executable supervision. Because generated Python programs are part of the construction pipeline, the benchmark encodes not only final answers but an operational path for deriving them. A plausible implication is that CarbonPDF-QA is meant to support evaluation of systems that combine retrieval with symbolic or programmatic reasoning, rather than relying only on free-form answer generation.

3. Question taxonomy and reasoning demands

CarbonPDF-QA organizes its questions into four types: Word Match, Max/Min, Top 3/5, and Calculation (Zhao et al., 5 Aug 2025). These types are defined by distinct reasoning demands rather than by superficial linguistic form.

Word Match questions are direct extraction tasks for explicitly stated values, such as total product carbon footprint or the percentage of a component. Max/Min questions require identifying the component with the highest or lowest footprint percentage. Top 3/5 questions require ranking the largest contributing components. Calculation questions require multi-step arithmetic based on values spread across the report, often combining lifecycle breakdowns and component percentages.

The benchmark includes multi-answer questions, and the output ordering must match the order of the components mentioned in the question. That requirement makes exact-match evaluation especially stringent, because correctness depends not only on set membership but on ordered output. The paper’s description of the task therefore combines retrieval, information extraction, ranking, and arithmetic reasoning in a single PDF-grounded setting.

4. Ground-truth generation and validation

Question generation is scripted from templates, but the answers are validated through a multi-stage workflow. Regular expressions extract product metadata and carbon values from parsed PDF text, and documents with missing expected patterns or multiple conflicting matches are discarded during extraction. Generated programs are then executed and compared against annotated CSV ground truth. The paper also reports a semester-long validation workflow involving students checking the data (Zhao et al., 5 Aug 2025).

Automated validation included checks for component percentages summing to 99%–101% and for product PCF values deviating by more than 2 × MAE from the dataset mean. Within that procedure, 24 samples failed the component-sum check and 56 samples failed the PCF deviation check. The 56 deviation cases were kept after manual review because they matched the source documents, while the 24 sum-check failures were manually corrected if needed and then revalidated.

This validation procedure is important because the benchmark is constructed from noisy PDF extractions rather than manually normalized documents. The paper’s workflow makes clear that CarbonPDF-QA is not a purely synthetic benchmark: its supervision is template-generated, but its answers are subjected to execution-based and human review. That design supports the claim that the benchmark is intended to preserve fidelity to messy source reports rather than to idealized document representations.

5. CarbonPDF as the reference method

The associated method, CarbonPDF, is a retrieval-augmented generation system designed specifically for this benchmark. At inference time, a retriever selects candidate PDF documents from the corpus, a critic model reranks or selects the most relevant document among the retrieved candidates, the selected reference text and question are passed to a reasoner, and the reasoner generates a Python program whose execution yields the final answer (Zhao et al., 5 Aug 2025).

The final retriever is TF-IDF, implemented with sklearn.TfidfVectorizer. The paper reports that TF-IDF performs best among the tested retrievers for this use case, with near-100% hit rate at top-10 and about 87% top-1 hit rate. A separate critic model is trained to choose the best document from the retrieved candidates. Instead of directly producing natural-language answers, the reasoner generates executable Python code enclosed in triple backticks. The paper states that this is especially important because many questions require arithmetic or ranking over extracted values, which is more reliable when computed programmatically than when generated as free-form text.

CarbonPDF is based on Llama 3 8B, with separate fine-tuning for the critic and the program-based reasoner. Training details reported in the paper are 3 epochs, 2.5 days each, two NVIDIA RTX 6000 Ada GPUs, learning rate 2.5e-5, per-device batch size 8, gradient accumulation steps 4, LoRA for parameter-efficient fine-tuning, 4-bit quantization, and paged Adam as optimizer. The benchmark therefore supports a system architecture in which retrieval quality, document selection, and symbolic computation are all first-class components rather than auxiliary implementation details.

6. Benchmark results, interpretation, and limitations

Under the reported evaluation protocol, the metrics are RMSE, MAE, and Exact Match (EM), with EM for multi-answer questions requiring the complete set of answers to match exactly, including order. The best reported system is CarbonPDF$_{Critic+Program}$ with RMSE 0.78, MAE 0.69, and EM 93.70%. Selected comparison systems include GPT-4o at RMSE 13.37, MAE 10.20, EM 49.20; GPT-4o$_{Program}$ at RMSE 7.33, MAE 5.97, EM 62.18; Gemini-2.0-flash$_{Program}$ at RMSE 6.84, MAE 5.60, EM 65.81; Llama 3$_{Program}$ at RMSE 2.45, MAE 2.19, EM 83.92; and TAT-LLM at RMSE 2.92, MAE 2.52, EM 64.13 (Zhao et al., 5 Aug 2025).

The ablation study reports Retriever + Few-shot at RMSE 833.09, MAE 668.10, EM 23.77; Retriever + Fine-tuned at RMSE 2.45, MAE 2.19, EM 83.92; Retriever + Critic + Fine-tuned w/o Program Reasoning at RMSE 2.39, MAE 1.96, EM 66.83; and Retriever + Critic + Fine-tuned (CarbonPDF) at RMSE 0.78, MAE 0.69, EM 93.70. Per-question-type EM is 94.41 for Word Match, 95.27 for Max/Min, 92.59 for Top 3/5, and 92.13 for Calculation. The paper also reports that, as answer count increases from 1 to 5, EM declines gradually, while numerical error stays low.

The paper’s central empirical claim is that GPT-4o struggles with data inconsistencies in PDF-extracted sustainability reports. It attributes CarbonPDF’s advantage to retrieval narrowing the context, the critic correcting retrieval mistakes, program generation making numerical reasoning explicit and executable, and fine-tuning on domain-specific question/program pairs. At the same time, the paper states several limitations: CarbonPDF relies primarily on text extraction and cannot effectively answer questions grounded in purely graphical content such as pie charts or bar charts; synonymy and semantic generalization remain difficult; PDF parsing limitations remain a bottleneck; and the dataset is built from four companies and focuses on product carbon footprint reports, so broader generalization beyond this domain is not yet established.

A common misconception would be to treat CarbonPDF-QA as simply another table-and-text QA benchmark. The benchmark’s explicit focus on inconsistent parsing artifacts, chart-derived textual content, and arithmetic over noisy PDF extractions indicates a different target: evaluation under document corruption and structural uncertainty. Another misconception would be to interpret its high EM figures as measuring only language-model fluency. The benchmark is constructed so that retrieval quality, execution fidelity, and answer ordering materially affect performance.