CarbonPDF: LLM-Based Carbon Footprint QA

• CarbonPDF is an LLM-based question-answering system that extracts carbon footprint data from PDF sustainability reports with noisy and inconsistent formats.
• It combines TF-IDF retrieval, a critic model fine-tuned from Llama 3, and Python program generation to achieve reliable arithmetic and compositional reasoning.
• Empirical results show significant improvements in RMSE, MAE, and exact match rates over state-of-the-art baselines in carbon footprint QA from real-world reports.

CarbonPDF is an LLM-based question-answering system for extracting and reasoning over carbon footprint information in product sustainability reports distributed as PDF files. It was introduced together with CarbonPDF-QA, an open-source dataset of question-answer pairs over 1,735 product report documents, and is designed specifically for the unstructured, inconsistent, and noisy text obtained after PDF parsing. Its core design combines retrieval, document criticism, program generation, and program execution, with both learned components fine-tuned from Llama 3. The system is positioned around a practical problem: sustainability reports often interleave prose and tables without standardized structure, so accurate carbon-footprint QA requires robustness to fragmented evidence rather than reliance on clean tabular inputs (Zhao et al., 5 Aug 2025).

1. Problem domain and motivation

CarbonPDF addresses question answering over product sustainability reports, especially reports containing carbon footprint information for electronics products. The reported difficulty is not merely document retrieval, but the compound effect of PDF parsing artifacts, inconsistent layouts, visually organized but structurally unencoded tables, and numerical content dispersed across noisy text fragments. In this setting, even strong general-purpose models can fail when asked to integrate evidence or perform arithmetic over imperfectly extracted contexts (Zhao et al., 5 Aug 2025).

The underlying corpus used for the work consists of 1,735 carbon footprint PDF reports from four major electronics manufacturers: HP, Dell, Acer, and Lenovo. The paper frames this as a domain where manual extraction is expensive and error-prone, and where prior QA systems trained on cleaner mixtures of tables and text are insufficiently robust to the post-parsing conditions encountered in real reports. GPT-4o is specifically reported to struggle with questions involving data inconsistencies (Zhao et al., 5 Aug 2025).

A central implication of this formulation is that CarbonPDF is not presented as a generic PDF understanding system. It is specialized for carbon footprint QA under noisy extraction conditions, and its design choices reflect that specialization.

2. CarbonPDF-QA dataset

CarbonPDF-QA is the dataset introduced alongside the method. It contains question-answer pairs for 1,735 product report documents, with human-annotated answers. Raw text was extracted from the PDFs using PyMuPDF, and regular expressions were used to tailor-extract relevant product and footprint data from the noisy text. The paper notes that extracted tables can span multiple paragraphs and may be interspersed with spurious or misaligned numbers, which directly motivates the downstream modeling choices (Zhao et al., 5 Aug 2025).

The dataset construction includes automated checks and human validation. One reported example is verifying that component percentages sum to nearly 100%, followed by manual review of anomalies. This is important because the benchmark is not based on sanitized gold tables; instead, it preserves the inconsistencies that arise from realistic parsing pipelines.

The question set comprises approximately 18,000 QA pairs and covers several families of tasks:

• Word Match: direct lookups such as total carbon or specific component percentages.
• Max/Min and Top 3/5: ranking-style questions over report contents.
• Calculation: questions requiring arithmetic or composition over extracted values.

A notable feature is that each QA pair is associated with a Python program template that returns the correct answer when run against the parsed document data. This makes answers checkable and operationalizes reasoning as executable program synthesis rather than direct string generation (Zhao et al., 5 Aug 2025).

3. System architecture

CarbonPDF uses a retrieve-and-generate framework with three main learned or algorithmic stages beyond parsing: a retriever, a critic model, and a program-based reasoner. The final answer is obtained by executing generated Python code rather than trusting the model’s free-form textual output (Zhao et al., 5 Aug 2025).

The retriever uses TF-IDF, implemented with sklearn’s TfidfVectorizer, and cosine similarity for document scoring. The reported hit rate is approximately 87% at top-1 retrieval, while top-10 retrieval approaches 100%. Retrieval therefore supplies a strong candidate set, but is not treated as sufficient by itself.

The critic model refines the retriever output by selecting the most relevant document or documents from the top-$k$ candidates. It is fine-tuned from Llama 3 and framed as a classification or ranking task over candidate references. Its training objective is to maximize

$\log p_{\mathcal{C}(d|q, r(q, k))}$

where $\mathcal{C}$ denotes the critic model, $d$ the correct document, $q$ the question, and $r(q,k)$ the retrieved references (Zhao et al., 5 Aug 2025).

The program-based reasoner is also fine-tuned from Llama 3. Given the question and reference text, it generates a Python program in a fixed template marked by triple backticks. That code is then executed by an interpreter to produce the final answer. The approach supports multi-answer outputs by returning lists in the required order, and is explicitly used to handle tasks such as traversal over components, sorting for top-$K$ questions, direct lookup, and arithmetic composition (Zhao et al., 5 Aug 2025).

Both the critic and the program reasoner are based on Llama 3 with 8B parameters. Each model was trained for 2.5 days on 2×NVIDIA RTX 6000 Ada GPUs using LoRA adaptation with 4-bit quantization and the Adam optimizer.

4. Operation on noisy PDF-derived evidence

A defining property of CarbonPDF is that it does not attempt to clean or standardize the extracted report text before question answering. Instead, it is trained and evaluated directly on noisy, fragmented, and inconsistently presented post-parsing content. This is an explicit methodological choice rather than an omission (Zhao et al., 5 Aug 2025).

The retriever provides recall over the messy corpus, but the critic model is reported as crucial for filtering spurious or less relevant references returned by TF-IDF. This is especially important when multiple nearby text fragments contain overlapping product metadata or duplicated numerical values.

The program-based answer generation stage addresses a different failure mode: arithmetic and compositional reasoning over noisy evidence. Rather than asking the model to emit final numeric answers directly, CarbonPDF asks it to produce executable Python. The reported rationale is that execution improves numerical precision and supports multi-hop reasoning in contexts where direct language-model generation is vulnerable to arithmetic errors (Zhao et al., 5 Aug 2025).

This suggests a broader design principle: robustness is achieved through decomposition. Retrieval handles scale, the critic handles relevance under ambiguity, and code execution handles compositional reasoning. The system’s gains are therefore not attributable to a single component.

5. Empirical performance

The reported evaluation compares CarbonPDF against a range of baselines, including ACT, CaML, Gemini without RAG, RAG-based prompting with DeepSeek, GPT-4o, and Gemini, program-generation variants of GPT-4o and Gemini, a fine-tuned direct Llama 3 model, and TAT-LLM. CarbonPDF achieves the best reported RMSE, MAE, and exact match (EM) (Zhao et al., 5 Aug 2025).

System	RMSE / MAE / EM	Notes
ACT, CaML	>400 / >200 / ~0–0.3%	Baseline QA systems
Gemini (no RAG)	76.6 / 71.9 / 0.5%	Direct model baseline
DeepSeek, GPT-4o, Gemini (RAG, few-shot, direct)	13.4–28.2 / 10.2–22 / 22–56%	Retrieval-assisted prompting
GPT-4o, Gemini Program	6.8–7.3 / 5.6–6.0 / 62–66%	Program generation without CarbonPDF pipeline
Llama 3 (fine-tuned, direct)	4.0 / 3.4 / 60%	Domain fine-tuning
TAT-LLM (fine-tuned)	2.9 / 2.5 / 64%	Strong structured-data baseline
CarbonPDF	0.78 / 0.69 / 93.7%	RAG + Critic + Program

The paper states that CarbonPDF outperforms current state-of-the-art techniques, including QA systems fine-tuned on table and text data. Relative to GPT-4o- or Gemini-based program generation approaches, the reported reduction in error is by factors of 8–11, and EM rises to 93.7% (Zhao et al., 5 Aug 2025).

Ablation results are used to isolate component contributions. Removing either the critic model or the program-based reasoning stage reduces EM by up to 27%, while MAE and RMSE increase by factors of 3–10. The system is also reported to generalize across direct lookup, ranking, calculation, and multi-answer questions, even as the number of required outputs increases.

6. Significance, interpretation, and limitations

CarbonPDF is significant primarily as a domain-specific architecture for trustworthy QA over messy sustainability PDFs. Its program-based output format makes answers transparent and auditable, which the paper identifies as valuable for high-stakes applications such as regulatory compliance and environmental reporting. Another reported implication is efficiency: relatively small fine-tuned Llama 3 8B models outperform much larger foundation models in this task domain (Zhao et al., 5 Aug 2025).

The work also clarifies a common misconception about PDF QA systems. CarbonPDF’s contribution is not a better PDF parser. The method begins from raw parsed text and is explicitly designed to operate under the failures of that parsing process. Its gains come from modeling the downstream consequences of those failures rather than eliminating them at source.

The main reported limitations are likewise specific. Purely graphical data, such as charts without adjacent text, remains unsolved. Semantic ambiguity, including synonymy or inferred component relations, is still challenging for the critic model. The paper identifies multimodal LLMs and synonym-aware retrieval as future directions (Zhao et al., 5 Aug 2025).

Within the carbon-accounting QA setting, CarbonPDF therefore occupies a distinct methodological niche: it treats carbon-footprint question answering as a noisy-evidence retrieval, disambiguation, and executable-reasoning problem, rather than as a conventional clean-table QA task.