PaperRecon: Evaluating Paper Reconstruction
- The paper introduces PaperRecon as a framework that quantitatively separates presentation quality from hallucination in AI-generated scientific papers.
- PaperRecon is defined as a reconstruction-based evaluation that uses structured summaries and limited auxiliary resources for grounded assessments.
- Empirical findings reveal a trade-off between high presentation scores and reduced factual hallucinations, with detailed inputs enhancing overall performance.
Paper Reconstruction Evaluation, usually abbreviated PaperRecon, is an evaluation framework for measuring how well modern coding agents can reconstruct a scientific paper from compressed source materials while separating two qualities that are often conflated: Presentation, which concerns how faithfully and effectively the paper is written, and Hallucination, which concerns whether the generated content contradicts the source paper. The framework begins from an existing paper, constructs a structured overview from it, asks an agent to regenerate the full paper from that overview and a limited set of auxiliary resources, and then compares the generated paper against the original along these two axes (Miyai et al., 1 Apr 2026).
1. Definition and rationale
PaperRecon was introduced as the first systematic evaluation framework for quantifying both the quality and the risks of papers written by modern coding agents (Miyai et al., 1 Apr 2026). Its motivation is the claim that existing assessments of AI-written papers, especially reviewer-style judgments, are inadequate because polished prose can conceal fabricated method details, experimental setups, or results. The framework therefore treats presentation quality and factual reliability as orthogonal dimensions rather than as a single notion of “good writing” (Miyai et al., 1 Apr 2026).
The core reconstruction setup is deliberately controlled. Starting from an original paper, the framework creates a structured summary—called overview.md in the abstract and research_overview.md in the main text—then provides that overview together with limited auxiliary resources to a coding agent. The agent is then asked to generate a full paper, which is subsequently compared to the original (Miyai et al., 1 Apr 2026). The paper argues that this approximates the writing component of AI Scientist-style systems while isolating core writing ability from unrelated problems such as reference discovery.
The resources given to the agent are explicitly enumerated. They include the research overview, original figures with simplified captions, original LaTeX table source with simplified captions, the original bibliography augmented with abstracts, the associated codebase if available, and, in benchmark execution, a template.tex that preserves the original section structure while leaving content empty (Miyai et al., 1 Apr 2026). The full original prose and the detailed original figure and table captions are withheld. This design is intended to prevent direct copying while still providing enough structure for a realistic reconstruction task.
A central premise is that reconstruction from compressed source materials creates a grounded comparison setting. Because there is a known original paper, presentation and hallucination can be measured against a concrete reference rather than through free-form impressions (Miyai et al., 1 Apr 2026). This suggests a broader methodological point: reconstruction-based evaluation is useful when the original artifact is available and direct fidelity, rather than standalone plausibility, is the object of interest.
2. Task structure and PaperWrite-Bench
PaperRecon normalizes both the ground-truth and generated papers into seven canonical section categories: Abstract, Introduction, Method, Benchmark Construction, Experiment, Related Work, and Conclusion (Miyai et al., 1 Apr 2026). Sections are first assigned by keyword rules and then by LLM classification when necessary. If multiple raw sections map to the same category, they are merged. This normalization is used consistently across the evaluation pipeline.
The framework is instantiated through PaperWrite-Bench, a benchmark of 51 papers (Miyai et al., 1 Apr 2026). The main paper states that these papers come from top-tier venues including ACL 2025, EMNLP 2025, CVPR 2025, CVPR 2026, ICCV 2025, ICLR 2025, NeurIPS 2025, ICLR 2026, and ACMMM 2025, while the appendix table gives a more detailed area-wise distribution: 21 ML papers, 21 CV papers, 5 multimedia papers, and 4 NLP papers (Miyai et al., 1 Apr 2026). The paper itself notes a small inconsistency between the venue list in the main text and the appendix table: the main text mentions EMNLP 2025, whereas the appendix lists NAACL25 and does not list EMNLP25.
The benchmark composition is also characterized by paper function. Of the 51 papers, 32 propose new methods, 12 introduce new benchmarks, and 7 do both (Miyai et al., 1 Apr 2026). The benchmark deliberately uses papers published after 2025, with the stated rationale that earlier related benchmarks were largely based on papers around 2024 and may not reflect the capabilities of modern agents.
For each benchmark paper, the construction pipeline creates research_overview.md using GPT-5 and then manually verifies it for sufficiency and faithfulness; the overview has an average length of 463 words (Miyai et al., 1 Apr 2026). Tables, figures, references, and code are extracted from arXiv source files; .bib entries are augmented with abstracts from the Semantic Scholar API; and if a code README contained an abstract or introduction, those sections are manually removed (Miyai et al., 1 Apr 2026). The original section structure is preserved in template.tex, and agents are instructed to write within that fixed structure so that comparison is more accurate across papers with heterogeneous organization.
3. Presentation evaluation
Presentation is evaluated with a rubric-based method rather than unconstrained LLM judging (Miyai et al., 1 Apr 2026). The paper reports that preliminary experiments found unconstrained judging to have low discriminative power, whereas rubrics produced finer-grained and more reliable assessment. Rubrics are initially generated from the ground-truth paper using GPT-5.4 and then reviewed and refined by the authors (Miyai et al., 1 Apr 2026).
For each section except the conclusion, the evaluator judges how well the generated section covers each rubric element on a 1–5 scale. The scale is explicitly defined as follows: 5 means the element is fully and accurately described with correct details; 4 means it is mostly described, with the core idea present but some details missing; 3 means it is partially described, with significant gaps or vagueness; 2 means it is barely mentioned; and 1 means it is completely absent (Miyai et al., 1 Apr 2026). The final section score is the average across all rubric elements, including figure and table items.
Figures and tables are incorporated directly into presentation scoring. Figures are aligned using the ground-truth TeX file. If both the original and generated paper reference a figure in the same section, the figure receives a full context score of 5; otherwise an LLM judges whether the figure is used in an appropriate context on the same 1–5 scale, and unreferenced figures receive a 1 (Miyai et al., 1 Apr 2026). Tables are matched through label matching, caption matching, and LLM-based matching, after which an LLM evaluates numerical accuracy, structural alignment, and content consistency on the same 1–5 scale (Miyai et al., 1 Apr 2026).
The paper reports the average numbers of rubric items per section: 10.3 for Abstract, 13.3 for Introduction, 12.6 for Related Work, 14.2 for Method, 14.6 for Benchmark Construction, and 14.3 for Experiment (Miyai et al., 1 Apr 2026). In this formulation, Presentation is fundamentally a measure of faithful coverage: it assesses whether the generated paper preserves the key scientific elements of the original in the appropriate sections, with correct use of figures and tables.
The framework’s human validation supports this rubric design. On 72 pairs of generated papers derived from 12 source papers and judged by three reviewers with top-tier reviewing experience, the ranking induced by rubric scores achieved Kendall’s with against human pairwise judgments (Miyai et al., 1 Apr 2026). The paper interprets this as evidence that rubric-based presentation scoring is reliable, while also noting some residual disagreement due to subjective preferences such as concise versus detailed writing.
4. Hallucination and citation evaluation
Hallucination is defined at the claim level and is evaluated through a two-stage agentic process grounded in the original paper (Miyai et al., 1 Apr 2026). The first stage uses GPT-5.4 to extract all concrete and verifiable claims from each generated section except the conclusion and classify each claim as Supported, Neutral, or Contradictory. Contradictory claims are then further labeled as major or minor (Miyai et al., 1 Apr 2026). A major contradictory claim is the framework’s primary hallucination unit.
The framework emphasizes that absence is not the same as contradiction. Neutral claims are those not present in the ground-truth paper but still reasonable supplementary detail that does not contradict it (Miyai et al., 1 Apr 2026). This distinction matters because the framework is intended to identify genuine fabrications and distortions rather than merely penalize every detail that was not explicitly stated in the original.
The second stage aggregates all claims marked contradictory and re-checks them with Claude Code using Sonnet 4.6, operating in read-only mode with access to the ground-truth LaTeX source, codebase, figures, and tables (Miyai et al., 1 Apr 2026). The verifier uses Read, Glob, and Grep tools and can revise claims from contradictory to supported or neutral. The reported hallucination metric is the count of major contradictory claims per paper after this verification stage.
Citation quality is evaluated separately by extracting citation keys from both the ground-truth and generated LaTeX and comparing them as sets (Miyai et al., 1 Apr 2026). The framework reports citation precision, citation recall, citation F1, and hallucinated citations, where hallucinated citations are citation keys used in the generated paper but absent from references.bib. The paper states the citation F1 in the standard form
where is precision and is recall (Miyai et al., 1 Apr 2026).
The hallucination detector is partially validated by manual inspection. The authors manually inspect 97 claims labeled major contradictory from GPT-5, GPT-5.4, and Sonnet-4.6 outputs and find that 96% are genuine contradictions or fabrications (Miyai et al., 1 Apr 2026). The paper presents this as evidence of high precision, while also acknowledging that it does not establish full recall.
5. Empirical findings
The main empirical result is a clear trade-off between presentation quality and hallucination (Miyai et al., 1 Apr 2026). Claude Code variants produce stronger presentation scores, whereas Codex—especially with GPT-5.4—produces fewer major contradictory claims. This trade-off is the paper’s central conclusion.
| Configuration | Avg. rubric score | Major contradictory claims per paper |
|---|---|---|
| Codex / GPT-5 | 3.26 | 10.2 |
| Codex / GPT-5.4 | 3.59 | 3.0 |
| Claude Code / Sonnet 4 | 3.49 | 12.0 |
| Claude Code / Sonnet 4.6 | 3.86 | 10.4 |
| ClaudeCode-Teams / Sonnet 4.6 | 3.82 | 9.8 |
Claude Code with Sonnet 4.6 achieves the highest overall presentation score, 3.86 (Miyai et al., 1 Apr 2026). Section-wise, its scores are reported as 4.37 for Abstract, 4.12 for Introduction, 3.08 for Related Work, 3.69 for Method, 3.84 for Benchmark Construction, and 4.00 for Experiment (Miyai et al., 1 Apr 2026). The paper notes that Abstract and Introduction are generally strongest across systems, whereas Related Work is relatively weak.
Hallucinations are concentrated particularly in Method and Experiment sections (Miyai et al., 1 Apr 2026). For example, Codex / GPT-5.4 averages 1.3 major contradictory claims in Method and 0.9 in Experiment, whereas Claude Code / Sonnet 4.6 averages 4.7 in Method and 3.6 in Experiment (Miyai et al., 1 Apr 2026). This concentration is significant because these sections contain the most technically consequential claims.
Citation metrics show a related trade-off. Claude Code / Sonnet 4.6 reaches citation precision 0.83, recall 0.58, F1 0.67, and 0.2 hallucinated citations, while Codex / GPT-5.4 reaches precision 0.86, recall 0.43, F1 0.56, and 0.0 hallucinated citations (Miyai et al., 1 Apr 2026). The Claude variants therefore achieve better citation coverage, whereas Codex produces fewer citation hallucinations.
The framework also captures improvements across model generations. GPT-5 to GPT-5.4 improves both presentation and hallucination, and Sonnet 4 to Sonnet 4.6 improves presentation while slightly reducing hallucinations (Miyai et al., 1 Apr 2026). The paper presents this as evidence that PaperRecon can function as a progress metric over time.
An especially notable additional analysis concerns the length of the research overview. When the overview is increased from roughly 463 words to roughly 1492 words, Sonnet 4’s rubric score improves from 3.49 to 3.64 and its hallucination count falls from 8.8 to 5.8; Sonnet 4.6 improves from 3.83 to 4.17 while hallucinations fall from 9.8 to 2.3 (Miyai et al., 1 Apr 2026). The paper interprets this as evidence that more detailed input improves both presentation and factual reliability.
Performance also varies by domain. Grouped by conference type, the reported averages are: ML rubric 3.58 with hallucination 8.3, CV rubric 3.63 with hallucination 10.1, multimedia rubric 3.47 with hallucination 10.7, and NLP rubric 3.77 with hallucination 6.0 (Miyai et al., 1 Apr 2026). The authors speculate that NLP papers may be easier because they tend to be more findings-based and involve fewer complex mathematical formulations or methods.
6. Limitations, interpretation, and related reconstruction-based frameworks
The paper identifies two explicit limitations (Miyai et al., 1 Apr 2026). First, PaperRecon assumes structured inputs such as figures, tables, references, and sometimes code, which isolates writing ability but does not cover harder settings where a system must retrieve or discover those materials itself. Second, scientific writing is highly diverse, and section-wise evaluation may not fully capture holistic writing quality or stylistic variation.
Additional validity constraints are also visible in the design. Rubric construction depends partly on LLM generation, though it is manually reviewed; hallucination validation measures precision but not recall; and the benchmark composition is uneven across fields, with relatively few NLP and multimedia papers relative to ML and CV (Miyai et al., 1 Apr 2026). The section-wise decomposition is analytically useful, but it may underrepresent global discourse properties such as overall coherence or rhetorical economy.
The broader significance of PaperRecon lies in its insistence that scientific writing systems must be evaluated both for how well they present a paper and for whether they remain faithful to source evidence (Miyai et al., 1 Apr 2026). The framework’s central warning is that fluent scientific prose is not a reliable proxy for truthfulness. A plausible implication is that any deployment of AI systems in paper writing, summarization, or scientific drafting requires source-grounded auditing rather than reviewer-style quality judgments alone.
In adjacent domains, related reconstruction-based evaluation ideas have appeared with different targets. RESCORE defines Paper-to-Simulation Recoverability for control-systems papers and evaluates whether an automated system can reconstruct executable simulations that match published plots (Bhat et al., 6 Apr 2026). RaV-IDP introduces Reconstruction as Validation for intelligent document processing by reconstructing extracted entities back into forms comparable to original source regions and scoring fidelity against the unmodified document crop (Jha, 26 Apr 2026). These frameworks operate on simulations and document entities rather than full papers, but they share with PaperRecon the principle that reconstruction against an original artifact can expose failures that plausibility-based evaluation misses.
PaperRecon’s specific contribution is to make that principle operational for AI-written scientific papers. It frames paper writing as a reconstruction task from compressed evidence, disentangles presentation from hallucination, grounds factual checking in original source materials, and supplies a benchmark—PaperWrite-Bench—on which improvements and trade-offs can be measured systematically (Miyai et al., 1 Apr 2026).