ReportEval: Report-Centric Evaluation Protocol

• ReportEval is a report-centric protocol that defines quality in agentic system reports through coherence, trace grounding, and actionable error analysis.
• It utilizes five key dimensions—overall structure, error analysis, root cause analysis, optimization, and overall impact—to score comprehensive reports.
• The approach aligns evaluations with real engineering workflows by emphasizing complete trace analysis rather than isolated outcome metrics.

to=arxiv_search.search 彩神争霸下载 玩大发快三json content='{"query":"id:(Yang et al., 28 Feb 2026) OR id:(Xie et al., 16 Apr 2026) OR id:(Fan et al., 9 Oct 2025) OR id:(Walden et al., 30 Sep 2025) OR id:(Xu et al., 22 Sep 2025) OR id:(Shi et al., 13 Jun 2026) OR id:(Li et al., 30 Sep 2025)", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}' to=arxiv_search.search _一本道 to=arxiv_search.search мундақ 微信的天天中彩票='(Yang et al., 28 Feb 2026) TraceSIR ReportEval (Xie et al., 16 Apr 2026) DR3-Eval (Fan et al., 9 Oct 2025) DeepResearch-ReportEval' to=arxiv_search.search 大发时时彩计划 重庆时时彩的 content='(Yang et al., 28 Feb 2026)' ReportEval is the report-centric evaluation protocol introduced with TraceSIR for judging how well an analysis report explains, diagnoses, and improves agentic execution traces in realistic engineering settings. Rather than scoring only final task outcomes or selected intermediate snippets, it evaluates end-to-end report quality and usability, with emphasis on coherence, trace-grounded error analysis, causal explanation, optimization usefulness, and practical debugging value across many failed traces. In the TraceSIR framework, ReportEval is paired with TraceBench and serves as the principal mechanism for comparing comprehensive scenario-level reports generated from agentic failures (Yang et al., 28 Feb 2026).

1. Definition and motivation

ReportEval is designed for settings in which practitioners must understand why agentic systems fail and how to fix them across many traces. Its stated purpose is to assess a report’s coherence and structure, correctness and trace-grounding of error analysis, depth and credibility of root-cause explanations, usefulness and feasibility of optimization suggestions, and overall practical impact for engineers and researchers. The protocol explicitly targets coherence and structure, accuracy and trace faithfulness, actionability, issue localization quality, root-cause analysis quality, optimization usefulness, and overall practical impact (Yang et al., 28 Feb 2026).

The protocol arises from a limitation in prior agent evaluation practice. Existing evaluations are described as largely outcome-oriented or based on partial intermediate signals such as selected states or summarized steps. Those approaches discard behavioral context embedded in long traces, hinder precise issue localization, and rarely produce consolidated, cross-case insights. Summarization and clustering approaches are characterized as losing trace-level evidence and therefore producing coarse or weakly grounded conclusions. ReportEval instead evaluates full reports that synthesize trace-grounded analyses across multiple cases (Yang et al., 28 Feb 2026).

Within TraceSIR, this emphasis matches the structure of the overall system. StructureAgent compresses traces into TraceFormat, InsightAgent performs fine-grained diagnosis, and ReportAgent aggregates insights across task instances into comprehensive reports. ReportEval evaluates the resulting report artifact rather than isolated predictions, which aligns the benchmark with the document that engineers actually consume during debugging and system improvement (Yang et al., 28 Feb 2026).

2. Dimensions, rubrics, and scoring

ReportEval uses five equally weighted dimensions, each scored on a 0–10 Likert-like scale. The anchors emphasize completeness, trace grounding, and actionability.

Dimension	Abbreviation	Focus
Overall Structure	OS	Organization, coherence, and explicit grounding in the analyzed execution traces
Error Analysis	EA	Correctness of identified errors and explicit linkage to trace steps
Root Cause Analysis	RCA	Insightfulness and justification of causal explanations
Optimization Analysis	OA	Relevance, feasibility, and actionability of proposed optimizations
Overall Impact	OI	Utility for understanding behavior and guiding decisions

The protocol reports an overall score on a 0–100 scale by averaging these five dimensions and normalizing. The worked example in the paper assigns a Deep Research report $OS=8$, $EA=7$, $RCA=9$, $OA=8$, and $OI=8$, producing an overall score of $80$. Judge aggregation is explicit: if $H$ human experts rate a report, then

$$S_{\text{human}} = \frac{1}{H}\sum_{h=1}^{H} S^{(h)}.$$

For LLM-as-a-judge with $R$ repeated runs,

$$S_{\text{LLM}} = \frac{1}{R}\sum_{r=1}^{R} S^{(r)}.$$

Relative improvement over a baseline is defined as

$EA=7$0

These formulas make the protocol explicitly comparative and scenario-level rather than example-level (Yang et al., 28 Feb 2026).

The five dimensions jointly encode a practitioner-centered notion of report quality. OS and EA force trace support and local evidence; RCA and OA require causal depth and implementable recommendations; OI captures whether the report materially helps debugging, decision-making, and system improvement. This structure makes ReportEval more than a textual quality metric: it is an engineering utility metric defined over diagnostic reports (Yang et al., 28 Feb 2026).

3. Benchmark protocol and evaluation procedure

ReportEval is instantiated on TraceBench, which covers three real-world agentic scenarios: BrowseComp for Deep Research, Tau2Bench for Function Calling, and SWE-bench for Agentic Coding. From GLM-4.6 runs on each benchmark’s official test split, 50 failure cases per scenario were randomly sampled with $EA=7$1, yielding 150 failed task instances in total. Each method then produces one comprehensive report per scenario over 50 cases. Although report generation can trigger every 10 processed cases, evaluation standardizes comparisons by using the final 50-case report (Yang et al., 28 Feb 2026).

The evaluation protocol uses both human and LLM judges. Human evaluation is performed by six engineers or researchers experienced with agentic systems, with each scenario rated by at least two experts. The LLM judge is GPT-5, run three times per report and averaged for stability. Both judge types follow the same five-dimension rubric centered on structure, correctness with trace support, causal depth, actionability, and practitioner impact (Yang et al., 28 Feb 2026).

Bias controls are explicit. Human evaluators are blinded to method identities and backbones; all methods operate on the same trace sets per scenario; and scores are averaged across experts or across multiple LLM runs. Inputs are identical TraceBench traces for all methods. The implementation details tied to the evaluation setting include TraceSIR instantiations with GLM-5 and Claude-4.6 backbones, ClaudeCode as baseline, GPT-5 as judge, and a StructureAgent abstraction threshold of $EA=7$2 words or 1,000 characters per element (Yang et al., 28 Feb 2026).

Aggregation is reported at several levels. Within a report, dimension scores are averaged and normalized to 0–100. Across judges, averages are computed across experts or LLM runs. Across scenarios, the paper compares scenario-level overall scores and method rankings. Results are presented in tables with per-dimension 0–10 scores, overall 0–100 scores, and rankings. No confidence intervals or statistical significance tests are reported (Yang et al., 28 Feb 2026).

4. Empirical findings in TraceSIR

Under human evaluation, TraceSIR outperforms ClaudeCode in all three scenarios and across both backbones. In Deep Research, TraceSIR scores 81.0 versus 66.0 with Claude-4.6 and 65.0 versus 55.0 with GLM-5. In Function Calling, it scores 77.0 versus 74.0 with Claude-4.6 and 53.0 versus 40.0 with GLM-5. In Agentic Coding, it scores 89.0 versus 77.0 with Claude-4.6 and 62.0 versus 57.0 with GLM-5 (Yang et al., 28 Feb 2026).

Under LLM-as-a-judge, the same ranking pattern holds. Deep Research scores are 91.3 versus 90.0 for Claude-4.6 and 88.0 versus 82.7 for GLM-5. Function Calling scores are 91.3 versus 88.0 for Claude-4.6 and 89.3 versus 80.7 for GLM-5. Agentic Coding scores are 90.7 versus 90.0 for Claude-4.6 and 84.7 versus 58.7 for GLM-5 (Yang et al., 28 Feb 2026).

Averaged across scenarios and backbones, TraceSIR achieves a $EA=7$3 relative improvement in human evaluation and a $EA=7$4 relative improvement under LLM judging. The largest human per-setting gains include $EA=7$5 in Function Calling with GLM-5 and $EA=7$6 in Deep Research with Claude-4.6; the largest LLM-judge gain is $EA=7$7 in Agentic Coding with GLM-5. Dimension-wise mean gains over ClaudeCode are $EA=7$8 in OS, $EA=7$9 in EA, $RCA=9$0 in RCA, $RCA=9$1 in OA, and $RCA=9$2 in OI on the 0–10 scale. The paper reports no ablations such as removing StructureAgent or InsightAgent (Yang et al., 28 Feb 2026).

The qualitative synthesis reported for usability is consistent with these numeric differences. In human studies, TraceSIR’s optimization suggestions and overall impact score higher, and experts prefer reports that link conclusions to specific Thought–Action–Observation steps and propose concrete process or model improvements. ReportEval’s OA and OI dimensions are the formal channel through which those preferences are captured (Yang et al., 28 Feb 2026).

5. Reliability, validity, and limitations

The paper does not report inter-annotator agreement coefficients such as Cohen’s $RCA=9$3 or Krippendorff’s $RCA=9$4. It also does not report numeric human–LLM correlation or agreement coefficients. What is reported is qualitative convergence: humans are described as more conservative and the LLM judge as more permissive, but both produce consistent rankings and the same qualitative conclusion that TraceSIR reports are more structured, comprehensive, and actionable across scenarios (Yang et al., 28 Feb 2026).

Validation against downstream engineering outcomes is indirect. ReportEval is intended to reflect practical engineering utility, but the paper validates that aim through expert preferences and scenario-wide improvements rather than objective downstream re-runs after optimization. This means the protocol assesses the perceived usefulness and groundedness of reports, not the realized effect of applying the recommendations in a subsequent system iteration (Yang et al., 28 Feb 2026).

Several threats to validity are acknowledged. Report quality depends on the underlying LLM backbone; coding-heavy or highly specialized domains are more challenging; the protocol targets moderate batch sizes such as 50 cases and may require hierarchical aggregation for much larger collections; and LLM-based analysis is token-expensive, slow, and subject to stochastic variability. Evaluation constraints include the absence of inter-annotator agreement coefficients, the absence of statistical significance tests, calibration differences between humans and LLM judges, and untested generalization beyond the three TraceBench scenarios (Yang et al., 28 Feb 2026).

These constraints position ReportEval as a practically motivated but not yet fully statistically characterized evaluation protocol. Its strength lies in aligning evaluation with engineering workflow; its main open methodological issue is the evidentiary gap between report-level judgment and downstream system improvement (Yang et al., 28 Feb 2026).

6. Broader report-centric evaluation landscape

ReportEval belongs to a broader shift from answer-only evaluation to report-centric evaluation. In deep research, DR$RCA=9$5-Eval operationalizes a multimodal, multi-file report-generation setting with five dimensions—Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality—over a static research sandbox, while “Understanding DeepResearch via Reports” defines DeepResearch-ReportEval around Quality, Redundancy, and Factuality for 100 curated queries (Xie et al., 16 Apr 2026, Fan et al., 9 Oct 2025). In citation-backed RAG, Auto-ARGUE evaluates long-form reports using sentence precision, nugget recall, weighted nugget recall, and F1-style overall scores, centered on content coverage and citation support (Walden et al., 30 Sep 2025).

In clinical report generation, analogous report-evaluation programs emphasize domain grounding rather than surface overlap. ReXrank provides a public leaderboard with eight metrics and section-aligned evaluation for chest X-ray report generation (Zhang et al., 2024). ReEvalMed argues that high metric scores do not necessarily translate into clinician trust and reframes evaluation around clinical alignment, discrimination, robustness, and monotonicity (Li et al., 30 Sep 2025). ReportQA treats the report as context for question answering and defines QAScore from positive consistency and negative false-positive control, with the goal of matching how clinicians actually use reports (Shi et al., 13 Jun 2026). RadEval unifies classic overlap metrics, concept-based metrics, and LLM-based evaluators into a standardized radiology framework (Xu et al., 22 Sep 2025).

A plausible implication is that ReportEval, in the narrow TraceSIR sense and in the broader report-centric sense, marks an evaluation shift toward artifacts that humans directly inspect and act upon. That shift is visible across agentic debugging, deep research, citation-grounded RAG, radiology, and financial report generation, even though the dimensions differ by domain (Yang et al., 28 Feb 2026, Jin et al., 10 Nov 2025).

The name should also be distinguished from “RepEval,” a different metric that evaluates text quality by projecting LLM representations along learned quality directions. RepEval is reference-free and representation-based rather than report-centric and practitioner-oriented (Sheng et al., 2024).