DailyReport: Benchmark for Daily Search Agents

• DailyReport is an open-ended benchmark designed to evaluate LLM-based search agents on daily search tasks, emphasizing realistic, report-style outputs.
• It decomposes each task into atomic subtasks that are evaluated through a cascade scoring system across instruction following, factuality, and rationality.
• The framework aggregates subtask scores into a user-centric metric, highlighting current gaps and paving the way for more actionable agent improvements.

DailyReport is an open-ended benchmark for evaluating search agents on daily search tasks. It targets LLM-based systems that autonomously browse the web and synthesize long, report-style answers for “daily, real-world information needs,” rather than specialized or static tasks. The benchmark contains 150 open-ended tasks with 3,546 associated rubrics, and its central methodological contribution is a cascade evaluation scheme built on subtask decomposition, disentangled dimensions, and user-centric aggregation. In this formulation, DailyReport measures not only whether an agent can produce a plausible long-form answer, but also whether it follows instructions, states correct facts under external verification, and presents rational analysis in a way that aligns with user priorities (Han et al., 11 Jun 2026).

1. Conceptualization and problem setting

DailyReport was introduced against two perceived deficiencies in prior search-agent evaluation. On the task side, earlier benchmarks are described as relying on “expert-crafted, specialized tasks” that are “overprocessed” or highly professional, often static, and at risk of becoming answerable from parametric memory rather than live retrieval. On the evaluation side, prior work is described as using coarse task-level rubrics with entangled dimensions such as overall correctness and completeness, yielding limited interpretability and weak alignment with user priorities. DailyReport addresses these issues by centering “daily, timely, practical tasks,” decomposing each task into atomic subtasks, and aggregating performance through a user-oriented scoring rule (Han et al., 11 Jun 2026).

The benchmark’s target systems are search agents that use LLMs to explore web sources and synthesize comprehensive answers. This aligns closely with the broader “DeepResearch” framing in which agents are organized into three stages—Interaction, Investigation, and Synthesis—and produce reports as their primary deliverable. That report-centric perspective is especially relevant because DailyReport evaluates end products intended to resemble what a user would read as a practical daily brief, rather than isolated retrieval or reasoning primitives (Fan et al., 9 Oct 2025).

DailyReport’s notion of “daily search tasks” is explicitly tied to widely discussed and timely information demands. The tasks are derived from trending topics and comments on platforms including Weibo, Xiaohongshu, Zhihu, Facebook, Reddit, and Twitter, and are meant to capture concrete user needs such as identifying ranked universities, summarizing policy changes, or analyzing recent events. This suggests a deliberate shift from benchmark instances defined by expert convenience toward query formulations grounded in actual public information-seeking behavior (Han et al., 11 Jun 2026).

2. Task construction, taxonomy, and coverage

The benchmark contains 150 tasks produced through a three-stage human-expert pipeline requiring over 500 annotation hours. First, annotators collect trending topics and associated user comments from major Western and Chinese platforms. Second, task writers convert selected discussions into realistic daily search tasks with clear scope, explicit factual and analytical requirements, and safety filtering. Third, tasks are labeled into 35 fine-grained categories and 10 high-level domains, with annotators chosen to be familiar with both Western and Chinese media ecosystems (Han et al., 11 Jun 2026).

Two task types are distinguished. “Retrieval-centric” tasks account for 100 tasks and emphasize objective information about specified entities with relatively light analysis. “Analysis-centric” tasks account for 50 tasks and require broader reasoning, autonomous identification of relevant information, and deeper synthesis. The task set is therefore broad in subject matter but structured in formulation, with each instance constrained by explicit user-facing requirements such as time, geography, quantity, format, or analytical function (Han et al., 11 Jun 2026).

Each original task is decomposed into subtasks derived from a taxonomy of constraint categories. The categories include content constraints, scope constraints, completeness constraints, quantity constraints, format constraints, setting constraints, attribute constraints, action and rule constraints, and function constraints. The decomposition follows three principles: atomicity, coverage, and traceability. Atomicity means each subtask typically corresponds to one constraint type; coverage means the subtasks jointly cover all explicit requirements in the task; traceability means every subtask must be grounded in the original task text (Han et al., 11 Jun 2026).

This decomposition is significant because DailyReport is designed around open-ended reports rather than short answers. In related report-level evaluation work, open-ended research-style queries are described as naturally inviting multi-page reports and resisting single ground-truth verification. DailyReport adopts that long-form orientation but makes it operational by attaching fine-grained rubrics to individual subtasks rather than judging only the report as an undifferentiated whole (Fan et al., 9 Oct 2025).

3. Rubric system and cascade evaluation

DailyReport provides 3,546 rubrics distributed across the 150 tasks. Each rubric is attached to a specific subtask and one of three dimensions: Instruction Following, Factuality, or Rationality. These dimensions are designed to be “strictly orthogonal.” Instruction Following evaluates whether the response fully executes the subtask’s instructions, including scope, coverage, and format. Factuality evaluates objectively verifiable claims through web-based external verification. Rationality evaluates whether the reasoning is logically coherent and contextually supported, excluding factual claims already handled under Factuality (Han et al., 11 Jun 2026).

Rubrics are initially generated by LLMs and then refined, corrected, and standardized by human experts. For Instruction Following, the judge scores criteria as $0$, $0.5$, or $1$. For Factuality, the score is the fraction of correct factual claims extracted for the subtask. For Rationality, the score is again $0$, $0.5$, or $1$, based on reasoning quality over the non-factual analytic portion of the response. The formal subtask evaluation is written as

$$\mathrm{dim}_i = \operatorname{Judge}_{\mathrm{dim}}\big(T_i, \mathrm{Res}, r_{\mathrm{dim}}(T_i)\big),$$

with $$\mathrm{dim} \in \{\mathrm{ins}, \mathrm{fac}, \mathrm{rat}\}$$ (Han et al., 11 Jun 2026).

The benchmark’s key innovation is cascade attribution. Factuality and Rationality are scored only when instruction following is non-zero. DailyReport formalizes overall dimension scores as

$$\text{Ins} = \frac{\sum_{k=1}^{n} \mathrm{ins}_k}{n},$$

$$\mathrm{Fac} = \frac{\sum_{k=1}^{n} \delta_k \cdot \mathrm{ins}_k \cdot \mathrm{fac}_k}{\sum_{k=1}^{n} \delta_k \cdot \mathrm{ins}_k},$$

$0.5$0

Here, $0.5$1 and $0.5$2 indicate whether the subtask has a factuality or rationality rubric. The interpretation given in the benchmark is explicitly user-aligned: if a system does not provide the requested content, it is not meaningful to score the factuality or rationality of content that never truly addressed the instruction (Han et al., 11 Jun 2026).

A concise summary of the evaluation dimensions is as follows.

Dimension	Subtask scoring	Role in cascade
Instruction Following	$0.5$3	Base condition for higher-level scoring
Factuality	$0.5$4	Scored only if instruction following is non-zero
Rationality	$0.5$5	Scored only if instruction following is non-zero

DailyReport’s cascade mechanism differs from purely report-level frameworks that directly score final outputs for quality, redundancy, and factuality. Report-level schemes such as DeepResearch-ReportEval are designed to measure holistic report properties, whereas DailyReport uses subtask-level disentanglement to localize failure modes more precisely. A plausible implication is that the two paradigms are complementary: report-level evaluation can reveal stylistic and structural properties of the final artifact, while DailyReport exposes whether the artifact actually satisfies the underlying user request at the granularity of individual constraints (Fan et al., 9 Oct 2025).

4. User-centric aggregation and benchmark metrics

DailyReport supplements the three core dimensions with a user-facing aggregate score called UserPref, defined on a 1–4 scale: 1 for Unhelpful, 2 for Deficient, 3 for Acceptable, and 4 for Perfect. To compute it, the benchmark introduces subtask importance levels based on an ablation-style thought experiment conducted by task creators. Each subtask is labeled as $0.5$6, $0.5$7, $0.5$8, or $0.5$9, depending on how severely overall user satisfaction would decline if that subtask alone were not satisfied (Han et al., 11 Jun 2026).

For each subtask $1$0, DailyReport defines an overall performance score

$1$1

The aggregation algorithm then computes means over different importance partitions and applies threshold logic. If all subtasks have $1$2, the report receives UserPref $1$3. If all core $1$4 subtasks fail, or if critical-but-not-core subtasks perform very poorly, the response is judged Unhelpful. Otherwise, the answer is assigned Deficient or Acceptable depending on whether importance-weighted conditions over $1$5, $1$6, and $1$7 groups are met (Han et al., 11 Jun 2026).

This scoring rule is designed to distinguish benchmark utility from simple rubric counting. DailyReport also reports SubTask Pass, defined as the proportion of subtasks that satisfy all rubrics. The distinction matters because a system can pass many low-importance subtasks while failing a single critical one. The benchmark explicitly notes that a system may have a higher SubTask Pass but lower UserPref, or the reverse, depending on which subtasks fail. This makes the benchmark diagnostic not only at the level of dimensions but also at the level of user-priority structure (Han et al., 11 Jun 2026).

Human involvement is unusually substantial for a benchmark of this type. The task and rubric construction required 500+ hours, and meta-evaluation used 300 randomly sampled subtasks. The selected judge model is Gemini-3-Flash, chosen through comparison with other strong models. Reported agreement figures are approximately 96.5% for Instruction Following, approximately 94.2% for Factuality, and approximately 95.3% for Rationality, while UserPref shows a Weighted Cohen’s Kappa of 0.859 against user judgments. These values are presented as evidence that the LLM-as-judge pipeline is reliable enough for benchmark-scale evaluation (Han et al., 11 Jun 2026).

5. Empirical results and diagnostic findings

DailyReport evaluates 17 agentic systems in three families: native Deep Research Agents, LLMs with search tools, and LLMs orchestrated via Claude Code. All systems receive a common system prompt instructing them to act as search assistants with web search and webpage reading, to avoid parametric knowledge, and to produce a 2,000+ word Markdown report with citations and a References section. Tool-augmented LLMs use google_search via Serper and fetch_webpage via Jina Reader; Claude Code configurations add file system and code execution via MCP; native agents are given longer timeouts of 1,800 seconds (Han et al., 11 Jun 2026).

The best overall performance is reported for GPT 5.4 with search, with UserPref $1$8, SubTask Pass $1$9, Instruction Following $0$0, Factuality $0$1, and Rationality $0$2. The strongest Claude Code configuration is CC-GPT 5.4, with UserPref $0$3, SubTask Pass $0$4, Instruction Following $0$5, Factuality $0$6, and Rationality $0$7. Native Deep Research Agents generally underperform those two families; among them, Grok 3 Deep Research is highest with UserPref $0$8, Factuality $0$9, and Rationality $0.5$0 (Han et al., 11 Jun 2026).

A central empirical conclusion is that even the best systems score below 3.0 on UserPref, which corresponds to the threshold for “Acceptable.” DailyReport therefore characterizes current search-agent performance as falling between Deficient and Acceptable on average. The benchmark further shows that Instruction Following is consistently strong across modern systems, with scores roughly in the $0.5$1–$0.5$2 range, whereas Factuality is the weakest dimension, ranging roughly from $0.5$3 to $0.5$4. Rationality is typically stronger than Factuality but remains imperfect (Han et al., 11 Jun 2026).

Task-type analysis shows that analysis-centric tasks often yield slightly higher Instruction Following and Rationality but lower Factuality than retrieval-centric tasks. The benchmark interprets this pattern as arising from the open-endedness of analytic writing: models can more easily appear to cover requested aspects and produce coherent-seeming analysis, but the source base is more heterogeneous and therefore harder to verify rigorously. Retrieval-centric tasks, being narrower and more enumerative, are easier to fact-check (Han et al., 11 Jun 2026).

Trace analysis introduces additional behavioral evidence. Systems that make more search calls tend to perform better overall. Claude Code–orchestrated systems use fewer search calls on average, which the authors associate with lower thoroughness for search-heavy tasks. Citation behavior also exhibits a notable failure mode. Although many systems maintain a high Reference_Ratio, Reference Accuracy and Refer-Claim Consistency are “far from perfect,” and the benchmark identifies “citation washing” as the case where reports appear well supported while citing low-quality, irrelevant, or semantically misaligned evidence (Han et al., 11 Jun 2026).

6. Position within report-oriented evaluation and daily report generation

DailyReport belongs to a larger methodological shift toward evaluating report synthesis rather than isolated QA skills. In “Understanding DeepResearch via Reports,” report-level evaluation is justified by the claim that DeepResearch systems must synthesize diverse sources, generate insights, and present coherent findings, and that these capabilities resist simple verification. That framework proposes three report-level dimensions—quality, redundancy, and factuality—and standardizes evaluation over 100 curated queries spanning 12 real-world categories. DailyReport shares the emphasis on end-to-end, long-form outputs, but its distinctive contribution is to decompose open-ended tasks into subtasks with cascade rubrics and user-priority weighting (Fan et al., 9 Oct 2025).

The benchmark also sits adjacent to domain-specific daily report generation systems. “WeatherSyn” defines a Weather Forecasting Report task in which a multimodal model generates a four-day, aspect-controlled forecast report from ERA5-based heatmaps and a structured prompt. Its evaluation emphasizes claim-based weighted F1 over meteorological aspects and LLM-based ranking for factual consistency and summary quality. This is a different operational setting from DailyReport—multimodal forecasting rather than web search—but it demonstrates a related design pattern: daily reports can be made machine-evaluable by imposing a structured output template, aspect taxonomy, and claim-level scoring regime (Zheng et al., 8 May 2026).

A similar domain specialization appears in “FinReport,” which constructs automated financial reports from stock-linked news, factor models, and EGARCH–VaR risk estimation. FinReport’s reports contain four sections—return forecasting, risk assessment, overall trend prediction, and summary—and are intended as explainable daily notes for ordinary investors. This suggests that DailyReport is not merely a benchmark name but part of a broader research trajectory in which “daily report” systems integrate heterogeneous evidence, structured decomposition, and user-facing summarization (Li et al., 2024).

The broader literature therefore places DailyReport at the intersection of three concerns: realistic web information seeking, interpretable evaluation, and report-centric synthesis. A plausible implication is that future systems may combine DailyReport-style subtask and user-preference evaluation with report-level criteria such as redundancy and with domain-specific claim taxonomies of the kind used in WeatherSyn and FinReport. The benchmark’s released dataset, code, evaluation scripts, prompts, and baseline configurations make it suitable not only as a leaderboard instrument but also as a reusable scaffold for training, diagnostics, and iterative improvement of search agents (Han et al., 11 Jun 2026).