DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle

Abstract: Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io

• The paper introduces a comprehensive benchmark (DAComp) designed to evaluate autonomous data agents across full-lifecycle enterprise data engineering and strategic analysis tasks.
• The paper details rigorous evaluation protocols using metrics like Component Score, Cascading Failure Score, and Success Rate, exposing significant performance gaps in current models.
• The paper underscores challenges in dependency management and strategic reasoning, and recommends advanced hierarchical agent architectures for improved data pipeline orchestration.

DAComp: A Comprehensive Benchmark for Data Agents in Enterprise Data Intelligence

Motivation and Scope

DAComp introduces a rigorously designed benchmark for evaluating autonomous data agents throughout the full enterprise data intelligence lifecycle, spanning both repository-level data engineering (DE) and open-ended data analysis (DA). The benchmark directly addresses the lacuna in existing agentic evaluation protocols that fail to model production-scale workflow complexity, focusing instead on isolated code generation tasks and deterministic Q&A paradigms. DAComp establishes realistic operational demands by leveraging industrial-scale schemas, multi-layer SQL pipeline construction, and open-ended strategic analysis tasks, thus differentiating itself in scale, scope, and evaluation paradigm.

Figure 1: DAComp aims to evaluate LLMs on full-lifecycle data intelligence workflows, encompassing repository-level data engineering (DE) and open-ended data analysis (DA).

Benchmark Design

DAComp comprises 210 tasks partitioned into DE-Arch (architecture specification), DE-Impl (pipeline construction), DE-Evol (system evolution), and DA (data analysis). DE tasks require agents to (1) plan high-level engineering blueprints, (2) implement entire DAG-style ETL pipelines over enterprise schemas, and (3) evolve existing repositories under changing requirements, demanding up to 4,000 lines of code and complex cross-file dependency management. DA tasks transcend deterministic analytics, posing open-ended business questions that require agents to autonomously select analytical strategies, code iteratively, interpret intermediate results, and synthesize multi-faceted recommendations.

Evaluation Protocols

Deterministic engineering tasks are scored using execution-based metrics:

• Component Score (CS): Measures individual node correctness via isolated evaluation.
• Cascading Failure Score (CFS): Sequentially nullifies component scores upon any upstream error, quantifying end-to-end DAG integrity.
• Success Rate (SR): Strict criterion requiring perfect execution for all components.

  Figure 2: Illustration of how CS, CFS, and SR differ in scoring a simple pipeline when an intermediate node fails.

For open-ended analysis, DAComp employs an LLM-judge guided by hierarchical rubrics, decomposing each task into explicit requirements and supporting multiple solution paths per sub-requirement. The evaluation admits diverse valid methodologies and aggregates scores bottom-up for robust assessment, while analytical dimensions include Completeness, Accuracy, Insightfulness, Readability, Analytical Depth, and Visualization. The Good-Same-Bad (GSB) protocol further benchmarks report quality relative to multiple baselines.

Experimental Results and Diagnostic Analyses

Repository-Level Data Engineering

State-of-the-art models such as GPT-5 and Gemini-2.5-Pro achieve only modest DE Scores (≈43%) and extremely low strict success rates (<20%), indicating a pronounced bottleneck in holistic pipeline orchestration. While specialized open-source models (DeepSeek-V3.1, Qwen3-Coder) rival proprietary ones in isolated component accuracy, all models experience severe attrition from CS to CFS metrics, highlighting the challenge of dependency management and cross-layer error propagation.

Analytical Task Complexity

Open-ended DA tasks expose model deficiencies in strategic reasoning and insight synthesis, with GPT-5 achieving the highest DA Score (≈56%) across dimensions but with clear gaps in Analytical Depth and Insightfulness for less capable models. Notably, smaller models fail to generate coherent analytical artifacts while code-specialized models underperform in report readability and visualization.

Figure 3: DA performance across five analytical objectives.

A fine-grained analysis reveals that agents perform well on concrete, descriptive tasks but struggle sharply with diagnostic and strategic problems, emphasizing the need for advanced planning and contextual interpretation capabilities.

Code Scale and Error Modalities

Performance degrades as code volume (node count, line count) increases, with medium-scale edits posing the greatest cognitive challenge, implicating nuanced business logic updates rather than simple repetition or trivial modifications.

Figure 4: Effect of node count and line count.

Error modes predominantly involve dependency errors, SQL omission, and cascading logic failures across pipeline layers. Circular dependencies and non-trivial joins in DE-Impl remain recurrent sources of failure, as illustrated in the following:

Figure 5: Examples of errors in DE-Impl: the red-crossed cases show mistakes such as joining on mismatched keys (account_id instead of campaign_id), and incorrect aggregation without respecting daily granularity, while the green-checked cases illustrate valid implementations with proper joins and staged aggregation.

Figure 6: Examples of errors in DE-Impl: a circular dependency in which int_campaign_perf.sql depends on campaign_summary.sql, creating a loop in the data pipeline.

Pipeline evolution introduces challenges in scope identification and downstream metric integrity, as demonstrated by instances of data leakage and improper row filtering:

Figure 7: DE-Evol pipeline-level error case. Layer 1 (Staging) contains duplicate “current” rows where the same account_id appears multiple times with is_most_recent = true. Layer 2 (Intermediate) joins to campaign stats while filtering on is_most_recent = true, causing double-counted spend.

Blueprint specification errors in DE-Arch include weak business enforcement, ambiguous boundaries, and insufficient constraint hardening:

Figure 8: DE-Arch error case. Key issues include: weak downstream enforcement, ambiguous boundaries, and missing tests or business logic constraints.

LLM-Judge Validation and Reliability

DAComp's rubric-driven LLM-judge methodology is extensively validated against human annotator agreement and multiple alternative judge models. Inter-rater reliability metrics (Item-level $\kappa_w > 0.83$, Case-level ICC > 0.89) demonstrate robust alignment, and leaderboard rankings are invariant under judge selection and aggregation hyperparameters. Grading stochasticity is negligible, supporting reproducibility.

Practical and Theoretical Implications

DAComp demonstrates that current LLM agents are fundamentally limited in performing full-stack autonomous data engineering and strategic business analysis, with core bottlenecks in global dependency tracking, codebase evolution, and open-ended synthesis. The distinction between engineering and analytical competence necessitates architectural advances in agent planning, workflow memory, long-range context management, and hierarchical reasoning, both for code and analytical output generation.

On the practical side, DAComp provides a rigorous framework for end-to-end system validation, laying the foundation for data agent deployment in real enterprise environments. The inclusion of a multilingual Chinese adaptation (DAComp-zh) further emphasizes the benchmark's relevance for global agent development.

Future Directions

The evidence from DAComp suggests several critical avenues for the community:

• Hierarchical agent architectures integrating symbolic reasoning and schema-level planning to overcome orchestration bottlenecks.
• Persistent tool-based memory and interaction protocols to improve pipeline evolution fidelity.
• Cross-modal communication for agents capable of seamlessly combining SQL, Python, and visualization tools under open-ended rubrics.
• Community adoption of rubric-based LLM-judge protocols to enable extensible, fair, and reproducible benchmarking of emergent agentic capabilities.

Conclusion

DAComp formalizes a comprehensive, modular, and end-to-end benchmark for evaluating autonomous data agents on both enterprise-scale data engineering and open-ended analytical workflows. The benchmark's strong empirical results and multi-dimensional evaluation expose acute limitations in current agent paradigms, particularly regarding system orchestration and strategic insight generation. DAComp is positioned to drive the evolution of agentic architectures and serves as a standardized testbed for the iterative refinement of real-world data intelligence agents (2512.04324).