InsightBench: Business Analytics Benchmark

• InsightBench is a curated benchmark that evaluates LLM-powered agents’ ability to perform comprehensive business analytics on realistic enterprise data.
• It employs a multi-step workflow including schema extraction, Python code generation, and recursive follow-up analysis to synthesize actionable insights.
• Comparative evaluations reveal that multi-step agents like AgentPoirot, particularly with gpt-4o, consistently outperform single-query approaches in key metrics.

InsightBench is a curated benchmark designed to evaluate LLM–powered agents on their ability to perform end-to-end business analytics, moving beyond single-query question answering to comprehensive insight discovery and summarization on realistic enterprise data. It represents a foundational resource in the landscape of automated analytics evaluation, targeting the synthesis, interpretation, and communication of business-relevant observations through multi-step, code-driven agent workflows (Sahu et al., 2024).

1. Benchmark Design and Dataset Construction

InsightBench comprises 31 tabular datasets derived from the ServiceNow enterprise–workflow platform, spanning five core business domains: Incident Management, User Management, Asset Management, Finance Expenses, and Enterprise Goal Management. Each dataset is standardized with 500 entries and incorporates authentic table schemas (e.g., fields such as incident_open_date, department, expense_amount), augmented with synthetic but plausible data values. Critical to its design, domain experts define SMART (Specific, Measurable, Achievable, Relevant, Time-bound) goals for each dataset instance.

To ensure the presence of ground-truth, expert-curated insights, specific patterns, trends, or anomalies are “planted” directly into the data. For instance, a linearly increasing time-to-resolution (TTR) for incidents is introduced using the model:

$$\mathrm{TTR} = 1 + \mathrm{slope} \times (\mathrm{incident\_open\_date} - \mathrm{data\_start\_date})$$

Text fields such as incident descriptions are generated by strong LLMs (e.g., GPT-4) to ensure linguistic naturalness and contextual realism (Sahu et al., 2024).

2. Multi-Step Insight Generation Workflow

InsightBench evaluates agent competence not merely on single-turn responses, but on their capacity to conduct an extended sequence of analytics steps. The reference workflow, as realized in the AgentPoirot baseline, adopts the following pipeline:

• Schema Extraction: Analyze the CSV for datatypes, summary statistics, and relevant feature ranges.
• High-Level Question Generation: Sample k root questions from the dataset schema and analytical goal.
• Code Generation and Execution: For each question, generate Python (pandas, matplotlib) code, execute, and collect outputs.
• Insight Extraction: Translate the code results into concise, natural-language insights grounded in the observed data.
• Follow-Up Loop: For each question, recursively propose, select, and execute follow-up questions to pursue deeper or broader lines of inquiry.
• Summary Synthesis: After iterating through root and follow-up questions, aggregate discovered insights into an overall summary and actionable recommendations (Sahu et al., 2024).

This approach is designed to emulate the process of an expert analyst by covering a spectrum of analysis types (descriptive, diagnostic, predictive, prescriptive).

3. Agent Benchmarks and Experimental Results

Two agent baselines are evaluated on InsightBench: the single-query Pandas Agent and the multi-step AgentPoirot. The experimental protocol involves multiple LLM backbones (e.g., gpt-4o, gpt-4-turbo, llama-3-70b, gpt-3.5-turbo) across the benchmark datasets. Results are reported using both ROUGE-1 and the LLaMA-3-Eval, an open-source evaluator that provides a learned similarity score for both insight-level and summary-level outputs.

Key findings include:

• AgentPoirot consistently outperforms the Pandas Agent in insight-level evaluations (e.g., +0.07 in LLaMA-3-Eval with gpt-4o backbone).
• The ranking of LLM backbones is: gpt-4o > gpt-4-turbo > llama-3-70b > gpt-3.5-turbo.
• All agents suffer significant performance drops on “hard” datasets, characterized by multi-step or heavily transformed analytical requirements (Sahu et al., 2024).

Agent / Backbone	Insight-ROUGE-1	Insight-L3-Eval	Summary-ROUGE-1	Summary-L3-Eval
PA (gpt-4o)	0.33±0.01	0.52±0.01	0.34±0.01	0.38±0.02
AgentPoirot (gpt-4o)	0.31±0.01	0.59±0.01	0.36±0.08	0.41±0.02
AgentPoirot (llama-3)	0.31±0.01	0.51±0.03	0.35±0.01	0.32±0.01

This table (adapted from (Sahu et al., 2024)) demonstrates the superiority of iterative, multi-step analytics workflows and the influence of LLM backbone selection on agent performance.

4. Evaluation Methodology

InsightBench adopts a two-level scoring protocol based on LLaMA-3-Eval:

• Insight-Level: Each ground-truth insight is matched to the agent-generated insight that maximizes the evaluator score, averaged over all ground-truth insights:

$$\mathrm{score}_{\mathrm{insight}} = \frac{1}{|GT|} \sum_{gt\in GT} \max_{a\in A} M(gt, a)$$

where $M$ is the LLaMA-3-Eval’s free-form similarity metric.

• Summary-Level: The agent-generated summary is scored against the expert summary.
• The final score is the mean of the insight-level and summary-level scores. ROUGE-1 is also optionally reported for comparison (Sahu et al., 2024).

5. Key Observations, Limitations, and Critique

InsightBench's core strengths are in its realistic dataset construction, diversity of business domains, explicit goal definition, and emphasis on end-to-end, multi-step agent workflows. However, a detailed investigation by the authors of InsightEval (Zhu et al., 28 Nov 2025) surfaces multiple critical flaws:

• Format and Content Inconsistencies: Missing goals, absent tables, misaligned question-answer pairs, and insight types not encompassed by the benchmark’s candidate set.
• Ambiguous Objectives: Many instance goals are overly broad, lacking actionable metrics or timeframes (“identify trends and underlying factors”).
• Schema Mismatches: Questions sometimes reference non-existent columns or undefined analytic categories, and insights occasionally emerge from incomplete or illogical premises.
• Inflated Recall via Redundancy: Paraphrased or repetitive insights artificially enhance recall without adding substantive coverage.
• Evaluation Limitations: Reliance on a single LLM judge (LLaMA-3-Eval) induces bias, fails to penalize spurious/irrelevant outputs, and credits only annotated insights, ignoring agent-discovered novel findings (Zhu et al., 28 Nov 2025).

InsightEval explicitly enumerates these deficiencies and proposes an alternative multi-evaluator, multi-metric protocol, incorporating recall, precision, $F_1$, and insight novelty to more accurately reflect agent performance.

6. Influence and Ongoing Developments

InsightBench catalyzed a shift from single-query QA to holistic, workflow-driven analytics agent evaluation and inspired a lineage of follow-on work refining dataset curation, taxonomy of insights (including evaluative and exploratory types), and evaluation methodologies (Zhu et al., 28 Nov 2025). Several lines of research now advocate:

• More tightly specified goals—quantifying metrics, cohorts, and time horizons in each instance.
• Multi-perspective question sets, ensuring full coverage of analytic types and viewpoints.
• Multi-evaluator protocols, balancing recall and precision, and identifying creative or novel discoveries beyond the annotated insight set.
• Expansion to multi-modal and retrieval-augmented settings (see MT-RAIG (Seo et al., 17 Feb 2025)), and integration of unstructured, visual, or adversarial data sources.

A plausible implication is that subsequent benchmarks will increasingly incorporate fine-grained, claim-based evaluation metrics and robust multi-step curation pipelines to avoid the pitfalls identified in the original InsightBench framework.

7. Related Benchmarks and Future Directions

The emergence of InsightBench prompted critical comparisons with contemporary frameworks, notably MT-RAIG (Seo et al., 17 Feb 2025) and InsightEval (Zhu et al., 28 Nov 2025). MT-RAIG generalizes insight generation to retrieval-augmented, multi-table settings and introduces faithfulness and completeness as evaluation dimensions, while InsightEval enforces rigorous goal specification, question-schema alignment, and a multi-evaluator metric suite.

Key anticipated directions for the field include:

• Enabling agents to process multi-modal, unstructured, and cross-domain datasets.
• Advancing robust evaluation, including precision, novelty, and human alignment.
• Supporting model development for complex, multi-hop reasoning and retrieval-augmented analytics workflows.

These developments collectively direct the field toward a more holistic, reliable, and insightful evaluation of LLM-based data analytics agents, supplying an evolving foundation for research and practical deployment in automated business analysis (Sahu et al., 2024, Zhu et al., 28 Nov 2025, Seo et al., 17 Feb 2025).