AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery (2504.07421v1)

Abstract: We introduce AgentAda, the first LLM-powered analytics agent that can learn and use new analytics skills to extract more specialized insights. Unlike existing methods that require users to manually decide which data analytics method to apply, AgentAda automatically identifies the skill needed from a library of analytical skills to perform the analysis. This also allows AgentAda to use skills that existing LLMs cannot perform out of the box. The library covers a range of methods, including clustering, predictive modeling, and NLP techniques like BERT, which allow AgentAda to handle complex analytics tasks based on what the user needs. AgentAda's dataset-to-insight extraction strategy consists of three key steps: (I) a question generator to generate queries relevant to the user's goal and persona, (II) a hybrid Retrieval-Augmented Generation (RAG)-based skill matcher to choose the best data analytics skill from the skill library, and (III) a code generator that produces executable code based on the retrieved skill's documentation to extract key patterns. We also introduce KaggleBench, a benchmark of curated notebooks across diverse domains, to evaluate AgentAda's performance. We conducted a human evaluation demonstrating that AgentAda provides more insightful analytics than existing tools, with 48.78% of evaluators preferring its analyses, compared to 27.67% for the unskilled agent. We also propose a novel LLM-as-a-judge approach that we show is aligned with human evaluation as a way to automate insight quality evaluation at larger scale.

• The paper introduces AgentAda, a novel LLM-powered analytics agent with a dynamic, skill-informed framework that tailors data insight discovery.
• It employs a three-stage pipeline—question generation, skill matching with a hybrid RAG approach, and robust code generation—to enhance analysis quality.
• AgentAda outperforms existing tools on the AdaBench benchmark, with human evaluations favoring its deeper, goal-aligned insights by nearly 49%.

AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery

Introduction

"AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery" introduces a novel LLM-powered analytics agent named AgentAda. The paper tackles the limitations in existing LLM-based analytics tools which often struggle with complex workflows and rely on basic analytical techniques. AgentAda aims to address these issues by introducing a dynamic, skill-informed framework that leverages a curated library of analytical skills for more structured and goal-oriented data analysis.

Approach and System Design

AgentAda's architecture comprises three pivotal components: question generation, skill matching, and code generation.

1. Question Generation:
   • This initial stage involves generating queries that relate directly to the user's objectives and persona. The goal is to formulate questions that are pertinent, thus facilitating meaningful data insights.
   • A dual-stage question generation process enhances the quality of questions by transitioning from basic to more advanced analytics queries.

     Figure 1: The distribution of tasks covered by AgentAda's framework.

2. Skill Matcher:
   • The system incorporates a hybrid Retrieval-Augmented Generation (RAG) approach to identify and apply the most suitable analytical skills from its library. This library includes techniques such as clustering, predictive modeling, and NLP methods like BERT.
   • Skills are matched with each generated question to ensure the best available techniques are employed for data analysis.

     Figure 2: The t-SNE embedding plots for the questions showing their diversity.

3. Code Generation:
   • Once a skill is matched to a question, AgentAda employs a code generation cascade to produce executable code tailored to that specific skill's documentation. This ensures that the analysis is both accurate and efficient.
   • The code generation module is adept at self-correction, incorporating error messages during regeneration attempts to improve robustness and reliability.

Benchmarking and Evaluation

AgentAda introduces a comprehensive benchmark named AdaBench that spans 700 examples across 49 domains and 28 task types. The diversity of AdaBench allows for robust evaluations of AgentAda's capabilities.

1. Human Evaluation:
   • The paper details human evaluation experiments that demonstrate AgentAda's ability to provide insights of greater depth and relevance than existing tools.
   • Evaluators preferred AgentAda-generated insights 48.78% of the time, significantly surpassing the unskilled baseline variant.
2. LLM-as-a-Judge Framework:
   • A novel evaluation strategy named SCORER (Structured Calibration Of Ratings via Expert Refinement) aligns LLM-evaluated insights with human judgments by optimizing LLM prompts based on human ratings, instead of model fine-tuning.
   • SCORER ensures scalability and reliability in evaluating insight quality.

Key Results and Insights

AgentAda outperformed existing agents such as Poirot and Pandas AI in critical areas, including depth of analysis and goal relevance. The adoption of skill guidance in data analytics tasks resulted in more profound insights and coherent analyses, as formalized numerically through benchmarks and qualitative feedback.

Figure 3: AgentAda's pipeline for automated data-driven insights.

Impact and Future Directions

AgentAda's significant contributions lie in its structured approach to LLM-powered data analytics, enabling goal-aligned, insightful, and advanced analysis across varied domains. This agent not only sets a new standard for LLM analytics but also opens new avenues for scalable evaluation using the LLM-as-a-judge framework.

Future works could expand AgentAda's scope beyond structured data to encompass unstructured data, multi-table analysis, and large-scale datasets, further pushing the boundaries of LLM capabilities in real-world applications.

Conclusion

The research showcases a significant leap in utilizing LLMs for advanced data analytics tasks, transitioning from basic statistical analysis to end-to-end, skill-informed data processing. AgentAda exemplifies the integration of LLMs with curated skill libraries and structured workflows to produce scalable, meaningful insights, setting the stage for future innovations in AI-driven data analytics.