Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists (2410.23331v1)

Abstract: We present a benchmark for LLMs designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure. The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.

• The paper introduces FeatEng, a benchmark focusing on evaluating LLM-generated feature engineering code on practical data science tasks.
• It assesses four key dimensions: practical usability, world knowledge application, complex skill integration, and resistance to exploitation.
• The experimental analysis shows that models like o1-preview and Gemini outperform traditional baselines, highlighting significant advances in automated feature engineering.

Evaluating LLMs as Data Scientists: An In-Depth Overview

The paper "Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists" introduces a new benchmark named FeatEng, specifically designed to assess the data engineering capabilities of LLMs. The benchmark focuses on a particularly intricate aspect of data science: feature engineering. The core of the evaluation rests on a model's ability to generate feature engineering code, which demands a robust understanding of both the domain-specific data and the overall intricacies of machine learning.

Key Contributions

The authors present a novel benchmarking approach that centers around practical usability, integration of world knowledge, complex skill integration, and resistance to exploitation. These are identified as the four crucial aspects that any benchmark should address to thoroughly evaluate the problem-solving capabilities of LLMs in real-world scenarios.

1. Practical Usability: The tasks in FeatEng are anchored in real-world data science problems, ensuring that any improvement in the model's output translates into a tangible functional value. By targeting the improvement of an XGBoost model’s performance after modification via the LLM-generated code, the benchmark embodies the pragmatist view on the instrumental value of knowledge.
2. World Knowledge Application: The benchmark mandates that LLMs dynamically employ their extensive knowledge bases to solve problems contextually. It necessitates the models to not only possess a wide range of knowledge but also effectively apply it to the given datasets, mimicking real-world problem-solving.
3. Complex Skill Integration: FeatEng evaluates LLMs on their ability to integrate multiple skills — such as data interpretation, reasoning, and code generation — to complete multifaceted tasks. This requirement is reflective of the need for models to combine various data processing functions seamlessly.
4. Resistance to Exploitation: The benchmark’s design inherently resists exploitation through memorization or reliance on dataset-specific biases. This is achieved by structuring tasks that require real improvements in the models’ capabilities, forcing them to generalize across different datasets.

Experimental Insights and Comparisons

The paper extensively evaluates several state-of-the-art LLMs using FeatEng, revealing distinct performance variability. These results indicate that models like o1-preview and Gemini exhibit superior performance, underscoring significant improvements over traditional baselines. However, the paper also highlights that existing open-source models show competitive capabilities in feature engineering tasks.

A noteworthy aspect of the paper is its comparison of FeatEng to existing benchmarks like MMLU, HumanEval, and more. The authors argue that while these existing benchmarks target individual competencies, they fall short in evaluating a model’s ability to synthesize these skills within a realistic problem-solving context.

Implications and Future Directions

From a practical standpoint, FeatEng presents a comprehensive tool for assessing LLM capabilities in data science, with potential applications in automated machine learning (AutoML) systems. The ability to autonomously generate sophisticated feature transformations could streamline the feature engineering process, saving time and enhancing the overall efficiency of data science workflows.

From a theoretical perspective, the FeatEng benchmark advances the discourse on LLM evaluation by emphasizing the integration of multiple skills in a practical context. Such an approach acknowledges the complexity of real-world data science tasks and sets a higher bar for what LLMs need to achieve to be considered genuinely capable.

Looking forward, the research suggests several avenues for future exploration. Enhancing the robustness of the benchmark by incorporating a larger and more diverse set of datasets could reduce potential biases. Additionally, there is scope for refining the evaluation metrics to offer an even more nuanced understanding of the models' performance.

In summary, the FeatEng benchmark represents a significant advancement in the domain of LLM evaluation, striving to provide a holistic view of a model's ability to perform realistic data science tasks. As LLMs continue to evolve, tools like FeatEng will be critical in developing models that are not only knowledgeable but also practically effective in real-world applications.