LightAutoDS-Tab: Multi-AutoML for Tabular Data

• The paper introduces LightAutoDS-Tab, a multi-AutoML system that fuses LLM agents with robust AutoML backends to automate tabular data pipelines.
• It employs specialized agents for task understanding, code generation, validation, and debugging to streamline Kaggle-style workflows.
• Empirical results demonstrate competitive performance across diverse datasets, validating the system’s flexible integration over single-backend solutions.

Searching arXiv for the main paper and closely related systems mentioned for accurate inline citation. LightAutoDS-Tab is a multi-AutoML, LLM-driven agentic system for tabular machine learning introduced in “LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Data” and published on 2025-07-17. It is designed to bridge classic AutoML frameworks with modern LLM code agents for Kaggle-style tabular tasks by using LLMs for task understanding, planning, code generation, configuration, debugging, and reporting, while delegating model training and search to mature AutoML backends when appropriate. The system takes a tabular dataset and a natural-language query as input and returns predictions, full Python pipeline code, a technical report, and a non-technical explanation (Lapin et al., 17 Jul 2025).

1. Problem setting and design rationale

LightAutoDS-Tab is motivated by a specific division in automated data science for tabular data. Classic AutoML frameworks such as AutoGluon, H2O, LightAutoML, and FEDOT are strong at systematic hyperparameter search, ensembling, and robust training routines, but they are weak in flexible pipeline design and high-level reasoning and typically require manual setup of task type, metric, target, constraints, and glue code. LLM-based data-science agents, including AutoKaggle, AIDE, and related code-generation scaffolds, are strong at understanding natural language, conducting EDA, and generating arbitrary Python pipelines, but are weaker in reliability: generated code can be brittle, context costs are high, and repeated trial-and-error may still fail to yield a stable, high-quality pipeline (Lapin et al., 17 Jul 2025).

The system therefore aims to combine these strengths. Its central design choice is to use LLM agents where semantic interpretation and workflow synthesis are needed, but to delegate model training and search to robust AutoML tools whenever possible. It also retains an LLM-only code-generation path for situations where AutoML is not desired or not appropriate. This design is explicitly framed as “multi-AutoML” rather than as an LLM wrapper around a single backend (Lapin et al., 17 Jul 2025).

A further design rationale concerns dependence on specific underlying tools. The paper argues that many existing LLM+AutoML integrations are tightly bound to one framework, which creates three problems: limited robustness across tasks, limited extensibility, and inherited limitations from the chosen backend. LightAutoDS-Tab responds by integrating several AutoML tools and letting the LLM configure them instead of hard-wiring the system to one library. In the Kaggle context emphasized by the paper, this is intended to reduce deep conversation trees, context bloat, and code fragility while exploiting mature tabular AutoML systems (Lapin et al., 17 Jul 2025).

2. Agentic architecture and execution flow

The architecture is organized around multiple specialized agents sharing a common state and coordinated by a supervisor through SelectAgent(S). The state includes conversation history, task reflection, latest code, execution logs, and validation status (Lapin et al., 17 Jul 2025).

Agent	Primary role
A_int-act	Conversational interaction and intent clarification
A_plan	Problem reflection and workflow decomposition
A_gen	Python code generation from skeletons
A_val	Execution checking and behavioral validation
A_impr	Error-driven debugging and refinement
A_auto	AutoML backend choice and configuration
A_int-pret	Human-readable reporting

The InteractorAgent handles user interaction and can terminate the loop if the request is purely informational. When a pipeline must be built, the PlannerAgent interprets the task description, identifies file roles, target, and metric through a “problem reflection” prompt, and decomposes the work into EDA, preprocessing, model training, evaluation, and submission creation. The GeneratorAgent then produces Python code from a predefined skeleton, either using standard ML libraries such as Scikit-learn, CatBoost, and TabPFN or filling in AutoML-specific skeletons for LightAutoML or FEDOT. The ValidatorAgent executes the code and checks success and behavioral adequacy; if the result is invalid, the ImproverAgent revises the code using the full error trace and previous solution. This Planner → Generator → Validator → Improver loop repeats until the Validator marks the solution as valid (Lapin et al., 17 Jul 2025).

In parallel, an AutoML route exists through the AutomlAgent. This route chooses a backend through an AutoML router, asks the LLM to generate configuration parameters, constructs the corresponding AutoML pipeline, and returns predictions, code, and report. The InterpreterAgent produces a non-technical summary of the problem, preprocessing steps, models, key hyperparameters, and final metrics. A verbal description of the architecture in the paper situates the user and UI on the left, routing and agentic orchestration in the middle, and predictions, exportable scripts, technical logs, and non-technical summaries on the right (Lapin et al., 17 Jul 2025).

3. LLM-mediated orchestration and multi-AutoML integration

The LLM is central to LightAutoDS-Tab, but its role is highly structured rather than unconstrained. It is used for task understanding and reflection, planning, code generation, AutoML configuration generation, routing decisions, debugging, and reporting. The paper highlights two experimental LLMs, GPT-4o and GigaChat2Max. In the code-generation path, the LLM writes into predefined solution.py skeletons with comments, tool-specific segments, and restricted “USER CODE” blocks; in the AutoML path, it emits structured configuration or completes a backend-specific script (Lapin et al., 17 Jul 2025).

Prompt design is a major control mechanism. The FEDOT-specific prompt prohibits deleting comments, modifying autogenerated segments between markers, or invoking methods and attributes not already present in the code or comments; only “USER CODE” regions may be changed. The LightAutoML configuration prompt enforces a fixed JSON format with "task_type", "target", and "task_metric", with a simple rule mapping regression to task_type="reg", task_metric="r2-score" and classification to task_type="binary", task_metric="auc". The AutoML router prompt is stricter still, producing only LAMA, FEDOT, or NO, depending on whether the user explicitly requests AutoML, LightAutoML, or FEDOT. The “Fix solution” prompt passes the prior code, task reflection, dataset path, stdout, and stderr and instructs the LLM to identify and correct the specific error without altering other logic (Lapin et al., 17 Jul 2025).

The integrated AutoML backends in the experiments are LightAutoML and FEDOT. LightAutoML is described as a strong tabular AutoML framework that won Kaggle AutoML Grand Prix 2024 and was the only solution to win 2 out of 5 tasks; FEDOT is described as an AutoML framework based on evolutionary composition of pipelines. Standard ML libraries remain available in the CodeGen route. The architecture is intended to be tool-agnostic: tools are wrapped through prompts, skeleton scripts, and a common Executor, allowing future backends such as AutoGluon to be plugged in without rewriting the agent logic (Lapin et al., 17 Jul 2025).

A common misconception would be to equate “multi-AutoML” with cross-tool ensembling. The paper explicitly states that the system does not currently ensemble outputs from different AutoML tools; instead, it treats each tool as a separate backend and compares their Normalized Performance Score. The multi-AutoML aspect is therefore one of flexible integration and backend choice rather than backend fusion (Lapin et al., 17 Jul 2025).

4. Methodological details and evaluation protocol

Algorithmically, the system is defined over an input query $\mathcal{Q}$, dataset $\mathcal{D}$, and LLM $\mathcal{L}$, with state $\mathcal{S}$ initialized from message history $\mathcal{M}=[\mathcal{Q}]$. After agent instantiation, SelectAgent(S) determines whether interaction remains conversational, whether the Planner-led code loop should run, or whether the AutoML route should be invoked. Termination occurs when the chat ends or a valid solution is returned (Lapin et al., 17 Jul 2025).

The system does not hard-code a single preprocessing pipeline. Instead, the LLM’s reflection identifies feature types—IDs, numerical, categorical, and datetime—and generated code or the chosen AutoML framework performs preprocessing. The paper references typical steps such as missing-value imputation, encoding of categoricals, and scaling or normalization, while noting that LightAutoML and FEDOT handle much of preprocessing internally on the AutoML routes. Feature engineering is not elaborated in depth, but the CodeGen path can define new features and the AutoML frameworks contribute their own feature processing and selection mechanisms (Lapin et al., 17 Jul 2025).

Model selection and optimization differ by route. In the CodeGen route, the LLM chooses models and hyperparameters directly, functioning as a one-shot or iterative manual tuner. In the AutoML route, LightAutoML and FEDOT perform their intrinsic hyperparameter optimization and model selection, with LightAutoML using internal search and ensembling strategies and FEDOT using evolutionary search to assemble composite pipelines. The system itself does not add another meta-optimization layer on top of the framework’s native capabilities (Lapin et al., 17 Jul 2025).

For evaluation, the paper adopts the Normalized Performance Score from AutoKaggle:

$$\mathrm{NPS}= \begin{cases} \frac{1}{1+s}, & \text{if } s \text{ is “smaller is better”} \ s, & \text{otherwise} \end{cases}$$

where $s$ is the public leaderboard score for a competition. This converts heterogeneous metrics such as AUC, accuracy, $R^2$, RMSE, and log-loss into a common direction in which higher is better, enabling averaging across tasks (Lapin et al., 17 Jul 2025).

The benchmark follows AutoKaggle’s setup with 8 Kaggle competition datasets: Titanic, Sp. Titanic, House Prices, Monsters, Academic Success, Bank Churn, Obesity Risk, and Plate Defect. These comprise 7 classification tasks and 1 regression task, with 1 multi-target task among them. The competitions are split evenly into classic tasks from before 2024 and modern tasks from after 2024, near or beyond GPT-4o’s training cutoff, to probe whether performance reflects memorization or generalization. Inside the pipeline, the provided training data are split into train/validation in an 8:2 ratio, while reported performance is based on normalized public leaderboard scores. Baselines include AutoKaggle, AIDE, and human Q25, Q50, and Q75 leaderboard percentiles (Lapin et al., 17 Jul 2025).

5. Empirical performance and comparative position

The principal quantitative result is that the best LightAutoDS-Tab configuration, LAMA+LLM, achieves an average NPS of 0.839 across the 8 competitions, compared with 0.835 for CodeGen, 0.835 for FEDOT+LLM, 0.816 for AutoKaggle, and 0.703 for AIDE. Human reference points are reported as Q25 = 0.820, Q50 = 0.836, and Q75 = 0.841. On this aggregate measure, LAMA+LLM is slightly above the reported human median and close to the reported human Q75 (Lapin et al., 17 Jul 2025).

The per-dataset results are used to support the multi-AutoML premise that no single backend dominates. On Titanic, FEDOT+LLM reaches 0.780 and is the strongest of the three LightAutoDS configurations. On Sp. Titanic, LAMA+LLM reaches 0.798 and is best among them. On Bank Churn, CodeGen reaches 0.885 and is slightly ahead of the AutoML routes. House Prices, Monsters, Academic Success, Obesity Risk, and Plate Defect also show variability across backends, reinforcing the claim that robustness across tabular tasks benefits from having several available routes rather than binding the entire system to a single framework (Lapin et al., 17 Jul 2025).

The paper also reports an LLM ablation for the CodeGen route. GPT-4o achieves an average NPS of 0.832, whereas GigaChat2Max reaches 0.835. Dataset-level differences are small but nonuniform: GigaChat2Max is higher on Titanic, House Prices, Monsters, Academic Success, Bank Churn, and Plate Defect, while GPT-4o is higher on Sp. Titanic and Obesity Risk. The paper interprets this as evidence that CodeGen performance is sensitive to the underlying LLM and that modular LLM adapters are therefore justified (Lapin et al., 17 Jul 2025).

In comparative terms, the paper positions LightAutoDS-Tab between classic AutoML and tree-search-heavy LLM AutoDS systems. Relative to classic AutoML, it lowers the manual burden of configuration and coding by mapping natural-language task descriptions and data context into backend parameters and executable pipelines. Relative to AutoKaggle, AIDE, and tree-search systems such as SELA or I-MCTS, it does not perform formal tree search and instead relies on a more directed plan → code → execute → fix loop or on direct AutoML execution. This suggests a design preference for targeted orchestration over broad search in code space (Lapin et al., 17 Jul 2025).

6. Limitations, implementation, and prospective development

The paper identifies two explicit future directions. The first is extension beyond tabular data to time series, sequential, and other structured-data settings such as recommendation tasks. The second is a more intelligent EDA system. The current EDA is characterized as relatively light: it supports feature typing and task reflection, but it does not provide deeply automated detection of issues such as data leakage, label leakage, or severe anomalies. A plausible implication is that stronger pre-model data diagnostics would improve robustness under leakage and distribution shift (Lapin et al., 17 Jul 2025).

Several additional limitations are discussed or implied. Performance and robustness depend on the quality of the base LLM, as shown by the CodeGen ablation. The paper does not report explicit wall-clock times or resource budgets, so runtime efficiency is argued indirectly rather than quantified. Very large datasets may be challenging depending on hardware, and the system’s interpretability layer is limited to high-level reporting rather than deep integration with tools such as SHAP. At the same time, robustness mechanisms are present: constrained prompts reduce API hallucinations, the Executor–Validator–Improver loop attempts focused repairs, and the coexistence of AutoML and CodeGen routes provides fallback options at least conceptually (Lapin et al., 17 Jul 2025).

Implementation is Python-based and includes LightAutoML, FEDOT, standard ML libraries such as Scikit-learn, CatBoost, and TabPFN, and LLM API clients for GPT-4o, GigaChat2Max, and potentially others. The user interface is built with Streamlit and supports dataset upload in CSV, XLSX, and Parquet formats, dataset preview, natural-language query input in technical or business language, and visualization of agent steps with technical details on the right and simplified explanations on the left. The LADS repository is described as containing agent implementations, prompts, tool-specific skeleton scripts, router and provider configuration files, Streamlit UI code, and an inference module that generates deployment-ready code with trained model artifacts and supports standalone prediction tasks (Lapin et al., 17 Jul 2025).

Within the literature on automated data science for tabular ML, LightAutoDS-Tab is therefore best understood as a tool-aware agentic shell that uses multiple AutoML backends, constrained code generation, and structured validation to automate Kaggle-style workflows while preserving exportable code and dual technical/non-technical reporting. Its empirical case rests less on a new search algorithm than on the integration strategy itself: task reflection and pipeline synthesis are delegated to the LLM, while training and search are delegated to mature backend systems when that is advantageous (Lapin et al., 17 Jul 2025).