---
title: 'AutoMLGen: LLM-Based MLE Coding Agent'
url: https://www.emergentmind.com/topics/automlgen
type: topic
---

# AutoMLGen: LLM-Based MLE Coding Agent

AutoMLGen denotes an LLM-based coding agent for machine learning engineering tasks such as Kaggle competitions and MLE-Bench. In its 2025 formulation, it integrates a curated ML domain knowledge base, Monte Carlo Graph Search (MCGS), and fine-grained operators to draft, debug, improve, fuse, and ensemble end-to-end pipelines under a fixed budget [2510.08511]. The term also overlaps with a broader AutoML discourse in which “generalized AutoML” means fully automated machine learning without expert involvement or expert knowledge; that stronger notion is explicitly distinguished from the intermediate, human-scaffolded systems that dominate current practice [1811.03822].

## 1. Conceptual scope and historical placement

AutoML literature has long separated incremental automation of parts of the machine-learning workflow from the stronger ideal of full autonomy. One influential taxonomy divides the field into **narrow AutoML** and **generalized AutoML**. Narrow AutoML covers intermediate techniques that automate portions of the pipeline but still depend on expert-designed outer loops, search spaces, priors, or meta-procedures. Generalized AutoML is defined as fully automated ML without expert involvement or expert knowledge, and is explicitly linked to AGI or “strong AI” [1811.03822].

Within that taxonomy, the named system AutoMLGen belongs to the narrow operational regime. It is specialized for **Machine Learning Engineering (MLE)** settings such as Kaggle and MLE-Bench, where success depends on iterative metric optimization, feature engineering, training strategies, and competition heuristics rather than merely producing runnable code. Its design assumes a curated knowledge base, a bounded operator set, and a search budget of fixed duration, all of which place it well inside the contemporary agentic-AutoML landscape rather than the stronger AGI-adjacent ideal of generalized AutoML [2510.08511].

This distinction matters because AutoMLGen is sometimes read as a claim to complete automation. The underlying research record suggests otherwise. The broader AutoML literature repeatedly argues that most existing systems remain dependent on human configuration at the outermost layer, and that additional automation often arrives with additional computational burden rather than with the elimination of expert priors [1811.03822]. A plausible implication is that AutoMLGen should be understood less as the endpoint of AutoML and more as a high-performance system for a demanding, code-centric subdomain of it.

## 2. System architecture and execution model

AutoMLGen is organized as an end-to-end coding agent for MLE tasks. Its inputs are a task \(T\), including problem description, data access pattern, evaluation metric, submission format, and constraints. Its core components are an **ML Domain Knowledge Base**, an **LLM backbone**, **Monte Carlo Graph Search**, and a sandboxed execution environment. The outputs are the best pipeline \(s^*\) found under budget, final submission files, and optionally ensembles of top-\(K\) solutions [2510.08511].

The execution loop begins with initialization. The task description is parsed; relevant priors \(R_{KB}(T)\) are retrieved from the knowledge base; and the LLM uses the task together with retrieved knowledge to draft initial pipeline candidates. Search then proceeds over a graph of candidate solutions. Each node stores a pipeline together with code and metadata. Each iteration performs selection, expansion, simulation, and backpropagation: a candidate node is selected; a fine-grained operator expands it into a new candidate; the new code is executed to obtain a task metric and validity signal; and rewards are propagated along the primary search path [2510.08511].

The search space is formalized over complete ML pipelines. States are full solutions, not isolated parameters. Actions are operator applications such as drafting, debugging, feature enhancement, competition-strategy modification, fusion, and ensembling. This makes AutoMLGen broader than classical configuration-only AutoML systems: it searches over code-level design decisions spanning preprocessing, model choice, training, inference, and submission generation rather than only over model families and hyperparameters.

The implementation used **DeepSeek-R1-0528** as the backbone model. The execution environment runs candidate code locally, performs validation through Kaggle-style evaluation scripts, records metrics and errors, and maintains branch-level and global memories for later reuse and ensembling. The resulting architecture is neither a monolithic planner nor a pure tree searcher; it is a graph-augmented search-and-execution system whose state is grounded in actual code runs rather than symbolic plans alone [2510.08511].

## 3. Domain knowledge base and fine-grained operator design

A central feature of AutoMLGen is its **domain knowledge base**, which is organized along three axes: **model-level knowledge**, **data-level knowledge**, and **strategy-level knowledge**. Model-level knowledge maps task and domain types to suitable model families and backbones, with usage guidelines. Data-level knowledge covers modality-specific preprocessing and feature-engineering principles. Strategy-level knowledge encodes “Kaggle-style” recipes such as test-time augmentation, ensembling strategies, pseudo-labeling, and cross-validation heuristics. The sources are described as open-source repositories, competition discussions, and manually curated materials [2510.08511].

The knowledge base is retrieved as \(R_{KB}(T)\) and injected into prompting both at initialization and during later refinement stages. This is not an incidental convenience layer. It is intended to mitigate cold-start failures, improve refinement quality, and stabilize convergence by supplying the LLM with domain priors that ordinary code-generation agents lack. In the reported ablation, a baseline MCTS+LLM agent without the knowledge base achieved a medal rate of **40.91%** on MLE-Bench-Lite, whereas adding the knowledge base increased medal rate to **50.00%** [2510.08511].

Search behavior is mediated by a set of **fine-grained operators** rather than by coarse whole-program rewrites. The operator set includes **Draft**, **Debug**, **Improve-Normal**, **Improve-FE**, **Improve-CS**, **Fusion**, **Code Review**, and **Ensemble**. Draft creates initial solutions or new branches. Debug repairs non-executable code with minimal localized edits. Improve-Normal adjusts standard performance-related settings such as learning rate or batch size. Improve-FE targets features and preprocessing. Improve-CS injects competition strategies such as pseudo-labeling or improved cross-validation. Fusion merges information from multiple nodes. Code Review checks for issues such as data leakage or metric-task mismatch. Ensemble combines top-\(K\) solutions near the end of search [2510.08511].

This operatorization is important because it decomposes MLE into semantically distinct edit types. A plausible implication is that AutoMLGen gains not only from stronger priors, but from a better factorization of the action space than generic code agents employ. The literature on adjacent systems reaches similar conclusions in other forms: LightAutoDS-Tab uses specialized planner, generator, validator, improver, and AutoML-router agents for tabular tasks, while NNGPT separates architecture synthesis, HPO, prediction, retrieval, and RL into distinct pipelines [2507.13413][2511.20333].

## 4. Monte Carlo Graph Search

AutoMLGen’s search algorithm is **Monte Carlo Graph Search**, a graph-augmented variant of MCTS. The search structure is a directed graph \(G = (V, E)\) with \(E = E_T \cup E_{\text{ref}}\). Nodes are candidate solutions. **Primary edges** \(E_T\) represent generative parent-child relations and form the tree backbone used for selection and backpropagation. **Reference edges** \(E_{\text{ref}}\) connect nodes across branches and levels for information flow and reuse, but do not participate in credit assignment [2510.08511].

Selection remains tree-based. It traverses only primary edges by UCT, using node visit counts, accumulated rewards, smoothing \(\varepsilon\), and an exploration constant set to **1.414**. The novelty lies in expansion. AutoMLGen implements four expansion modes: **primary expansion**, **intra-branch evolution**, **cross-branch reference**, and **multi-branch aggregation**. Primary expansion behaves like standard MCTS. Intra-branch evolution lets the operator inspect recent nodes in the same branch, enabling reflection over local trajectory history. Cross-branch reference exposes the current branch to globally strong nodes from elsewhere in the search graph. Multi-branch aggregation creates a new branch from a union of top trajectories across branches, enabling explicit multi-solution fusion [2510.08511].

Backpropagation is intentionally restricted to primary edges. This preserves the stability and interpretability of MCTS-style credit assignment while allowing the graph structure to influence generation through references. The result is a hybrid between tree-guided exploration and graph-mediated knowledge sharing. The design is explicitly motivated by limitations of linear or tree-structured search, where information transfer is local and strong solutions discovered in one branch remain isolated from others [2510.08511].

The contrast with earlier AutoML search paradigms is sharp. AlphaD3M casts pipeline synthesis as a single-player game with edit operations over pipeline primitives and uses MCTS guided by a learned policy-value model, but its search remains tree-structured over pipeline edits [2111.02508]. GAMA exposes pluggable random search, ASHA, and asynchronous evolutionary search over scikit-learn pipelines, emphasizing transparency and modularity rather than cross-branch recombination [2007.04911]. AutoMLGen extends the search space to code-level MLE decisions and extends the search structure from trees to graphs.

## 5. Benchmarking and empirical performance

AutoMLGen was evaluated on **MLE-Bench**, described as a benchmark of **75 Kaggle competitions** spanning NLP, vision, signal processing, and tabular tasks, with **22 low-complexity**, **38 medium**, and **15 high** complexity tasks. The reported environment used **32 Intel Xeon vCPUs**, **230GB RAM**, **1× NVIDIA A800 GPU**, and a **12-hour** budget per task, averaged over **3 seeds** [2510.08511].

On the full MLE-Bench under that 12-hour budget, AutoMLGen achieved an **average medal rate of 36.4 ± 1.2 %**, with **62.1 ± 3.0 %** on low-complexity tasks, **26.3 ± 2.6 %** on medium, and **24.4 ± 2.2 %** on high. It also achieved a **valid submission rate of 96.4 ± 0.4 %**, a **Median+ rate of 48.4 ± 1.2 %**, and a **gold medal rate of 18.7 ± 0.8 %**. The reported comparison places it above the second-best average medal rate, **Neo at 34.2%** with **36h** runtime, and above **ML-Master at 29.3%** average medal rate under the same **12h** budget [2510.08511].

On **MLE-Bench-Lite**, AutoMLGen reached **62.1 ± 3.0 %** medal rate, ahead of reported baselines including **MLZero: 36.4%**, **MLE-Star: 43.9 ± 6.2%**, **AIRA-dojo: 47.7%**, and **KompeteAI: 51.5 ± 1.5%** [2510.08511]. For high-complexity tasks, the paper notes that medal rates are similar to ML-Master and Neo at **24.4%**, but AutoMLGen shows a higher win rate in average scores, suggesting stronger average solutions across that subset [2510.08511].

The ablation study isolates the contributions of the knowledge base and the graph search design. On MLE-Bench-Lite with a single seed, the baseline MCTS+LLM agent achieved **40.91%** medal rate, **68.18%** Median+, and **65.33%** beat ratio. Adding the knowledge base raised these to **50.00%**, **77.27%**, and **68.59%**. Adding intra-branch MCGS raised them further to **59.09%**, **81.82%**, and **73.20%**. Full AutoMLGen with knowledge base and full MCGS reached **68.12%** medal rate, **86.36%** Median+, and **78.33%** beat ratio [2510.08511].

These results matter because they attribute AutoMLGen’s gains to structural components rather than to the base LLM alone. A common misconception is that the system’s performance is reducible to a stronger model backend. The ablation record indicates that curated priors, trajectory reuse, cross-branch reference, and multi-branch aggregation are all substantial contributors.

## 6. Related systems, misconceptions, and limits

AutoMLGen sits at the intersection of several research trajectories. One line emphasizes transparent, modular AutoML frameworks such as **GAMA**, which exposes pluggable search algorithms and post-processing for scikit-learn pipelines [2007.04911]. A second line treats pipeline synthesis as structured search, exemplified by **AlphaD3M**, which uses self-play, sequence models, and MCTS over pipeline edit operations [2111.02508], and by grammar-constrained variants that prune the search space for valid pipelines [1905.10345]. A third line combines LLMs with existing AutoML tools, as in **LightAutoDS-Tab**, which routes among LLM-generated code, LightAutoML, and FEDOT for tabular tasks [2507.13413]. A fourth line pushes toward autonomous ML-engineering agents, such as **KompeteAI**, which adds a merging stage, RAG, and a predictive scoring model that accelerates pipeline evaluation **6.9 times** [2508.10177], and **NNGPT**, which uses a closed loop of generation, assessment, and self-improvement for neural-network development [2511.20333].

AutoMLGen’s distinctive contribution within that landscape is its explicit use of graph-structured search for code-level MLE optimization. It does not merely search over pipeline templates, nor does it only orchestrate external AutoML frameworks. Instead, it treats candidate solutions as nodes in a graph, permits cross-branch knowledge flow through reference edges, and combines those search mechanics with a domain knowledge base and operator-specialized prompting [2510.08511].

The strongest conceptual caution concerns its name. In the broader AutoML literature, generalized AutoML denotes the final goal of fully automated ML without expert involvement or expert knowledge, and that goal is tied to unresolved AGI-scale obstacles [1811.03822]. The 2025 system called AutoMLGen is not that. It relies on a curated knowledge base, task-specific operator design, MLE-Bench-style local evaluation, and a fixed 12-hour wall-clock search. This suggests that the term functions more as a productively ambitious label than as a literal realization of generalized AutoML.

Its reported limitations are correspondingly concrete. Performance depends on foundation-model quality. The system incurs substantial compute and time cost, requiring up to **12h per task** with GPU-backed execution. Its knowledge base is tuned to Kaggle-like ML engineering, and the paper notes possible brittleness on very novel domains or on tasks insufficiently covered by the curated priors. Future directions include extending beyond MLE-Bench, introducing more explicit multi-step and decomposed code generation, and broadening the domain scope beyond the current competition-centric setting [2510.08511].

Source: https://www.emergentmind.com/topics/automlgen