Evolutionary Feature Engineering for Structured Data

Abstract: LLMs are increasingly used as open-ended search operators in evolutionary optimization. We introduce Evolutionary Feature Engineering (EFE), a framework for using LLM-based evolution to discover preprocessing transformations for structured data. EFE represents transformations as Python programs with a standardized fit/transform interface, allowing them to be inserted directly into existing machine learning pipelines. During evolution, candidate programs are refined using dataset context, summary statistics, and downstream performance feedback on validation set. We instantiate EFE in two settings. For time-series forecasting, EFE-Time learns invertible, dataset-specific normalizations that improve off-the-shelf time-series foundation models. It reduces forecasting errors (MASE, WQL, MAE) 3% or more when averaged across datasets and improvements are as much as 19% on the COVID-Deaths dataset. Notably, these improvements occur with recent TSFMs such as Chronos-2. For tabular prediction, EFE-Tab evolves compact feature programs that add useful interpretable features and remove redundant ones, improving or matching existing LLM-based feature-engineering methods. We found EFE-Tab to be particularly effective on classical decision trees, where small sets of evolved features yield competitive accuracy while preserving interpretability. Overall, EFE demonstrates that LLM-based evolution can improve both accuracy and interpretability when automatically tackling structured data.

• The paper introduces a novel framework that uses LLMs as evolutionary optimizers to create dataset-specific, executable preprocessing programs.
• The method evolves stateful fit/transform pipelines with explicit parsimony penalties, achieving performance gains such as improved MASE and AUC.
• Key implications include model-agnostic transferability and additive benefits with fine-tuning, supporting enhanced interpretability and scalability.

Evolutionary Feature Engineering for Structured Data: An Expert Review

Introduction

This paper introduces Evolutionary Feature Engineering (EFE), a framework that leverages LLMs as evolutionary optimizers to automate the discovery of dataset-specific preprocessing programs for structured data. EFE evolves executable, stateful feature transformations, represented as Python programs with a fit/transform interface, to augment downstream predictive models without modifying their architecture or training protocol. The framework is specialized in two distinct domains: time-series forecasting (EFE-Time), where it learns invertible data normalizations, and tabular classification (EFE-Tab), where it generates compact, interpretable feature transformations. By combining LLM-driven program synthesis and selection logic based on direct validation efficacy and explicit parsimony penalties, EFE proposes a scalable and interpretable approach to feature engineering.

Figure 1: Given a dataset and optional context, EFE uses metadata, statistics, and past feedback to prompt an LLM to propose fit/transform programs. Candidates are evaluated against the identity baseline, and scores are fed back into an evolutionary loop.

Methodology

The EFE framework conducts search over the space of valid, executable preprocessing programs for structured data, distinguishing itself by evolving full-featured fit/transform (and inverse_transform for time-series) pipelines rather than stateless or fixed-library transformations. The iterative evolutionary loop is operationalized with the following cycle:

1. The current population of program candidates is paired with feedback from prior evaluations, dataset context (including summary statistics and metadata), and select inspiration programs.
2. An LLM, prompted with this information, proposes a modified candidate program.
3. The candidate is verified for validity (executability, schema preservation, leakage prevention) and inserted before the fixed downstream model.
4. The candidate is evaluated on a hold-out validation protocol; its score, along with feedback and diagnostics, is returned to the evolutionary loop.
5. The selection strategy is driven by downstream validation performance, with penalties for complexity, runtime, and failure cases, using composite scoring functions specialized for each setting.

EFE-Time evolves invertible normalizations for time-series inputs, requiring the programs to include fit, transform, and inverse_transform methods. The downstream model, typically a time-series foundation model (TSFM), operates in the transformed space, and predictions are mapped back to the original space for evaluation.

EFE-Tab, in contrast, evolves compact feature engineering programs for tabular data, enforcing a parsimony penalty on additional or dropped features. Candidate programs are scored by the improvement in predictive performance (typically AUC) relative to the raw-feature baseline, after being processed by simple, interpretable models like decision trees in the inner-loop, but downstream evaluation is also performed for gradient-boosted trees and foundation models.

Time-Series Forecasting (EFE-Time): Results and Analysis

Synthetic and Real-World Performance

The authors construct a robust experimental framework for EFE-Time utilizing the GIFT-Eval benchmark suite, with Chronos-2 as the primary TSFM. Notably, they illustrate that even state-of-the-art TSFMs struggle with sequences exhibiting smooth exponential growth—a setting where standard normalizations (e.g., RevIN, arcsinh scaling) are inadequate. EFE-Time evolves invertible normalizations tailored to the underlying dynamic structure of each dataset.

Empirically, EFE-Time achieves mean improvements of 3–4% in MASE, WQL, and MAE across multiple real-world and synthetic datasets, with up to 19% improvement on the challenging Covid-Deaths dataset. Performance gains are dataset-dependent, with no significant improvement observed for datasets that are already regular or not normalization-sensitive.

Figure 2: The change of validation and test MASE during EFE-Time evolution for Covid-Deaths.

Figure 3: The change of validation and test MASE during EFE-Time evolution for Solar-hourly.

Figure 4: MASE $10.68 \rightarrow 1.63$, wQL $0.038 \rightarrow 0.006$, MAE $4810 \rightarrow 735$.

Program evolution is visualized as an incremental, sometimes non-monotonic process. Evolution-tree analyses demonstrate that high-performing programs tend to stabilize through robust affine or nonlinear history normalization and increasing sophistication such as branching and winsorization, with certain runs converging to strong solutions while others plateau in suboptimal basins.

Generalization and Additive Effects with Fine-Tuning

A salient finding is the cross-model transferability of EFE-Time-evolved transformation programs. Programs optimized for Chronos-2 transfer without modification to other TSFMs (TimesFM-2.5, Moirai-2-Small, Reverso-Nano), consistently providing mean improvements in MASE and related metrics regardless of model architecture. This supports the claim that forecasting failures are often rooted in raw data characteristics rather than model-specific artifacts.

An additional interaction term is uncovered when combining EFE-Time with model fine-tuning. The resulting additive effect outperforms either approach in isolation. In domains where the data normalization is highly effective, evolved transformations can even exceed fine-tuning alone in improving forecast accuracy, highlighting the practical relevance of advanced preprocessing in model deployment workflows.

Numerical Results: Per-Series and Aggregate Effects

Highly detailed per-series analyses corroborate aggregate score improvements, with examples of individual series where MASE is reduced by an order of magnitude after EFE-Time normalization. Nevertheless, improvement is not uniformly distributed: datasets such as restaurant visitor counts or resource allocation traces, dominated by exogenous or non-Markovian factors absent from input history, exhibit negligible or no gain.

Figure 5: Evolution trees for the three $\mathtt{covid\_deaths}$ runs (seeds $1$, $2$, $3$ from left to right). All three seeds locate programs that cross the 10% improvement line.

Tabular Feature Engineering (EFE-Tab): Results and Analysis

EFE-Tab is evaluated using nine datasets from the TabArena benchmark. The key design principle is parsimony: programs are penalized for generating extraneous features or dropping informative ones, focusing evolution on transformations with demonstrable benefit. Comparison is made against state-of-the-art LLM-driven feature generation methods, CAAFE and LLM-FE.

EFE-Tab achieves the strongest mean rank (1.39 versus 2.44 for LLM-FE) when evaluated with shallow decision trees, demonstrating that compact feature programs can match or outperform larger feature sets in predictive gain while maintaining transparency. This effect is especially pronounced in low-data and shallow-model regimes—scenarios where classical feature expansion leads to overfitting or signal dilution. The effectiveness of EFE-Tab decreases modestly for TabPFN, likely due to foundation models' robustness against redundant features.

Figure 6: Test AUC vs. tree depth.

This parsimony is not simply theoretical: inspection reveals that EFE-Tab-induced improvements arise from interpretable feature constructs (ratios, thresholds, binned encodings) that align with domain knowledge, a property desirable for real-world deployment and auditability. EFE-Tab is also robust in low-sample scenarios and exhibits optimal trade-offs in data partitioning during evolution.

Implications and Future Directions

Practical Implications

EFE demonstrates that LLM-driven program evolution is not only feasible but beneficial when applied to the open-ended space of executable preprocessing pipelines. For practitioners deploying models in environments where transparency and model stability are essential, EFE-time provides a rigorously validated framework for automating data normalization—often neglected or heuristically addressed in typical MLOps workflows.

On tabular data, EFE-Tab represents a credible path toward interpretable, data-efficient feature selection—a constraint increasingly emphasized in regulated domains. Both specializations of EFE clearly outperform current state-of-the-art LLM-based feature engineering frameworks in their respective regimes.

Theoretical and Methodological Implications

The formalization of feature engineering as evolutionary search over the set of valid, stateful preprocessing programs is a notable contribution. The modularity of the approach—fixing the downstream routine and optimizing only the data representation—provides strong causal attribution for performance improvements and facilitates ablation studies.

The demonstration of additive benefits between evolved preprocessing and downstream model adaptation (such as fine-tuning) suggests that feature space reconfiguration and parameter-space adaptation are orthogonal, opening the way for hybrid optimization pipelines.

Limitations and Prospective Work

• EFE relies on powerful LLMs to design valid, nontrivial fit/transform (and inverse) pipelines; non-frontier models show instability, particularly on more complex time-series domains.
• The evolutionary process uses fixed hyperparameters and feedback structures; future work should address adaptive control for exploration–exploitation balance and systematically ablate feedback mechanisms.
• Although leakage and validity constraints are checked, further study is required to ensure that automatically synthesized preprocessing does not inadvertently encode dataset idiosyncrasies or bias.

Conclusion

Evolutionary Feature Engineering (EFE) advances the state of automated model-agnostic feature engineering by casting the process as LLM-driven evolutionary search over fit/transform program space. EFE-Time and EFE-Tab validate the paradigm's efficacy in both time-series and tabular domains, yielding interpretable, transferable, and high-utility preprocessing pipelines. Through comprehensive scoring, parsimony enforcement, and extensive benchmarking, the paper establishes EFE as a strong baseline for future program-synthesis-guided feature engineering, with robust implications for both theory and practice in structured data learning. The results motivate further exploration into adaptive evolutionary heuristics, LLM resource efficiency, and regulatory-driven auditing of synthesized preprocessing pipelines.