Evolutionary Feature Engineering (EFE)

• Evolutionary Feature Engineering (EFE) is a method that uses iterative evolutionary search to automatically select, construct, and refine features in machine learning pipelines.
• EFE integrates various candidate representations—from variable subsets to executable code—enabling tailored transformations and improved interpretability.
• Empirical results show that EFE can enhance predictive metrics across data types, though it often demands substantial computational resources due to iterative evaluation.

Evolutionary Feature Engineering (EFE) is the use of evolutionary search to automate the selection, construction, and refinement of features under downstream fitness signals. In the literature, the searched object may be a subset of variables, a transformation tree, a traversal string, a symbolic expression, a Python feature program, or a full preprocessing module with fit/transform and inverse_transform interfaces. Across these variants, EFE replaces one-shot or purely enumerative feature construction with iterative proposal, evaluation, and retention of candidates according to validation performance, while often preserving explicit feature semantics or executable code artifacts (Zhu et al., 2020, Namakin et al., 2021, Ouyang et al., 29 Oct 2025, Taga et al., 2 Jul 2026).

1. Conceptual scope and research lineage

EFE is broader than automated feature construction alone. In some formulations it denotes evolutionary search over constructed features; in others it includes feature selection, wrapper-based subset optimization, or symbolic regression over transformed representations. The defining commonality is not a single operator set, but the presence of an evolutionary loop in which candidate representations are generated, scored, and updated using fitness feedback.

System	Data regime	Salient formulation
MDR-EFE (Bulinski et al., 2016)	binary-response stratified samples	cross-validated estimation of weighted prediction error for relevant factor subsets
DIFER (Zhu et al., 2020)	tabular classification and regression	encoder-predictor-decoder refinement of traversal-string features in continuous embedding space
Correlation-aware EDA (Namakin et al., 2021)	classification feature selection	pairwise probability model over subsets with compact two-individual evolution
ELATE (Murray et al., 20 Aug 2025)	time-series forecasting	LM-generated feature code guided by Granger causality, mutual information, and SHAP pruning
LLM-FE (Abhyankar et al., 18 Mar 2025)	tabular learning	feature-program search with island memory and iterative LLM mutation
FELA (Ouyang et al., 29 Oct 2025)	industrial event logs	multi-agent idea/code/critic evolution with hierarchical idea–feature organization
EFE-Time / EFE-Tab (Taga et al., 2 Jul 2026)	structured time series and tabular data	evolution over executable fit/transform programs

This lineage also shows a shift in candidate representation. Earlier systems emphasize subsets, grammars, and symbolic programs; later systems increasingly evolve free-form executable code or stateful preprocessing modules. This suggests that recent EFE work is moving from fixed operator grammars toward open-ended program search, while still retaining explicit validation-based selection and, in several systems, strong pressure toward interpretability.

2. Representations, objectives, and search spaces

A central design choice in EFE is how a feature candidate is represented. In DIFER, a feature is a traversal string of a transformation tree. These strings are encoded into continuous embeddings $z = E(f)$, scored by a predictor $s = P(z)$, and locally improved by gradient ascent,

$$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$

before decoding back to valid traversal strings with grammar and type constraints enforced by feat_tree.py (Zhu et al., 2020).

In time-series EFE, the object of search is often a set of engineered transformations. ELATE formalizes multivariate forecasting with

$\hat{y}_{t+h} = f_{\theta}\big(\phi(x_{1:t})\big),$

and poses automated feature engineering as

$$\min_{\mathcal{T}} E\big(\phi(\mathcal{T}, X), y\big).$$

Its candidate features are executable Python code strings that map an input dataframe to new series, constrained by a causal rule that $[f(X)]_t$ only depends on past rows $X_{t',:}$ with $t' < t$ (Murray et al., 20 Aug 2025).

More recent systems search over richer program objects. FELA uses a two-layer knowledge base

$$K = \{ (I_i, \{ d_{i,j} \}_{j=1}^{M_i}) \}_{i=1}^{N_I},$$

where $I_i$ are human-readable ideas and $s = P(z)$0 are concrete feature implementations. Its global objective is written as

$s = P(z)$1

EFE-Time and EFE-Tab extend this programmatic view by representing candidates as Python classes with standardized fit/transform interfaces; for time series, inverse_transform is also required so that forecasts can be mapped back to the original scale (Ouyang et al., 29 Oct 2025, Taga et al., 2 Jul 2026).

MDR-EFE uses a different mathematical object: a prediction rule $s = P(z)$2 evaluated by weighted misclassification error,

$s = P(z)$3

There the search target is a subset of relevant factors and the induced MDR decision function, with consistency established for stratified K-fold estimates of $s = P(z)$4 (Bulinski et al., 2016).

3. Evolutionary operators, feedback signals, and search control

Classical EFE uses population-based selection, mutation, crossover, and fitness evaluation. The correlation-aware estimation-of-distribution algorithm for feature selection replaces explicit crossover with sequential sampling from a learned pairwise probability model. Given an already selected set $s = P(z)$5, subsequent feature choice is governed by

$s = P(z)$6

where $s = P(z)$7 is a significance vector and $s = P(z)$8 is an interaction matrix. The method generates only two individuals per iteration, evaluates them with an SVM-based fitness $s = P(z)$9, and updates the probability model toward the winner and away from the loser (Namakin et al., 2021).

ELATE combines evolutionary search with lightweight statistical guidance. Its addition loop samples high-scoring features using probabilities proportional to $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$0, where the temperature $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$1 decays within each generation. Candidate code is scored by Granger causality and mutual information, while generation boundaries invoke a model-based SHAP filter with recursive elimination and correlation pruning at $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$2. The system does not perform explicit crossover of code; instead it relies on LM-guided mutation and generation conditioned on sampled examples, dataset metadata, and prior scores (Murray et al., 20 Aug 2025).

LLM-FE introduces an island-based memory model for tabular program search. Feature programs are clustered within islands by score signature, few-shot exemplars are sampled by Boltzmann selection, and the LLM generates new modify_features programs by mutating successful antecedents. The result is an explicitly evolutionary prompt loop rather than one-shot prompting (Abhyankar et al., 18 Mar 2025).

FELA generalizes this further into a multi-agent search system. Idea Agents operate in three sub-modes—Feature Proposer, Idea Synthesizer, and Idea Creator—using an analyze $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$3 self-reflect $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$4 reconstruct pattern. Code Agents translate pseudocode into executable transformations. Critic Agents validate novelty, schema consistency, and technical correctness. Idea selection is governed by a UCB bandit,

$$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$5

with $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$6 the visit count of idea $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$7 and $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$8 the exploration parameter. Long-term and short-term memory support global steering and local retrieval-augmented refinement, respectively (Ouyang et al., 29 Oct 2025).

A separate line of work augments GP-based EFE with domain knowledge before evolution begins. In the LLM-augmented symbolic-regression pipeline built around M3GP and M6GP, GPT-4o supplies constructed terminals using only feature names and the target objective; these features are appended once to the terminal set $$z^{(t+1)} = z^{(t)} + \eta \nabla_z P(z^{(t)}), \qquad \eta = 10^{-4},$$9, increasing the probability that early random individuals contain informative terminals (Batista, 27 Mar 2025).

4. Domain-specific instantiations

Industrial event logs provide a setting in which EFE operates over heterogeneous, non-i.i.d. data with temporal and relational structure. FELA models logs as temporally ordered user–action events $\hat{y}_{t+h} = f_{\theta}\big(\phi(x_{1:t})\big),$0, and explicitly targets session-based features, time-windowed aggregations, sequence features such as order and inter-event time, and relational or graph features such as teammate statistics and user–item bipartite metrics. Its Code Agents can implement multi-table joins, time-windowed aggregations, clustering, and graph-derived constructs without being restricted to a predefined operator library (Ouyang et al., 29 Oct 2025).

Time-series EFE spans two distinct but related traditions. ELATE engineers covariate features for forecasting through lags, differences, rolling statistics, exponentially weighted moving averages, Fourier or spectral terms, calendar indicators, and domain-specific operators such as group-wise rolling aggregates or oscillators. By contrast, EFE-Time searches over invertible normalization programs placed directly in front of time-series foundation models. Representative transforms include robust affine scaling, arcsinh variance stabilization, log-offset mappings for positive series, and history-only detrending or seasonal adjustment; the inverse map is part of the candidate program, so the transformation is evaluated as an integrated preprocessing module rather than as a simple derived column (Murray et al., 20 Aug 2025, Taga et al., 2 Jul 2026).

Tabular EFE includes grammar-based symbolic construction, differentiable refinement, and LLM-based program evolution. The GE-plus-lasso methodology for cloud power modeling evolves engineered features through a BNF grammar and then fits a linear, convex lasso head over the resulting feature matrix. LLM-FE treats feature engineering as a program search problem over executable Python functions. EFE-Tab evolves compact feature programs that can add useful interpretable features and remove redundant ones, and is reported to be particularly effective on classical decision trees (Arroba et al., 2024, Abhyankar et al., 18 Mar 2025, Taga et al., 2 Jul 2026).

Scientific-discovery-oriented EFE has a different objective: not only predictive gain, but interpretable recovery of governing relations. In sparse-sensor fluid settings, tailored feature libraries of derivatives, invariants, and nonlinear interactions are combined with Gene Expression Programming and Sequential Threshold Ridge Regression. The system recovers canonical PDEs, truncation-error terms, hidden source terms, and a Smagorinsky closure with $\hat{y}_{t+h} = f_{\theta}\big(\phi(x_{1:t})\big),$1, showing how EFE can function as an operator-discovery pipeline rather than merely a predictor enhancer (Vaddireddy et al., 2019).

In genetics and biomedical classification, MDR-EFE and related selection methods emphasize interaction structure rather than raw feature abundance. Stratified MDR-EFE is designed for binary response variables with class imbalance, while the correlation-aware EDA addresses complementary and correlated features by learning pairwise co-selection propensities (Bulinski et al., 2016, Namakin et al., 2021).

5. Interpretability, leakage control, and governance

A recurring claim in the literature is that EFE need not be opaque. FELA makes interpretability an architectural primitive through its hierarchical idea–feature knowledge base and the structured tuple (reason, summary, pseudocode). Knowledge-base snapshots provide traceable links among original, synthesized, and newly created ideas, and correlation analysis on Taobao shows strong intra-idea and weak inter-idea correlations, consistent with semantically coherent feature families (Ouyang et al., 29 Oct 2025).

Several systems encode transparency directly in executable artifacts. ELATE stores each feature as validated Python code, while LLM-FE returns explicit transformation programs such as modify_features(df). In the physics-oriented pipeline, symbolic expressions and sparse coefficient vectors are the final output. This means interpretability is often realized not by post hoc explanation, but by retaining the actual engineered transformation or recovered governing equation (Murray et al., 20 Aug 2025, Abhyankar et al., 18 Mar 2025, Vaddireddy et al., 2019).

Leakage control is equally central. ELATE enforces a causal constraint, exposes Target_Tminus1 rather than future targets, validates generated code with an AST allow-list, and restricts rolling and group-wise operators to past indices. EFE-Time and EFE-Tab impose stage-1 validity checks for schema preservation, determinism, absence of Inf or NaN creation, and prohibition of access to validation targets or Pool B during fitting. These design choices directly counter the common misconception that LLM-based EFE is inherently unconstrained (Murray et al., 20 Aug 2025, Taga et al., 2 Jul 2026).

Governance considerations also appear in privacy-preserving LLM integration. The GP-based symbolic-regression pipeline that augments the terminal set with GPT-4o features shares only the list of feature names and the target objective, not sample values or train/test splits. This preserves privacy while still allowing domain-specific constructs such as ratios, category sums, or nonlinear age transforms to be suggested before evolutionary search begins (Batista, 27 Mar 2025).

6. Empirical evidence, limitations, and research directions

Empirical results show that EFE can improve predictive performance, but the gains are task- and formulation-dependent. On industrial event logs, FELA improves Taobao AUC from 0.630 with raw features to 0.653, and from 0.641 for LLM-FE to 0.653; on Tencent it improves AUC from 0.683 to 0.701, and from 0.686 for LLM-FE to 0.701. In the same study, FELA generated 22 features from 5 ideas over 16 hours on Taobao, and ablations removing critics or long–short-term memory reduced AUC by approximately 0.01 (Ouyang et al., 29 Oct 2025).

For time-series forecasting, ELATE reports average reductions versus Base of 8.4% RMSE and 9.6% MAE across seven domains, while EFE-Time reports average improvements of 3.0% in MASE, 3.6% in wQL, and 3.7% in MAE over 10 GIFT-Eval datasets with Chronos-2. On CovidDeaths, EFE-Time improvements reach up to 19% in wQL, 17.2% in MAE, and 10.1% in MASE. The same learned normalization programs transfer without re-evolution to TimesFM-2.5, Moirai-2-Small, and Reverso-Nano, with average MASE gains of 3.06%, 2.25%, and 2.42%, respectively (Murray et al., 20 Aug 2025, Taga et al., 2 Jul 2026).

For tabular data, LLM-FE achieves mean rank 1.54 on classification and 1.00 on regression in the reported XGBoost comparisons. EFE-Tab achieves the best mean rank for single decision trees, 1.39 versus 2.44 for LLM-FE, 3.00 for CAAFE, and 3.17 for no feature engineering, while remaining competitive on LightGBM and TabPFN. DIFER also reports substantial gains, including Ionosphere accuracy improving from 0.8520 to 0.9289 with LogisticRegression and SpectF accuracy improving from 0.7078 to 0.8200 (Abhyankar et al., 18 Mar 2025, Taga et al., 2 Jul 2026, Zhu et al., 2020).

Selection-oriented EFE likewise shows strong sparsity–performance trade-offs. The correlation-aware EDA achieved the best average accuracy on 8 of 13 datasets, selected the fewest features on 12 of 13 datasets, and dominated ACC$\hat{y}_{t+h} = f_{\theta}\big(\phi(x_{1:t})\big),$2PDF on 12 of 13 datasets, with a Friedman test $\hat{y}_{t+h} = f_{\theta}\big(\phi(x_{1:t})\big),$3. In the LLM-augmented GP pipeline, consistent improvements to test performance were observed for one-third of the datasets, while only 1 of 77 test cases showed a statistically significant decrease with LLM augmentation (Namakin et al., 2021, Batista, 27 Mar 2025).

The limitations reported across the literature are also consistent. Repeated downstream training, LLM calls, and feature computation remain expensive; FELA’s Taobao run took 16 hours, and ELATE’s 4o+SHAP averages about 3 hours. Sparse or noisy rewards degrade search efficiency; ELATE notes overfitting on ETTh1 and ILI, DIFER depends on surrogate fidelity in $\hat{y}_{t+h} = f_{\theta}\big(\phi(x_{1:t})\big),$4, and the correlation-aware EDA models only pairwise interactions. Time-series systems can fail when held-out dynamics are driven by exogenous, non-Markovian factors, and smaller LLMs are reported to be less stable for synthesizing invertible preprocessing code (Ouyang et al., 29 Oct 2025, Murray et al., 20 Aug 2025, Zhu et al., 2020, Taga et al., 2 Jul 2026).

Future directions stated in the literature include richer reinforcement learning for idea and feature search, parallel island evolution, human-in-the-loop steering, multimodal and streaming settings, adaptive prompting with domain ontologies or RAG, and more explicit resource-aware scheduling. A plausible implication is that future EFE systems will continue to combine open-ended code generation with increasingly strict validity, auditability, and parsimony controls, rather than reverting to unrestricted black-box search (Ouyang et al., 29 Oct 2025, Batista, 27 Mar 2025, Taga et al., 2 Jul 2026).