EFE-Tab: Evolutionary Feature Engineering

• EFE-Tab is a framework for static binary classification that evolves compact, stateful Python feature transformers using a standardized fit/transform interface.
• It employs an evolutionary search using a large language model to propose and optimize interpretable feature programs, balancing improvements in ROC-AUC with penalties for complexity and runtime.
• Empirical results show that EFE-Tab enhances performance for shallow decision trees and standard learners, providing human-readable transformations that integrate seamlessly with scikit-learn pipelines.

Searching arXiv for the specified topic and source paper. arxiv_search.query({"search_query":"all:EFE-Tab OR ti:\"Evolutionary Feature Engineering for Structured Data\" OR id:(Taga et al., 2 Jul 2026)","start":0,"max_results":5}) I found the primary relevant paper on "Evolutionary Feature Engineering for Structured Data" (Taga et al., 2 Jul 2026), which introduces EFE-Tab. EFE-Tab is the tabular prediction instantiation of Evolutionary Feature Engineering (EFE), a framework that treats feature engineering for structured data as evolutionary search over executable preprocessing programs. In EFE-Tab, a LLM acts as an open-ended variation operator that proposes Python programs with a standardized fit/transform interface; each candidate is inserted immediately upstream of a fixed downstream learner and scored by held-out ROC-AUC improvement, regularized by feature-set complexity and runtime. The explicit target is not unrestricted feature synthesis, but compact, interpretable feature programs that add useful features, remove redundant ones, and remain leakage-free under fold-based evaluation (Taga et al., 2 Jul 2026).

1. Definition and problem setting

EFE-Tab is formulated for static tabular binary classification. Given a dataset $D=\{(x_i,y_i)\}_{i=1}^n$, a candidate program $p$ is evaluated across $K$ validation instances, where each fold defines a train/validation split denoted Pool A and Pool B. On fold $j$, the program first fits state $\sigma_{j,p}$ on Pool A only, then transforms both pools with that same fitted state, and finally trains a fixed learner $A$ on the transformed Pool A representation before computing ROC-AUC on transformed Pool B. The objective optimized during evolution is the gain over the identity program $p_{id}$, with explicit penalties for parsimony and runtime (Taga et al., 2 Jul 2026).

This formulation places EFE-Tab in a different design space from automated feature engineering systems based on fixed grammars or latent embeddings. The search object is a full Python program implementing a feature transformer, not a symbolic tree constrained by a domain-specific language. The use of a standardized fit/transform interface makes the method directly compatible with existing scikit-learn-style pipelines and permits stateful preprocessing, including transformations whose parameters must be estimated from data.

A recurrent misconception is that EFE-Tab is primarily a model-selection framework. In the reported setup, the downstream learner is fixed during search; the optimization target is the preprocessing program. Another misconception is that it is intended only for maximizing predictive performance regardless of feature-set size. The scoring function explicitly penalizes both added and dropped columns, so compactness is part of the optimization target rather than a post hoc preference.

2. Program representation and feature transformations

Each candidate is a Python program implementing a standardized class, termed FeatureProgram, with methods

• fit(self, X: pd.DataFrame, y: Optional[pd.Series], context: dict) -> None or state dict
• transform(self, X: pd.DataFrame) -> pd.DataFrame

This representation is deliberately executable rather than symbolic. The evaluator runs the emitted Python code in a sandbox, allowing rich stateful operations while preserving a uniform insertion point in the machine-learning pipeline. Because the interface mirrors scikit-learn’s fit/transform pattern, a program can be wrapped as a transformer or used in a precompute-and-pass workflow.

Allowed operations include real-valued transforms such as arithmetic, powers, sqrt, log/exp, clipping or winsorization, and robust scaling by median and MAD; interactions and ratios such as $x_i \times x_j$ and $x_i / (x_j + \epsilon)$; nonlinear transforms and binning including quantile bins, equal-width bins, and threshold indicators; categorical handling through one-hot encoding, target encoding fitted on Pool A and reused on Pool B, and rare-level grouping; grouped aggregates keyed by categorical columns; column dropping; and basic sanitization including missing-value imputation and dtype consistency (Taga et al., 2 Jul 2026).

Several constraints govern admissible programs. The transform must return a DataFrame with the same number of rows as the input, preserve row order, avoid in-place mutation, and produce a consistent schema across Pool A and Pool B after fitting on Pool A. Data-dependent statistics, such as target-encoding maps or grouped rates, must be computed strictly on Pool A and then frozen. Access to Pool B labels or evaluator internals is disallowed, and candidate programs must be deterministically executable and numerically safe.

The emphasis on interpretable code is central. Feature programs are human-readable sequences of transformations such as ratios, threshold indicators, bins, grouped means, and target encodings. This differs from methods that construct opaque latent representations, because the final transformation can be inspected, version-controlled, and audited directly.

3. Evolutionary search, context, and evaluator safeguards

EFE-Tab uses OpenEvolve with a MAP-Elites-style archive keyed by code diversity, three islands, and fixed selection ratios: exploitation $0.70$, exploration $p$0, and elite selection $p$1. At each iteration, the system selects parent code and inspiration examples from the archive, prompts the LLM with parent source code, inspiration snippets, dataset context, and aggregate evaluator feedback, and then validates and scores the newly proposed program before archive update (Taga et al., 2 Jul 2026).

The search loop can be summarized as follows. First, an identity program initializes the archive. At iteration $p$2, the controller selects a parent program $p$3 and inspiration examples conditioned on the search history $p$4. The LLM receives the parent code, inspiration snippets, recent feedback, and Pool-A-derived dataset context, and emits a complete modified program $p$5. Static validation checks function signatures, schema consistency, determinism, and other constraints. If valid, the program is fitted on Pool A, used to transform both pools, and evaluated by training the fixed learner on transformed Pool A and scoring on transformed Pool B. The resulting score, feedback, and code are inserted into the archive, which retains the current best-so-far program.

The evaluator deliberately limits information exposure. At each iteration, EFE-Tab resamples a $p$6 split of the training data into Pool A and Pool B. These pools vary across iterations to induce diversity and reduce overfitting. The LLM is exposed only to Pool A summaries and aggregate scoring numbers from Pool B; it never sees raw Pool B rows or labels. Pool-A context includes dataset name, shape, target column, problem type, class balance, column-level statistics, target correlations, top inter-feature correlations, and a small number of sample rows.

The safeguards are designed to enforce leakage-free feature construction. fit may access $p$7 and context, but transform may not access $p$8 on either pool. The evaluator rejects programs that mutate inputs in place, generate inconsistent schemas, or attempt to inspect forbidden information. The feedback returned to the LLM is restricted to aggregate AUCs and Pool-A-only interpretability artifacts such as permutation importance, a shallow decision tree trained on transformed Pool A, and correlation summaries.

4. Objective function, regularization, and pruning logic

For binary classification, the per-fold improvement of program $p$9 over the identity baseline is

$K$0

Averaging over folds gives

$K$1

The combined score optimized during evolution is

$K$2

where $K$3 counts both generated and dropped features, $K$4 is the number of training rows, and $K$5 is runtime (Taga et al., 2 Jul 2026).

The use of $K$6 is significant. EFE-Tab does not treat pruning as a separate postprocessing stage. Instead, parsimony is built into the objective itself: every added or dropped column carries cost. The resulting search pressure favors programs whose transformed representation improves validation ROC-AUC enough to justify the complexity it introduces or removes.

A common misconception is that redundant-feature removal is governed by a fixed correlation rule. That is not the reported mechanism. EFE-Tab does not impose a global correlation threshold, nor does it use mutual information or Gini impurity in the evaluator. Pruning emerges from the parsimony penalty together with Pool-A-only feedback, particularly permutation importance, top inter-feature correlations, and a small human-readable decision tree fitted on the transformed representation. When correlation is reported, it is the standard Pearson correlation

$K$7

The AUC used in scoring is the area under the ROC curve and may be written as

$K$8

Because the objective is explicitly relative to the identity program, EFE-Tab measures marginal utility over the unmodified feature set rather than absolute performance alone.

5. Empirical performance and interpretability profile

The reported experiments evaluate EFE-Tab on nine TabArena binary-classification datasets spanning telecom churn, HR analytics, e-commerce, banking, healthcare, and sports. During evolution, the fixed learner is TabPFN-v2. Final evaluation is conducted with TabPFN-v2, LightGBM, and single decision trees. Baselines are No FE, CAAFE, and LLM-FE, all run under the same 100-iteration LLM-query budget per fold; results are averaged across the three folds from the first official repetition (Taga et al., 2 Jul 2026).

The strongest reported effect appears with single decision trees. EFE-Tab achieves the best mean rank across nine datasets, with mean rank $K$9, compared with $j$0 for LLM-FE, $j$1 for CAAFE, and $j$2 for No FE. On individual datasets, it attains the best average ROC-AUC on Churn, HRAnalyticsJobChange, E-CommerceShippingData, in_vehicle_coupon_recommendation, BankMarketing, Diabetes, and Fitness_Club. Online_shoppers_intention and Bank_Customer_Churn are exceptions in which LLM-FE slightly wins, while EFE-Tab remains competitive.

The reported learner-level ranking pattern is:

Downstream learner	EFE-Tab mean rank	Reported characterization
Single decision tree	1.39	Best; particularly effective
LightGBM	1.94	Best; separation smaller
TabPFN	2.22	Best; methods closer

The analysis attributes this profile to model alignment. Simple interactions, discretizations, and compact engineered variables can yield decision boundaries that shallow trees exploit efficiently. A depth-sensitivity study shows that the gains are largest on shallower decision trees; with depth $j$3, EFE-Tab surpasses no-feature-engineering baselines that require substantially deeper trees to match the same AUC. This indicates that the evolved programs often trade raw feature complexity for more easily separable transformed representations.

With TabPFN, the lift is largest in low-data regimes and shrinks as the training set grows. This is consistent with the claim that useful feature construction can compensate for limited data when the downstream learner is strong but not fully invariant to redundant or poorly structured inputs. The pool-fraction sweep further shows an inverted-U behavior: too little Pool A yields noisy or unrepresentative fitted statistics, whereas too much Pool A reduces iteration-to-iteration diversity.

The interpretability claim is not merely rhetorical. Programs are code rather than latent embeddings, and the final selected transformations are compact in practice relative to CAAFE and LLM-FE. The paper reports that EFE-Tab programs were compact yet matched or exceeded the baselines’ ROC-AUC, especially in settings where small readable trees are a deployment requirement.

6. Practical adoption, limitations, and broader context

The reported default configuration for adoption is approximately 100 iterations per fold with a $j$4 Pool A/Pool B split. If interpretability is paramount, single decision trees at small depth pair best with EFE-Tab. For a compromise between accuracy and readability, LightGBM also benefits. In low-data regimes, TabPFN benefits most from the evolved features. The parsimony weight $j$5 is the main control for the performance-versus-feature-count trade-off, and the evaluator’s permutation-importance report, shallow decision tree, and correlation summaries are intended to guide pruning and refinement (Taga et al., 2 Jul 2026).

Several limitations are explicit. Not every dataset benefits. Gains can be small when raw features already align well with the task or when the downstream learner is robust to redundancy. The largest lifts occur in low-data regimes; the gap narrows as data volume increases. Some generated features may be numerically fragile, for example under division by near-zero, although validator checks and iterative feedback mitigate this. The paper also notes bias risk: column semantics may induce features that encode sensitive attributes, so human review is recommended before deployment.

From a systems perspective, EFE-Tab is a form of leakage-aware evolutionary program synthesis for preprocessing. It inherits the flexibility of executable Python while constraining state estimation to Pool A and using only aggregate Pool-B feedback. This suggests a division of labor in which the downstream learner remains conventional and off-the-shelf, while representational adaptation is concentrated in a compact, auditable preprocessing layer.

Within the broader EFE framework, EFE-Tab is paired with EFE-Time, which targets invertible, dataset-specific normalizations for time-series forecasting. The contrast clarifies the tabular specialization: EFE-Time searches over history-only normalizations for time-series foundation models, whereas EFE-Tab searches for compact feature programs for static tabular prediction. In that sense, EFE-Tab is best understood not as a general-purpose AutoML system, but as a program-search method for interpretable feature construction under explicit complexity and leakage constraints.