Automated Feature Engineering

• Automated Feature Engineering is the process of automatically creating, transforming, and selecting features to improve downstream predictive performance on tabular data.
• It encompasses diverse methodologies such as transformation-graph exploration, reinforcement learning, interaction-aware search, and neural-based extraction.
• AutoFE reduces manual effort by mitigating dimensionality issues and balancing interpretability with statistical reliability in model building.

Searching arXiv for recent and foundational AutoFE papers to ground the article. tool call: arxiv_search {"query":"automated feature engineering tabular interpretable feature engineering federated knowledge graph reinforcement learning arXiv", "max_results": 10, "sort_by": "relevance"} tool output not shown in this interface? Searching for the specific papers and adjacent AutoFE work mentioned in the source material. {"query":"\"Lifting Interpretability-Performance Trade-off via Automated Feature Engineering\" OR SAFE OR AutoMAN OR OpenFE arXiv", "max_results": 10, "sort_by": "relevance"} Automated Feature Engineering (AutoFE) is the process of automatically building and selecting new features that help improve downstream predictive performance. Within the standard AutoML taxonomy, it occupies the part of the pipeline concerned with data imputation, feature transformation, and feature selection, and its practical importance is especially pronounced for tabular learning, where representation quality often determines whether linear and other classical models remain competitive without extensive manual trial-and-error (Chen et al., 2019, Wang et al., 2022, Overman et al., 2024).

1. Conceptual foundations

AutoFE is most naturally viewed as a search over transformations of the input space. In one explicit formulation, the objective is to learn an interpretable model class $G$ together with a transformation class $H$, so that

$$g = \argmin_{g \in G} \min_{h \in H} \mathcal{L}(g(h(x)), y),$$

with AutoFE corresponding to the search for useful $h$ (Gosiewska et al., 2020). In the broader AutoML survey literature, the same idea appears as a bilevel problem: choose a feature set, possibly including original and generated variables, so as to minimize validation loss after training a downstream learner on the transformed data (Chen et al., 2019).

This framing is important because AutoFE is not merely unsupervised transformation of columns. The generated representation is always judged through a downstream predictive task. Several papers therefore describe the problem as a nested process of creating new features and controlling feature set size through selection, rather than as generation alone (Azim et al., 2023). The distinction matters operationally: a system that produces many transformed columns but does not manage redundancy, instability, or selection pressure is not yet solving the full AutoFE problem.

The literature also converges on a shared motivation. Manual feature engineering is repeatedly described as expensive, time-consuming, and dependent on domain knowledge; conversely, a well-designed representation can mitigate the curse of dimensionality, improve generalization, alleviate sparsity, and make classical models more competitive (Azim et al., 2023, Wang et al., 2022). This recurring motivation explains why AutoFE remains central even in settings where the downstream learner is already strong.

2. Search spaces and feature representations

A defining axis of AutoFE research is how candidate features are represented. Classical systems typically follow an expand-and-select or expand-and-reduce paradigm: define a library of unary, binary, or compositional transforms, generate a large candidate pool, and then rank or prune (Dong et al., 2024, Park et al., 29 Apr 2026). This representation is explicit and symbolic, but it faces combinatorial growth as transform order and feature count increase.

A second representation family is model-guided but still explicitly interpretable. SAFE, for example, does not generate arbitrary symbolic expressions or latent embeddings; it learns interpretable binary features obtained by piecewise-constant transformations of numerical variables and merged-level encodings of categorical variables, extracted from a surrogate model’s response surface (Gosiewska et al., 2020). Here the engineered representation remains additive, rule-like, and human-readable.

A third family replaces explicit candidate enumeration with learned parameterizations over the transform space. AutoMAN learns local masks over raw features for each transform and a global mask over transformed outputs, thereby exploring feature-transform relationships without explicitly manifesting the full combinatorial feature pool (Dong et al., 2024). In this view, the search object is not primarily a finished feature expression, but a sparse transform graph induced by learned importance masks.

A fourth family is neural feature extraction or construction. FeatGeNN uses a CNN-style local extractor, correlation-based pooling, and an MLP-based global generator to create new tabular features, with the resulting engineered columns appended to the original dataset (Silva et al., 2023). The representation here is neither purely symbolic nor purely latent: the generated features are exported as explicit columns, but they arise from learned neural transformations.

Recent LLM-based systems push the representation further toward executable artifacts. Eureka explicitly recasts feature engineering as a code generation problem in which a feature is an executable Python program generated from a structured plan (Li et al., 24 May 2026). Rogue One likewise treats feature construction as a sequence of symbolic transformations implemented in code and stored with explanations in a feature pool (Bradland et al., 19 Nov 2025). This suggests a broader shift from “feature as transformed column” to “feature as program,” while retaining explicit formulas and provenance.

3. Methodological families

One major methodological family treats AutoFE as sequential decision making. The transformation-graph framework models feature engineering as budgeted exploration of a directed acyclic graph whose nodes are transformed datasets and whose actions apply transformations to graph nodes; the search policy is then learned with approximate Q-learning from prior datasets (Khurana et al., 2017). Later reinforcement-learning systems extend this idea in different directions: E-AFE assigns one agent per original feature, uses a Feature Pre-Evaluation model to reduce the sample size and candidate feature size seen by the expensive evaluator, and applies a two-stage policy training strategy (Wang et al., 2022); InHRecon decomposes the generation step into cascading Markov Decision Processes for operation choice, first-feature choice, and second-feature choice, with rewards informed by downstream utility and H-statistics (Azim et al., 2023).

A second family relies on information-theoretic or interaction-aware search. IIFE ranks feature pairs by interaction information,

$$\tau_{ij} = I(F_i,F_j,Y) = I(F_i,F_j \mid Y) - I(F_i,F_j),$$

and expands only the highest-scoring pairs, which turns interaction strength into a prefilter for candidate construction (Overman et al., 2024). InHRecon uses Friedman’s H-statistics to reward binary crossings whose parent features exhibit interaction strength relevant to prediction (Azim et al., 2023). These methods do not search blindly over all pairs; they use estimated synergy to decide where expansion is warranted.

A third family improves expand-and-reduce systems by controlling the search frontier before full generation. SCOPE-FE starts from the observation that unrestricted unary and binary expansion yields a candidate space of order $\mathcal{O}(p \cdot d^2)$, then reduces this space by pruning operators through OperatorProbing and restricting admissible feature pairs through FeatureClustering (Park et al., 29 Apr 2026). The methodological emphasis is upstream control of operator-feature and pairwise combinations, rather than better downstream pruning alone.

A fourth family is surrogate-guided feature extraction. SAFE first trains a surrogate black-box model, then extracts one-dimensional response summaries with partial dependence profiles, discretizes numerical variables by changepoint detection, merges categorical levels by hierarchical clustering, and finally trains a glass-box model on the transformed features (Gosiewska et al., 2020). The downstream model is trained on the original labels rather than distilling the surrogate’s predictions, so the method is better understood as surrogate-guided feature engineering than response mimicry.

Domain-specific AutoFE also appears in sequence-aware and generative settings. For credit-card fraud detection, multi-perspective HMMs are trained on eight supervised sequence partitions and their per-transaction log-likelihoods are used as additional features for a downstream classifier (Lucas et al., 2019). The essential idea is that a generative model can be used as a feature constructor rather than as the final predictor.

4. Interpretability, semantics, and transparent feature generation

Interpretability in AutoFE is not a single notion. In autofeat, the objective is to improve linear regression and linear classification through a large pool of nonlinear candidate features followed by aggressive feature selection, so that the final predictor remains a linear model on explicit symbolic expressions (Horn et al., 2019). In SAFE, interpretability is operationalized through binary interval indicators and grouped categorical levels whose coefficients remain directly attached to readable conditions such as age intervals or merged credit-history levels (Gosiewska et al., 2020).

A notable development is the move from post hoc interpretability to interpretability-aware generation. KRAFT formalizes feature interpretability with a binary knowledge-graph-based function

$I_{KG}: Dom^F \rightarrow \{0,1\},$

and constrains the selected feature set so that every retained feature is judged interpretable by Description Logics and SWRL rules (Bouadi et al., 2024). In this framework, adding features with incompatible units or applying semantically invalid aggregations can be rejected during feature generation rather than merely explained afterward. SMART adopts a related but softer view: feature interpretability is the cognitive effort required by domain experts to understand generated features and align them with domain knowledge, and the overall objective is scalarized as a weighted combination of predictive performance and interpretability (Bouadi et al., 2024).

These knowledge-informed systems enlarge the meaning of “interpretable AutoFE.” The focus is no longer only sparsity or symbolic simplicity, but semantic validity with respect to units, ontology concepts, and domain rules. In SMART, the state seen by the DQN is a semantic vectorization derived from a knowledge graph, and the reward includes both model improvement and feature interpretability (Bouadi et al., 2024). In KRAFT, the discriminator is not a neural critic but a symbolic reasoner, so semantic admissibility becomes part of the search process itself (Bouadi et al., 2024).

The literature also treats the interpretability-performance trade-off as contingent rather than fixed. SAFE explicitly argues that extracting information from complex models may improve the performance of linear models and questions the common myth that complex machine learning models outperform linear models (Gosiewska et al., 2020). This does not mean the trade-off disappears universally, but it does mean AutoFE can sometimes move a model to a better point on the interpretability-performance plane than either raw linear modeling or opaque high-capacity models alone.

5. Efficiency, scalability, and federated execution

Efficiency is a first-class concern in current AutoFE research. E-AFE argues that the dominant cost in modern AutoFE is not feature generation itself but repeated feature evaluation: in its timing example, only about $0.1\%$ of time is spent on generation, whereas about $90\%$ is spent on evaluation (Wang et al., 2022). Its response is to reduce both the number of rows used in pre-evaluation and the number of candidate features that reach the expensive downstream evaluator. AutoMAN pursues a different route: it avoids explicit candidate-pool materialization altogether, learns transform-conditioned masks end-to-end, and claims overall feature-engineering complexity $O(Nmk)$, with especially large latency advantages on high-dimensional data such as Isolet (Dong et al., 2024).

Expand-and-reduce systems have motivated another efficiency line based on search-space control. SCOPE-FE identifies the candidate explosion of OpenFE-style pipelines as $H$0, then reduces the pairwise term to $H$1 for fixed target cluster size $H$2 by clustering features before pair generation (Park et al., 29 Apr 2026). Its empirical results report large runtime reductions relative to OpenFE, especially on high-dimensional datasets such as MI, while maintaining competitive predictive performance. IIFE reaches a related conclusion from a different direction: better pair preselection via interaction information can materially reduce search without broad candidate explosion (Overman et al., 2024).

Time-varying and distributed settings extend the same systems question. AutoMAN adds temporal masks and temporal transforms so that the framework can operate on time series without flattening each time step into an ordinary tabular column (Dong et al., 2024). Federated Automated Feature Engineering goes further by asking how AutoFE should operate when raw data are partitioned across clients and cannot be centralized. It introduces Horizontal-FLAFE, Vertical-FLAFE, and Hybrid-FLAFE, and reports that in the horizontal setting the downstream test scores of the federated AutoFE algorithm are close in performance to the case where data are held centrally and AutoFE is performed centrally (Overman et al., 2024). In that setting, a central idea is to separate local candidate generation from communication-aware federated selection.

LLM-based AutoFE introduces another efficiency profile. Eureka uses an LLM-driven three-stage loop—Expert Agent, LLM Feature Factory, and Self-Evolving Alignment Engine—but the reported training budget is 100 epochs over 6 hours on 4 NVIDIA A100 GPUs (Li et al., 24 May 2026). Rogue One likewise incurs substantial runtime because repeated XGBoost evaluation dominates total cost, with the appendix reporting a strong correlation between runtime and dataset size (Bradland et al., 19 Nov 2025). This suggests that agentic and code-generating AutoFE trades search flexibility and semantic richness for higher systems overhead.

6. Evaluation, statistical reliability, and usability

AutoFE evaluation is methodologically delicate because the feature space is itself selected using the data. One line of work therefore focuses on statistical validity rather than search alone. The selective-inference framework for tree-search-based AutoFE shows how to test generated features in a downstream linear model while accounting for the fact that the same response vector guided feature generation. Conditioning on the event that the AutoFE algorithm returned the observed feature set yields selective $H$3-values with exact finite-sample type-I error control under the stated assumptions (Matsukawa et al., 2024). This directly addresses the “double-dipping” problem that ordinary post-selection significance tests ignore.

A different methodological critique concerns empirical benchmarking. IIFE identifies several recurrent issues in the AutoFE literature: reporting cross-validation scores as final performance, allowing transductive leakage in feature generation, and failing to retune hyperparameters before and after feature engineering (Overman et al., 2024). The broader AutoML survey also notes that AutoFE lacks a standard protocol with a fixed set of transformation operations and identical training settings for generated solutions (Chen et al., 2019). These observations help explain why direct comparison across papers is often less stable than leaderboard-style summaries suggest.

Usability has emerged as a separate, and sobering, evaluation axis. The survey “How Usable is Automated Feature Engineering for Tabular Data?” studies 53 methods and reports that 26/53 had no code, 27/53 were open source, only 11/27 open-source methods were usable in the authors’ test pipeline, only 10/27 provided documentation, only 7/27 had an active community, and 0/27 supported the time-and-memory constraints the survey regards as necessary for usable automation (Schäfer et al., 19 Aug 2025). The conclusion is unusually strong: under that definition, no surveyed method qualified as a truly usable AutoFE system. This suggests that AutoFE remains, to a large extent, a research field with fragmented software rather than a mature engineering ecosystem.

Taken together, these evaluation results indicate that AutoFE is now shaped by three distinct questions: whether the generated features improve prediction, whether the search process is statistically reliable and leakage-free, and whether the resulting system can actually be installed, controlled, and trusted in practice. The current literature offers substantial progress on the first question, growing attention to the second, and a clear warning on the third (Matsukawa et al., 2024, Overman et al., 2024, Schäfer et al., 19 Aug 2025).

AutoFE now spans a wide methodological spectrum: transformation-graph exploration, interaction-aware search, surrogate-guided discretization, differentiable masking, knowledge-graph-constrained generation, sequence-model-derived features, LLM-based feature programs, and federated search. This suggests that “automating feature engineering” is no longer a single algorithmic problem, but a family of design choices about representation, search, semantics, systems constraints, and statistical validation. A plausible implication is that the next advances will come less from any single search heuristic than from tighter integration across these axes: interaction-aware generation, semantic constraints, inference-aware evaluation, and usable software.