Transformer-Based AutoEFT for Feature Engineering

• The paper presents a novel approach that integrates a lightweight, decoder-only GPT-FT with reinforcement learning to automate feature transformation for enhanced model performance.
• It employs a continuous embedding space and gradient-ascent search, significantly reducing parameter count and inference time compared to traditional LSTM or encoder-decoder methods.
• Empirical results on diverse tabular datasets demonstrate up to 6% improvement in predictive metrics, confirming the method's scalability and practical utility.

Transformer-Based AutoEFT encompasses a class of algorithms for automated feature transformation (AutoEFT) that integrate transformer architectures—specifically, decoder-only variants such as GPT—with reinforcement learning (RL) and continuous embedding optimization. The core objective is to discover feature transformation sequences that enhance downstream model performance (e.g., F1, 1–RAE) via efficient, scalable, and parameter-light frameworks. Recent work, notably the "GPT-FT" framework, addresses the computational limitations of previous encoder-decoder or LSTM-based approaches by leveraging a single-layer, multi-head transformer design for both sequence reconstruction and model performance estimation, while supporting direct gradient-based search in a learned transformation embedding space (Gao et al., 28 Aug 2025).

1. High-Level Problem Formulation

Feature transformation aims to optimize data representations by synthesizing new features from base inputs via functionally rich, compositional expressions (often in postfix form, e.g., "x1 x2 + x3 *"), thereby maximizing a downstream objective $R(\mathcal{M}(\widetilde{\mathcal{F}}), y)$, where $\mathcal{M}$ is a predictive model and $\widetilde{\mathcal{F}}$ is the transformed feature set. The combinatorial search space of valid transformation sequences, and the non-trivial mapping from feature engineering to improved predictive accuracy, renders discrete search strategies inefficient and sample-inefficient.

Transformer-based AutoEFT recasts this as a continuous, differentiable process: first, by constructing a joint embedding space for transformation sequences, and second, by enabling direct optimization of these embeddings with respect to an internal proxy for downstream performance, as estimated by the model's auxiliary head (Gao et al., 28 Aug 2025).

2. Four-Stage Transformer-Based AutoEFT Workflow

The GPT-FT methodology operationalizes AutoEFT in four distinct but interlocked stages:

1. Transformation Records Collection: The framework decomposes sequence generation into three collaborative Markov Decision Processes (MDPs)—a head-feature agent, operation agent, and tail-feature agent—that incrementally build syntactically valid postfix transformation sequences. An RL loop maximizes downstream performance, collecting $(\gamma_i, v_i)$ pairs where $\gamma_i$ encodes a transformation sequence and $v_i$ the associated model metric.
2. Embedding Space Construction via Revised GPT: Using a parameter-reduced, decoder-only GPT variant, termed GPT-FT, transformation sequences are embedded via a single self-attention layer and projected into a compact $d$-dimensional latent. This architecture supports two parallel output heads: the Text Predictor $ψ$ (for sequence reconstruction) and the Task Classifier $δ$ (for predicted metric estimation), jointly optimized with a weighted sum of negative log-likelihood (NLL) and mean squared error (MSE).
3. Gradient-Ascent Search in Embedding Space: The top-$k$ embedding vectors from collected high-performing sequences serve as seeds. For each, the embedding $\mathcal{M}$0 is updated iteratively via $\mathcal{M}$1, leveraging the differentiable performance head to identify regions of the embedding manifold associated with higher predicted task performance.
4. Autoregressive Sequence Reconstruction: Refined embeddings $\mathcal{M}$2 are processed by $\mathcal{M}$3 to generate candidate transformation sequences, which are validated, executed, and scored by a downstream model. The best-performing transformation is then selected for deployment.

This closed-loop approach replaces discrete search with a learned, continuous optimization process, integrating model-based RL, compact transformer-based sequence models, and explicit performance-driven search (Gao et al., 28 Aug 2025).

3. Architectural Modifications and Model Design

GPT-FT departs significantly from canonical GPT-1-type models:

Component	Standard GPT-1	GPT-FT (AutoEFT)
Transformer Depth	12 layers	1 layer
Embedding Dimensionality	768	64
Attention Heads	12	1
Output Heads	1 (Text)	2 (Text + Task Classifier)
FFN Block per Head	2-layer MLP	1-layer linear

The architecture eschews encoders, using a decoder-only stack to minimize parameter count and computational overhead. The Text Predictor $\mathcal{M}$4 is responsible for autoregressive sequence output, while the Task Classifier $\mathcal{M}$5 models $\mathcal{M}$6 given latent embedding $\mathcal{M}$7. Both are trained end to end using the composite loss:

$\mathcal{M}$8

with $\mathcal{M}$9 controlling the tradeoff between reconstruction fidelity and performance regression.

This design achieves a 50% reduction in model memory footprint compared to LSTM-based seq2seq models (e.g., MOAT) and full-scale GPT-1, with per-transformation models ranging from 0.08 MB to 7.08 MB (Gao et al., 28 Aug 2025).

4. Mathematical Formulation and Optimization

Let $\widetilde{\mathcal{F}}$0 be a transformation sequence. The embedding process produces $\widetilde{\mathcal{F}}$1, where $\widetilde{\mathcal{F}}$2 represents the learned embedding generator plus positional encoding and attention.

Continuous improvement is achieved via gradient-based search in embedding space:

$\widetilde{\mathcal{F}}$3

where $\widetilde{\mathcal{F}}$4 provides a differentiable proxy for downstream performance.

The sequence generation process is defined autoregressively as

$\widetilde{\mathcal{F}}$5

Decoding uses greedy or beam search to maximize $\widetilde{\mathcal{F}}$6.

The loss function for model training:

• Sequence reconstruction: $\widetilde{\mathcal{F}}$7
• Performance regression: $\widetilde{\mathcal{F}}$8

$\widetilde{\mathcal{F}}$9 is selected by sensitivity sweeps, with optimal values empirically found in $(\gamma_i, v_i)$0 (Gao et al., 28 Aug 2025).

5. Computational Efficiency and Scalability

Transformer-based AutoEFT leverages several efficiency improvements:

• Parameter Efficiency: GPT-FT is $(\gamma_i, v_i)$150% smaller than LSTM-seq2seq counterparts.
• Attention Complexity: $(\gamma_i, v_i)$2 per self-attention layer vs. $(\gamma_i, v_i)$3 in encoder-decoder setups.
• Inference Time: Yields 18–41% faster throughput relative to competitive baselines—for example, 27.9 s vs 34.1 s on Ozone, 39.4 s vs 67.2 s on Tecator.
• Scalability: The single-layer, decoder-only design parallelizes efficiently and enables practical real-time or batch use on large tabular datasets.

Ablation studies document that both RL-based record generation and gradient-ascent in embedding space are critical; omitting either degrades performance, confirming the need for both discrete exploration and continuous optimization (Gao et al., 28 Aug 2025).

6. Empirical Performance

Experiments on fifteen public tabular datasets (Kaggle, OpenML, UCI; 9 classification, 6 regression) establish that Transformer-Based AutoEFT (GPT-FT):

• Outperforms all baselines (RDG, ERG, LDA, AFAT, NFS, TTG, GRFG, DIFER, MOAT) in F1-score and 1–RAE;
• Achieves up to +6% relative improvement versus MOAT in predictive metrics across tasks;
• Achieves consistent parameter and inference-time reductions;
• Maintains performance for $(\gamma_i, v_i)$4 and a single transformer layer, eliminating need for deeper stacks;
• Remains robust across a wide range of datasets and task complexities.

This confirms the scalability and practical strength of transformer-based AutoEFT for automated feature engineering in tabular domains (Gao et al., 28 Aug 2025).

7. Future Directions

Promising avenues for follow-on research include:

• Scaling transformer-based AutoEFT to massive or streaming datasets via sparse-attention or Mixture-of-Experts augmentations;
• Integration of privacy-preserving computation (e.g., secure multiparty computation or homomorphic encryption) to address data sensitivity requirements;
• Advancing cross-domain generalization of transformation prompts and benchmarking robustness in sequence reconstruction tasks.

By unifying parameter-efficient transformer architectures, differentiable embedding optimization, and explicit task performance modeling, Transformer-Based AutoEFT establishes a generalizable and computationally tractable paradigm for next-generation automated feature engineering pipelines (Gao et al., 28 Aug 2025).