FeatGeNN: Evolutionary Auto Feature Engineering

• The paper demonstrates that correlation-pooling in convolution architectures outperforms traditional max-pooling by preserving linear relationships in tabular data.
• It integrates CNN, MLP, MRMR feature selection, and a genetic algorithm to efficiently generate and select informative features.
• Empirical results show improved F1-scores and reduced feature bloat across multiple UCI datasets compared to standard AutoFE methods.

Searching arXiv for the FeatGeNN paper and closely related context. FeatGeNN is an Automated Feature Engineering (AutoFE) method for tabular data that introduces a convolutional architecture in which correlation is used as a pooling function rather than max-pooling. It is presented as Feature Generation with Evolutionary Convolutional Neural Network and is motivated by the view that linear relationships between features are critical in tabular data and should be preserved during feature extraction. The method combines convolutional feature extraction, correlation-based pooling, a Multi-Layer Perceptron (MLP) for global feature generation, MRMR Feature Selection, and a Genetic Algorithm (GA) that evolves candidate CNN models according to downstream model performance (Silva et al., 2023).

1. Motivation and problem setting

Automated Feature Engineering aims to remove the bottleneck and subjectivity in manual feature creation for machine learning on tabular data. In the formulation associated with FeatGeNN, existing AutoFE approaches include expansion-reduction methods, reinforcement-learning agents, and neural architectures. These are described as either generating large numbers of redundant features, which can increase computational cost and risk overfitting, or failing to capture high-order, informative feature interactions effectively (Silva et al., 2023).

A central premise of FeatGeNN is that most pooling operations in previous neural approaches use max-pooling because of its success in computer vision, even though tabular data have a relational, orderless structure. In this framing, max-pooling may discard subtle but important relationships among features. FeatGeNN is therefore motivated by the hypothesis that linear relationships, expressed through correlations, should be explicitly leveraged during feature extraction for tabular learning tasks (Silva et al., 2023).

This positioning places FeatGeNN within the AutoFE literature as an alternative to feature generation pipelines that depend on exhaustive expansion and pruning. A plausible implication is that the method is intended not only to improve predictive performance but also to control feature proliferation, since the stated criticisms of prior approaches concern both redundancy and computational burden.

2. Architectural composition

FeatGeNN consists of two consecutive blocks followed by an evolutionary feature engineering loop. The first block is a Local Feature Extraction Block built from convolution layers and a novel pooling operation. Its stated role is to extract high-level local feature representations while preserving linear and strong relationships between different features. The key innovation in this block is correlation-based pooling, or correlation-pooling, implemented using the Pearson correlation coefficient instead of max-pooling (Silva et al., 2023).

The second block is a Global Feature Generation Block. It takes the pooled local features and processes them through an MLP to generate global and higher-level features. These generated features are then combined, by concatenation, with the original feature set for downstream model learning. In the reported experiments, the downstream learner is a Random Forest classifier (Silva et al., 2023).

The third component is the Evolutionary Feature Engineering Loop. A GA iteratively evolves a population of CNN models to generate increasingly better feature sets, with fitness measured by the impact of those features on model performance. The summary specifies that performance is measured by an external learning algorithm, exemplified by Random Forest and F1-score. Before model evolution, MRMR Feature Selection is applied to preselect relevant, non-redundant features in order to reduce noise and computational cost (Silva et al., 2023).

Taken together, these components define a pipeline in which feature selection precedes neural feature generation, and neural feature generation is itself optimized through evolutionary search. This suggests a hybrid methodology rather than a purely differentiable end-to-end system.

3. Correlation-based pooling

The distinctive mechanism in FeatGeNN is the replacement of max-pooling with correlation-pooling. The rationale given is that max-pooling retains only the maximum value in a neighborhood or subset and ignores correlations, whereas correlation-pooling groups and selects features that are linearly related. This is presented as better aligned with the dependencies that are crucial in tabular data (Silva et al., 2023).

The method uses the Pearson correlation coefficient between features $x$ and $y$:

$$r = \frac{n\sum xy - (\sum x)(\sum y)}{\sqrt{[n\sum x^{2}-(\sum x)^{2}][n\sum y^{2}-(\sum y)^{2}]}}$$

where $x$ and $y$ are feature vectors, interpreted as columns of the data matrix for two features $X$ and $Y$, and $n$ is the number of samples. This quantity is computed for all pairs of features, yielding a correlation matrix (Silva et al., 2023).

To quantify how correlated a feature is with all others, FeatGeNN defines a Correlation Score for feature $f$:

$CS_f = \frac{\sum_{k}^{N} r_{fk}}{N}$

where $y$0 is the Correlation Score for feature $y$1, $y$2 is the Pearson correlation coefficient between feature $y$3 and each other feature $y$4, and $y$5 is the total number of features. Features with the highest $y$6 within a local group are selected, or pooled, for the next layer (Silva et al., 2023).

The implementation details reported for scalability are specific. For computational efficiency, only a subset of data samples, for example 70–80%, is used to estimate correlations. An iterative computation is also proposed to make the process scalable. For multivariate relationships beyond simple pairs, the correlation matrix is used to inform selection (Silva et al., 2023).

The method does not claim that correlation exhausts all useful feature dependence structures. Rather, within the reported framework, it is treated as a pooling criterion more suitable than max-pooling for tabular data. The stated future direction of exploring pooling functions inspired by information theory reinforces the interpretation that correlation-pooling is one point in a broader design space.

4. Feature selection and evolutionary search

MRMR Feature Selection is used before the evolutionary process begins. Its stated purpose is to preselect relevant, non-redundant features and thereby reduce both noise and computational cost. Within FeatGeNN, this step functions as a front-end compression stage before convolutional feature extraction and GA-driven model evolution (Silva et al., 2023).

The GA is used to iteratively evolve a population of CNN models for generating increasingly better sets of features. Candidate models are evaluated according to their impact on downstream model performance, measured externally rather than by an internal reconstruction or proxy objective. In the experiments summarized for FeatGeNN, Random Forest and F1-score serve as the evaluation mechanism (Silva et al., 2023).

This arrangement differentiates FeatGeNN from AutoFE systems that rely primarily on random transformations or deterministic feature expansion followed by pruning. The combination of MRMR, correlation-aware convolutional extraction, and evolutionary search is explicitly identified as a promising paradigm for automated feature engineering on tabular data (Silva et al., 2023).

A plausible implication is that the evolutionary loop acts as a search procedure over feature-generating neural architectures rather than merely over downstream predictive models. Under that interpretation, the system is designed to optimize the representation-generating process itself.

5. Experimental protocol and empirical results

FeatGeNN is evaluated on six classification datasets from the UCI Repository that are described as popular in the AutoFE literature: SpamBase with 4601 samples and 57 features, Megawatt1 with 253 samples and 37 features, Ionosphere with 351 samples and 34 features, SpectF with 267 samples and 44 features, Credit_Default with 30,000 samples and 25 features, and German Credit with 1001 samples and 24 features (Silva et al., 2023).

The evaluation protocol uses a Random Forest classifier for all feature methods, F1-score as the metric, and 5-fold cross-validation repeated 30 times to assess stability, with averages and maxima reported. The comparisons include Random, DFS, AutoFeat, LFE, NFS, and DIFER (Silva et al., 2023).

Three research questions are highlighted in the reported results. First, FeatGeNN with correlation-pooling consistently outperformed its own max-pooling variant across almost all datasets. The clearest numerical example given is SpamBase, where F1 increased from 0.9102 on Raw features to 0.9530 with correlation-pooling, exceeding the max-pooling variant at 0.9422 (Silva et al., 2023).

Second, reducing the sample size used for correlation estimation causes only a slight loss in performance. The summary states that even with as little as 60–80% of samples, FeatGeNN remains robust and competitive (Silva et al., 2023).

Third, relative to the literature baselines, FeatGeNN achieves state-of-the-art or highly competitive results in F1-score on all datasets. Across all six datasets, it achieved the best or second-best F1-score. The reported average and maximum F1 values for FeatGeNN are 0.9530 and 0.9644 on SpamBase, 0.9151 and 0.9171 on Megawatt1, 0.9644 and 0.9713 on Ionosphere, 0.8776 and 0.8802 on SpectF, 0.8095 and 0.8102 on Credit_Default, and 0.7814 and 0.7827 on German Credit (Silva et al., 2023).

The summary also emphasizes feature economy. FeatGeNN produced the smallest or tied-for-fewest number of new features for most datasets. A specific example is provided for SpamBase, where only one new feature was added, contrasting with methods such as AutoFeat that create more features (Silva et al., 2023).

These findings support two linked claims made in the source: that correlation-based pooling is better suited than max-pooling for tabular data, and that competitive or superior model performance can be obtained while creating fewer new features.

6. Significance, limitations, and outlook

The main findings are framed around four themes. First, correlation-based pooling is described as far superior to max-pooling for tabular data because it directly leverages inter-feature relationships. Second, computational efficiency is attributed to iterative correlation computation and sub-sampling. Third, the method reduces feature bloat by creating fewer but more informative features. Fourth, it generalizes across a variety of representative tabular datasets by outperforming or matching best-in-class methods (Silva et al., 2023).

Within machine learning on tabular data, the reported implications are explicit. Neural architectures for tabular data should use correlation rather than only orderless, image-inspired pooling operations to extract high-level representations. Efficient feature creation is presented as important for predictive performance with fewer features, with possible relevance to interpretability, resource-constrained applications, and regulatory compliance. The combination of feature selection, correlation-aware convolutional feature generation, and evolutionary search is characterized as a promising paradigm for AutoFE (Silva et al., 2023).

The article’s claims should nevertheless be read in the scope of the reported evaluation: six UCI classification datasets, Random Forest as the downstream learner, and F1-score as the principal metric. This suggests that the demonstrated advantages are strongest within that benchmark regime. The authors’ stated future work is to explore pooling functions inspired by information theory beyond Pearson correlation, potentially capturing richer feature interactions for tabular machine learning problems (Silva et al., 2023).

FeatGeNN thus occupies a specific place in AutoFE research: a tabular feature generation framework built around the proposition that pooling should preserve inter-feature dependence rather than merely select local maxima. Its reported contribution is not only a new pooling rule, but a compound methodology in which MRMR filtering, correlation-aware convolution, MLP-based global feature generation, and GA-based model evolution are jointly used to produce compact and effective feature sets (Silva et al., 2023).