GANDALF: Gated Adaptive Network for Deep Automated Learning of Features

Abstract: We propose a novel high-performance, interpretable, and parameter & computationally efficient deep learning architecture for tabular data, Gated Adaptive Network for Deep Automated Learning of Features (GANDALF). GANDALF relies on a new tabular processing unit with a gating mechanism and in-built feature selection called Gated Feature Learning Unit (GFLU) as a feature representation learning unit. We demonstrate that GANDALF outperforms or stays at-par with SOTA approaches like XGBoost, SAINT, FT-Transformers, etc. by experiments on multiple established public benchmarks. We have made available the code at github.com/manujosephv/pytorch_tabular under MIT License.

• The paper introduces the GFLU module, which adaptively selects and refines features for enhanced interpretability in tabular data analysis.
• It demonstrates competitive performance against models like XGBoost by achieving high accuracy with fewer parameters and efficient computation.
• The architecture employs a gating mechanism that automatically generates feature masks, providing clear insights similar to traditional feature importance metrics.

An Analysis of GANDALF: Gated Adaptive Network for Deep Automated Learning of Features for Tabular Data

The paper presents GANDALF, a novel deep learning model specifically crafted for tabular data analysis. This architecture introduces the Gated Feature Learning Unit (GFLU), a component designed to enhance feature selection and representation learning. By addressing some of the limitations faced by other deep learning approaches for tabular data, GANDALF aims to bridge the gap between deep learning models and traditional methods like Gradient Boosted Decision Trees (GBDTs).

Architectural Overview

GANDALF innovatively leverages the gating mechanism, typically used in temporal data models like GRUs, for non-temporal tabular datasets. The GFLU processes tabular data by selecting relevant features and forming complex feature interactions. This unit employs a gating mechanism that adjustably filters input features, resulting in a refined feature representation. Notably, the architecture simultaneously ensures interpretability and computational efficiency.

The architecture stacks multiple GFLUs in a sequence, where each unit contributes to refining the feature representation gradually. GFLU's design draws inspiration from GRUs, but it uniquely adapts them by employing stage-specific weight matrices and feature masks. This capability of automatic feature selection through learned masks provides GANDALF with interpretability akin to feature importance scores seen in models like XGBoost.

Empirical Evaluation

The authors conducted extensive evaluations on two prominent benchmarks: the Tabular Benchmark and TabSurvey. Across a selection of datasets, GANDALF demonstrated performance at par with, or superior to, leading models including XGBoost and SAINT. Importantly, GANDALF achieved these results while maintaining parameter and computational efficiency, showcasing fewer parameters and Multiply-Accumulate Operations (MACs) compared to alternative deep learning models.

Moreover, GANDALF's superiority was evident in specific scenarios where feature interactions and selection played a pivotal role, which could be attributed to its specialized architecture.

Interpretability and Hyperparameters

GANDALF's feature selection mechanism allows for straightforward interpretability through aggregated feature masks, closely resembling the feature importance metrics familiar to researchers using GBDTs. The study analyzed these feature attributions using metrics like Normalized Discounted Cumulative Gain (NDCG) and MoRF/LeRF ablations, which affirmed the fidelity and reliability of GANDALF's interpretability.

When it comes to hyperparameter tuning, GANDALF is relatively simplistic, focusing primarily on the number of GFLU stages and dropout rates, contrasting sharply with the extensive hyperparameter space required by other models.

Implications and Future Directions

By outperforming other deep learning approaches while retaining computational efficiency, GANDALF represents a notable contribution to tabular data analysis. Its architecture showcases potential applications where feature selection is paramount, opening doors for more efficient and interpretable AI models in this domain.

Future research could explore optimizing GFLU configurations or integrating GANDALF with ensemble methods to further enhance performance. Additionally, exploring variations of the gating mechanism and incorporating advanced feature interaction techniques could yield deeper insights and performance gains.

In conclusion, GANDALF is a significant advancement in the field of deep learning for tabular data, merging interpretability, efficiency, and accuracy in a coherent framework. It holds promising implications for practical applications, and ongoing research may continue to expand its capabilities and applicability.