MambaTab: A Plug-and-Play Model for Learning Tabular Data (2401.08867v2)

Abstract: Despite the prevalence of images and texts in machine learning, tabular data remains widely used across various domains. Existing deep learning models, such as convolutional neural networks and transformers, perform well however demand extensive preprocessing and tuning limiting accessibility and scalability. This work introduces an innovative approach based on a structured state-space model (SSM), MambaTab, for tabular data. SSMs have strong capabilities for efficiently extracting effective representations from data with long-range dependencies. MambaTab leverages Mamba, an emerging SSM variant, for end-to-end supervised learning on tables. Compared to state-of-the-art baselines, MambaTab delivers superior performance while requiring significantly fewer parameters, as empirically validated on diverse benchmark datasets. MambaTab's efficiency, scalability, generalizability, and predictive gains signify it as a lightweight, "plug-and-play" solution for diverse tabular data with promise for enabling wider practical applications.

• The paper introduces MambaTab, a plug-and-play model for tabular data that reduces parameter count by over 99% compared to transformer-based methods.
• The paper demonstrates MambaTab's efficacy in both vanilla supervised learning and feature incremental learning through extensive benchmarking across eight datasets.
• The paper highlights MambaTab's low preprocessing requirements and scalability, offering a robust solution for dynamically evolving datasets across various domains.

Introduction

In the landscape of machine learning, tabular data persists as the central format across various domains: industrial, healthcare, academic, among others. Deep learning models, including CNNs and transformers, have been extensively adopted for tabular data, leading to remarkable performances. Nonetheless, these techniques entail significant computational resources, extensive preprocessing, and hyperparameter tuning, creating accessibility and scalability constraints. Existing solutions also typically fall short when confronted with feature incremental learning (FIL)—a scenario where features within the dataset increase over time. This highlights the demand for innovative solutions that support the continuous evolution of datasets.

State-of-the-Art and Motivation

The paper contextualizes the challenge by surveying the present solutions, which fall into different categories: classical machine learning models, deep learning approaches based on CNNs and transformers, and more recently, the adoption of self-supervised learning strategies. Deep learning models like TabNet, AutoInt, and TabTransformer represent the most recent advancements for tabular data, leveraging attentions and embeddings to manage categorical and numerical features. Yet, the divergence from simpler models towards these complex architectures exacerbates the need for extensive tuning and data manipulation. Notably, almost all current methods operate under vanilla supervised learning, with limited capacity to handle FIL. The research necessitates an architecture that can operate efficiently in dynamic feature environments without retraining from scratch when new data enters the scene.

Novel Approach: MambaTab

The authors propose a novel solution: MambaTab, based on structured state-space models, specifically exploiting the Mamba SSM variant for handling tabular data in an end-to-end supervised learning setting. MambaTab stands out due to its parameter efficiency, low preprocessing requirements, and innate support for FIL. The ability of the Mamba framework to deal with long-range dependencies and its linear scalability sets MambaTab apart from the conventional deep learning models, notably reducing the number of parameters, typically by more than 99%, in comparison to transformer-based solutions. The paper's empirical evaluation across various benchmark datasets illustrates MambaTab's superior performance, evidencing its capability as a lightweight and adaptable methodology for practitioners dealing with tabular data.

Benchmarking MambaTab

In an extensive empirical paper spanning eight public datasets and two different learning contexts—vanilla supervised learning and FIL—MambaTab consistently outstripped state-of-the-art baselines. For vanilla supervised learning, MambaTab provided superior or competitive performances across datasets while utilizing a fraction of the parameters required by other models. In the domain of FIL, the methodology demonstrated a seamless adaptation without the need for complex restructuring or significant parameter tuning.

Conclusion and Future Work

MambaTab is set forth as a transformative approach for tabular data, offering not only reduced complexity but also an 'out-of-the-box' solution for environments where datasets continually evolve. Its ability to deliver across a spectrum of domains and dataset structures without the encumberment of labor-intensive preprocessing establishes it as a robust candidate for broad applications. Looking ahead, the authors aspire to extend their work into regression tasks, further broadening the scope of MambaTab's utility. Through continued refinement and extension, MambaTab offers the potential to mitigate current challenges and propel the next wave of machine learning for tabular data.