Embeddings for Tabular Data: A Survey (2302.11777v1)

Abstract: Tabular data comprising rows (samples) with the same set of columns (attributes, is one of the most widely used data-type among various industries, including financial services, health care, research, retail, and logistics, to name a few. Tables are becoming the natural way of storing data among various industries and academia. The data stored in these tables serve as an essential source of information for making various decisions. As computational power and internet connectivity increase, the data stored by these companies grow exponentially, and not only do the databases become vast and challenging to maintain and operate, but the quantity of database tasks also increases. Thus a new line of research work has been started, which applies various learning techniques to support various database tasks for such large and complex tables. In this work, we split the quest of learning on tabular data into two phases: The Classical Learning Phase and The Modern Machine Learning Phase. The classical learning phase consists of the models such as SVMs, linear and logistic regression, and tree-based methods. These models are best suited for small-size tables. However, the number of tasks these models can address is limited to classification and regression. In contrast, the Modern Machine Learning Phase contains models that use deep learning for learning latent space representation of table entities. The objective of this survey is to scrutinize the varied approaches used by practitioners to learn representation for the structured data, and to compare their efficacy.

• The paper surveys methodologies for embedding tabular data, categorizing them into classical (SVM, tree-based) and modern deep learning (CNN, GNN, Transformer) phases, detailing their benefits and limitations.
• It identifies key challenges in tabular data embedding, including data quality issues, complex feature dependencies, heterogeneous data types, and the need for domain-specific adaptations.
• The survey discusses how models view tables as images, graphs, or sentences, leveraging techniques from other domains to address challenges and support downstream tasks like classification, retrieval, and question answering.

The paper "Embeddings for Tabular Data: A Survey" provides a comprehensive analysis of methodologies and challenges associated with embedding tabular data, which is a crucial and widespread data format within numerous industries, such as finance, healthcare, logistics, and climate science. Given the growing complexity and size of databases, effectively embedding tabular data has become essential for executing various computational tasks within databases.

The paper categorizes the development in the field into two distinct phases:

1. Classical Learning Phase: This includes traditional machine learning paradigms such as Support Vector Machines (SVMs), linear and logistic regression, and decision-tree-based methods including Random Forests, AdaBoost, Gradient Boosting, and XGBoost. These methods are typically effective for small to medium-sized datasets and are primarily used for tasks like classification and regression. Limitations of these methods include the need for significant feature engineering and their limited application scope, being mostly centered around structured data problems.
2. Modern Machine Learning Phase: This phase harnesses the power of deep learning, offering increased flexibility and the ability to handle large datasets. Techniques discussed include embedding models that treat tables in various modalities—such as text, images, and graphs—and leverage advanced architectures like Graph Neural Networks (GNNs), Convolutional Neural Networks (CNNs), and self-attention-based Transformers. The paper specifically examines models like EmbDi, URLNet, TaBERT, and TURL, which utilize deep learning for capturing latent representations of tabular data entities across different conceptions.

The paper identifies distinct challenges related to learning from tabular data:

• Data Quality: Issues such as imbalanced data distributions, missing values, and noisy data sources hamper learning models.
• Complex Feature Dependencies: Tabular data often involves intricate dependencies both within columns and across different columns, complicating the learning process.
• Heterogeneous Data Types: Managing data that consists of mixed types, such as numerical, categorical, and text fields, poses significant pre-processing and interpretability challenges.
• Domain-Specific Vocabulary: Contextual and semantic differences across various industries require specialized model adaptations to effectively learn meaningful embeddings from tables.

Several methodologies have emerged to address these challenges by viewing tables as images, graphs, or collections of sentences:

• Image-Based Models: These transform tables into image-like structures that can be processed through CNNs. While capable of capturing spatial correlations, they may fail to capture long-term dependencies.
• Graph-Based Models: These represent tables as graphs, using relational structures to learn embeddings. Challenges include complex feature engineering requirements and potential difficulties in managing heterogeneous data.
• Sentence-Based Models: These leverage NLP-based techniques, viewing tables as linear text sequences, to utilize powerful pre-trained transformer models such as BERT and its variants for enhanced representational learning.

The survey also elucidates a variety of downstream tasks that benefit from these embeddings, such as classification, regression, link prediction, question answering, table retrieval, semantic parsing, and metadata discovery. It highlights datasets commonly used for these tasks and underscores their importance in understanding model performance and applicability.

In conclusion, the paper serves as a thorough review of current methods employed to learn embeddings for tabular data, discussing both traditional approaches and advanced deep learning techniques. It provides insights into the benefits and limitations of different methods, illustrating the critical aspects of effectively exploiting structured data for a range of computational tasks.