RoBERTa-Tab: Transformer for Tabular Data

• RoBERTa-Tab is a tabular data modeling framework that converts rows into token sequences using label encoding for categorical features and quantile-discretization for continuous ones.
• It employs a two-stage training process with masked token pretraining followed by classification or regression fine-tuning, leveraging transformer contextual capacity.
• The framework unifies heterogeneous datasets and offers transparent instance-level interpretability, achieving competitive AUC scores compared to traditional tree-based models.

RoBERTa-Tab is a tabular data modeling framework that applies the principles and architectures of pre-trained LLMs (PLMs), most prominently RoBERTa, to the representation and prediction challenges inherent in tabular datasets. By leveraging modality transformation, masked token pretraining, and classification fine-tuning, RoBERTa-Tab exploits semantic textualization and transformer-based contextual capacity to enable unified, flexible, and competitive performance for supervised and semi-supervised tabular learning (Liu et al., 2022, Le et al., 13 Dec 2025).

1. Architectural Foundations

RoBERTa-Tab frames each tabular row as a structured sequence suitable for transformers. Input representation proceeds as follows: for each feature (column), categorical variables are label-encoded and assigned unique token indices; continuous features are quantile-discretized (TP-BERTa’s C4.5-inspired binning) and indexed. Thus, a row $i$ maps to the token sequence $$x_i = [\text{CLS}, t_{1,b_{i1}}, \ldots, t_{D,b_{iD}}, \text{SEP}]$$, with embeddings $E[x_i]$ and position embeddings $P[1:|x_i|]$. Initial hidden states $H^{(0)}$ are the sum $E[x_i] + P[1:|x_i|]$.

The backbone is a standard RoBERTa-base transformer: 12 layers, 12 heads, hidden dimension $d=768$. At the final layer, the row-level representation $z_i$ is extracted from the [CLS] token ($z_i = H^{(L_\tau)}_{[\text{CLS}]}$). No architectural modification is made to either token or self-attention structure; semantic alignment arises from input encoding and finetuning stages.

2. Training Workflow: Masked Token Pretraining and Fine-Tuning

RoBERTa-Tab employs a two-stage training pipeline:

• Masked-token pretraining: Randomly mask 15% of tokens per sequence. The objective is to reconstruct masked tokens via cross-entropy loss: $$\mathcal{L}_{\text{mask}} = - \sum_{i} \sum_{k\in\mathcal{M}_i} \log p(x_{i,k} | x_{i,\setminus\mathcal{M}_i}; \theta_{\text{enc}})$$.
• Classification/regression fine-tuning: The final [CLS] features serve as instance representations. For classification, outputs pass through a linear layer into softmax probabilities; for regression, through a linear projection. Training uses cross-entropy (classification) or MSE (regression) loss. Only the prediction head is trainable in post-pretraining steps, while the transformer encoder remains frozen.

The modality transformation (as in PTab) is applied before tokenization: tabular rows are encoded as pseudo-sentences, with each $p_i = h_i : x_i$ for feature header $h_i$ and value $x_i$, concatenated with [SEP] delimiters (Liu et al., 2022).

3. Data Mixing, Unification, and Semi-Supervised Learning

Textualization unifies disparate tables (heterogeneous headers, categorical domains), leveraging PLM token space. Masked-token pretraining allows for natural mixing of sets from different domains—instances from various sources become “sentences” in one vocabulary, facilitating the growth of training sets without explicit feature alignment.

During semi-supervised learning, unlabeled tables from related domains are combined and transformed into textual form for mask prediction pretraining. Only small labeled subsets are used for downstream fine-tuning. Mixed-dataset masked pretraining increases target-domain AUC by approximately 0.4–0.6 points; mixing with out-of-domain tables degrades performance (Liu et al., 2022).

4. Interpretability and Feature Analysis

RoBERTa-Tab provides transparent, instance-level interpretability via its attention mechanisms. Attention maps from [CLS] to token-fields identify features driving predictions (e.g., "credit yes/no" prioritization over uninformative tokens). Visualization of embeddings shows meaningful clustering as feature values (e.g., date fields) are varied, reflecting input-sensitive, instance-based similarity analyses. These properties enable token-level explanations and relate closely to classical notions of feature importance while maintaining the resolution and expressiveness afforded by transformer attention (Liu et al., 2022).

5. Performance Benchmarks and Comparative Evaluation

RoBERTa-Tab demonstrates competitive supervised and semi-supervised performance on canonical tabular benchmarks. In five-fold cross-validation over eight binary classification datasets (BM, AD, IN, OS, ST, BC, SB, QB), average AUC scores are as follows:

Model	Supervised AUC	Semi-supervised AUC (500 labels)
XGBoost	0.861	N/A
TabTransformer	N/A	0.815
SAINT	N/A	0.815
GBDT+pseudo	N/A	0.793
RoBERTa-Tab/PTab	0.866	0.836

Mixed-dataset pretraining yields substantial AUC gains for in-domain mixing; out-of-domain mixing is detrimental (Liu et al., 2022). RoBERTa-Tab’s approach rivals or surpasses tree-based algorithms (XGBoost) and dedicated tabular neural architectures, particularly when annotation is costly or sparse.

6. Extension via Graph Priors: BOLERO

BOLERO augments RoBERTa-Tab with a static bipartite graph head, enabling explicit modeling of inter-instance relationships through graph neural network (GNN) message passing (Le et al., 13 Dec 2025). Instance nodes connect to anchor nodes representing categorical values (A_cat), continuous feature bins (A_cont), and optionally aggregated relations (A_feat). Edge weights incorporate instance-feature alignments and feature-feature pointwise mutual information (PPMI), built transductively across train/val/test splits.

Initial node features for instances are frozen [CLS] embeddings from RoBERTa-Tab; anchor embeddings are trainable. A TransformerConv GNN propagates and refines representations, with attention-based neighbor aggregation. Only graph head parameters (anchor embeddings, GNN weights, prediction head) are trainable in finetuning. No additional masked loss is used during this step.

Extensive benchmarking on 80 classification and 64 regression datasets confirms statistically significant improvements. BOLERO achieves a win rate of 62.7% in classification and 62.1% in regression against all tested baselines (including RoBERTa-Tab, TP-BERTa, TabPFN-v2, XGBoost, CatBoost), with median F1 improvements up to +0.110 and RMSE reductions up to 27.5%. These results affirm the value of lightweight graph priors in tabular transformer models (Le et al., 13 Dec 2025).

7. Context, Significance, and Limitations

RoBERTa-Tab operationalizes PLM adaptability for tabular data, exploiting semantic textualization to circumvent the rigid constraints of archaic feature alignment and numeric encoding schemes. The backbone is demonstrably robust but must be tuned carefully: mixing disparate datasets is beneficial only within sufficiently aligned domains. Furthermore, BOLERO’s bipartite graph overlay introduces inductive bias that regularizes representations and enables inter-row reasoning absent from vanilla transformers.

A plausible implication is the continued relevance of transformer models for heterogeneous tabular tasks, provided semantic, contextual, and relational priors are efficiently encoded. The typical frozen-backbone + trainable head paradigm promotes scalability across tasks and data domains. Limitations remain in terms of out-of-domain generalization, dependency on text-to-token mappings, and the necessity for rigorous multi-dataset evaluation protocols for fair comparison.

References:

• “PTab: Using the Pre-trained LLM for Modeling Tabular Data” (Liu et al., 2022)
• “Can Graphs Improve Tabular Foundation Models?” (Le et al., 13 Dec 2025)