REaLTabFormer: Relational & Tabular Transformer

• REaLTabFormer is a Transformer-based synthetic data generator that handles both flat and relational tabular data with explicit overfitting safeguards.
• It employs a two-stage architecture using a GPT-2 autoregressive model for parent tables and a Seq2Seq decoder for child tables with column-aware tokenization.
• Evaluations demonstrate that it sustains strong utility and fidelity on larger datasets despite higher computational demands compared to alternatives.

REaLTabFormer, short for Realistic Relational and Tabular Transformer, is a Transformer-based synthetic data generator for both non-relational tabular data and relational tabular data organized as a parent table with a one-to-many child table linked by a key. It was introduced to address a gap between strong single-table synthesizers and comparatively weaker methods for relational synthesis, while also incorporating explicit mechanisms for overfitting detection and reduced data copying (Solatorio et al., 2023). Subsequent work positioned it within the broader class of Transformer-based tabular data synthesis (TDS) models and examined how changes in the underlying decoder-only LLM affect runtime, machine-learning utility, and similarity to real data distributions, particularly under prosumer hardware constraints (R et al., 25 Sep 2025).

1. Definition and problem setting

REaLTabFormer is a machine-learning based, generative, explicit density, and tractable density model in the taxonomy used by the sensitivity study (R et al., 25 Sep 2025). Its purpose is to generate synthetic tabular data that preserves statistical structure and, in the relational setting, dependency structure across tables. The original formulation emphasizes two target regimes. In the first, rows are treated as independent observations in an ordinary tabular dataset. In the second, synthesis must respect a parent-child relationship, which requires modeling both the distribution within each table and the dependence structure across linked tables (Solatorio et al., 2023).

The motivating problem is not only realism but also controlled generalization. Tabular data often contains mixed data types, missing values, skew, and complex feature interactions, while relational datasets add one-to-many mappings that are difficult to reproduce with methods designed for a single flat table. The original paper therefore frames REaLTabFormer as a unified transformer framework for tabular and relational synthesis, whereas the later sensitivity analysis frames it as a representative Transformer-based TDS tool whose computational burden may be reducible without sacrificing data quality (Solatorio et al., 2023, R et al., 25 Sep 2025).

A common misconception is that REaLTabFormer is only a single-table generator similar to GReaT. The published description does not support that interpretation. GReaT is used as a comparison point for row-to-text autoregressive generation, but REaLTabFormer extends beyond a single table through a second-stage conditional generator for relational data (R et al., 25 Sep 2025).

2. Two-stage generative architecture

The core design is a two-part framework. For a parent table or any non-relational table, REaLTabFormer uses a GPT-2-style autoregressive transformer. For relational synthesis, it adds a Seq2Seq Transformer that generates the child table conditioned on the parent table (Solatorio et al., 2023).

In the parent-table stage, each row

$o_i = [x_{i1}, x_{i2}, \ldots, x_{in}]$

is treated as a token sequence, and generation follows an autoregressive factorization of the form

$$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$

with row likelihood conceptually expressed as

$$P(o_i) = \prod_{j=1}^{n} P(x_{ij} \mid x_{i1}, \ldots, x_{i,j-1}).$$

The original implementation uses GPT-2 with 768-dimensional embeddings, 6 decoder layers, and 12 attention heads, trained as a causal LLM in Hugging Face Transformers (Solatorio et al., 2023).

For relational data, child rows linked to a given parent are concatenated into a sequence

$s_i = [o_i^1, o_i^2, \ldots, o_i^n],$

and the child generator models

$$x_{ij}^n \sim P(X \mid o_i^1, \ldots, x_{i1}^n, x_{i2}^n, \ldots, x_{ij-1}^n, C_k).$$

The pretrained parent GPT-2 model is reused as the encoder, its weights are frozen, and a transformer decoder is trained to autoregressively emit child-table values conditioned on the parent context. The original paper argues that freezing the encoder speeds training, transfers parent-table knowledge directly, reduces the number of parameters that must be trained, and allows the encoder to act as a learned contextual representation of the parent table (Solatorio et al., 2023).

In the later sensitivity study, this overall REaLTabFormer pattern is preserved, but the underlying LLM is varied by architecture type, number of layers, hidden dimension, and overall parameter count. The study evaluates decoder-only LLM families from Hugging Face, including GPT-2, GPT-Neo, GPT-NeoX, GPT-J, GPT-BigCode, and LLaMA, with lighter and heavier layer configurations such as GPT-2 with 6 and 12 layers, GPT-Neo with 2, 4, 6, 8 layers, GPT-NeoX with 1 layer, GPT-J with 1 layer, GPT-BigCode with 6 and 12 layers, and LLaMA with 1 and 2 layers (R et al., 25 Sep 2025).

3. Tabular representation, vocabulary design, and constrained sampling

A defining feature of REaLTabFormer is its fixed column-aware tokenization rather than the use of a large generic LLM vocabulary. Each unique token in a column receives its own token id in a column-specific fixed vocabulary. The stated consequences are reduced irrelevant vocabulary, faster training and sampling, and fewer invalid outputs (Solatorio et al., 2023).

The preprocessing strategy is intentionally minimal and mostly text-based. Categorical values become tokens directly. Missing values are left as-is so that the model can learn their distribution and generate or impute them. Datetime values are converted to Unix timestamps and then treated like numeric values. Numeric values undergo a text-based normalization pipeline: round to a fixed precision, cast to strings, left-align using the magnitude of the largest value, right-pad shorter values with zeros, move the negative sign to the leftmost position, and split into fixed-length string partitions (Solatorio et al., 2023). This is presented as an alternative to lossy numeric discretization schemes such as GMMs or quantization.

During generation, REaLTabFormer uses the column-specific vocabulary to restrict token choices at each step. A hash map tracks which tokens are valid for each column, and only tokens valid for the current column are considered. The paper reports $\ll 1\%$ invalid samples in experiments (Solatorio et al., 2023). This constrained generation is not merely an implementation convenience; it is part of the model’s strategy for preserving syntactic validity in heterogeneous tabular domains.

The sensitivity study provides an indirect efficiency context for these design choices. It notes that Transformer-based TDS models leverage self-attention and can achieve strong data quality, but often at high computational cost. REaLTabFormer’s fixed-vocabulary tabular design therefore occupies an intermediate position: it is still a Transformer-based method with nontrivial runtime demands, yet it avoids some of the inefficiencies associated with broad language-model vocabularies (R et al., 25 Sep 2025).

4. Overfitting detection and privacy-related controls

REaLTabFormer includes two explicit anti-memorization mechanisms: target masking and $Q_{\delta}$-based overfitting detection (Solatorio et al., 2023).

Target masking applies masking to target or label tokens rather than to inputs in the BERT style. Some target tokens are randomly replaced with a special mask token during training, with a reported mask rate of 10\%. The purpose is to encourage prediction of masked versions rather than memorization of exact training rows. The paper presents this as a practical anti-copying regularizer. It does not provide formal differential privacy (Solatorio et al., 2023).

The second mechanism is a training-time monitor based on distance to closest record (DCR) distributions. Given training data $T_r$ and generated data $G$, the model computes a distance matrix

$\mathbf{D} = L(T_r, G),$

then forms DCR distributions for nearest synthetic-to-training and training-to-synthetic matches. If generated data is too close to training data, the DCR distribution shifts toward zero, which is treated as a signal of possible memorization (Solatorio et al., 2023).

To detect such shifts, the paper introduces the asymmetric quantile-based discrepancy statistic

$$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$0

where the quantiles are taken from a reference sample and evaluated against another sample. The stated motivation is that this is especially useful for detecting changes in the left tail, where DCR values near zero indicate dangerous copying. A bootstrap procedure with

$$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$1

is used to compute a threshold $$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$2, and training is evaluated every

$$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$3

epochs with an early-stopping rule triggered after

$$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$4

consecutive violations (Solatorio et al., 2023).

A common misunderstanding is to equate these controls with a formal privacy guarantee. The published description does not justify that reading. The paper explicitly positions target masking and $$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$5 as empirical safeguards against copying and overfitting rather than as differential privacy mechanisms (Solatorio et al., 2023).

5. Evaluation methodology and reported performance

The original paper evaluates non-relational synthesis on six real-world datasets—Abalone, Adult income, Buddy, California Housing, Diabetes, and Facebook Comments—and relational synthesis on Rossmann store sales and Airbnb new user bookings (Solatorio et al., 2023). For non-relational tabular data, the baselines are TVAE, CTABGAN+, Tab-DDPM, and GReaT. For relational data, the main open-source baseline is HMA from SDV (Solatorio et al., 2023).

Three evaluation families are reported. Machine Learning Efficacy (MLE) measures whether synthetic data can replace real data for downstream prediction, using $$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$6 for regression and macro $$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$7 for classification. Discriminator Measure (DM) trains a random forest to distinguish real from synthetic data, with accuracy near 50\% interpreted as higher realism. For relational data, the paper uses Logistic Detection (LD) with ROC-AUC-based scoring and reports that higher LD, closer to 100, indicates more realistic synthetic data (Solatorio et al., 2023).

The empirical conclusions in the original study are differentiated by data regime. For ordinary tabular data, REaLTabFormer is reported to achieve competitive or best downstream ML utility and strong discriminator scores, with especially good performance on the large Facebook Comments dataset; the paper additionally notes that GReaT was too slow to train on Facebook Comments (Solatorio et al., 2023). For relational data, REaLTabFormer is reported to substantially outperform SDV’s HMA in logistic detection on Rossmann and Airbnb, and the qualitative analysis attributes this to better preservation of joint distributions, groupwise relationships, and inter-table associations (Solatorio et al., 2023).

The later sensitivity study adopts a different evaluation protocol centered on four real-world datasets—Stroke_healthcare, Customer, House, and Adult—preprocessed by encoding categorical columns, normalizing continuous columns, removing incomplete records, and shuffling and splitting to avoid temporal or positional bias (R et al., 25 Sep 2025). It evaluates runtime, ML utility, and similarity to real data distributions. Runtime is defined as total time for training and synthetic data generation, measured in seconds, collected via a Linux monitoring shell script, with each experiment repeated five times and the average runtime reported. ML utility is assessed with Logistic Regression and Random Forest for classification using Accuracy and Macro-F1, and Linear Regression and Random Forest for regression using $$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$8, again with five repetitions and mean score. Similarity is measured using a random-forest discriminator whose accuracy near 0.5 indicates that real and synthetic data are hard to distinguish (R et al., 25 Sep 2025).

6. Sensitivity to LLM hyperparameters, comparative positioning, and limitations

The 2025 study investigates whether the computational burden of Transformer-based TDS can be reduced without sacrificing data quality by varying model family/type, number of layers $$x_{ij} \sim P(X \mid x_{i1}, x_{i2}, \ldots, x_{i,j-1}),$$9, hidden dimension $$P(o_i) = \prod_{j=1}^{n} P(x_{ij} \mid x_{i1}, \ldots, x_{i,j-1}).$$0, and indirectly the number of parameters through the approximation

$$P(o_i) = \prod_{j=1}^{n} P(x_{ij} \mid x_{i1}, \ldots, x_{i,j-1}).$$1

where $$P(o_i) = \prod_{j=1}^{n} P(x_{ij} \mid x_{i1}, \ldots, x_{i,j-1}).$$2 is a family-specific constant. The study notes that model size grows linearly with the number of layers and quadratically with hidden dimension, implying that deeper or wider models are much more expensive (R et al., 25 Sep 2025).

Across the tested configurations, the study reports that runtime is proportional to the number of hyperparameters, with shallower configurations completing faster. GReaT consistently achieves lower runtimes than REaLTabFormer, and only on the largest dataset do they have comparable runtime. It also reports that REaLTabFormer did not converge on the larger datasets when using LLaMA (R et al., 25 Sep 2025). These findings place REaLTabFormer on the costlier side of Transformer-based synthesis even when lightweight LLMs are used.

On data quality, however, the same study reports a different pattern. On small datasets, both GReaT and REaLTabFormer achieve synthetic data with high utility and optimal similarity, while on larger datasets only REaLTabFormer sustains strong utility and similarity. It further reports that REaLTabFormer’s utility and similarity remain relatively stable across layer reductions, and that reducing the number of layers does not significantly harm downstream performance (R et al., 25 Sep 2025). This suggests that the model’s relational and distributional fidelity is less sensitive to moderate reductions in architectural depth than its runtime is.

In comparative terms, the two studies together position REaLTabFormer as a quality-oriented Transformer-based synthesizer. Relative to GReaT, it is generally slower but more robust on larger datasets and more suitable when relational/tabular fidelity is a central requirement (R et al., 25 Sep 2025). Relative to SDV’s HMA, it captures parent-child structure more effectively in the reported relational benchmarks (Solatorio et al., 2023). The principal limitations stated in the source material are equally clear: REaLTabFormer is not formally differentially private; child-table generation is constrained by computational resources and sequence length; the relational model is limited to the parent/child setup described; the $$P(o_i) = \prod_{j=1}^{n} P(x_{ij} \mid x_{i1}, \ldots, x_{i,j-1}).$$3 method is empirical and heuristic; and freezing the parent encoder, while efficient, may limit end-to-end adaptation (Solatorio et al., 2023).

A plausible implication is that REaLTabFormer is most appropriate when the objective is not maximum speed but a balance between fidelity and feasible compute. The sensitivity study states that REaLTabFormer with lightweight LLMs provides the best balance because it preserves data quality while reducing computational requirements, although its runtime remains higher than that of GReaT and other TDS tools, so efficiency gains are possible only up to a certain level (R et al., 25 Sep 2025).