MultiTab-Bench: Synthetic Tabular MTL Benchmark

• The paper introduces MultiTab-Bench, a synthetic multitask tabular benchmark generator that enables controlled evaluation of multitask learning through adjustable task structure and complexity.
• It employs a configurable data synthesis protocol to generate datasets with precise control over task count, correlation, and complexity, facilitating direct measurement of multitask gains.
• The benchmark supports rigorous ablation studies and scaling analyses, serving as a crucial tool for diagnosing and comparing multitask architectures like MultiTab-Net.

MultiTab-Bench is a synthetic multitask tabular benchmark generator introduced in conjunction with the MultiTab-Net architecture, addressing the systematic evaluation of multitask learning (MTL) methods on tabular data with controllable task structure and complexity. MultiTab-Bench provides a standardized mechanism to probe and diagnose multitask dynamics, serving as an essential resource for the rigorous comparison and ablation of multitask architectures for tabular domains (Sinodinos et al., 13 Nov 2025).

1. Motivation and Role in Multitask Learning

Tabular data permeates applications across finance, healthcare, e-commerce, and the sciences. As real-world tabular datasets grow in size and start to incorporate multiple targets—often related but distinct—effective multitask modeling becomes essential. Historically, multitask evaluation for tabular MTL has been limited by the lack of standardized benchmarks that allow explicit control over foundational properties such as the number of tasks, degree of correlation among tasks, and task complexity.

MultiTab-Bench directly addresses this gap by generating synthetic tabular datasets where such factors are not only tunable but also precisely known to the experimenter. This synthetic benchmarking enables dissecting the behavior of multitask models, differentiating architectural improvements from dataset artifacts, and precisely quantifying multitask gain across a broad spectrum of task relationships.

2. Dataset Generation Protocol

MultiTab-Bench acts as a generalized synthetic multitask tabular dataset generator. The core design principle is to allow systematic modulation of parameters that critically affect multitask dynamics:

• Task Count ($t$): The number of tasks to be learned jointly can be set from a small to a large value, enabling assessment of scaling properties.
• Task Correlations ($\rho_{ij}$): The inter-task correlation structure is directly specified, typically via covariance matrices over target variables, allowing for datasets with independent, partially correlated, or highly redundant tasks.
• Relative Task Complexity: The structural complexity of each task (e.g., regression vs. classification, nonlinear vs. linear, feature relevance) can be tuned independently per task.

Data synthesis proceeds by first sampling feature vectors $\mathbf{x}$, typically from a multivariate distribution (e.g., Gaussian) with specified covariance. Task targets $y_i$ are generated via user-specified functions—linear, nonlinear, categorical, or hybrid—potentially sharing feature subsets or noise sources according to the desired correlation structure.

3. Evaluative Scope and Research Applications

MultiTab-Bench provides a controlled, reproducible environment for evaluating multitask architectures under varied and stress-tested regimes. It enables:

• Quantitative analysis of multitask gain: By measuring the improvement in performance (e.g., reduction in loss) when learning tasks jointly vs. separately, and relating this to true underlying correlations and complexity differentials.
• Ablation of attention and masking strategies: MultiTab-Bench supports systematic testing of architectural interventions—such as token masking, inter-feature and inter-task attention configurations, and cross-task competition mitigation—under known ground-truth data conditions.
• Scaling and generalization diagnostics: With the ability to scale task count and feature dimensionality, MultiTab-Bench fosters exploration of whether gains persist or fade with problem size and whether models degrade gracefully with increasing heterogeneity.

4. Integration with MultiTab-Net and Experimental Findings

MultiTab-Bench operates as the principal synthetic benchmark for MultiTab-Net, the first multitask transformer designed for large tabular data. In the seminal experiments, MultiTab-Bench was used to show:

• The superiority of MultiTab-Net in multitask gain compared to both shared and split MLP baselines and earlier transformer approaches, specifically under varying inter-task correlation structures.
• The empirical effectiveness of multitask masked attention, particularly the variant that prevents task tokens from attending to each other (T $\not\rightarrow$ T), as measured on datasets synthesized to include both highly correlated and uncorrelated tasks.

These results demonstrate the diagnostic power of MultiTab-Bench: it enables rigorous, automated stress-testing of architectural design decisions, unconfounded by heterogeneous real-world data idiosyncrasies.

5. Benchmark Configuration Parameters

The generator exposes, at minimum, the following configurable parameters:

Parameter	Description	Typical Range
Task count ($t$)	Number of targets/tasks jointly modeled	$2 \leq t \leq 100$
Task correlation	Covariance/correlation structure among target variables	$-1 \leq \rho_{ij} \leq 1$
Task complexity	Function mapping $\mathbf{x}$ to $y_i$ per task (linear, nonlinear, categorical)	Variable
Feature count ($d$)	Number of input columns/features	$10 \leq d \leq 10^4$
Sample size ($n$)	Number of rows/samples	$10^4 \leq n \leq 10^7$

This direct exposure of low-level properties stands in contrast to most non-synthetic multitask tabular benchmarks, where such axes are fixed by the originating application domain and are not directly manipulable.

6. Distribution, Use, and Significance

MultiTab-Bench is released as open-source code at the MultiTab repository (https://github.com/Armanfard-Lab/MultiTab) (Sinodinos et al., 13 Nov 2025), supporting Python-based reproducibility and extensibility. The benchmark is designed specifically for the evaluation and research of multitask tabular models, but the underlying principles are applicable to broader synthetic multitask learning research.

Its significance lies in providing, for the first time in the tabular domain, the kind of benchmark control that has proven indispensable in vision and language (e.g., synthetic noise-controlled datasets, procedurally generated language), thereby accelerating method development and principled assessment of multitask methods.

7. Limitations and Prospective Extensions

As a synthetic generator, MultiTab-Bench abstracts away some complexities of real tabular data, such as missing values, categorical encodings, rare event dynamics, and intricate inter-feature dependence present in particular scientific or industrial domains. While it enables controlled analysis, it is not a surrogate for large-scale real-world benchmarks. A plausible implication is that rigorous benchmarking should pair MultiTab-Bench analyses with evaluations on actual tabular datasets to ensure real-world transferability.

Prospective extensions include the incorporation of realistic data corruption processes (e.g., missingness, heavy-tailed feature marginals), domain-driven noise models, and multi-modal target distributions, broadening the ecological validity of synthetic MTL evaluations.