TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data

Abstract: Synthetic tabular data support use cases like data sharing, model development under access restrictions, and rapid prototyping of analytical workflows. Modern generative models are evaluated by their statistical similarity, correlation structure, privacy, and downstream machine-learning utility. However, such evaluations leave a gap: they rarely test the structure that matters for analytical queries. We present TabQueryBench, a query-centric benchmark that uses SQL-shaped analytical queries as structural assessors for synthetic data fidelity. It provides an extensible foundation for query-centric synthetic-data evaluation. From 12 public sources of analytical queries, TabQueryBench taxonomizes recurring cross-domain logic into 44 reusable query templates and grounds them to each dataset via a policy-guided template-to-SQL pipeline. This makes queries schema-aware while preserving comparability across generative models. Across 49 datasets and 11 generative models, it activates 10-12 templates per dataset, producing more than 100 executable SQL queries per dataset. Our systematic experiments show five main patterns. First, current tabular generative models can have good distance-based fidelity, but they still fall short on query-centric fidelity: RealTabFormer achieves the highest query-centric fidelity, but it only reaches 0.75 +/- 0.15 (REAL data score is 1.00). Second, tabular generative models struggle with very high-cardinality discrete support. Third, SOTA generative models preserve good global conditional query-centric fidelity, but fail more on local queries. Fourth, tail fidelity deteriorates as queries move toward the extreme tail; even the best model recovers only about 40.7% of real rare values. Finally, there is a fidelity-cost tradeoff in tabular generation: BayesNet offers the strongest tradeoff, with slightly lower query-centric fidelity but much lower generation cost.

• The paper introduces a query-centric benchmarking framework that evaluates synthetic tabular data based on its ability to support SQL analytical queries.
• It details a two-stage methodology using reusable query templates and dataset-specific mappings to assess various generative models across 49 datasets.
• Findings reveal that state-of-the-art models struggle with high-cardinality, tail queries, and local conditional query fidelity despite high distance-based scores.

TabQueryBench: Advancing Query-Centric Benchmarking for Synthetic Tabular Data

Motivation and Background

Synthetic tabular data generation is increasingly critical for scenarios such as privacy-preserving data sharing, model development under data access restrictions, and rapid prototyping of analytics workflows. Historically, the evaluation of synthetic tabular data has centered on aggregate statistical fidelity—distributional similarity at the column or table level, dataset-wide predictive utility, and privacy metrics. However, these metrics often provide an incomplete picture of downstream usability: they do not assess whether generated tables support the actual analytical queries that practitioners run on the real datasets.

TabQueryBench (2607.03926) addresses this misalignment by introducing a query-centric paradigm for benchmarking synthetic tabular data, reframing synthetic data fidelity in terms of preserving analytical query answers. This approach reflects the operational needs of data consumers who are less interested in record-level mimicry, and more concerned with whether synthetic data correctly supports subgroup analyses, conditional aggregations, tail/rarity exploration, missingness evaluation, and cardinality-sensitive queries as expressed naturally in SQL workloads.

Benchmark Design and Query Taxonomy

TabQueryBench systematically constructs a benchmark grounded in recurring analytical SQL query patterns. The benchmark is instantiated with 49 real-world tabular datasets and 11 generative models, spanning Bayesian, tree-based, adversarial (GAN/VAE), diffusion-based, transformer, and flow-matching architectures. The benchmarking pipeline operates in two main stages:

1. Template Construction: Recurring analytical logics are taxonomized from 12 public sources—including TPC-H, TPC-DS, ClickBench, and OLAP analytics benchmarks—into a library of 44 reusable query templates grouped into five principal query families:
   • Subgroup: Aggregate analyses over subpopulations (e.g., grouped means, counts, ranks).
   • Conditional: Aggregations over groups with/without filter predicates (global structure vs. local slice).
   • Tail/Rarity: Coverage and behavior of rare/outlier regions.
   • Missingness: Marginal and conditional patterns of missing data.
   • Cardinality/Range: Preservation of discrete support and value ranges.
2. Dataset Grounding: For each dataset, schema-aware policies map template placeholders to eligible columns, ensuring type and support alignment without manual curation. This results in over 100 executable, dataset-specific SQL queries per dataset, collectively yielding thousands of benchmark problems across the suite.

The entire toolchain—template library, SQL realization, runtime validation, and evaluation—is open-sourced for extensibility and reproducibility.

Experimental Evaluation and Findings

TabQueryBench is used to evaluate 11 prominent generative models on 49 datasets. Key findings include:

• Distance-based fidelity overestimates analytical utility: All generative models exhibit substantially lower query-centric fidelity than distance-based scores suggest. For example, RealTabFormer, the best-performing model, achieves $0.75 \pm 0.15$ query fidelity relative to a real-data reference of $1.00$, despite high distance-based scores.
• Systematic failure on high-cardinality attributes: Generative models struggle to preserve the full discrete support for columns with >$10^3$ unique values. For instance, with a real column cardinality of 96,777, BayesNet outputs 242, and CTGAN produces 19,630. This support loss causes analytical queries (group-by, filter) over such attributes to operate on an inconsistent domain.
• Global conditional structure is preserved better than local slices: SOTA models perform well on aggregate conditional queries, but fidelity degrades markedly under local filters. For example, TabDiff scores 0.632 globally but drops to 0.276 with a subpopulation filter, indicating a lack of reliable substructure modeling.
• Tail and rarity queries highlight poor rare-category support: Rare-region fidelity is consistently low, deteriorating as tail thresholds become more restrictive. Even the best models recover only ≈41% of real rare values for ultra-rare queries, limiting reliability for risk and outlier analysis in synthetic data.
• Cost-fidelity tradeoff is substantial: While RealTabFormer offers maximal analytical fidelity, it incurs high computational cost and runtime ($\sim$150 minutes/dataset). BayesNet, conversely, provides near-top query fidelity with an order-of-magnitude lower cost, representing the optimal fidelity-cost balance for typical settings.

Practical and Theoretical Implications

The findings of TabQueryBench have direct implications for both synthetic data research and its application:

• Benchmark design: Query-centric evaluation exposes failure modes (e.g., local conditional support, high-cardinality, tails) invisible to standard distributional metrics, urging the community to adopt or integrate query-focused benchmarks to improve model realism and practical utility.
• Model development: The fine-grained breakdown enables targeted improvements—addressing high-cardinality dropout, rare-event underfitting, and conditional subgroup distortions. Future model design may integrate query families directly into loss objectives or use them for validation/model selection.
• Usability tradeoffs: Practitioners must consider the tradeoff between fidelity and computational/executional cost in choosing generative models based on deployment constraints.
• Privacy assessment: The methodology offers a complementary view to formal privacy metrics; however, preservation of analytical structure may inadvertently intensify privacy leakage for rare subgroups or high-cardinality attributes, highlighting the tension between utility and privacy protections in synthetic data release.
• Towards multi-table and querying LLMs: While TabQueryBench focuses on single-table settings, the methodology naturally extends to multi-table/relational synthetic data and provides a scaffold for benchmarking synthetic data utility in LLM-driven text-to-SQL pipelines.

Future Directions

TabQueryBench points to several avenues of research:

• Query-aware generative modeling: Incorporating query templates or family-level constraints into training and evaluation loops to directly optimize for query fidelity.
• Multi-table, relational query benchmarking: Extending to evaluation scenarios where join patterns, foreign-key consistency, and relational analytics play a central role.
• Public benchmark extension and standardization: Open-source assets enable community-wide extensions, facilitating comprehensive, structural, and reproducible synthetic data evaluation.
• Analysis of stability and LLM-induced variance: Further characterization of benchmark stability under different SQL realizers or LLM configurations is warranted.

Conclusion

TabQueryBench establishes query-centric fidelity as a critical axis for the evaluation of synthetic tabular data and demonstrates that prevailing generative models fall short on structural properties essential for practical analytics. This work compels a methodological shift from aggregate statistical resemblance toward query-oriented and structure-preserving synthetic data generation. Its public release provides the community with a robust, extensible framework for benchmarking analytical query fidelity, catalyzing targeted advances in tabular generative modeling and synthetic data utility research.