SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation

Abstract: Generating high-fidelity synthetic tabular data remains a critical challenge for enhancing data availability in privacy-sensitive and low-resource domains. Recent approaches leverage LLMs by representing table rows as sequences, yet suffer from two fundamental limitations: (1) they model feature dependencies densely, introducing spurious correlations; and (2) they assume static relationships between features, ignoring how these dependencies vary with feature values. To overcome these limitations, we introduce SAGE (Sparse Adaptive Guidance), a novel LLM-based generation framework that enforces sparse and dynamic dependency guidance. SAGE discretizes features into value-aware pseudo-features and constructs a mutual information-based sparse dependency graph. This graph adaptively guides generation through explicit context selection or implicit logit correction, enabling LLMs to focus on truly relevant information during synthesis. Our extensive experiments across six datasets and multiple tasks reveal that SAGE not only improves data fidelity and downstream utility, boosting F1 scores by 10% compared to previous LLM-based methods, but also reduces policy violations by one point. These results highlight the importance of adaptive structure in tabular data generation and provide new insights into context-sensitive control of LLMs.

• The paper introduces a novel MI-based sparse and adaptive guidance mechanism to generate semantically coherent synthetic tabular data.
• It employs Feature Selector and Logit Correction techniques to dynamically adjust LLM attention based on value-conditioned dependencies.
• Empirical results demonstrate up to a 10% F1 improvement and lower violation rates, enhancing utility, realism, and privacy across domains.

Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation: An Expert Review

Motivation and Problem Statement

Synthetic tabular data generation is central to privacy-sensitive and low-resource domains. Conventional deep tabular generators—VAEs, GANs, and recent diffusion models—struggle to encode rich, context-dependent feature dependencies, often leading to illogical or implausible records. Recent advances leveraging LLMs by textualizing tabular rows unlock world knowledge and reduce certain inconsistencies, but suffer from two key deficiencies: (i) dense dependency modeling introduces spurious correlations and computational overhead, and (ii) static dependency assumptions fail to capture value-conditioned dynamics. The “SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation” (2604.24368) addresses those limitations by enforcing both sparse and adaptive dependency modeling within LLM-based generators, dynamically modulating guidance according to feature values and relationships.

Figure 1: Value-conditioned dynamic dependencies in tabular data illustrate how, e.g., loan purpose modulates the age and income distribution, a structure missed by static dependency models.

Methodological Contributions

The SAGE framework introduces a preprocessing pipeline for value-aware pseudo-feature discretization, constructing a mutual information (MI)-based sparse dependency matrix. Each tabular record is represented as a binary vector of pseudo-features, corresponding to value bins (for numeric attributes) or categories (for categorical ones). The MI matrix quantifies fine-grained statistical dependencies between pseudo-features, capturing both sparse and value-conditioned interactions. During generation, two guidance strategies are utilized:

Feature Selector (Explicit Context Pruning): The MI matrix is used to prune the input context based on a threshold $\tau$, limiting the LLM’s attention to relevant, information-rich prefixes during each feature value prediction. This context adapts dynamically as feature values are generated, supporting value-dependent sparsity.

Logit Correction (Implicit Confidence Adjustment): The generation logits for each feature are rescaled according to the average MI between the current target and previously generated pseudo-features. This adaptive sharpening/smoothing adjusts the model's confidence in value prediction according to the informativeness of the active context, rather than relying on a fixed structure.

Figure 2: Overview of the SAGE pipeline. Preprocessing computes an MI-based dependency matrix; generation is guided via Feature Selector or Logit Correction mechanisms.

Through random shuffling of feature-value ordering and supervised fine-tuning (computing loss only over value tokens), SAGE discourages position-based spurious correlations, reduces computational cost, and enhances alignment between training and inference objectives.

Empirical Results

SAGE was evaluated across six datasets spanning binary/multiclass classification and regression, including spatially-constrained (California Housing), high-dimensional medical (MIC), and small-scale (Iris) domains. Downstream evaluation involved training classifiers and regressors on synthetic data, then assessing performance on real data (utility metric), alongside fidelity measures (violation rates under domain constraints), realism (discriminator accuracy), and privacy (DCR).

Utility: SAGE consistently outperforms state-of-the-art baselines (GReaT, GraDe, SPADA, TVAE, CTGAN, TabSyn), with statistically significant increases up to +10% in F1—especially in small datasets (Iris)—demonstrating robust dependency modeling and reduced overfitting.

Fidelity: SAGE’s violation rates under spatial and semantic constraints (e.g., housing coordinates, education consistency) are markedly lower than baselines. Logit Correction, in particular, achieves 1-point lower violation on California Housing, while Feature Selector reduces semantic violations on Adult Income by almost 3-fold.

Figure 3: SAGE-generated samples for California Housing exhibit accurate spatial distributions, with almost no points outside the true boundary.

Figure 4: Violation rate of SAGE versus baselines on California Housing; lower rates demonstrate improved constraint fidelity.

Realism: Discriminators trained to distinguish real from synthetic achieve lower accuracy on SAGE outputs (Table results, see paper), indicating closer alignment between synthetic and real data distributions.

Privacy: DCR distributions confirm that SAGE achieves a balance between fidelity and privacy; higher DCR values correspond to less overlap with real instances, while maintaining downstream task utility.

Figure 5: DCR for California Housing. SAGE exhibits enhanced privacy preservation while maintaining distributional fidelity.

Distributional Fidelity: On Iris, SAGE closely reproduces the density shapes for all features, confirming preservation across both categorical and numeric attributes.

Figure 6: Distributional density visualization for Sepal/Petal length and width on Iris: synthetic and real distributions are closely matched.

Model Robustness & Ablation: Downstream performance is robust to MI threshold variations. Feature Selector and Logit Correction display complementary strengths: explicit pruning yields stable utility and low MAPE in high-dimensional and semantic domains; implicit adjustment excels in spatial fidelity.

Figure 7: SAGE performance across different LLM architectures demonstrates method-agnostic robustness.

Figure 8: Ablation study: impact of MI thresholds on downstream performance (classification/regression); stability until aggressive pruning removes critical dependencies.

Theoretical Implications

SAGE leverages information-theoretic perspectives: MI as proxy for statistical dependence, information bottleneck representation for Feature Selector, and KL divergence minimization rationale for Logit Correction. The dynamic adaptation of the dependency graph achieves approximate conditional relevance akin to Bayesian networks, without requiring brittle expert-annotated graphs or explicit logic rules.

Practical and Future Implications

The practical benefits include improved synthetic data quality for downstream ML tasks, enhanced privacy preservation, robust generalization across diverse domains, and computational efficiency (order-of-magnitude reductions in sampling latency and training time). SAGE’s sparse dynamic guidance also sets a foundation for further advancements:

• Higher-order dependency modeling: Current pairwise MI may miss multivariate dependencies; autoregressive LLMs partially mitigate this, but explicit modeling remains open for innovation.
• Adaptive hybrid guidance: Combining explicit pruning and implicit logit adjustment adaptively, possibly with learned thresholds, could further improve constraint adherence and generalization.
• Scalable preprocessing: Technical optimizations make SAGE feasible for high-dimensional data, but further work on incremental MI estimation or parallelization could broaden applicability.

The demonstrated ability to control synthetic data generation with value-sensitive, context-aware guidance informs both theoretical understanding and practical implementation of LLM-driven synthetic tabular data methodologies—a critical capability for healthcare, finance, and scientific applications facing stringent privacy and data availability demands.

Conclusion

SAGE delivers sparse, dynamically adaptive context-sensitive guidance for LLM-based tabular data generation, advancing the state of the art in fidelity, downstream utility, and privacy. By leveraging value-aware pseudo-features and MI-based dependency graphs, SAGE achieves semantically coherent, constraint-abiding synthetic data across diverse domains. Both explicit and implicit guidance mechanisms offer complementary benefits, and empirical evidence demonstrates performance gains and theoretical soundness. Future work should focus on higher-order dependency modeling, adaptive hybrid guidance, and further scalability.