When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning

Abstract: Medical tabular data are ubiquitous in clinical research, but deep learning for tables remains underexplored because reliable labels often require costly expert adjudication, even though structured clinical variables are routinely available in tabular form. Self-supervised learning can leverage these unlabeled tables, and recent binning-based pretexts offer a promising inductive bias, but existing objectives fix a single global quantile discretization and apply feature-agnostic supervision. We propose Adaptive Binning, a training-adaptive discretization pretext for tabular SSL that couples discretization to learning through a feature-wise coarse-to-fine curriculum. Motivated by the spectral bias of neural networks and the principles of curriculum learning, our method progressively refines discretization per feature upon plateau detection and selects representation-aware splits to jointly improve value-space concentration and representation-space coherence. A heterogeneity-aware objective unifies categorical reconstruction with ordinal supervision for numerical features, and experiments on public medical tabular datasets under unified evaluation protocols show consistent gains for linear probing and fine-tuning without dataset-specific discretization tuning. We further introduce a medical tabular SSL benchmark with standardized protocols to support reproducible progress in this underexplored domain. Our code is available at https://github.com/labhai/Adaptive-Binning.

• The paper introduces Adaptive Binning, a method that refines discretization dynamically using per-feature plateau detection.
• It employs a Dispersion-Informed Gain-based Splitting technique to improve variance reduction and representation coherence with median-based partitioning.
• Experimental results show significant gains in AUC, accuracy, and RMSE on clinical tabular benchmarks with minimal tuning requirements.

Adaptive Binning for Tabular Self-Supervised Learning

Motivation and Background

Tabular data is pivotal in clinical and biomedical domains, underpinning decision systems and outcome prediction. However, representation learning for tabular data is underutilized in deep learning compared to imaging and language, due to inherent structural heterogeneity (mixed categorical/numerical types), absence of spatial/temporal context, and limited label availability. The standard reliance on tree ensembles (XGBoost, CatBoost) is justified by their natural handling of mixed types via recursive partitioning, producing piecewise-constant decision surfaces. Recent advances have adapted self-supervised learning (SSL) strategies for tabular domains, where binning-based pretext tasks discretize continuous features and reconstruct bin indices, enabling tree-like inductive bias. However, prior approaches adopt a fixed global quantile discretization, which is feature-agnostic and static, limiting adaptability, representation granularity, and type-aware supervision.

The paper proposes Adaptive Binning: a training-adaptive, feature-wise, coarse-to-fine discretization curriculum for tabular SSL. This approach leverages the spectral bias of neural networks (preference for learning coarse structure before finer details) and curriculum learning principles to realize feature-specific, learning-driven refinement of binning, combined with heterogeneity-aware objectives for categorical and ordinal supervision.

Methodological Framework

The Adaptive Binning framework introduces three tightly coupled innovations:

When: Feature-Wise Plateau Trigger (FPT)

FPT autonomously detects per-feature stagnation in reconstruction loss during pretraining, identifying when a feature’s current discretization is saturated and eligible for refinement. The trigger monitors normalized epoch-wise reconstruction loss per feature, updating a patience counter and best metric to initiate refinement on plateau detection. This granular adaptation circumvents the inefficiency of globally synchronized bin updates.

Where: Dispersion-Informed Gain-based Splitting (DIGS)

DIGS algorithmically determines which bin of a numerical feature to split and where to establish new boundaries. Splitting is motivated by both variance reduction in the value-space and coherence improvement in the representation-space. Within each bin, splitting candidates are generated at the median, maintaining equal-frequency partitioning. The split is accepted if it jointly improves variance and dispersion in encoder-induced embeddings, using a composite score threshold.

Figure 1: Overview of the Adaptive Binning framework, displaying how FPT and DIGS drive feature-wise bin refinement and HORD delivers type-aware reconstruction objectives.

How: Heterogeneity-aware ORDinal Loss (HORD)

HORD unifies categorical and numerical feature supervision in a type-aware manner. Categorical features are reconstructed via standard cross-entropy, while numerical features are supervised with a soft-target cross-entropy (SORD) that penalizes prediction errors proportionally to bin distance, preserving ordinal relationships. Additional mean-variance regularization encourages predicted distributions to be concentrated and locally accurate. Feature-wise losses are averaged and weighted by feature count to ensure balanced optimization.

Experimental Analysis

Datasets and Protocols

A curated medical tabular benchmark spans binary, nominal, and ordinal multiclass classification, and regression tasks. Pretraining employs 1000 epochs on standard MLP encoders, followed by two downstream evaluation protocols: linear probing of frozen representations and full fine-tuning with multiple tabular architectures (MLP, ResNet, TabNet, FT-Transformer, T2G-Former).

Linear Probing and Ablation Studies

Adaptive Binning achieves the lowest average rank across masking, binning, and objective variants, consistently outperforming fixed binning and mask-based reconstructions. Strong numerical results demonstrate margin improvements in AUC, accuracy, QWK, and RMSE across datasets. Ablation experiments reveal that removing any single component (FWA, FPT, DIGS, or HORD) degrades performance, validating the necessity of their integration.

Figure 2: Hyperparameter sensitivity sweeps for HORD and refinement parameters, evidencing method robustness to hyperparameter choices and endorsing the default configuration.

Hyperparameter sweeps of HORD regularization, patience, and the DIGS threshold confirm that performance is stable for a wide range and centered at the default, reducing tuning burden in clinical application.

Fine-tuning and Transferability

Fine-tuning experiments show the transferability of Adaptive Binning: SSL-pretrained initializations with Adaptive Binning maintain or improve over both supervised and previous SSL objectives across model families and tasks. The method demonstrates robustness to architecture selection and downstream optimization, supporting its deployment in diverse medical tabular modeling scenarios.

Implications and Future Directions

Adaptive Binning transforms discretization into a dynamic, representation-driven pretext mechanism rather than a static engineering parameter. This provides better inductive bias, granularity, and type-aware supervision, leading to improved representation quality and downstream transfer in medical tabular domains. The benchmark protocol and codebase further support reproducibility and rigorous comparison.

Practically, Adaptive Binning reduces the requirement for dataset-specific tuning, increases model robustness, and mitigates clinical deployment risk. Theoretically, it bridges curriculum learning and SSL for tabular domains, opening pathways to extend adaptive pretexts to other biomedical modalities and non-tabular domains.

Anticipated future developments include: (i) evaluation in broader clinical endpoints (e.g., survival, multi-outcome prediction), (ii) cross-dataset transfer with adaptation scenarios, and (iii) extension to multimodal (tabular-image-text) SSL frameworks.

Conclusion

Adaptive Binning advances tabular self-supervised learning by constructing a feature-wise, learning-coupled discretization curriculum leveraged via plateau-triggered refinement, representation-aware splitting, and heterogeneity-aware reconstruction objectives. This yields robust representations and transferability across architectures and clinical tasks, with reproducible evaluation protocols supporting continued progress in the domain of medical tabular SSL (2606.19827).