PFN-Boost: Integrating PFNs with GBDTs

• PFN-Boost is a hybrid approach that integrates Prior-Fitted Networks and gradient boosting to enhance tabular prediction using Bayesian priors and residual modeling.
• It initializes gradient boosting with centered and scaled PFN logits, enabling effective correction of prediction errors especially in small to medium datasets.
• The method achieves robust state-of-the-art performance, scaling to large datasets without requiring additional PFN fine-tuning.

PFN-Boost refers to methodologies that integrate Prior-Fitted Networks (PFNs)—notably Transformer-based tabular models such as TabPFN—with gradient boosting frameworks to surmount the scalability and performance limitations of standalone PFNs and tree ensembles on tabular data. PFN-Boost approaches enable strong, pretrained Bayesian priors to inform scalable, residual-based learning, achieving robust state-of-the-art results from small to large sample regimes by fusing the inductive biases, representational advantages, and statistical strengths of both model classes (Jayawardhana et al., 4 Feb 2025, Wang et al., 3 Mar 2025).

1. Theoretical Motivation

TabPFNs and related PFNs can leverage large-scale pretraining for in-context tabular prediction, yielding near-Bayesian inference, especially for small $n \leq 10^3$, but they do not scale to larger datasets due to quadratic complexity in the number of input tokens. Conversely, gradient-boosted decision trees (GBDTs) are computationally efficient and effective for medium to large $n$ ($n\gg10^3$), but lack transferable priors and, by design, cannot leverage prior knowledge from other datasets or semantics in table structure. PFN-Boost injects the PFN prior into the training dynamics of GBDTs by initializing the boosting process with the predictive scores of a pretrained PFN, seeding the boosting with Bayesian-informed soft predictions and enabling subsequent trees to directly model the residuals for improved performance (Jayawardhana et al., 4 Feb 2025). This fusion is justified by the observation that ensembling multiple PFN predictors can improve accuracy, but only boosting can systematically correct errors that show up as pseudo-residuals challenging for the initial PFN (Wang et al., 3 Mar 2025).

2. Mathematical Framework

Consider a tabular classification task with labeled data $\{(x_i, y_i)\}_{i=1}^n$, $y_i\in\{1,\dots,C\}$. The PFN (e.g., TabPFN) produces for each sample $x$ a logit vector $z(x)\in\mathbb{R}^C$. Define the centered, scaled PFN initialization: $$\tilde{z}(x) = s \left(z(x) - \frac{1}{n}\sum_{i=1}^n z(x_i)\right),$$ where $s\geq0$ is a scale hyperparameter.

For PFN-Boost, $F_0(x) = \tilde{z}(x)$ initializes the prediction. Subsequent steps fit weak learners $n$0 (e.g., regression trees) to the pseudo-residuals: $n$1 using, e.g., multiclass logistic loss

$n$2

Predictions are then updated: $n$3 with either a fixed or line-searched learning rate $n$4 (Jayawardhana et al., 4 Feb 2025).

For BoostPFN, PFNs serve as weak learners, with each PFN inference conditioned on a sampled subset of training data, where sample weights are adaptively updated to emphasize high-residual ("hard") examples. The ensemble prediction after $n$5 rounds is: $n$6 with $n$7 a size-$n$8 weighted subsample and $n$9 the fixed PFN. Sampling weights are updated according to heuristics such as the Exp–Hadamard, Hadamard, or AdaBoost-style updates based on residual magnitude or misclassification (Wang et al., 3 Mar 2025).

3. Algorithmic Workflow and Implementation

The canonical PFN-Boost workflow proceeds as follows (Jayawardhana et al., 4 Feb 2025):

1. Pretrained PFN scoring: Compute PFN logits for all train/test samples.
2. Centering/scaling: Adjust logits to produce $n\gg10^3$0 via chosen $n\gg10^3$1.
3. Initialization: Set initial boosting prediction $n\gg10^3$2.
4. Iterative boosting: For $n\gg10^3$3:
   • Compute residuals $n\gg10^3$4.
   • Fit weak learner $n\gg10^3$5 (small tree) to residuals.
   • Update $n\gg10^3$6.
5. Tuning: Hyperparameters (rounds $n\gg10^3$7, max depth, learning rate, regularization, subsample ratios, $n\gg10^3$8) are tuned sequentially, with $n\gg10^3$9 tuned after tree parameters.

For BoostPFN (Wang et al., 3 Mar 2025), the algorithm employs sampling weights and multiple rounds, each time drawing a $\{(x_i, y_i)\}_{i=1}^n$0-sized subset, running PFN inference, updating the ensemble via line-searched step sizes, and updating sample weights based on residuals, as detailed in Algorithm 1 of (Wang et al., 3 Mar 2025). No PFN fine-tuning is required; all PFN inferences are with fixed parameters. Subsample sizes and batch parameters are chosen so as to fit within GPU memory bounds.

Summary Table: PFN-Boost Family Approaches

Approach	Weak Learner	PFN Usage	Scalability
PFN-Boost	Decision tree	PFN as initializer	GBDT scalability ($\{(x_i, y_i)\}_{i=1}^n$1)
BoostPFN	PFN itself	PFN as weak learner	Extends PFN up to $\{(x_i, y_i)\}_{i=1}^n$2 pretraining size

4. Scalability and Complexity

For small datasets, TabPFN and similar PFNs dominate, with $\{(x_i, y_i)\}_{i=1}^n$3 complexity due to attention over all input tokens. GBDTs, scalable to $\{(x_i, y_i)\}_{i=1}^n$4, have $\{(x_i, y_i)\}_{i=1}^n$5 time and $\{(x_i, y_i)\}_{i=1}^n$6 memory. PFN-Boost costs one $\{(x_i, y_i)\}_{i=1}^n$7 TabPFN forward pass (feasible for $\{(x_i, y_i)\}_{i=1}^n$8), followed by standard GBDT costs for subsequent rounds, so the overall complexity inherits that of the trees for large $\{(x_i, y_i)\}_{i=1}^n$9. For large-scale applications, either subsampling or the BoostPFN variant is applied, the latter drawing $y_i\in\{1,\dots,C\}$0-sized subsets for each round to keep PFN compute quadratic in $y_i\in\{1,\dots,C\}$1. Empirical results confirm that BoostPFN is practical up to datasets 50 times PFN's pretraining size (up to $y_i\in\{1,\dots,C\}$2) (Wang et al., 3 Mar 2025).

5. Empirical Performance and Comparative Evaluation

Empirical benchmarks on 16–30 real tabular datasets of varying size confirm that:

• For extremely small $y_i\in\{1,\dots,C\}$3, PFN or TabPFN alone performs best.
• In $y_i\in\{1,\dots,C\}$4, PFN-Boost consistently outperforms both standalone PFN and GBDT baselines, gaining 1–2 AUC points on average.
• For $y_i\in\{1,\dots,C\}$5, PFN-Boost matches or slightly exceeds GBDT performance, owing to the initialization from a prior-informed PFN prediction.
• PFN-Boost surpasses stacking (PFN logits appended as features) and selection (val AUC best-pick) ensembles, improving mean AUC by up to 0.5 points over stacking.
• BoostPFN achieves AUC parity or superiority relative to LightGBM, CatBoost, XGBoost, and Bagging of PFNs for subsample sizes up to 50,000, with time-to-accuracy benefits (approximately 60s per million samples to reach top mean AUC in one regime, compared with hundreds of seconds for GBDTs and AutoGluon) (Jayawardhana et al., 4 Feb 2025, Wang et al., 3 Mar 2025).

6. Ablation, Analysis, and Practical Considerations

• The scale parameter $y_i\in\{1,\dots,C\}$6 in PFN-Boost is instrumental: $y_i\in\{1,\dots,C\}$7 defaults to a standard GBDT; $y_i\in\{1,\dots,C\}$8 recovers PFN-only prediction. Optimal performance typically arises for intermediate $y_i\in\{1,\dots,C\}$9.
• Centering PFN logits when initializing is essential to avoid spurious tree compensations.
• For $x$0, random $x$1-point subsampling for PFN remains robust to subsample seed, maintaining performance.
• In BoostPFN, three sample-weight updating rules (Exp–Hadamard, Hadamard, AdaBoost-style) are empirically comparable; best choice is selected by validation fold.
• Both approaches require no PFN fine-tuning; zero training time is preserved for the PFN component.
• To exploit PFN’s benefits for very large datasets, users should adjust subsample size to available hardware, select round counts heuristically according to $x$2, and conduct tuning of all GBDT hyperparameters before scaling (Jayawardhana et al., 4 Feb 2025, Wang et al., 3 Mar 2025).

7. Impact and Future Directions

PFN-Boost and BoostPFN methodologies decisively expand the applicability of PFNs to larger datasets and strengthen probabilistic ensemble models for tabular data. By bridging pretrained transformer priors and classic tree-ensemble scalability, they enable consistent state-of-the-art prediction across regimes. A plausible implication is that further research may generalize these boosting strategies to additional pretrained tabular, multimodal, or language-driven models, or develop hardware-aware PFN inference frameworks optimized for even higher $x$3. Robust theory (such as $x$4 convergence for BoostPFN under smooth loss assumptions) provides guidance for further development and deployment in AutoML and industrial machine learning settings (Wang et al., 3 Mar 2025, Jayawardhana et al., 4 Feb 2025).