PCAForest: Linear Diffusion Oversampling

• PCAForest is a latent-space diffusion model that uses linear PCA and gradient-boosted trees to generate synthetic minority examples in imbalanced tabular data.
• It embeds data into a low-dimensional space via PCA and learns a conditional flow-matching vector field through tree-based regression for efficient, deterministic reverse ODE generation.
• Empirical results across 11 datasets show competitive minority recall and F1 scores, with up to 50% faster generation compared to full-dimensional diffusion methods.

Searching arXiv for the specified paper and closely related uses of the name "PCAForest" to ground the article and disambiguate it from "Randomized PCA Forest". I’ll look up the arXiv entries for the cited papers to ensure the terminology and citation context are current. PCAForest is a latent-space, tree-driven diffusion method for minority oversampling in severely imbalanced tabular learning. It was introduced as one of three variants—alongside EmbedForest and AttentionForest—in a family of methods that use conditional flow matching with gradient-boosted trees as the vector-field learner, while operating in compact latent spaces to preserve tabular structure and reduce computation (Ihsan et al., 20 Nov 2025). Within that family, PCAForest is the variant that uses a linear PCA embedding, couples a GBT-based flow with a decoder back to the original feature space, and generates synthetic minority examples by solving a deterministic reverse ODE in the reduced latent space.

1. Problem setting and position within the method family

PCAForest is designed for the setting in which severe class imbalance is common in real-world tabular learning and rare but important minority classes are essential for reliable prediction. The motivating context is that existing generative oversampling methods such as GANs, VAEs, and diffusion models can improve minority-class performance, but they often struggle with tabular heterogeneity, training stability, and privacy concerns (Ihsan et al., 20 Nov 2025).

The broader method family introduced with PCAForest consists of three latent-space variants. PCAForest uses linear PCA embedding; EmbedForest uses a learned nonlinear embedding; and AttentionForest uses an attention-augmented embedding. All three methods learn a diffusion process in latent space and decode back to the original feature space. In comparative results across 11 datasets from healthcare, finance, and manufacturing, AttentionForest achieves the best average minority recall while maintaining competitive precision, calibration, and distributional similarity, whereas PCAForest and EmbedForest reach similar utility with much faster generation, yielding favorable accuracy-efficiency trade-offs (Ihsan et al., 20 Nov 2025).

In that formulation, PCAForest occupies the most explicit linear end of the design space. It replaces a learned encoder with principal component analysis and replaces a neural vector-field parameterization with gradient-boosted-tree ensembles. The resulting model is therefore characterized by a closed-form linear encoder/decoder, a conditional flow in the reduced space, and deterministic latent-space generation.

2. PCA embedding and conditional flow-matching formulation

Let each original example be $x\in\mathbb{R}^d$. PCAForest first computes a linear PCA embedding onto a $k$-dimensional latent space with $k\ll d$. If

$$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$

is the sample covariance and $\{(\lambda_j,v_j)\}_{j=1}^d$ are its eigenpairs ordered by descending $\lambda_j$, the projection matrix is

$$W=[\,v_1\;v_2\;\cdots\;v_k]\in\mathbb{R}^{d\times k}.$$

The PCA encoding of $x$ is then

$z=W^\top(x-\mu)\in\mathbb{R}^k,$

and the associated decoding uses the pseudo-inverse

$x\approx \mu+Wz.$

The diffusion component is defined in this latent space. For each latent datum $k$0 and Gaussian target $k$1, PCAForest adopts the conditional flow-matching objective restricted to $k$2 by forming the straight-line interpolation

$k$3

Along this path, the ideal vector field is

$k$4

A time-dependent field $k$5 is learned by minimizing

$k$6

Under this objective, the learned field satisfies the deterministic reverse-time ODE

$k$7

which transports the prior $k$8 back to the data manifold at $k$9 (Ihsan et al., 20 Nov 2025).

This construction makes PCAForest a latent-space diffusion model in a precise sense: the data distribution is modeled after linear dimensionality reduction, and transport from a Gaussian prior back to minority-class latent patterns is learned through conditional flow matching rather than through a stochastic reverse diffusion process.

3. Gradient-boosted-tree vector-field learner

Rather than using a neural network, PCAForest uses a gradient-boosted-tree ensemble to approximate $k\ll d$0. The unit interval is discretized into $k\ll d$1 time steps $k\ll d$2. For each $k\ll d$3, the method collects training pairs $k\ll d$4 across all $k\ll d$5 samples and trains a regression forest $k\ll d$6 to minimize

$k\ll d$7

Training proceeds by standard boosting: at each iteration a new tree is fit to the residuals of the previous ensemble, aggregating until a fixed number of iterations or until convergence. At inference, the time-indexed approximation is given by $k\ll d$8 (Ihsan et al., 20 Nov 2025).

Two structural consequences are emphasized in the method description. First, training is embarrassingly parallel across the discretized time indices $k\ll d$9. Second, each tree ensemble operates in the compact $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$0-dimensional space rather than in the original $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$1-dimensional feature space. The combination of temporal discretization and low-dimensional regression is central to PCAForest’s computational profile.

The choice of GBTs is also method-defining. In PCAForest, the vector field is represented by a collection of time-specific boosted regressors rather than by a shared continuous-time neural parameterization. This makes the model tree-driven in an exact sense: the flow-matching target is learned through boosted trees at each time slice.

4. Decoder, reverse ODE integration, and sample generation

After a latent sample has been transported to $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$2, PCAForest reconstructs it to the original feature space by inverse PCA:

$$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$3

No further learned decoder is required, because PCA provides a closed-form inverse so long as one retains the mean $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$4 and principal axes $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$5 (Ihsan et al., 20 Nov 2025).

Generation begins from the Gaussian prior in latent space. For each synthetic point, the method samples

$$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$6

then integrates the reverse ODE by Euler steps over the discretized time grid. Writing $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$7, the update is

$$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$8

for $$\Sigma=\frac{1}{n}\sum_{i=1}^n (x^{(i)}-\mu)(x^{(i)}-\mu)^\top$$9. The resulting latent vector at $\{(\lambda_j,v_j)\}_{j=1}^d$0 is then decoded by inverse PCA, and the minority label is assigned to the reconstructed point.

Two procedural details delimit the model class. No stochasticity is added in the reverse pass; the method solves an ODE, not an SDE. In addition, there is no extra conditioning on class beyond operating exclusively on minority patterns in training. The synthetic data are therefore produced by deterministic reverse transport from a latent Gaussian prior through a field learned only from minority-class structure.

5. Hyperparameters, complexity, and efficiency profile

The key hyperparameters reported for PCAForest are the latent dimension $\{(\lambda_j,v_j)\}_{j=1}^d$1, the number of diffusion time steps $\{(\lambda_j,v_j)\}_{j=1}^d$2, and the GBT parameters: number of boosting iterations $\{(\lambda_j,v_j)\}_{j=1}^d$3, tree depth, and learning rate for residual updates. The latent dimension is chosen so that the top-$\{(\lambda_j,v_j)\}_{j=1}^d$4 eigenvalues retain, for example, $\{(\lambda_j,v_j)\}_{j=1}^d$5 of variance, and the number of diffusion time steps may be, for example, $\{(\lambda_j,v_j)\}_{j=1}^d$6 (Ihsan et al., 20 Nov 2025).

Training complexity is dominated by fitting $\{(\lambda_j,v_j)\}_{j=1}^d$7 gradient-boosted-tree regressors in $\{(\lambda_j,v_j)\}_{j=1}^d$8. Each round costs $\{(\lambda_j,v_j)\}_{j=1}^d$9 per time step. Sampling $\lambda_j$0 examples requires $\lambda_j$1 operations. The method description states that the small latent dimension $\lambda_j$2 and short $\lambda_j$3 yields a substantial speed-up compared to diffusion in $\lambda_j$4 or neural ODE solvers (Ihsan et al., 20 Nov 2025).

This efficiency profile is part of the rationale for PCAForest within the broader latent-space family. Because the encoder and decoder are linear and closed-form, reconstruction does not introduce an additional learned network. Because the flow is learned in the reduced latent manifold, both training and generation are tied to $\lambda_j$5 rather than to the ambient dimension $\lambda_j$6. The paper’s ablation studies further report that smaller embeddings tend to improve minority recall, while aggressive learning rates harm stability, indicating that the principal hyperparameters affect both efficiency and predictive behavior.

6. Empirical performance, trade-offs, and nomenclatural disambiguation

Across 11 imbalanced tabular datasets, PCAForest matched or exceeded baseline oversamplers in minority recall and F1 while running up to 50% faster than full-dimensional Forest-Diffusion (Ihsan et al., 20 Nov 2025). A specific benchmark reported for Mammography gives recall $\lambda_j$7, recall $\lambda_j$8, and F1 $\lambda_j$9, versus full Forest-Diffusion’s $$W=[\,v_1\;v_2\;\cdots\;v_k]\in\mathbb{R}^{d\times k}.$$0 in RF/XGB. The Wasserstein distance between real and synthetic points in latent space was reported as only $$W=[\,v_1\;v_2\;\cdots\;v_k]\in\mathbb{R}^{d\times k}.$$1, far below GAN-based CTGAN or plain Forest-Diffusion, indicating tight distributional similarity. Privacy proxies—nearest-neighbor distance ratio and distance-to-closest-record—remained on par with or better than baselines.

The broader empirical summary in the same work states that PCAForest yields near-state-of-the-art minority-class recall and F1, strong distributional fidelity, and substantially reduced generation time. Within the three-model family, AttentionForest attains the best average minority recall while maintaining competitive precision, calibration, and distributional similarity, whereas PCAForest offers an accuracy-efficiency trade-off through much faster generation.

The name “PCAForest” can invite confusion with a distinct line of work based on randomized PCA trees. That alternative method, “Randomized PCA Forest for Outlier Detection,” is an unsupervised outlier detection method that utilizes RPCA Forest for outlier detection and combines leaf-depth statistics with mean in-leaf distances to produce an outlier score (Rajabinasab et al., 18 Aug 2025). It addresses a different task, uses a different tree construction, and is not the latent-space, conditional-flow-matching oversampler introduced in (Ihsan et al., 20 Nov 2025).