Imbalanced Regression Framework

• Imbalanced regression is defined by skewed continuous target distributions where rare values are underrepresented, challenging conventional learning algorithms.
• Data-level techniques, such as weighted resampling and synthetic augmentation, enrich sparse target regions to improve overall model performance.
• Advanced loss functions and regionally weighted evaluation metrics ensure robust, fair treatment of minority targets across diverse application domains.

Imbalanced regression (IR) describes the scenario in supervised learning where the distribution of the continuous target variable is highly skewed, with certain ranges or key events substantially underrepresented. This phenomenon endangers model performance, as learning algorithms are prone to focus on densely sampled regions of the target space, while systematically neglecting rare but potentially critical values. Recent research has provided rigorous definitions, theoretical analyses, new metrics, and a growing toolset of principled algorithms to address this challenge across domains such as time-series forecasting, tabular and graph-based regression, and deep representations.

1. Theoretical Foundations and Problem Formalization

The IR framework is grounded in the observation that regression models, in minimizing the expected loss

$$\mathbb{E}[\mathcal{L}] = \int_\mathcal{Y} \hat{\mathcal{L}}(y) p_Y(y) dy$$

(where $\hat{\mathcal{L}}(y)$ is the per-target expected loss), are inherently shaped by the target distribution $p_Y(y)$ (Kowatsch et al., 19 Feb 2024). Densely populated regions in $\mathcal{Y}$ dominate optimization, causing standard learning algorithms to underfit rare (minority) targets. Unlike the discrete imbalance in classification, regression presents a continuum of target values; “imbalance” is not categorical but measured via a relevance measure $\mu$ on events in $\mathcal{Y}$. A domain is $\mu$-balanced if, for any sets $A,B \in \mathcal{A}$, $\mu(A) \leq \mu(B) \Rightarrow P_Y(A) \leq P_Y(B)$. This generalizes classical class imbalance and underscores challenges unique to regression.

Quantifying imbalance in regression necessitates new metrics. The Kolmogorov and Wasserstein metrics compare $p_Y$ with a reference or relevance distribution $\mu$, measuring the maximum or integrated deviations, respectively, to summarize imbalance severity. These metrics aid in both benchmarking and guiding experimental design (Kowatsch et al., 19 Feb 2024).

2. Algorithmic Strategies: Data-Level Techniques

2.1. Data Augmentation and Resampling

Data augmentation remains pivotal for counteracting IR. Methods split into those manipulating sample weights, resampling the covariates, or generating synthetic target values via parametric or neural generative models:

• Weighted Resampling and Data Augmentation (WR + DA): Observed in actuarial and survey contexts, the WR-DA pipeline first estimates the covariate density (via kernel estimation) and then samples or weights data in proportion to the inverse empirical density relative to a target distribution $f_0$. Synthetic augmentation (e.g., through Gaussian noise, SMOTE, GAN, Copula, or random forests) broadens support before weighted resampling ensures comprehensive coverage, limiting overfitting due to limited observations in rare regions (Stocksieker et al., 2023).
• Generative Modeling with VAEs and Smoothed Bootstrap: To tackle the complexity and sparsity of IR, recent approaches integrate VAEs trained with loss functions upweighting rare target values according to inverse kernel density (with power $\alpha$):

$$\mathcal{L}(\theta, \phi, x, y) = \beta_x \, \mathbb{E}_q[\log p_\theta(x|z)] - \beta_{KL} D_{KL}(q_\theta(z|x) \| p(z)) + \frac{\beta_y}{\hat{f}_Y(y)^\alpha} \mathbb{E}_q[\log p_\theta(y|z)]$$

Synthetic samples are drawn by smoothed bootstrap from the latent means, leveraging kernel mixtures to reliably populate rare regions (Stocksieker et al., 9 Dec 2024Stocksieker et al., 19 Aug 2025). Disentanglement penalties can be imposed to enhance latent independence, facilitating nonparametric bootstrapping.

• Cluster-Based and GMM-Based Segmentation: Other augmentation frameworks employ k-means or Mahalanobis-Gaussian mixture modeling to segment the joint feature–target space, automatically isolating rare groups without arbitrary thresholds. Oversampling is then performed via KDE-driven synthetic generation or GAN-based adversarial refinement, with GANs trained specifically on minority samples to capture the nonlinear joint structure (Alahyari et al., 19 Apr 2025Alahyari et al., 2 Aug 2025).
• Tree-Based Synthetic Generation: CART-based augmentation is adapted for IR by weighting samples (with kernel-based, DenseWeight, or relevance functions) inversely to local target density, enabling threshold-free augmentation focused on sparse regions. Conditional columnwise feature generation within tree leaves ensures the preservation of feature-target dependencies while maintaining computational efficiency and interpretability (Pinheiro et al., 3 Jun 2025).

2.2. Distribution Smoothing and Error-Aware Sampling

• Error Distribution Smoothing (EDS): EDS introduces a complexity-to-density ratio (CDR) by combining local function complexity (Frobenius norm of the Hessian) and data density. Data are adaptively selected to ensure that regions of high function complexity (and low density) are sufficiently represented, smoothing the distribution of prediction errors and reducing sample redundancy (Chen et al., 4 Feb 2025).
• Local Adaptive Oversampling: LDAO decomposes the dataset into a mixture of local clusters, learns the local density in each, and oversamples each region in proportion to its sparsity. This sidesteps the need for any binary rare/common threshold, enabling seamless balancing across the entire target distribution (Alahyari et al., 19 Apr 2025).

3. Loss Function Engineering and Model-Level Approaches

• Balanced MSE: Theoretical analysis reveals that standard MSE is statistically suboptimal for imbalanced regression, as it reflects the bias of $p_\text{train}(y)$. The Balanced MSE loss introduces a correction, statistically “converting” the prediction to match a balanced evaluation distribution:

$$\mathcal{L} = -\log N(y;\hat{y},\sigma^2) + \log \int_Y N(y;\hat{y},\sigma^2) p_\text{train}(y) dy$$

Implementation via Gaussian Mixture Models (GMM), batch-based Monte Carlo, or bin-approximate integration makes this approach scalable for high-dimensional tasks (Ren et al., 2022).

• Regularization and Representation Constraints: RankSim forces global ranking consistency between the label and feature space, maximizing Spearman correlation of neighbor ranks—thus ensuring structured representation of continuous targets in the feature space even under heavy imbalance (Gong et al., 2022). Surrogate-driven Representation Learning (SRL) employs enveloping and homogeneity losses to enforce uniform coverage and smoothness (equal spacing) in the latent space, particularly effective for operator learning and continuous regression (Dong et al., 2 Mar 2025).
• Mixture-of-Experts and Ensemble Methods: Uncertainty Voting Ensembles (UVOTE) leverage MoE architectures with negative log-likelihood (Laplace) losses and density-driven expert specialization, aggregating predictions based on estimated aleatoric uncertainty. This approach improves estimation quality, particularly in few-shot regimes, and provides calibrated uncertainty for interpretability (Jiang et al., 2023).
• Probabilistic Regression with Uncertainty Calibration: Methods such as HypUC apply kernel density–based weighting in the loss function to upweight rare targets and employ two-stage (global and hyperfine/binwise) uncertainty calibration, producing reliable error estimates especially relevant for high-stakes domains such as clinical ECG regression (Upadhyay et al., 2023).

4. Evaluation Metrics and Taxonomy

New evaluation metrics reflect the continuous, region-focused goals of IR:

• Regionally Weighted Metrics: SERA (Squared Error-Relevance Area) integrates squared error over the relevance-weighted target domain, emphasizing accuracy for critical rare regions. Likewise, Relevance-Weighted RMSE (RW-RMSE) incorporates a continuous relevance function $\phi(y)$:

$$\text{RW-RMSE} = \sqrt{\frac{\sum_i \phi(y_i) (y_i - \hat{y}_i)^2}{\sum_i \phi(y_i)}}$$

(Pinheiro et al., 3 Jun 2025).

• Pipeline Recommendation and Meta-Learning: The Meta-IR framework leverages meta-features (simple statistics and measures of data complexity) to recommend optimal combinations of balancing and learning methods for new tasks, greatly accelerating the model selection process over brute-force or standard AutoML approaches (Avelino et al., 16 Jul 2025).
• Taxonomies and Surveys: Recent work offers comprehensive taxonomies for IR methods, categorizing strategies by their focus on model type, learning process (e.g., data-level, loss-level, representation-level), and the evaluation metric, facilitating systematic comparison and benchmarking (Avelino et al., 16 Jul 2025).

5. Application Domains and Case Studies

The IR framework has been instantiated in diverse domains:

• Industrial Time-Series Forecasting: Frameworks using sample-specific weight functions to target rare, high-variation events (such as abrupt temperature increases in industrial machinery) have demonstrated the trade-off between performance on rare versus common regimes, with key gains measured by upper-bound test set RMSE (Silvestrin et al., 2021).
• Tabular and Deep Visual Regression: Balanced MSE, LDAO, and GAN-augmented methods have improved performance in age estimation, pose estimation, energy consumption, and fault diagnosis through explicit handling of rare value prediction (Ren et al., 2022Alahyari et al., 19 Apr 2025Alahyari et al., 2 Aug 2025).
• Graph-Based Molecular and Materials Prediction: Semi-supervised frameworks on graphs employ confidence-aware, reverse-sampled pseudo-labeling and label-anchored mixup on GNN representations, producing notable MAE improvements in the few-shot regime for properties of molecules and polymers (Liu et al., 2023).
• Medical Prognostics: Weighted regression with uncertainty calibration enables both accurate continuous estimation (e.g., of potassium or LVEF) and practical uncertainty quantification, with tested improvements over previous deterministic and probabilistic baselines in large ECG datasets (Upadhyay et al., 2023).
• Operator and System Identification: Geometric constraints in latent space, complexity-driven data selection, and error smoothing inform representation learning and data acquisition policies for system identification and scientific computing tasks, extending IR tools beyond conventional “regression” into operator learning.

6. Limitations, Open Questions, and Future Directions

The field of IR remains rapidly evolving and presents outstanding challenges and avenues for research:

• Many data-level methods require hyperparameter tuning (e.g., rarity exponent $\alpha$, kernel bandwidth, number of clusters), with success dependent on robust density estimation. Automated or adaptive parameterization is a promising direction.
• GAN and VAE approaches can be computationally expensive and exhibit training instability; hybrid strategies (e.g., combining fast tree-based methods with deep generative refinement) may offer practical balance.
• The design of continuous, interpretable relevance functions for real-world use remains domain-dependent and merits further standardization.
• There is a need for further theoretical work connecting the CDR, Kolmogorov/Wasserstein imbalance metrics, and their implications for generalization error bounds.
• Expanding IR solutions to multimodal, high-dimensional, or non-tabular domains (e.g., image, text, graph data) and unstructured outputs is an ongoing research frontier.

7. Summary Table: Major Methodological Families in Imbalanced Regression

Category	Core Principle	Example Papers
Data-Level Resampling	Reweight, resample, or generate new samples	(Stocksieker et al., 2023, Alahyari et al., 19 Apr 2025, Pinheiro et al., 3 Jun 2025)
Deep Generative Models	VAE/GAN synthetic data, often with weighted losses	(Stocksieker et al., 9 Dec 2024, Stocksieker et al., 19 Aug 2025, Alahyari et al., 2 Aug 2025, Alahyari et al., 29 Apr 2025)
Loss Function Engineering	Balanced MSE, uncertainty-weighted NLL, etc.	(Ren et al., 2022, Jiang et al., 2023, Upadhyay et al., 2023)
Distribution/Representation	Smoothing, geometric regularization, rank similarity	(Chen et al., 4 Feb 2025, Dong et al., 2 Mar 2025, Gong et al., 2022)
Meta-Learning & Pipeline Rec.	Meta-feature driven selection of optimal approach	(Avelino et al., 16 Jul 2025)

The IR framework thus encompasses a set of theoretical and algorithmic advances aimed at ensuring fairness, generalizability, and effectiveness when learning under target imbalance—a ubiquitous condition in regression problems across scientific, industrial, and decision-support settings.