Modified Support Vector Regression (GBSVR)

• The paper demonstrates how aggregating data into granular balls reduces quadratic programming complexity from O(m^3) to O(n^3) while preserving predictive performance.
• The modified SVR formulation replaces individual data points with Granular Regression Balls, employing a split-and-refine procedure that enhances noise robustness and interpretability.
• Empirical results show that GBSVR achieves higher R², lower RMSE, and significantly faster training times compared to traditional SVR on both synthetic and real-world datasets.

Modified Support Vector Regression (SVR) Formulation refers to a family of approaches that adapt the standard SVR—whose quadratic programming (QP) complexity and sensitivity to outliers can inhibit large-scale or robust regression—in part by aggregating input data points into "granular balls" and performing regression on these coarse-grained summaries. The central objective is to improve computational efficiency, robustness to noise, and interpretability without significant loss in predictive power. The most prominent instantiation is Granular Ball Support Vector Regression (GBSVR), as documented in recent literature, notably "GBSVR: Granular Ball Support Vector Regression" (Rastogi et al., 13 Mar 2025).

1. Definitions and Mathematical Notation

Let the training data be $$D = \{ (x_i, y_i) \mid x_i \in \mathbb{R}^l,\, y_i \in \mathbb{R},\, i=1,\dots,m \}$$. GBSVR replaces individual data points with $n \ll m$ "granular regression balls" (GRBs):

$\text{GRB}_j = (c_j, r_j, \hat{y}_j)$

where:

• $c_j$ (center): $(1/u_j) \sum_{i \in G_j} x_i \in \mathbb{R}^l$ for $G_j \subset \{1,\dots,m\}$, $u_j = |G_j|$,
• $r_j$ (radius): either $\max_{i \in G_j} \|x_i-c_j\|$ or $(1/u_j) \sum_{i \in G_j} \|x_i - c_j\|$,
• $\hat{y}_j$ (aggregated target): $(1/u_j) \sum_{i \in G_j} y_i$.

These GRBs constitute an adaptive cover of the data space, coarsening representation for regions with low target-variance.

2. Construction of Granular Balls

GRBs are generated through a recursive split-and-refine procedure:

• Discretization: The continuous target $y$ is quantized into $k$ equal-frequency (quantile) bins. Each $y_i$ is assigned a discrete label $\ell_i \in \{1,\dots,k\}$.
• Purity Metric: For any GRB$_j$,

$$\text{quality}(\text{GRB}_j) = \frac{|\{i \in G_j: \ell_i = \text{majority label of }G_j\}|}{|G_j|}$$

• Splitting: If $\text{quality} < T$ (threshold) or $|G_j| > p$ (cardinality bound), split $G_j$ via 2-means clustering on $x_i$, update $c_j$, $r_j$, $\hat{y}_j$.
• Termination: Continue until all balls satisfy $\text{quality} \geq T$ and $|G_j| \leq p$.

The final set of $n$ balls succinctly captures both the geometry and local output structure of the data (Rastogi et al., 13 Mar 2025).

Parameter	Role	Tuning Range/Guideline
$k$	Target discretization granularity	Typically $[10, 100]$
$T$	Purity threshold	$[0.7,\ 0.99]$
$p$	Maximum ball cardinality	$[2,\, 10]$

3. Modified SVR Formulation on Balls

Classic $\epsilon$-SVR solves:

$$\min_{w,b,\xi,\xi^*} \frac{1}{2} \|w\|^2 + C \sum_{i=1}^m (\xi_i + \xi_i^*)$$

subject to

$$y_i - w^\top x_i - b \leq \epsilon + \xi_i \ w^\top x_i + b - y_i \leq \epsilon + \xi_i^* \ \xi_i, \xi_i^* \geq 0$$

In GBSVR, data points $(x_i, y_i)$ are replaced by balls $(c_j, r_j, \hat{y}_j)$:

• Each $x$ in $\text{GRB}_j$ can be expressed as $x = c_j + \Delta$, $\|\Delta\| \leq r_j$.
• The worst-case ball-wise constraints for each ball $j$ become:

$$\|w\| r_j + \hat{y}_j - w^\top c_j - b \leq \epsilon + \xi_j \ w^\top c_j + b - \hat{y}_j - \|w\| r_j \leq \epsilon + \xi_j^* \ \xi_j, \xi_j^* \geq 0$$

leading to the primal objective:

$$\min_{w,b,\xi,\xi^*} \frac{1}{2} \|w\|^2 + C \sum_{j=1}^n (\xi_j + \xi_j^*)$$

Thus, the QP size is reduced from $O(m)$ to $O(n)$, providing significant computational benefits (Rastogi et al., 13 Mar 2025).

4. Algorithmic Workflow

The modified SVR workflow proceeds as follows:

1. Granular Ball Generation.
   • Input: $D$, thresholds $T$, $k$, $p$.
   • Output: $\{\text{GRB}_j\}_{j=1}^n$.
2. Quadratic Programming.
   • Solve the dual of the GBSVR problem in $2n$ variables:

$$\max_{\alpha, \alpha^*} -\frac{1}{2}\|A\|^2 - \frac{1}{2}B^2 + \|A\|B + \sum_{j=1}^n (\alpha_j - \alpha_j^*) \hat{y}_j - \epsilon \sum_{j=1}^n (\alpha_j + \alpha_j^*)$$

where $A = \sum_{j=1}^n (\alpha_j - \alpha_j^*) c_j$ and $B = \sum_{j=1}^n (\alpha_j - \alpha_j^*) r_j$, subject to

$$\sum_{j=1}^n (\alpha_j - \alpha_j^*) = 0, \quad 0 \leq \alpha_j, \alpha_j^* \leq C$$

1. Model Extraction.
   • Recover $w = \tfrac{\|A\| - B}{\|A\|}A$; $b$ by complementary slackness.
2. Prediction.
   • For a test point $x$, assign to the nearest ball (usually by Euclidean metric), then predict using $w^\top x + b$.

This process incurs a complexity of $O(n^3)$ for solving the QP (versus $O(m^3)$ for standard SVR), with $n$ being the number of balls (Rastogi et al., 13 Mar 2025).

5. Empirical Performance and Robustness

Empirical results demonstrate the following:

• On synthetic datasets with heteroscedastic noise:
  • GBSVR achieves higher $R^2$ and lower RMSE than SVR and NuSVR. For example, on a type A noise function, GBSVR achieves $R^2=0.9823$, RMSE $=0.0459$ (compared to SVR $R^2=0.9760$, RMSE $=0.0528$).
• On real-world benchmarks (UCI, time series, wind prediction):
  • GBSVR produces a modest number of balls ($n\approx 50$–$150$ for $m=159$–$414$).
  • GBSVR maintains $<5\%$ $R^2$ loss under $20\%$ added Gaussian noise, while SVR loses $>20\%$.
  • Mean training time reduces from $\sim 50$s (SVR) to $\sim5$s (GBSVR).
• In ablation studies:
  • Purity threshold ($T$) and ball size ($p$) control tradeoffs between accuracy and granularity.
  • GBSVR robustly captures non-linear regimes—superior to con-MGSVR in sharp-variation scenarios (Rastogi et al., 13 Mar 2025).

6. Advantages, Limitations, and Interpretability

Advantages:

• Significant QP complexity reduction ($O(m^3) \to O(n^3)$).
• Enhanced robustness to outliers and noise due to aggregation at the ball level and splitting by purity/variance.
• Interpretability is increased: clusters of input-output pairs are directly observable via balls in joint $X \times Y$ space (Xia et al., 2023).
• Empirically fast: both training and inference times decrease by one or two orders of magnitude relative to pointwise methods; robust under heavy label noise (Xia et al., 2023, Rastogi et al., 13 Mar 2025).

Limitations:

• Hyperparameters $k$, $T$, and $p$ require tuning (typically by grid search or cross-validation).
• Discretization of $y$ may result in loss of target granularity, possibly limiting modeling of fine-variation.
• No current regression-ball variant of granular-ball twin SVM exists; GBTSVM and LS-GBTSVM focus solely on binary classification (Quadir et al., 2024).

7. Research Directions and Related Granular Approaches

Key avenues for further work include:

• Designing continuous or adaptive-purity metrics to remove dependence on fixed $k$-level discretizations.
• Extension to kernelized versions, multi-output regression, and scalable online training.
• Incorporation within neural and deep architectures to exploit both efficiency and interpretable multi-granularity structure (Xia et al., 2023).
• Adapting twin-SVM approaches to regression with granular balls is an open research area; current literature lacks such extensions (Quadir et al., 2024).

Granular-ball regression and GBSVR embody a paradigm shift in SVR methodology by aligning model complexity and prediction resolution with natural data granularity, yielding scalable, robust, and interpretable regression frameworks (Xia et al., 2023, Rastogi et al., 13 Mar 2025).