Feature Scaling in Machine Learning

• Feature Scaling is a set of data transformation techniques that standardize feature ranges and dispersions to ensure consistent algorithm behavior.
• It improves performance in scaling-sensitive models like SVM, KNN, and neural networks by mitigating issues of feature dominance and convergence lag.
• Advanced approaches such as supervised and dynamic scaling adapt to data distributions, label information, or temporal drift to enhance predictive accuracy.

Feature scaling comprises a set of transformations applied to the columns of feature matrices in machine learning, pattern recognition, and signal processing. Its objectives are (1) to standardize the ranges, central tendencies, or dispersions of features; (2) to enable efficient algorithmic behavior in downstream tasks such as regression, classification, clustering, and ranking; and (3) in certain advanced settings, to integrate supervised or structural information into the scaled representation. Classical techniques include “unsupervised” methods (min–max, z-score, robust scaling, etc.), while recent works introduce supervised, task-adaptive, and dynamic frameworks that adjust scaling based on data distribution, feature importance, label, or temporal drift. Model sensitivity to feature scaling is highly algorithm-dependent, with instance-, kernel-, and gradient-based learners strongly affected, and tree-based ensembles typically invariant under most scaling schemes.

1. Mathematical Definitions and Classical Scaling Techniques

Feature scaling is formally defined as the application of a transformation $f_j$ to each feature vector $x_{:,j}$ in the dataset $X\in\mathbb{R}^{n\times d}$. The twelve principal scaling techniques evaluated in (Pinheiro et al., 9 Jun 2025) are:

• Min–Max Normalization (MM): $x_{norm} = (x - x_{min})/(x_{max} - x_{min})$
• Max Absolute Scaling (MA): $x_{norm} = x / \max(|x|)$
• Z‐score Normalization (ZSN): $x_{norm} = (x - \mu)/\sigma$
• Variable Stability Scaling (VAST): $x_{norm} = [(x - \mu)/\sigma]\cdot(\mu/\sigma)$
• Pareto Scaling (PS): $x_{norm} = (x - \mu)/\sqrt{\sigma}$
• Mean Centering (MC): $x_{norm} = x - \mu$
• Robust Scaler (RS): $x_{norm} = (x - median)/IQR$
• Quantile Transformation (QT): $x_{norm} = F_{target}^{-1}(F_{empirical}(x))$
• Decimal Scaling (DS): $x_{norm} = x / 10^j$, $j = \min\{k: \max |x|/10^k < 1\}$
• Tanh Transformation (TT): $$x_{norm} = \frac{1}{2}[\tanh(0.01(x-\mu)/\sigma)+1]$$
• Logistic Sigmoid (LS): $x_{norm} = 1/(1+e^{-q})$, $q=(x-\mu)/\sigma$
• Hyperbolic Tangent (HT): $x_{norm} = (1-e^{-q})/(1+e^{-q})$, $q=(x-\mu)/\sigma$

These transformations mitigate issues of feature dominance in Euclidean distance, gradient magnitude instability, and convergence lag in iterative optimization. Classic preprocessing fits the scaler on training data and applies it to validation/test splits to avoid leakage. Robust scaling (RS, QT) is preferred for outlier-affected or skewed distributions; standardization (ZSN) is default for most tabular data.

2. Model-Specific Scaling Sensitivities and Empirical Evaluation

Pinheiro et al. (Pinheiro et al., 9 Jun 2025) conducted exhaustive empirical analyses with fourteen learning algorithms across sixteen UCI benchmarks. Sensitivity to scaling is strongly dependent on model family:

• Scaling-Invariant Models: Random Forest, gradient boosting (XGBoost, CatBoost, LightGBM), AdaBoost, Naive Bayes—performance variation $\Delta\text{accuracy}<1\%$ across all scaling regimes.
• Scaling-Sensitive Models:
  • SVM/SVR: Significant performance uplift under ZSN or RS; e.g., SVM (Dry Bean) jumps from 58.0% (NO) to 92.6% (ZSN).
  • KNN: Large benefit from ZSN/MM; e.g., accuracy increases from 71.1% (NO) to 92.2% (ZSN).
  • MLP/TabNet: Substantial gains via ZSN/PS; e.g., MLP (Dry Bean) moves from 29.8% (NO) to 93.1% (RS).
  • Linear/Logistic Regression: Marked accuracy improvements (e.g., +2.3 pp on Breast Cancer).
• Regression metrics: Scaling can shift MSE by 20–60% for sensitive models; robust scaling (RS, QT) effective for outliers/skew.

In contrast, tree-based ensembles perform equally across all scalers, suggesting scaling may be omitted to reduce memory and runtime overhead for these models.

3. Supervised and Dynamic Feature Scaling Approaches

Recent literature advances supervised and dynamic scaling methodologies that incorporate label or loss information, feature importance, or temporal adaptation:

a. Decision-Tree–Driven Scaling: DTization

Islam et al. (Islam, 2024) introduced DTization, which combines decision tree feature-importance assignment and robust scaling. For each feature $j$, a tree is constructed; the earliest depth at which $j$ is split in the tree determines a weight $w_j=\exp(x\cdot depth_j)$, $x=\log(2)/d$. Each column is robust-scaled $(x_{ij}-Q1_j)/(Q3_j-Q1_j)$ and then multiplied by $w_j$. Empirical studies across ten datasets show DTization consistently improves classification MCC and regression $R^2$ over unsupervised methods.

Classification Example:

Dataset	MCC (Other)	MCC (DTization)
Wine	0.6173	0.9157
Credit Card	0.2872	0.8332

Regression Example:

Dataset	$R^2$ (Other)	$R^2$ (DTization)
House Price	0.126	0.676

b. Feature-Importance/Dynamic Scaling for Instance-Based Learners

The FIDS scheme (Bhardwaj et al., 2018) uses out-of-bag (OOB) error permutation with Random Forest to compute raw importance scores $e_j$ for features. After normalization and thresholding, weights $w_j$ are used for axis scaling before KNN or other distance-based inference:

$$w_j = \frac{e_j - \mu_e}{\sigma_e}, \qquad w_j \leftarrow \max(0, w_j)$$

This approach automatically down-weights noisy features and captures non-linear interactions, yielding incremental but stable accuracy gains over standard Z-score normalization in instance-based tasks.

c. Online Dynamic Feature Scaling

Kaya et al. (Bollegala, 2014) demonstrated that static scaling fails under online or concept-drift regimes. Their dynamic feature scaling (DFS) method learns per-feature transforms (linear or sigmoid) jointly with classifier weights via SGD as data streams in, adapting to shifts without revisiting historical data. Convex DFS (FS-2 variant) reliably outperformed both static scaling and state-of-the-art passive-aggressive learners in one-pass accuracy, especially on binary classification (e.g., Heart dataset from 57% to 82%).

4. Feature Scaling in Structured and Invariant Learning Frameworks

a. Scale-Invariant Learning-to-Rank

As per (Petrozziello et al., 2024), scale-invariant LTR architectures explicitly partition features into “fixed-scale” and “scalable.” The scalable subset is processed via a log-linear wide path whose additive offset under $x\to c\cdot x$ cancels in pairwise ranking differences:

• For scalable features $x^{S}$, $$f_w(x^\mathsf{q}, x^{S}) = w^\top(q \otimes \log x^{S})$$.
• Under $x^{S} \rightarrow c x^{S}$, $\log(c x^{S}) = \log x^{S} + \log c$, and the ranking is preserved.

Experimental perturbations (price × 10, rating × 10, etc.) showed near-zero performance drop for the scale-invariant model compared to standard LTR, demonstrating robustness to train-test scaling inconsistencies in large-scale production deployments.

b. Spectral Feature Scaling for Dimensionality Reduction

Matsuda et al. (Matsuda et al., 2018) introduced supervised feature scaling for spectral clustering and dimensionality reduction. By fixing entries of the target Fiedler vector to known labels, the method derives feature-wise scales from the solution of a $(m+1) \times (m+1)$ generalized eigenproblem. The scaled data is then used in standard Laplacian eigendecomposition, improving linear separability and clustering robustness in high-dimensional, low-sample regimens, notably outperforming kernel LDA/LPP and unsupervised spectral clustering in both toy and gene-expression tasks.

5. Computational Costs, Implementation Guidelines, and Best Practices

Feature scaling introduces memory, preprocessing, and computational overheads but is crucial for enabling convergence and performance in scaling-sensitive models (Pinheiro et al., 9 Jun 2025):

• Memory: Lightweight scalers (DS, MA, MM) consume $\sim$0.2–4 kB; RS/QT/TT up to 400 kB per dataset.
• Preprocessing time: Simple scalers execute in $<1$ ms per feature; quantile-based methods notably higher.
• Model training cost: For MLP, overhead up to $2.2\times$ baseline; for tree ensembles, negligible ($\sim1\%$).
• KNN: Feature scaling dominates inference drift and should be consistently applied.
• For online/streaming: Estimate and adapt scale parameters on-the-fly as in DFS.
• Supervised scaling: Requires a pre-pass through supervised model (tree or RF) for weight determination.
• Avoid leakage: Fit scalers only on training data and apply to holdout sets for reproducibility.

Recommended practice is: use Z-score or robust scaling for SVM, KNN, linear/MLP/TabNet, skip scaling for tree ensembles or NB, employ supervised techniques (DTization, FIDS, DFS) when label or feature importance is accessible, and integrate scale-invariant design in structured models when production consistency cannot be guaranteed.

6. Limitations, Open Questions, and Future Directions

Limitations and deployment caveats include:

• Supervised scaling (DTization, FIDS) incurs extra model training cost and can overfit label-specific noise without regularization or cross-validation.
• Dynamic/online scaling (DFS) requires careful hyperparameter selection to avoid instability; theoretical global convergence guaranteed only for convex variants.
• Certain nonlinear or quantile-based scaling mechanisms inflate memory and preprocessing costs, limiting applicability for very large datasets.
• Scale-invariant frameworks require upfront feature partitioning and cannot auto-detect mis-scaled features. Negative or complex scaling factors (e.g. in spectral scaling) require careful regularization.
• Open questions include joint optimization of scaling and hyperparameters, scaling under severe imbalance, and extension to multi-class or multi-modal settings.

A plausible implication is that as machine-learning workflows move further into real-time, streaming, multi-modal, and production environments, future research will focus on robust, adaptive, and supervised scaling strategies that minimize preprocessing cost while maximizing predictive consistency and generalization. Recent developments consistently demonstrate that model-specific and context-specific scaling unlocks substantial performance improvements in most non-ensemble algorithms, and scale-invariance or dynamic adjustment is increasingly necessary in streaming and distributed applications.