Ratio-based Loss Functions

Abstract: Algorithms in machine learning and AI do critically depend on at least three key components: (i) the risk function, which is the expectation of the loss function, (ii) the function space, which is often called the hypothesis space, and (iii) the set of probability measures, which are allowed for the specified algorithm. This paper gives a survey of a certain class of loss functions, which we call ratio-based. In supervised learning, margin-based loss functions for classification tasks depending on the product of the output values $y_i$ and the predictions $f(x_i)$ as well as distance-based loss functions depending on the difference of $y_i$ and $f(x_i)$ for regression are common. Distance-based loss functions are in particular useful, if an additive model assumption seems plausible, i.e. the common signal plus noise assumption. However, in the literature, several loss functions proposed for regression purposes have a multiplicative error structure in mind and pay attention to relative errors, i.e. to the ratio of $y_i$ and $f(x_i)$. In this survey article, we systematically investigate such ratio-based loss functions and propose a few new losses, which may be interesting for future research. We concentrate on investigating general properties of ratio-based loss functions like continuity, Lipschitz-continuity, convexity, and differentiability, because these properties play a central role in most machine learning algorithms. Therefore, we do not focus on some specific machine learning algorithm to derive universal consistency, learning rates, or stability results. Instead, we want to enable future research in this direction.

• The paper presents a systematic framework for ratio-based loss functions that leverage the ratio between prediction and target to achieve intrinsic scale invariance.
• It rigorously analyzes key properties such as continuity, differentiability, convexity, and Lipschitz continuity, essential for effective empirical risk minimization.
• The work catalogs various loss families, including logarithmic, hyperbolic cosine, and robust variants, providing practical guidance for applications in regression, survival analysis, and beyond.

Ratio-based Loss Functions: A Systematic Framework

Introduction

The paper "Ratio-based Loss Functions" (2605.05808) introduces and systematically investigates a generalized class of loss functions for supervised learning where the penalization structure is defined via the ratio between prediction and target, instead of their difference. In contrast to the classical distance-based or margin-based losses (which target additive error models under a signal-plus-noise assumption), ratio-based losses are natural for multiplicative error models and provide intrinsic scale-invariance, which is invaluable in regression problems involving positive targets (e.g., survival analysis, economics, or physical measurements constrained to $\mathbb{R}_{>0}$). The paper establishes a unifying definition, investigates fundamental mathematical properties essential for application in empirical risk minimization, and provides an extensive catalog of both existing and novel ratio-based losses. The analysis is complemented by a detailed study of their continuity, differentiability, convexity, and robustness.

Formal Definition and Motivation

A ratio-based (rb) loss is defined as any measurable mapping $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$ that can be written, up to a monotonic link function $u: \mathbb{R} \to Y$ and an offset $c\geq 0$, as

$$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$

where $Y$ is the codomain of the targets (typically, an open interval of positive reals), and $\ell$ is a representing function with $\ell(1) = 0$. The mapping $u$ ensures compatibility between possibly unconstrained predictors and positive-valued targets. This definition generalizes previous "relative error" constructions and systematizes their connection to common statistical modeling contexts (e.g., log-link GLMs, accelerated failure time models, Poisson regression).

The rationale for ratio-based losses is clear: in contexts where the magnitude of the data is meaningful only up to a scale or when distributions exhibit multiplicative variability, ratio-based penalization provides scale-invariant evaluation and model fitting, which additive/difference-based loss functions fundamentally fail to capture.

Figure 1: Plots of the representing functions $\ell$ using the logarithm; Huber-type logarithmic relative loss with parameter $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$0.

Structural Properties

Continuity, Differentiability, and Transformations

The paper establishes formal results on when these loss functions inherit standard regularity required for optimization and statistical analysis. Continuity and differentiability of $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$1 with respect to predictions are inherited from $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$2 and $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$3, as encoded in Lemmas 1 and 2. Notably, the design enables the utilization of well-known link functions (e.g., softplus, log, logistic) to satisfy problem-specific codomain restrictions.

Convexity

Convexity analysis reveals critical differences from the distance-based regime. The construction of convex ratio-based losses requires explicit symmetrization and careful crafting of $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$4 to ensure convexity is not lost due to the nonlinear transformation $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$5, as shown via the general family:

$$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$6

where $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$7 is constructed to satisfy suitable monotonicity of its first and second derivatives. The paper provides several representatives—some convex, some non-convex—clarifying structural constraints.

Figure 2: Plots of the representation functions $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$8 using logarithm and hyperbolic cosine.

Lipschitz Continuity

Lipschitz properties, critical for generalization and robust empirical risk minimization (ERM), are established whenever both $$L: X \times \mathbb{R} \times \mathbb{R} \to [0,\infty)$$9 and $u: \mathbb{R} \to Y$0 are (locally/globally) Lipschitz, with the domain offset $u: \mathbb{R} \to Y$1 ensuring boundedness below, which can be necessary for global control.

Scale Symmetry and Ratio-Symmetry

A salient feature of ratio-based design is the possibility of encoding ratio-symmetry: losses where over- and under-estimation by reciprocal factors are penalized equally; formalized as $u: \mathbb{R} \to Y$2. The paper also makes clear situations where asymmetry is preferable, and designs parameterized weighting to enable controlled asymmetry (e.g., as in quantile regression's pinball loss).

Catalog of Ratio-based Losses

A core contribution is a comprehensive catalog of specific ratio-based representation functions, which enriches the options available for application-optimized model design. The losses are grouped according to analytical structure and desirable statistical properties.

• Logarithmic family: squared log-loss $u: \mathbb{R} \to Y$3, absolute log-loss $u: \mathbb{R} \to Y$4, and their Huber-type hybrid. These tie to multiplicative error models and link naturally with log-transformed regression.
• Hyperbolic cosine family: log-cosh and cosh-log variants provide robust, smooth, and symmetric penalization.
• Absolute and squared relative error: $u: \mathbb{R} \to Y$5 and $u: \mathbb{R} \to Y$6, with Huber-type smoothing for robust interpolation.
• Inverse/Anti-underestimation focus: $u: \mathbb{R} \to Y$7, $u: \mathbb{R} \to Y$8 penalize underestimation more heavily.
• Least Absolute/Least Product Relative Error (LARE/LPRE): $u: \mathbb{R} \to Y$9 and $c\geq 0$0 generalize prominent robust regression criteria.
• General Relative Loss: Employs arbitrary symmetric/weighted penalties via $c\geq 0$1.
• Insensitive and robust variants: Analogues of $c\geq 0$2-insensitive support vector regression, as well as piecewise robust (Hampel-type) and smooth bounded losses, offering tunable resistance to outliers and local insensitivity.

Figure 3: Plots of the representing functions $c\geq 0$3 of absolute relative, squared relative, and Huber-type relative loss functions; Huber-type relative loss for parameter $c\geq 0$4.

Figure 4: Plots of the representing functions $c\geq 0$5 of LARE loss functions; Huber-type least absolute relative loss function's representation $c\geq 0$6 with parameter $c\geq 0$7.

All examples are presented with precise mathematical definitions and graphical plots, displaying their shape and regularity properties.

Figure 5: Plot of LPRE's representing function $c\geq 0$8.

Theoretical and Practical Considerations

Relationship to Distance-based Losses

A detailed analysis establishes that, while any (nontrivial) ratio-based loss can be written—after suitable logarithmic or exponential transformations—as a distance-based loss, and vice versa, this correspondence does not preserve interpretability with respect to the untransformed variables, nor does it trivialize the choice between the two in practical modeling. The essential differences in invariance structures (shift-invariance for distance, scale-invariance for ratio) and in their sensitivity to systematic multiplicative versus additive model errors are thoroughly characterized. This has direct implications for the design of learning objectives in regression settings where scale and units are meaningful.

Nemitski Loss and Regularization

The paper demonstrates that ratio-based losses often fulfill the Nemitski property, which is a technical requirement ensuring that the expected risk is well defined for broad classes of measurable hypotheses, important for kernel-methods, empirical risk minimization over Banach/Hilbert spaces, and existence of minimizers.

Robustness and Generalization

By cataloging robust, insensitive, and Hampel-type variants, the paper provides guidance on constructing loss functions with bounded influence and well-controlled sensitivity to outliers and small errors—key both for robust statistics and modern deep learning.

Figure 6: Plots of the representing functions $c\geq 0$9 of robust loss based on maximum loss function; Left: loss without insensitivity ($$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$0) for various $$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$1; Right: robust loss functions with insensitivity values $$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$2 and parameter $$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$3.

Implications and Future Research Directions

Practical consequences of this framework include improved design of regression losses in positive-valued or multiplicative error domains, alternative risk structures for survival analysis/time-to-event modeling, and richer avenues for robust regression and deep learning under scale or measurement invariance. The theoretical analysis sets the ground for further study, particularly:

• Characterizing representer theorems, sample complexity, and rates for ERM under ratio-based losses.
• Deep integration with kernel methods or other nonparametric hypotheses spaces.
• Extension of ratio-based principles to structured outputs or other ranges.
• Investigation of empirical performance and calibration in regression and probabilistic forecasting tasks.
• Analytical characterization of convex and Lipschitz regions for parameterized ratio-based loss classes.

Figure 7: Plots of the representing functions $$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$4 of weighted loss functions for various weights $$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right),$$5.

Conclusion

This paper provides a rigorous and comprehensive theoretical foundation for the use of ratio-based loss functions in supervised learning. The general definition, analysis of mathematical properties, and systematic cataloging of both known and novel loss families establish the groundwork required for their principled deployment in statistical learning. While distance-based and ratio-based losses are interchangeable in certain transformed variable spaces, their interpretative and modeling characteristics make ratio-based losses more suitable in multiplicative or scale-variant settings. The work formalizes theory sufficient for immediate application in robust, scale-invariant regression and prompts further research in convex, Lipschitz, and robust loss design for advanced learning systems.

Reference:

"Ratio-based Loss Functions" (2605.05808)