HL-Gauss: Gaussian Histogram Loss for Regression

Updated 14 July 2026

HL-Gauss is a histogram-based regression loss that transforms scalar targets into truncated Gaussian distributions to achieve smoother, more stable gradient optimization.
It leverages a softmax output layer and minimizes KL-divergence between the target and predicted histograms, reframing regression as a distribution matching problem.
Empirical results across benchmarks like CT Position and Bike Sharing demonstrate that HL-Gauss offers improved accuracy with minimal hyperparameter tuning.

Gaussian Histogram Loss, usually abbreviated HL-Gauss, is a histogram-based distributional regression loss in which a scalar target is replaced by a truncated Gaussian target distribution centered at the observed label, while the model predicts a categorical histogram over a bounded output range and is trained by cross-entropy or, equivalently up to an additive constant, by minimizing a KL-divergence from target to prediction. The method reframes scalar regression as distribution matching: the network learns an estimate of the conditional distribution $Y \mid \mathbf{x}$ , and the final regression output is obtained from the mean of the predicted histogram rather than from a directly regressed scalar (Imani et al., 2018, Imani et al., 2024).

1. Historical emergence and problem setting

HL-Gauss emerged from work on distributional losses for regression, motivated by the observation that supervised learners can benefit when hard targets are converted to soft targets. The 2018 paper “Improving Regression Performance with Distributional Losses” introduced the method as a novel distributional regression loss and argued that its gains are associated less with learning richer semantics than with producing better-behaved gradients during optimization (Imani et al., 2018). The 2024 paper “Investigating the Histogram Loss in Regression” revisited the method as a “recent approach to regression” and undertook a more detailed theoretical and empirical analysis of why and when the performance gain appears (Imani et al., 2024).

The general setting assumes that the output variable lies in a bounded support $[a,b]$ . Instead of directly fitting $f(\mathbf{x}) \approx y$ with $\ell_2$ or $\ell_1$ , the model emits a probability vector over $k$ bins, and each scalar label $y_j$ is converted into a target distribution whose bin masses can be computed through its CDF. In the HL-Gauss instantiation, this target distribution is a Gaussian-smoothed neighborhood around the label, truncated to the histogram support (Imani et al., 2024).

A central conclusion shared across the regression analyses is that the advantage of HL-Gauss is not adequately explained by representation learning, by a generic auxiliary-task effect, or by simply “learning a distribution.” The reported evidence instead points to the geometry of the loss and the associated optimization dynamics as the main source of the gain (Imani et al., 2024).

2. Mathematical formulation

Let the support $[a,b]$ be partitioned into $k$ bins, with bin $i$ occupying $[a,b]$ 0. The model outputs a softmax vector

$[a,b]$ 1

which is interpreted as a histogram density: the probability mass in bin $[a,b]$ 2 is $[a,b]$ 3, and the corresponding density value inside that bin is $[a,b]$ 4 (Imani et al., 2024).

With a softmax parameterization, the bin probabilities are

$[a,b]$ 5

Here $[a,b]$ 6 denotes the learned feature representation and $[a,b]$ 7 the output-layer parameters for bin $[a,b]$ 8 (Imani et al., 2024).

For a target distribution with CDF $[a,b]$ 9, the target mass assigned to bin $f(\mathbf{x}) \approx y$ 0 is

$f(\mathbf{x}) \approx y$ 1

The Histogram Loss is then

$f(\mathbf{x}) \approx y$ 2

which is the discrete cross-entropy between the target histogram and the predicted histogram (Imani et al., 2018, Imani et al., 2024).

For HL-Gauss, the target distribution is a truncated Gaussian centered at the label $f(\mathbf{x}) \approx y$ 3, with variance $f(\mathbf{x}) \approx y$ 4, over $f(\mathbf{x}) \approx y$ 5. Its truncation normalizer is

$f(\mathbf{x}) \approx y$ 6

and the target mass in bin $f(\mathbf{x}) \approx y$ 7 is

$f(\mathbf{x}) \approx y$ 8

This construction spreads target mass across neighboring bins in a controlled way through $f(\mathbf{x}) \approx y$ 9, rather than collapsing the label into a one-hot categorical target (Imani et al., 2018, Imani et al., 2024).

At inference time, the regression prediction is the mean of the predicted histogram. In the RL formulation this is written explicitly as an expectation over bin centers $\ell_2$ 0: $\ell_2$ 1 which is the same principle specialized to value estimation (Chang et al., 2 Jul 2026).

3. HL-Gauss within the histogram-loss family

A generic histogram loss requires only three ingredients: a histogram output layer, a target distribution, and cross-entropy between target and predicted histograms. HL-Gauss is therefore a specific instantiation rather than the entire class (Imani et al., 2024).

Its distinctive features are the use of a truncated Gaussian target distribution, the explicit control of smoothing via $\ell_2$ 2, and the fact that bin masses are computed by integrating the Gaussian CDF over bin intervals rather than by heuristic assignment. The 2024 analysis also stresses the role of padding: because the Gaussian is truncated to $\ell_2$ 3, the support should be enlarged so that tail mass outside the histogram is negligible (Imani et al., 2024).

Variant	Target construction	Reported behavior
HL-OneBin	All mass in the single bin containing the label	Often much worse than HL-Gauss; associated with higher bias and more extreme gradients
HL-Uniform	Mixture of a delta target and a uniform distribution with smoothing parameter $\ell_2$ 4	Generally hurts performance; can bias the mean toward the center of the output range
HL-Projected	Mass assigned to the two nearest bins so the expected value equals the label exactly	Performs much better than HL-OneBin and close to HL-Gauss
HL-Gauss	Truncated Gaussian centered at the label with variance $\ell_2$ 5	Main recommended variant; usually performs best

The contrast with HL-OneBin is especially important. HL-OneBin is effectively standard cross-entropy over bins, with

$\ell_2$ 6

for the label bin $\ell_2$ 7. The empirical finding that HL-OneBin is often markedly worse than HL-Gauss indicates that the gain is not produced by the histogram representation alone, but by the particular form of Gaussian target smoothing (Imani et al., 2024).

A related point concerns HL-Uniform. Uniform smoothing might appear to be a generic label-smoothing substitute for Gaussian smoothing, but the reported results indicate that it generally hurts performance because it biases the histogram mean toward the center of the support. By contrast, HL-Projected removes discretization bias by design and comes close to HL-Gauss, although HL-Gauss still tends to win empirically (Imani et al., 2024).

4. Theoretical interpretation and optimization properties

The principal theoretical claim in the regression literature is that histogram losses help primarily through optimization effects, not because the network learns “extra information” in a richer probabilistic sense (Imani et al., 2018, Imani et al., 2024).

For the softmax-parameterized histogram, the last-layer gradient has the familiar cross-entropy form $\ell_2$ 8, and the 2024 analysis gives the bound

$\ell_2$ 9

This is contrasted with squared error, whose gradient scale depends on $\ell_1$ 0, which can be much larger and more variable. The reported interpretation is that HL-Gauss keeps gradients in a smaller, more stable range, so gradient descent makes steadier progress (Imani et al., 2024).

The same body of work also analyzes bias introduced by histogram discretization. When the target distribution is Gaussian and the bins are sufficiently wide and padded, the discretization bias in the histogram mean is bounded by half the bin width: $\ell_1$ 1 This matters because HL-Gauss is not merely a softer version of a one-bin classifier; with appropriate binning and smoothing, the mean of the best histogram approximation remains close to the label (Imani et al., 2024).

A further theoretical result links histogram matching to scalar prediction accuracy. If the KL divergence between target and prediction is small, then the squared error between their means is bounded: $\ell_1$ 2 This provides a direct formal connection between minimizing histogram cross-entropy and improving the quality of the decoded scalar mean (Imani et al., 2024).

The literature also notes a connection to entropy-regularized search or maximum-entropy RL, in the sense that HL can be interpreted as minimizing divergence to an exponentiated reward distribution. The 2024 study presents this as an intuition for the improved optimization behavior rather than as a complete explanatory theory (Imani et al., 2024).

5. Empirical behavior and implementation practice

Across the reported regression benchmarks, HL-Gauss often improves on plain $\ell_1$ 3 loss. The 2024 study reports gains on CT Position, Song Year, Bike Sharing, Pole, time-series forecasting datasets, and Atari value prediction, and characterizes the recommended recipe as broadly viable without costly hyperparameter tuning (Imani et al., 2024).

The 2018 results provide concrete examples. On CT Position, the reported test MAE is 19.110 for $\ell_1$ 4 and 8.992 for HL-Gaussian, while test RMSE moves from 29.512 to 19.980. On Bike Sharing, the reported test MAE changes from 2899.84 to 2495.21, and RMSE from 4601.21 to 4182.06. On Song Year, the gain is described as smaller, with HL-Gaussian slightly outperforming or matching $\ell_1$ 5 (Imani et al., 2018).

The empirical analysis in 2024 systematically examines several alternative explanations and largely rejects them. The performance gain is reported to be not just representation learning, not just an auxiliary task, not just robustness to outliers, and not just label augmentation. A softmax output layer helps, but $\ell_1$ 6+softmax is usually still worse than HL-Gauss. This pattern reinforces the view that the decisive factor is the loss geometry / optimization behavior (Imani et al., 2024).

Training curves and diagnostic measurements are consistent with that view. HL-Gauss is reported to reduce training error faster early in training, to exhibit more stable gradients than $\ell_1$ 7, and to benefit in some settings from annealing $\ell_1$ 8. The 2018 work likewise reports smaller, more stable gradients and faster and smoother convergence than $\ell_1$ 9 (Imani et al., 2018, Imani et al., 2024).

The reported hyperparameter behavior is structured rather than arbitrary. Too few bins increase bias. Larger bin counts generally do not cause overfitting. Varying $k$ 0 yields a U-shaped / v-shaped error curve, with performance degrading when $k$ 1 is either too small or too large. The analysis does not strongly support a classical bias–variance tradeoff story, because train and test errors often move together; this suggests that $k$ 2 acts mainly through smoothing, optimization, and truncation effects (Imani et al., 2024).

The practical recipe emphasized in the 2024 paper is deliberately simple: 100 evenly sized bins, $k$ 3 about 2 bin widths, and padding to enlarge the support, in some experiments described as about $k$ 4 on each side. The main regression experiments use Adam, minibatch size 256, standard neural nets with ReLU hidden layers, and a softmax over bins. The method is explicitly presented as not requiring expensive dataset-specific hyperparameter tuning to work reasonably well (Imani et al., 2024).

6. Extension to reinforcement learning and dynamic supports

HL-Gauss has also been applied to RL critics by reframing scalar Bellman targets as Gaussian-smoothed categorical targets over a support interval $k$ 5. The critic predicts a categorical distribution $k$ 6, the target bin probabilities $k$ 7 are obtained by integrating a truncated Gaussian over the bins, and the loss is

$k$ 8

The predicted value can then be decoded by taking the expectation over bin centers (Chang et al., 2 Jul 2026).

In this setting, a fixed support interval creates a specific difficulty. If the support is too narrow, target mass is truncated, causing truncation bias. If it is too broad, the same number of bins covers a larger range, so each bin is broader and quantisation bias increases. The problem is especially acute in RL because return scales are non-stationary as the policy changes during training (Chang et al., 2 Jul 2026).

The 2026 paper “Learning the Supports for Categorical Critic in Reinforcement Learning” addresses this limitation by proposing Dynamic Support Endpoint Learning (DySEL), which learns $k$ 9 and $y_j$ 0 jointly with the critic. The method derives an upper bound on the mean-squared Bellman error,

$y_j$ 1

and converts the support-selection problem into a constrained objective balancing a width penalty, the standard HL-Gauss cross-entropy term, and a coverage constraint based on the Gaussian mass $y_j$ 2 inside the support (Chang et al., 2 Jul 2026).

The resulting Lagrangian min-max objective is

$y_j$ 3

with the two principal forces interpreted as opposing: the width penalty favors a narrow support for tighter bounds and better resolution, while the mass constraint forces the support to remain wide enough to include most of the target Gaussian mass (Chang et al., 2 Jul 2026).

Empirically, DySEL is evaluated on 11 DeepMind Control Suite continuous-control tasks using TD3 as the base algorithm, against TD3 and TD3 + HL-Gauss with fixed support $y_j$ 4. The reported configuration uses $y_j$ 5 bins and sigma-to-width ratio $y_j$ 6. The reported finding is that TD3+DySEL is competitive with TD3+HLG on most tasks and improves on a subset, with especially strong gains on humanoid tasks. Two support-evolution patterns are observed: some tasks converge quickly to a stable interval, while others gradually expand the support during training. Ablations show that removing the width penalty causes support explosion and divergence, whereas removing the mass constraint yields supports that are too narrow and degrades performance (Chang et al., 2 Jul 2026).

A recurrent source of confusion is the term Histogram Loss itself. The 2016 paper “Learning Deep Embeddings with Histogram Loss” defines a batch-level ranking loss for metric learning by estimating positive and negative similarity distributions with 1D histograms and minimizing the probability that a negative pair has higher similarity than a positive pair. Its histogram estimator uses linear interpolation and is explicitly equivalent to a triangular kernel, not a Gaussian-smoothed regression target. It is therefore conceptually related to HL-Gauss at the level of differentiable histograms, but it is not HL-Gauss (Ustinova et al., 2016).

The same distinction applies to “Continuous Histogram Loss” (CHL), which generalizes the embedding-style histogram loss to continuous-valued pairwise similarities by constructing a 2D joint histogram over distances and similarities. CHL also uses triangular / piecewise linear binning, not Gaussian kernels, and its objective is the probability of reversed ordering in the joint $y_j$ 7 space. It is a histogram-based loss, but not the Gaussian regression formulation identified with HL-Gauss (Zholus et al., 2020).

A second disambiguation concerns Gaussian losses outside histogram regression. The 2022 rotated-detection paper “Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization” models boxes as Gaussians and trains with KLD, GWD, or BCD between predicted and target Gaussian distributions. Although this is Gaussian-distribution learning, it is not a histogram loss and should not be conflated with HL-Gauss (Yang et al., 2022).

The most important substantive misconception within the HL-Gauss literature is that its gain derives mainly from “learning the full conditional distribution.” The 2024 regression analysis argues against that interpretation and concludes that the benefits appear to come mainly from optimization effects, not from modeling extra information. A plausible implication is that HL-Gauss is best understood not as a generic density-estimation upgrade to regression, but as a distributionally parameterized surrogate objective whose Gaussian target smoothing and softmax histogram output create a more stable optimization landscape (Imani et al., 2024).