--- title: Bidirectional Earth Mover Distance Loss url: https://www.emergentmind.com/topics/bidirectional-earth-mover-distance-emd-loss type: topic --- # Bidirectional Earth Mover Distance Loss Bidirectional Earth Mover Distance (EMD) loss denotes a family of optimal-transport-based training objectives built around the Earth Mover’s Distance, i.e., the 1-Wasserstein distance, but interpreted in more than one way across the literature. In the balanced setting, where the compared measures have equal total mass, EMD is symmetric, so a formally “bidirectional” loss obtained by summing both argument orders is redundant and reduces to a constant multiple of the same metric. In practice, however, the phrase is also used for asymmetric generalizations that penalize opposite transport directions differently, for unbalanced variants that relax mass conservation, and for learned or regularized surrogates that average both input orders to enforce symmetry or improve matching fidelity in neural training pipelines [2106.02051, 1611.07573, 2306.04712, 2311.09998, 2403.13206]. ## 1. Formal definition and relation to Wasserstein geometry For discrete measures \(P\) and \(Q\) with equal total mass, classical balanced EMD is the minimum-cost transport problem \[ \min_{T \ge 0}\;\sum_{i=1}^{K}\sum_{j=1}^{K} T_{ij}\, d(i,j) \quad \text{s.t.} \quad T\,\mathbf{1} = P,\qquad T^{\top}\mathbf{1}=Q, \] with \(d(i,j)\) given by a ground metric. In continuous form, for probability measures \(\mu,\nu\), the same object is the 1-Wasserstein distance \[ W_1(\mu,\nu)=\inf_{\gamma\in\Pi(\mu,\nu)} \int \|x-y\|\,d\gamma(x,y). \] This establishes EMD as an optimal transport metric rather than a binwise discrepancy, which is why it encodes output-space geometry directly into the loss [1611.07573, 2306.04712]. In one dimension, EMD admits a closed form in terms of cumulative distribution functions. For probability measures \(u,v\) on \(\mathbb{R}\) with CDFs \(U,V\), \[ W_1(u,v)=\int_{\mathbb{R}} |U(t)-V(t)|\,dt. \] For discrete ordered histograms \(p=(p_1,\dots,p_B)\) and \(q=(q_1,\dots,q_B)\), with cumulative histograms \(C_p(i)=\sum_{k=1}^{i}p_k\) and \(C_q(i)=\sum_{k=1}^{i}q_k\), the standard discrete 1D EMD is \[ \mathrm{EMD}(p,q)=\sum_{i=1}^{B}|C_p(i)-C_q(i)|. \] With non-uniform bin widths \(\Delta z_i\), the corresponding discretization becomes \[ \mathrm{EMD}(p,q)=\sum_i |P_i-Q_i|\,\Delta z_i. \] This \(O(\#\text{bins})\) formulation is the basis of several practical losses for histogram regression and depth-distribution supervision [2106.02051, 2403.13206]. For chain-connected output spaces, the same structure appears as cumulative mass imbalance along adjacent edges. If \(\varphi_i=\sum_{j=1}^{i}(P_j-Q_j)\) and \(\hat{M}_i\) is the consecutive edge cost, then \[ W_1(P,Q)=\sum_{i=1}^{K-1}\hat{M}_i\,|\varphi_i|. \] Tree-connected label spaces admit an analogous decomposition over subtree mass differences, which allows hierarchical EMD losses to respect ontology structure such as WordNet [1611.07573]. ## 2. Symmetry, redundancy, and the meaning of “bidirectional” The central fact behind the term is that balanced EMD is symmetric: \[ \mathrm{EMD}(P,Q)=\mathrm{EMD}(Q,P). \] Accordingly, if predictions and targets are normalized to equal mass, a bidirectional loss of the form \[ L_{\text{bi}}=\mathrm{EMD}(P,Q)+\mathrm{EMD}(Q,P) \] reduces to \[ L_{\text{bi}}=2\,\mathrm{EMD}(P,Q). \] For 1D histograms, this means that explicitly computing “both directions” adds no information. The same observation is made in settings as different as histogram regression, detector-image compression, NeRF depth supervision, and chain- or tree-structured classification [2106.02051, 1611.07573, 2306.04712, 2403.13206]. This symmetry depends on equal-mass transport. The cited works therefore normalize outputs to the probability simplex, to charge fractions, or to per-ray termination distributions summing to one. In NeRF supervision, the standard weights \(w_i=T_i\alpha_i\) are normalized as \[ p_i=\frac{w_i}{\sum_j w_j} \] precisely so that symmetric EMD applies along each ray. In compression for HGCAL, inputs are normalized by total charge so that \[ \sum_i x_i=\sum_i \hat{x}_i=1, \] again ensuring balanced transport [2403.13206, 2306.04712]. The term “bidirectional” becomes substantive only when symmetry is absent or only approximately realized. Three situations recur. First, mass mismatch motivates unbalanced OT. Second, asymmetric losses deliberately weight opposite discrepancy directions differently. Third, approximate neural surrogates may not satisfy exact transport constraints, so averaging both input orders can reduce bias or enforce metric symmetry. DeepEMD is a clear example: its predicted attention is row-normalized but not column-constrained, so it is trained and evaluated in both directions. Likewise, a learned EMD CNN for calorimeter images enforces symmetry by construction through \[ f(x_1,x_2)=\frac{1}{2}\big(g(x_1,x_2)+g(x_2,x_1)\big). \] These are architectural uses of bidirectionality, not changes to the underlying balanced metric [2306.04712, 2311.09998]. ## 3. Asymmetric extensions: Earth Mover’s Pinball loss and unbalanced transport A principal asymmetric generalization is the Earth Mover’s Pinball (EMP) loss, introduced for histogram-valued regression. It applies the scalar pinball loss \[ \rho_\tau(u)=u(\tau-\mathbf{1}\{u<0\}) \] to cumulative histogram discrepancies \(u_i=C_p(i)-C_q(i)\), yielding \[ L_{\mathrm{EMP}}^\tau(p,q)=\sum_{i=1}^{B}\rho_\tau\!\big(C_p(i)-C_q(i)\big). \] At \(\tau=0.5\), \[ L_{\mathrm{EMP}}^{0.5}(p,q)=0.5\sum_{i=1}^{B}|C_p(i)-C_q(i)|, \] so median EMP is equal to EMD up to a constant factor. The paper states that this constant factor does not change the optimum and can be ignored or folded into the learning rate; with that understanding, median EMP recovers the standard EMD [2106.02051]. EMP is asymmetric for \(\tau\neq 0.5\). Specifically, \[ L_{\mathrm{EMP}}^\tau(p,q)\neq L_{\mathrm{EMP}}^\tau(q,p) \] in general, while \[ L_{\mathrm{EMP}}^\tau(p,q)=L_{\mathrm{EMP}}^{1-\tau}(q,p). \] This replaces redundant bidirectional EMD with a controlled directional preference. The cited formulation interprets \(\tau<0.5\) as penalizing underestimation of cumulative mass more heavily, and \(\tau>0.5\) as penalizing overestimation more heavily. A symmetrized EMP can be formed by averaging the two directions, \[ \frac{L_{\mathrm{EMP}}^\tau(p,q)+L_{\mathrm{EMP}}^\tau(q,p)}{2} = \frac{L_{\mathrm{EMP}}^\tau(p,q)+L_{\mathrm{EMP}}^{1-\tau}(p,q)}{2}, \] which recovers EMD at \(\tau=0.5\) [2106.02051]. A different departure from symmetry arises when the compared objects do not have equal total mass. In that case classical EMD is undefined, and unbalanced OT augments transport with marginal relaxations, for example \[ W_{\tau}(P,Q)=\min_{\gamma\ge 0,\,\pi_P,\,\pi_Q} \int c(x,y)\,d\gamma(x,y) +\tau\Big(\mathrm{KL}(\pi_P\|P)+\mathrm{KL}(\pi_Q\|Q)\Big). \] This construction is motivated in settings with leftover background mass, occlusion-induced truncation, or non-normalized energy. The cited NeRF work does not use unbalanced OT, because it normalizes per-ray weights and uses a single symmetric EMD/Sinkhorn term; the detector-compression work similarly recommends normalization unless preserving absolute energy is part of the objective, in which case an additional energy penalty may be added [2403.13206, 2306.04712]. ## 4. Gradients, relaxations, and differentiable optimization In one dimension, exact EMD and EMP are amenable to direct backpropagation through cumulative sums. For EMP, with \(u_i=C_p(i)-C_q(i)\), the pinball subgradient is \[ \frac{\partial \rho_\tau(u_i)}{\partial u_i}=\tau-\mathbf{1}\{u_i<0\}, \] with subgradient at \(u_i=0\) any value in \([\tau-1,\tau]\). Propagated to per-bin predictions, \[ \frac{\partial L_{\mathrm{EMP}}^\tau}{\partial p_k} = \sum_{i=k}^{B}\big[\tau-\mathbf{1}\{u_i<0\}\big]. \] For discrete 1D EMD with bin widths \(\Delta z_i\), \[ \frac{\partial}{\partial p_j}\,\mathrm{EMD}(p,q) = \sum_{i\ge j}\mathrm{sign}(P_i-Q_i)\,\Delta z_i \] almost everywhere. These formulas explain why cumsum-based implementations are both exact and computationally light in ordered spaces [2106.02051, 2403.13206]. Nonsmoothness is a recurrent obstacle. One response is smoothing. EMP admits a softplus approximation \[ \rho_\tau^\alpha(u_i)\approx -\tau\delta_i+\alpha\log(1+\exp(\delta_i/\alpha)), \] where \(\delta_i=C_q(i)-C_p(i)\). As \(\alpha\to 0\), the gradients recover pinball subgradients. Another response is relaxation. For chain-connected or tree-connected outputs, the relaxed loss \[ W_1^\rho(P,Q)=\sum_{i=1}^{K-1}\hat{M}_i|\varphi_i|^\rho \] is proposed, with \(\rho=2\) recommended as EMD\(^2\). The cited analysis argues that \(\rho=2\) yields smooth, real-valued gradients with nonzero Hessians, improving convergence while preserving mass. In reduced-data ImageNet-1k experiments, this relaxed EMD loss achieved better Top-1 accuracy than cross entropy loss [1611.07573]. A second response is entropic regularization. Sinkhorn Distance replaces the linear program with \[ \operatorname{SD}_\lambda(P,Q) = \inf_{T\in U(P,Q)} \Big(\langle M,T\rangle-\frac{1}{\lambda}\langle T,\log T\rangle\Big), \] which is differentiable and solved by iterative matrix scaling. The cited analysis, however, emphasizes \(O(K^2)\) memory and per-iteration cost, sensitivity to the iteration limit, float32 numerical issues caused by the exponential kernel, and practical gradient asymmetry unless \(\lambda\) is large and iterations are ample. Bidirectional averaging can damp numerical asymmetry, but the underlying stability problem remains [1611.07573]. A third response is to learn a differentiable surrogate. In detector compression, a CNN is trained to regress exact POT EMD values and then frozen as a loss; in point clouds, DeepEMD trains a Transformer to predict the matching matrix itself rather than the scalar EMD, because this gives accurate gradients with respect to point coordinates. Both strategies are explicitly designed to make EMD-like training feasible when exact optimal transport is too slow or insufficiently differentiable for repeated use inside a training loop [2306.04712, 2311.09998]. ## 5. Neural implementations across domains In histogram-valued regression, the standard workflow is: normalize predicted outputs with softmax to obtain a density histogram, compute cumulative sums, form cumulative residuals, apply pinball per bin, and sum over bins. The paper further describes quantile conditioning by feeding \(\tau\) as an extra input and sampling \(\tau\sim U(0,1)\) per batch element so that all quantiles can be learned simultaneously. Because cumulative histograms are monotone and bounded in \([0,1]\), this formulation yields monotone cumulative outputs within \([0,1]\) and exposes uncertainty via quantile bands rather than only a single median prediction [2106.02051]. In NeRF, EMD is applied not to rendered depth expectations but to the full ray termination distribution. Along a ray, \[ p(t)=T(t)\sigma(t), \] with discretized weights \[ \alpha_i=1-\exp(-\sigma_i\delta_i),\qquad T_i=\exp\!\Big(-\sum_{j