Compute-Equivalent Loss: Theory & Applications

• Compute-equivalent loss is a class of loss functions defined by preserving key optimization metrics rather than identical finite-sample values.
• It unifies approaches in causal uplift, deep reinforcement learning, and accuracy measures by leveraging constant and asymptotic equivalence criteria.
• Applications include dynamic compute allocation in ASR and bias correction in prioritized replay, offering insights into efficiency and performance trade-offs.

Compute-equivalent loss denotes a family of equivalence constructions rather than a single canonical object. In the cited literature, the term is used for losses that are not literally identical to a reference objective but are treated as equivalent under a specified criterion: optimization-equivalence up to an additive constant in causal uplift learning, expected-gradient equivalence under alternative sampling in deep reinforcement learning, and almost-sure asymptotic equivalence between level-based and share-based accuracy measures (Yang, 2020, Fujimoto et al., 2020, Coleman, 17 Nov 2025). Related work also places loss design in a compute-aware setting by adding a FLOP-based penalty to predictive loss in streaming ASR, while recent theory on loss shift shows that apparent equivalence between losses can fail even when the data distribution is fixed, because different losses may require different Bayes-relevant information (Xie et al., 2022, Sevetlidis, 11 Jun 2026).

1. Conceptual scope and criteria of equivalence

The common feature across these uses is that equivalence is defined relative to a preserved quantity, not by literal identity of formulas. In causal machine learning, the preserved quantity is the optimizer of mean-square-error had the unobserved individual lifts been available (Yang, 2020). In prioritized replay, the preserved quantity is the expected gradient on the parameters under a change of sampling distribution (Fujimoto et al., 2020). In the asymptotic theory of level-based and share-based losses, the preserved quantity is the large-sample ranking behavior of average losses up to a deterministic constant factor (Coleman, 17 Nov 2025).

This immediately excludes a frequent misconception: compute-equivalent loss does not generally mean that two procedures produce identical finite-sample objectives, identical updates at every step, or identical numerical values. The causal uplift construction is equivalent to MSE only up to a constant independent of the model parameters. The prioritized replay construction is equivalent only in expected gradient. The level/share construction is asymptotic and almost sure, not exact at finite $n$ (Yang, 2020, Fujimoto et al., 2020, Coleman, 17 Nov 2025).

A second misconception is that equivalence is necessarily preserved when one task is replaced by another. The loss-shift framework shows that this need not hold: if two losses induce different Bayes quotients, then a representation sufficient for one loss may be intrinsically insufficient for the other, even under the same joint law $P(X,Y)$ (Sevetlidis, 11 Jun 2026).

2. Optimization-equivalent MSE in causal uplift learning

In causal machine learning, the difficulty is that the individual lift $l_i=y_i^T-y_i^C$ is never directly observed, because each unit is observed either in treatment or in control, never both. Standard MSE,

$${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$

therefore cannot be computed as written (Yang, 2020).

The proposed remedy is to bin model predictions into groups and define the loss on those groups. If the model outputs discrete prediction values

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$

the subsets assigned to bin $n$ are

$$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$

The subset-average lift and global average lift are estimated from treatment and control outcomes: $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$

$$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$

The resulting loss is

$$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$

Its status as a compute-equivalent loss is established by decomposing MSE by bins and combining that decomposition with total variance: $P(X,Y)$0 The final term does not depend on $P(X,Y)$1, so minimizing $P(X,Y)$2 is optimization-equivalent to minimizing MSE (Yang, 2020).

The construction is described as universally applicable because the grouping structure is induced by the model’s own prediction values rather than by tree nodes, internal architecture, hidden intermediate variables, or feature-specific heuristics. The same paper also shows that gradient descent can be applied directly. The derivative contains a “bias” term,

$P(X,Y)$3

and an effective finite-difference correction for bin migration,

$P(X,Y)$4

The broader significance is methodological. Rather than training separate treatment and control outcome models and subtracting them, one can directly optimize a single parameterized model $P(X,Y)$5 against the causal loss. The loss therefore functions both as an evaluation criterion and as a trainable surrogate for unavailable counterfactual MSE (Yang, 2020).

3. Expected-gradient-equivalent loss in prioritized replay

In deep reinforcement learning, compute-equivalent loss is defined through expected gradients under different sampling distributions. If a loss $P(X,Y)$6 is trained on data sampled from distribution $P(X,Y)$7, its expected gradient is

$P(X,Y)$8

Using importance sampling, this can be rewritten under another distribution $P(X,Y)$9 as

$l_i=y_i^T-y_i^C$0

A new loss $l_i=y_i^T-y_i^C$1 is then defined so that

$l_i=y_i^T-y_i^C$2

which yields equality of expected gradients (Fujimoto et al., 2020).

The paper states the theorem in prioritized replay form. Given a data set $l_i=y_i^T-y_i^C$3 of $l_i=y_i^T-y_i^C$4 items, loss functions $l_i=y_i^T-y_i^C$5 and $l_i=y_i^T-y_i^C$6, and priority scheme $l_i=y_i^T-y_i^C$7, the expected gradient of $l_i=y_i^T-y_i^C$8 when $l_i=y_i^T-y_i^C$9 is sampled uniformly is equal to the expected gradient of $${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$0 when $${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$1 is sampled with priority $${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$2, if

$${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$3

If priorities are treated as constants with respect to backpropagation via a stop-gradient operator, a sufficient condition is

$${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$4

This equivalence is used to reinterpret Prioritized Experience Replay. For the family

$${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$5

the compute-equivalent uniform loss changes the effective exponent from $${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$6 to $${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$7. The paper uses this transformation to argue that PER can change the effective objective being optimized; for example, PER with MSE changes the effective loss exponent to $${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$8, which can increase outlier sensitivity (Fujimoto et al., 2020).

The same framework motivates LAP and PAL. LAP is a corrected prioritized replay variant with clipped low priorities, and PAL is its uniform-sampling equivalent. The key theoretical statement is that PAL and LAP have the same expected gradient. Empirically, the paper reports that LAP and PAL both match or outperform vanilla TD3 across MuJoCo tasks, that they perform nearly identically in MuJoCo, and that in MuJoCo environments non-uniform sampling can be replaced by adjusting the loss function instead. On Atari, LAP improves over PER on 9 of 10 games, while PAL improves over vanilla DDQN but underperforms PER on 6 of 10 games, indicating that actual prioritization remains more valuable in that setting (Fujimoto et al., 2020).

The paper also proves a minimum-variance result: among all loss/prioritization pairs with the same expected gradient, the minimum-variance gradient estimator is obtained by using the L1 loss and priority

$${\rm MSE}(\lambda)=\frac{1}{\|S\|}\sum_i (p_i-l_i)^2,$$9

This gives compute-equivalence a second role beyond reinterpretation: it becomes a design principle for jointly choosing loss and sampling scheme (Fujimoto et al., 2020).

4. Asymptotic equivalence between level-based and share-based losses

A third usage concerns level-based and share-based accuracy measures. Let $f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$0 be level values with totals

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$1

and shares

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$2

A level-based loss is $f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$3, whereas a share-based loss is $f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$4. Their sample averages are

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$5

Asymptotic equivalence is defined by

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$6

for some constant $f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$7, or equivalently

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$8

(Coleman, 17 Nov 2025)

The main theorem is proved for weighted exponentiated difference losses,

$f(\lambda,x_i)=p_i \in \{P_n\}_{n=1}^N,$9

with $n$0. The class includes absolute difference, squared difference, absolute percentage error, Webster/Saint-Laguë or $n$1, Huntington–Hill / equal proportions, and Cobb–Douglas-type loss. The bridge between level-space and share-space is the identity

$n$2

Under the paper’s regularity conditions—finite moments, weighted Cesàro boundedness or uniform integrability, stable total means, asymptotic regularity of weights, and sparse deviations—the average level-based loss and average share-based loss converge almost surely to a constant ratio. If

$n$3

then

$n$4

with $n$5 for $n$6-weights or $n$7 for $n$8-weights (Coleman, 17 Nov 2025).

The paper interprets this as almost sure asymptotic convergence of numerical accuracy and distributive accuracy. Its practical claim is that when losses are averaged over many units, differences in ratios and ranks become negligible when the average or summed difference between target values and realizations is near zero. In that sense, level-based and share-based formulations form an asymptotic compute-equivalent class: they can look different but become interchangeable for large-scale ranking and evaluation (Coleman, 17 Nov 2025).

5. Compute-cost regularization in streaming Transformer ASR

In streaming ASR, the relevant loss construction is compute-aware rather than equivalence-to-an-unobserved target. The objective is

$n$9

where $$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$0 is the standard neural transducer or RNN-T loss, $$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$1 is a differentiable penalty tied to cumulative FLOPs across the network for a streaming sequence, and $$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$2 is a tunable scalar balancing accuracy against compute (Xie et al., 2022).

The architecture introduces dynamic compute allocation through a lightweight arbitrator

$$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$3

with outputs at timestep $$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$4

$$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$5

These decisions toggle feed-forward execution, query computations in multi-headed attention, and key/value computations in multi-headed attention. The resulting sparse computation pathways are optimized end-to-end by Gumbel-Softmax or Gumbel-Sigmoid relaxation, with hard binary decisions used at evaluation time (Xie et al., 2022).

The compute term is defined conceptually as a FLOP-based cost over the selected sparse execution path. The paper does not provide a closed-form expanded equation for $$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$6 in the extracted material, but it describes it as being “accounted for by the cumulative number of FLOPs across the components of the network for a streaming sequence.” Differentiable optimization proceeds through sampled sparse masks. For key-based masking, the attention mask is

$$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$7

and attention is updated as

$$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$8

Training uses a two-stage schedule. During pre-training, $$S_n \equiv S_n^T \cup S_n^C, \qquad S_n^T=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^T\}, \qquad S_n^C=\{x_i \mid f(\lambda,x_i)=P_n,\ x_i\in S^C\}.$$9 for 120 epochs. During fine-tuning, the compute penalty is activated for the final 80 epochs, with $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$0 initialized to $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$1 and gradually increased to $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$2. The Gumbel temperature is annealed from $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$3 to $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$4, and the contribution scaling factor from $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$5 to $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$6 (Xie et al., 2022).

On LibriSpeech, the baseline full causal Transformer-Transducer has WER $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$7. The best reported trade-off uses a Dual-RNN arbitrator with query-key-based toggling, achieving $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$8 compute cost reduction and $$\bar{l}_n \equiv \frac{1}{\|S_n\|}\sum_{x_i\in S_n} l_i \approx \bar{y}_n^T - \bar{y}_n^C,$$9 WER, described as about a $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$0 relative WER increase versus the baseline. Random toggling achieves similar compute reduction, around $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$1, but drives WER above $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$2. This establishes that compute reduction alone is insufficient; the compute penalty must be optimized jointly with predictive loss (Xie et al., 2022).

6. Loss shift, Bayes quotients, and the limits of compute-equivalence

The loss-shift framework places an important limit on any broad reading of compute-equivalence. Transfer learning is usually studied as a consequence of distribution shift, but the framework identifies an orthogonal failure mode in which the data distribution is fixed and the loss changes. Under a fixed joint law $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$3, one loss may require a coarser representation than another. The formal object is the Bayes quotient $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$4, the statistic of $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$5 that exactly captures the Bayes-relevant information for loss $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$6 (Sevetlidis, 11 Jun 2026).

A representation $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$7 is Bayes-sufficient for $$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$8 when

$$\bar{l} \equiv \frac{1}{\|S\|}\sum_i l_i \approx \bar{y}^T - \bar{y}^C.$$9

Losses are ordered by refinement: $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$0 Strict refinement,

$$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$1

means that $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$2 distinguishes strictly more than $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$3. The immediate consequence is a qualitative obstruction: if the target loss is strictly finer than the source loss, then a source-minimal representation cannot be sufficient for the target. If $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$4 is Bayes-minimal for $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$5, then

$$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$6

but strict refinement gives

$$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$7

so $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$8 cannot be Bayes-sufficient for $$L(\lambda) = \sum_{n=1}^N \frac{\|S_n\|}{\|S\|} \bigg[ (P_n-\bar{l}_n)^2 - (\bar{l}_n-\bar{l})^2 \bigg].$$9 (Sevetlidis, 11 Jun 2026).

For finite-output log loss, the paper derives an exact identity. If

$P(X,Y)$00

then

$P(X,Y)$01

and equivalently

$P(X,Y)$02

The paper explicitly identifies this as the precise quantitative meaning of “compute-equivalent loss” failure: two losses are effectively compute-equivalent only if they induce the same Bayes quotient, or at least if the source representation still retains everything the target quotient needs (Sevetlidis, 11 Jun 2026).

The binary classification example clarifies the point. Zero-one loss needs only the Bayes class

$P(X,Y)$03

whereas binary log loss needs the full $P(X,Y)$04. An accuracy-minimal representation that stores only $P(X,Y)$05 is therefore sufficient for classification but insufficient for probabilistic prediction, with excess log risk

$P(X,Y)$06

The experiments reinforce the theoretical distinction. In controlled binary, learned bottleneck, dSprites, and CIFAR-10H settings, representations that were classification-equivalent or majority-label-accuracy-equivalent nonetheless differed sharply in downstream log-loss or soft-label performance under a fixed data distribution. This suggests that compute-equivalence across losses is not guaranteed by unchanged data, unchanged accuracy, or similar upstream training dynamics; it depends on preservation of the Bayes-relevant quotient for the target loss (Sevetlidis, 11 Jun 2026).