Misaligned by Design: Incentive Failures in Machine Learning

Abstract: The cost of error in many high-stakes settings is asymmetric: misdiagnosing pneumonia when absent is an inconvenience, but failing to detect it when present can be life-threatening. Because of this, AI models used to assist such decisions are frequently trained with asymmetric loss functions that incorporate human decision-makers' trade-offs between false positives and false negatives. In two focal applications, we show that this standard alignment practice can backfire. In both cases, it would be better to train the machine learning model with a loss function that ignores the human's objective and then adjust predictions ex post according to that objective. We rationalize this result using an economic model of incentive design with endogenous information acquisition. The key insight from our theoretical framework is that machine classifiers perform not one but two incentivized tasks: choosing how to classify and learning how to classify. We show that while the adjustments engineers use correctly incentivize choosing, they can simultaneously reduce the incentives to learn. Our formal treatment of the problem reveals that methods embraced for their intuitive appeal can in fact misalign human and machine objectives in predictable ways.

• The paper demonstrates that incorporating human utility directly into training loss degrades performance by suppressing incentive for learning.
• It proposes a two-stage framework that separates information acquisition from decision making using ex post weighting of predictions.
• Empirical evaluations in medical imaging and computer vision show that unweighted training with ex post transformation outperforms direct utility-weighted training.

Incentive Failures in Machine Learning: A Formal Analysis of Misalignment

Overview

"Misaligned by Design: Incentive Failures in Machine Learning" (2511.07699) offers a formal and empirical study of the mechanisms underlying incentive misalignment in supervised learning models, especially in settings with asymmetric costs of error such as medical diagnosis. The core finding contests the "aligned learning premise" (ALP)—the prevailing design practice of training models using utility-weighted (asymmetric) loss functions encoding human preferences about error types. Through theoretical analysis rooted in Bayesian and incentive design models, accompanied by robust empirical results on deep learning tasks (chest X-ray classification, CIFAR image classification), the paper demonstrates that direct incorporation of these human objectives into the training loss often degrades performance on the very metric it aims to optimize. The authors advocate a separation between learning and decision-making: train on a strictly proper, standard (unweighted) loss function, and implement human objectives exclusively through ex post transformation of predictions.

Theoretical Foundation

The theoretical framework treats model training as a two-stage incentive problem: (1) learning (information acquisition mapping features to calibrated posterior distributions over labels), and (2) choice (transforming these posteriors into decisions or predictions, possibly under asymmetric human objectives). The key insight is that utility-weighted loss functions—while optimal for incentivizing choice decisions—distort the gradients and thus suppress the marginal value of learning, often causing suboptimal information acquisition.

Let $u(a, y)$ denote the utility of action $a$ in state $y$; for class-weighted cross-entropy, $u^w(y',y) = w_y\mathbf{1}\{y' = y\}$, with $w_y$ expressing the utility asymmetry. The standard ALP prescribes training with a loss $\ell^w(p, y) = -w_y\log p_y$. However, the authors’ analysis shows that such weighting flattens the utility landscape, making it optimal for the model to inflate predictions for the high-weight class across much of the posterior space. This suppression of the marginal gradient (learning incentive) means that the model becomes less sensitive to improvements in information.

Figure 1: Utility weighting distorts optimal predictions, suppressing incentives for learning and flattening marginal learning utility across most posterior probabilities.

The paper formalizes this with the Learning Identity proposition: the expected empirical prediction loss can be decomposed such that the residual learning loss $\tilde{\ell}^u(q)$—under optimal post-processing and utility-weighting—is a concave function whose gradient is given by the utility-weighted prediction losses. For class-weighted cross-entropy, optimal prediction becomes $p^w_y(q) = \frac{w_yq_y}{\langle w, q \rangle}$, and the learning incentive gradient (Equation 17) is suppressed for large $w_y$. In the limit, overwhelming emphasis on a single class leads to a model that disregards input information altogether.

Empirical Results

Empirical evaluation spans chest X-ray diagnosis (DenseNet, CheXNeXt) and CIFAR image classification (ViT-B16). Models are trained (1) with class-weighted cross-entropy loss (Weighted Training) and (2) with standard cross-entropy (Unweighted Training) whose predictions are ex post transformed according to the analytical optimal utility weighting. Both regimes are compared under machine-centric and human-centric metrics: weighted loss (internal alignment) and downstream classification utility (external alignment).

Across tasks and classes—e.g., pneumonia diagnosis with a $99:1$ weighting ratio—the Ex Post Weighting regime consistently achieves lower weighted loss and equal or higher classification utility compared to direct Weighted Training.

Figure 2: Comparison of weighted loss across training regimes in pneumonia diagnosis, showing consistent superiority of ex post weighting versus direct utility-weighted training.

In the chest X-ray task, mean reduction in weighted loss using Ex Post Weighting is $6.96\%$ over direct Weighted Training, and performance is robust across all training epochs. Similar findings hold for other disease targets (cardiomegaly, infiltration, pneumothorax), inverse probability weighting (class imbalance correction), and for the most difficult classes in CIFAR-10 ("cat") and CIFAR-100 ("maple tree").

Figure 3: Weighted loss in pneumonia diagnosis disaggregated by training run; ex post weighted predictions (blue) always outperform weighted training (red) when evaluated on the weighted objective.

Figure 4: Classification utility curves reinforce that ex post weighted predictions do not harm—and often improve—downstream decision performance.

Figure 5: Weighted loss when emphasizing "cat" class on CIFAR-10, supporting generality of findings to standard computer vision benchmarks.

Implementation and Practical Recommendations

Analytical Ex Post Transformation

Given a strictly proper loss function (e.g., cross-entropy), model outputs calibrated posterior probabilities $q \in \Delta(\mathcal{Y})$. To implement ex post utility-weighting, transform predictions for each sample using:

$p^w_y(q) = \frac{w_yq_y}{\sum_{y'} w_{y'}q_{y'}}$

where $w$ is the desired class weighting vector encoding the human utility or cost asymmetry. The mapping is always well-defined for $w_y > 0$.

In code:

import numpy as np def ex_post_weighted_predictions(q, w): numerator = w * q denom = np.dot(w, q) return numerator / denom

This mapping should be applied to the predicted class probabilities before making downstream decisions or computing utility-sensitive metrics.

Training Strategy

• Always train with strictly proper, unweighted loss (e.g., cross-entropy).
  • This ensures maximal information acquisition, robust gradients, and calibrated posterior predictions.
• For asymmetric human objectives or class imbalance:
  • Transform predictions ex post using the analytical formula above.
  • Implement decision rules (e.g., thresholding, conditional risk minimization) on the transformed posteriors.
• Do not integrate utility-weighting (class or sample weights) directly into the training loss unless learning incentives are proven unaffected.
• For multi-class problems, ex post transformation generalizes; for cost matrices, inverting the utility cost matrix provides the recalibration mapping (see Theorem "Analytical Recalibration").

Resource and Scaling Considerations

• Training with unweighted loss typically exhibits better convergence properties due to well-behaved gradients.
• Ex post analytical transformation is computationally trivial and compatible with batch inference; scaling to large datasets or high-class-count tasks poses no bottleneck.
• The approach is architecture-agnostic: applies to deep CNNs, transformers, decision trees, and any model outputting calibrated probabilities.

Implications and Future Directions

The findings invalidate a fundamental premise underlying widespread ML practice in high-stakes domains. The decoupling of learning and decision incentives is shown to be critical for effective alignment. From a theoretical perspective, the work establishes formal equivalence with classical economic and information design models (e.g., Blackwell ordering, Bayesian persuasion), paving the way for principled incentive-based training objectives.

Contrary to intuition, greater alignment demands resisting the urge to directly encode human preference in training objectives—a result that may diverge markedly from prior alignment and cost-sensitive learning literature. Moreover, the internal- vs. external-alignment distinction echoes emergent themes in AI safety and mesa-optimizers: designing objectives that align not just outputs, but also the learning process itself ([hubinger2019mesa]).

Practical deployment of ML models in medicine, finance, and other asymmetric-cost settings should adopt the ex post utility-weighting pipeline. For fairness or distribution shift concerns, further theoretical and empirical work is needed to extend this framework to sequential and dynamic decision problems.

Conclusion

The separation of learning and decision incentives is vital to avoid incentive failures and maximize alignment in machine learning systems. Training on strictly proper, unweighted loss, and imposing downstream objectives ex post, consistently outperforms direct utility-weighted training, both in terms of machine-centric and human-centric metrics. Recognition of this principle should inform the design of high-stakes, decision-support AI and drive future research in incentive-aligned machine learning.