AUM-Based Pseudo-Labeling Methods

• AUM-based pseudo-labeling is a technique that uses time-averaged margin dynamics to distinguish clean from mislabeled examples for robust label correction.
• It leverages training dynamics by tracking logits over epochs, enabling selective relabeling, removal, or downweighting of unreliable samples.
• Empirical studies on datasets like WebVision50 and CIFAR demonstrate error reductions, validating its effectiveness in uncertainty-guided learning.

Searching arXiv for the cited AUM and related pseudo-labeling papers to ground the article. AUM-based pseudo-labeling denotes a family of label-selection, relabeling, or sample-weighting procedures that use the Area Under the Margin (AUM) as a temporal reliability signal for a label assignment. In the arXiv literature, AUM is defined from training dynamics, not from a single prediction: for a training example $x_i$ with label $y_i$, the per-step margin is the gap between the logit of the provided label and the largest competing logit, and the AUM is the average of that margin over training (Pleiss et al., 2020). Although AUM was introduced for identifying mislabeled data, the same signal can be repurposed for pseudo-labeling: high-AUM samples are treated as label-consistent, while low-AUM samples can be suppressed, removed, or relabeled using model predictions. Related work in unsupervised domain adaptation, semi-supervised learning, and crowdsourced annotation situates AUM within a broader class of uncertainty-aware pseudo-labeling and filtering methods (Zheng et al., 2020).

1. Formal definition of AUM and its variants

The canonical AUM construction in the mislabeled-data literature is based on logits. For each training example $x_i$ with provided label $y_i$, the model outputs logits $f_\theta(x_i)$, and the margin at training step $t$ is

$$m_i^{(t)} = f_\theta(x_i)_{y_i} - \max_{j \neq y_i} f_\theta(x_i)_j.$$

The corresponding AUM is

$$\mathrm{AUM}_i = \frac{1}{T}\sum_{t=1}^{T} m_i^{(t)}.$$

Equivalently, AUM is the time-averaged signed confidence in the labeled class. A large positive AUM means the labeled class consistently outranks all alternatives; a negative AUM means the labeled class is usually below some other class. In this sense, AUM ranks examples by label plausibility rather than by instantaneous confidence (Pleiss et al., 2020).

A crowdsourced extension replaces the single-label setting by worker-specific labels and a worker-weighted aggregation. In that setting, the paper on Weighted Areas Under the Margin (WAUM) defines the margin using softmax probabilities rather than raw logits:

$$M^{(t)}(x,y)=\sigma^{(t)}_{y}(x)-\sigma^{(t)}_{[2]}(x),$$

with

$$\mathrm{AUM}\left(x, y; \mathcal{D}_{train}\right) = \frac{1}{T}\sum_{t=1}^T \big[\sigma^{(t)}_{y} (x)- \sigma^{(t)}_{[2]}(x)\big].$$

WAUM then forms a trust-weighted average over worker labels for task $y_i$0:

$y_i$1

This extension preserves the AUM idea—measuring how consistently training supports a label—while making the score sensitive to worker heterogeneity and task-dependent trust (Lefort et al., 2022).

2. Why margin dynamics can support pseudo-label decisions

The central empirical observation behind AUM is that cleanly labeled examples and mislabeled examples exhibit systematically different margin dynamics during SGD training. Clean samples are eventually learned correctly; their labeled-class margins tend to become positive and remain positive for much of training, producing relatively large positive AUM values. Mislabeled samples behave differently because the provided label disagrees with the underlying visual or semantic pattern; their labeled-class margin is often small, near zero, or negative for much of training, producing much lower AUM values (Pleiss et al., 2020).

The separation arises because SGD tends to learn common, consistent patterns first. Clean examples reinforce their assigned labels, whereas mislabeled examples produce unstable or contradictory gradients. Consequently, the trajectory of $y_i$2 over time is a better indicator of label quality than a single final prediction. The reported overall finding is that AUM values for clean and mislabeled examples form distinct distributions, enabling effective thresholding and ranking (Pleiss et al., 2020).

For pseudo-labeling, this suggests a specific interpretation of AUM as a confidence-over-time signal. A label is not trusted merely because the model is confident at one epoch; it is trusted because the label is supported persistently and strongly across training. A plausible implication is that AUM-based pseudo-labeling is particularly relevant in noisy-label regimes, where single-pass confidence can be high for an incorrect target.

3. From mislabeled-data detection to AUM-based pseudo-labeling

The original AUM method is a mislabeled-data detector, not a pseudo-labeling algorithm. Its practical decision rule is to train the model, record per-example margins over time, compute $y_i$3 for each example, compare those values to a learned threshold, and flag examples with unusually low AUM as likely mislabeled (Pleiss et al., 2020).

The bridge to pseudo-labeling is explicit in the methodological implication of that procedure. If a sample is flagged as mislabeled, a downstream system can remove it, downweight it, or relabel it with the model’s current best guess. In that form, AUM becomes a selection rule for deciding when a model-generated label should replace a provided label. A plausible AUM-based pseudo-labeling loop is:

1. Train on current labels or pseudo-labels.
2. Track the margin $y_i$4 for each sample.
3. Compute $y_i$5.
4. If $y_i$6 is below a learned mislabeled threshold, replace the label with $y_i$7 or postpone its use.
5. Retrain or continue training using the updated labels.

This adaptation uses AUM as a gating variable for relabeling rather than only as a detector. The resulting policy is naturally tri-partite: high AUM, positive margin implies retention of the original label; low AUM with strong alternate-class margin implies replacement by a pseudo-label; intermediate AUM implies deferral or exclusion until later epochs. The key distinction from standard self-training is that the decision is based on trajectory consistency across training, not just instantaneous posterior mass (Pleiss et al., 2020).

4. Thresholding, weighting, and related uncertainty-guided frameworks

AUM-based pseudo-labeling belongs to a broader class of methods that estimate pseudo-label reliability and then either filter or reweight the training signal. Four representative formulations are the original AUM detector, the UNRN framework for UDA person re-identification, the UES framework for semi-supervised learning, and WAUM for crowdsourced ambiguity filtering (Pleiss et al., 2020, Zheng et al., 2020, Wu et al., 13 Mar 2025, Lefort et al., 2022).

Approach	Reliability signal	Training action
AUM	Time-averaged margin over training	Flag, remove, downweight, or relabel suspicious samples
UNRN	Teacher–student KL divergence over soft multilabels	Reweight ID, triplet, and contrastive losses
UES	Multi-head disagreement relative to a mean prediction	Apply long-tailed nonzero sample weights
WAUM	Worker-weighted average of per-label AUMs	Prune ambiguous tasks before aggregation

The original AUM method avoids a universal hand-tuned cutoff by learning a data-dependent upper bound for mislabeled-like behavior. It adds an extra class populated with purposefully mislabeled threshold samples, then uses the resulting AUM distribution to isolate suspicious examples. This makes the threshold more principled than arbitrary percentile selection (Pleiss et al., 2020).

UNRN adopts a different uncertainty proxy. It uses a mean teacher framework in which teacher weights are an EMA of student weights, computes teacher and student soft multilabel distributions over target cluster centers and source class centers, and defines uncertainty as the KL divergence $y_i$8. Credibility is then $y_i$9, and the method uses this weight to suppress unreliable pseudo-labeled samples in the identity classification, triplet, and contrastive losses. The core rationale is that wrong pseudo labels tend to show weaker teacher–student consistency than correct ones (Zheng et al., 2020).

UES addresses a different limitation: the brittleness of hard confidence thresholds in semi-supervised learning. It estimates uncertainty from disagreement among multiple prediction heads relative to a mean reference prediction, then converts uncertainty into long-tailed weights rather than a keep/discard decision. The method explicitly argues that threshold selection is open/problematic, confidence can be overconfident and unreliable, and valuable low-confidence samples are discarded by hard gating. In contrast to AUM-style ranking, UES keeps all samples but assigns smaller weights to more uncertain pseudo-labels (Wu et al., 13 Mar 2025).

WAUM extends the AUM principle to crowdsourced annotation, where the object of interest is not only a label but also an ambiguous task. It computes AUM for each worker-provided label, weights each worker contribution by a trust factor derived from the worker’s confusion matrix diagonal and the model’s class distribution, and prunes tasks with low WAUM before any standard aggregation method is applied. In this setting, AUM-like filtering is task-level rather than sample-label-level (Lefort et al., 2022).

5. Empirical evidence and operational regimes

Direct empirical evidence in the original AUM paper concerns mislabeled-data detection rather than pseudo-label correction. The reported gains nonetheless indicate that the underlying reliability signal is practically useful. On the noisy real-world WebVision50 classification task, the method removes 17% of training data, yielding a 1.6% (absolute) improvement in test error. On CIFAR100, removing 13% of the data leads to a 1.2% drop in error. The paper also states that AUM consistently improves upon prior work on synthetic and real-world datasets (Pleiss et al., 2020).

In UDA person re-identification, uncertainty-guided pseudo-label weighting yields state-of-the-art benchmark results in the reported setting. UNRN achieves, for example, Duke → Market: mAP 78.1, R1 91.9; Market → Duke: mAP 69.1, R1 82.0; Market → MSMT17: mAP 25.3, R1 52.4; and Duke → MSMT17: mAP 26.2, R1 54.9. Ablations further show that adding UID improves mAP by +1.9 on Duke→Market and +3.4 on Market→Duke, and that combining UID + UTRI + UCT gives the best result (Zheng et al., 2020).

In semi-supervised learning, UES reports modest but consistent classification gains and larger gains on pose estimation. When combined with FixMatch, it achieves a 0.2% accuracy improvement on CIFAR-10 with 40 labeled data points and a 0.26% accuracy improvement on CIFAR-100 with 400 labeled data points. When combined with DualPose, it reports a 3.47% improvement in PCK on Sniffing, a 7.29% improvement in PCK on FLIC, and a 3.91% improvement in PCK on LSP under the specified low-label settings (Wu et al., 13 Mar 2025).

For crowdsourced learning, WAUM is evaluated on simulated datasets and on CIFAR-10H, LabelMe, and Music. The reported pattern is that WAUM generally improves accuracy after pruning, often improves or preserves calibration (ECE), and consistently outperforms AUMC when task ambiguity is heterogeneous across workers. The gains are especially clear on sparse-label datasets such as LabelMe, while on CIFAR-10H they are more modest because the dataset is already relatively curated (Lefort et al., 2022).

These results do not establish a single universal AUM-based pseudo-labeling recipe. They do, however, support a common conclusion: uncertainty signals that incorporate either training dynamics, cross-model consistency, or ensemble disagreement can improve the quality of pseudo-supervision relative to naive confidence thresholding.

6. Limitations, failure modes, and common misconceptions

AUM is not a one-shot statistic. It requires tracking training dynamics, since the score depends on margins accumulated over many epochs or steps. It also depends on the training regime: optimizer, architecture, learning-rate schedule, augmentation, and number of epochs can affect AUM because the quantity is defined through SGD dynamics (Pleiss et al., 2020).

A common misconception is that low AUM definitively identifies a wrong label. The cited literature is explicit that not all low-AUM examples are mislabeled. Some hard-but-correct examples may also have low margins, so the method is probabilistic rather than definitive. The same concern carries over to pseudo-labeling: replacing labels for low-AUM examples can amplify mistakes if the model’s current prediction is wrong, which is why a conservative strategy is usually needed (Pleiss et al., 2020).

Threshold calibration remains a substantive issue. The extra-class thresholding strategy in the original AUM method makes calibration more principled, but the choice of threshold samples and their construction influences results. Related SSL work takes a different stance: UES argues that threshold selection is itself an open problem and therefore replaces hard gating with continuous long-tailed weights (Wu et al., 13 Mar 2025).

Another misconception is that uncertainty estimates are equally informative throughout training. UES explicitly notes that the correlation between uncertainty and prediction accuracy is not high in the early stages of training, and that it improves later. This supports using uncertainty as a soft weighting signal rather than as a strict binary gate from the beginning of optimization (Wu et al., 13 Mar 2025).

Task pruning can also be harmful. WAUM reports a limitation case on two_moons, where pruning the hardest points degrades performance because those points are important for learning the data structure. More generally, over-pruning changes the training distribution, can violate the i.i.d. assumption, and may remove useful hard examples rather than harmful ambiguous ones (Lefort et al., 2022).

In teacher–student weighting schemes, the converse failure mode is trivial suppression: if the model could simply declare all target samples uncertain, it would avoid learning from them. UNRN addresses this by including an uncertainty regularization term that penalizes consistently large uncertainty, preventing the optimization from collapsing into indiscriminate downweighting (Zheng et al., 2020).

Taken together, these limitations indicate that AUM-based pseudo-labeling is best understood not as a standalone oracle for label correctness, but as a temporal reliability statistic that can be integrated with threshold learning, soft weighting, or task-level pruning. Its main contribution is to move pseudo-label selection away from instantaneous confidence and toward the consistency of label support across training.