Hard Label Criteria Overview

• Hard Label Criteria (HLC) is a term denoting the use of discrete, one-hot or majority-vote labels in place of full probabilistic distributions.
• It is applied across supervised classification, knowledge distillation, and black-box attacks to study how reducing label granularity influences model training, accuracy, and security.
• Empirical results show that the effectiveness of hard labels depends on data ambiguity, annotator density, and imposed constraints, underscoring the need to balance precision with information fidelity.

Across recent literature, “Hard Label Criteria” does not have a single technical meaning. In machine learning, it commonly denotes regimes in which supervision or oracle access is limited to discrete labels rather than logits, probabilities, or full label distributions; in other domains, the same acronym denotes distinct constructs such as an acoustic labeling algorithm, the High-quality Lightweight Codec, and the Hard Lefschetz Condition (Grossmann et al., 2022, Zucatelli et al., 8 Aug 2025, He et al., 31 Mar 2026, Hind et al., 22 Jun 2026). The unifying motif in the machine-learning usage is discreteness: majority-vote targets, top-1 API responses, or logically constrained label assignments replace richer probabilistic objects, and the central technical question becomes what is lost, preserved, or even improved by that restriction.

1. Terminological scope and formal senses

The term is used non-uniformly across the cited corpus. In supervised classification, a hard label is the single class obtained by collapsing annotations, while a soft label is the empirical class distribution. For a $k$-class problem with one-hot annotations $a_i(x)\in\{0,1\}^k$, the hard label is

$l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$

whereas the soft label is

$l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$

This distinction underlies the ambiguity-focused analysis of label distributions and the later work on hard-label delivery for annotator disagreement (Grossmann et al., 2022, Gheibi et al., 20 May 2026).

Sense	Operational meaning	Representative source
Hard-label supervision	One-hot or majority-vote targets replace distributions	(Grossmann et al., 2022, Mandal et al., 2024)
Hard-label-only access	Oracle/API returns only top-1 class	(Sanyal et al., 2022, Yuan et al., 2023, Yu et al., 2022)
Hard logical constraints	Predictions must satisfy explicit rules or hierarchy	(Giunchiglia et al., 2021)
HLC as named algorithm	Binary non-stationarity labeling from INS regions	(Zucatelli et al., 8 Aug 2025)
HLC as unrelated acronym	High-quality Lightweight Codec; Hard Lefschetz Condition	(He et al., 31 Mar 2026, Hind et al., 22 Jun 2026)

This terminological spread matters because the phrase “hard label” can refer either to the representation of supervision, the restriction on model queries, or the enforcement of symbolic constraints. A plausible implication is that any technical discussion of HLC must fix the access model, label semantics, and decision rule before comparing results across papers.

2. Representational adequacy: when hard labels are sufficient and when they fail

The strongest direct critique of hard-label sufficiency comes from work on ambiguous data. The core claim is that hard labels are sufficient only when each image has a single, unambiguous class and the annotation process can reliably collapse all disagreement into one correct target; once ambiguity is genuine rather than merely noisy, the one-hot target ceases to represent the underlying label distribution faithfully (Grossmann et al., 2022). The synthetic benchmark in that study was designed to isolate ambiguity from noise: $15{,}000$ images, split $60\%/20\%/20\%$, with $40\%$ from six base classes and $60\%$ from interpolations. Because the interpolation process is known, there is no noise, only ambiguity. Even in that noise-free setting, soft labels yielded $ACC=0.9096\pm0.0137$ and $KL=0.0394\pm0.0009$, whereas hard labels yielded $a_i(x)\in\{0,1\}^k$0 and $a_i(x)\in\{0,1\}^k$1 (Grossmann et al., 2022).

The same pattern appears on the real-world MiceBone dataset, where about $a_i(x)\in\{0,1\}^k$2 annotations per image were collected for $a_i(x)\in\{0,1\}^k$3 images in three classes. There, hard-label training produced $a_i(x)\in\{0,1\}^k$4 and $a_i(x)\in\{0,1\}^k$5, while soft-label training produced $a_i(x)\in\{0,1\}^k$6 and $a_i(x)\in\{0,1\}^k$7 (Grossmann et al., 2022). The paper further reports that soft labels yield a more regular internal feature space: on the synthetic data they induce six distinct clusters plus a cyclic interpolation region, and on MiceBone they produce better-structured transitions between ambiguous classes. The stated conclusion is not that hard labels are universally unusable, but that ambiguity alone is enough to make them insufficient.

A different failure mode of hard-label assumptions appears in gradient inversion. Earlier analytical attacks such as iDLG rely on the sign structure of final-layer gradients under one-hot labels. Once labels are softened by label smoothing or mixup, the sign cue can disappear, and the usual “recover the one-hot label first, then reconstruct the image” pipeline no longer holds. The proposed alternative is a scalar-optimization framework that can simultaneously recover the augmented label and the input feature of the last fully connected layer from single-input gradients, together with a theorem showing that the simplex constraint alone is too weak for analytical label recovery (Wang et al., 2024). In that setting, the hidden hard-label assumption is not merely limiting; it is structurally incorrect.

3. Soft versus hard supervision in optimization and distillation

The sharpest theoretical comparison between soft- and hard-label training is given for knowledge distillation in a two-layer ReLU student trained by projected gradient descent. Under a neural tangent kernel / RKHS separability assumption with margin $a_i(x)\in\{0,1\}^k$8, soft-label training from teacher logits requires roughly

$a_i(x)\in\{0,1\}^k$9

neurons to reach classification loss $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$0, whereas hard-label training requires roughly

$l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$1

The practical implication is stated explicitly: when $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$2, soft-label training can be significantly more efficient in neuron count, so distillation is especially beneficial on difficult datasets (Mandal et al., 2024). The same paper reports motivating experiments on MNIST and CIFAR-10 showing that, as the task becomes harder, the gap between soft-label and hard-label training widens, with soft labels retaining accuracy much better at small student widths.

The explanation offered there is geometric rather than heuristic. Soft-label training only needs to match teacher logits up to moderate tolerance, allowing the student to remain near a useful random initialization; hard-label training must drive outputs into the extreme $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$3 regime, which requires a larger parameter move and therefore a wider network to preserve stability (Mandal et al., 2024). This shifts the usual intuition about distillation from “regularization” toward explicit neuron-complexity reduction.

However, later work shows that hard labels can be advantageous when the issue is not class margin but the fidelity of the soft target estimate. On CIFAR-10H, “hard-label delivery” methods—multipass and stochastic label sampling (SLS)—treat the annotator distribution as the target while delivering one hard label at a time during optimization. SLS and soft-label cross-entropy optimize the same expected objective,

$l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$4

with the same expected gradient but different gradient covariance. Empirically, when only a small number of annotations per example is available, hard-label delivery improves over direct soft-label training; when full annotator distributions are available, both hard-label methods match soft-label training. The paper also reports flatter basins for SLS, including $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$5 and trace $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$6 for SLS versus $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$7 and trace $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$8 for soft labels in the main comparison (Gheibi et al., 20 May 2026).

A third regime appears in large-scale dataset distillation with few stored crops per image. There, crop-level teacher supervision can exhibit Local-View Semantic Drift (LVSD), formally associated with nonzero covariance $l_h(x)=\arg\max_k \sum_{i=0}^N a_i(x),$9 of teacher predictions across crops. The proposed HALD schedule uses hard labels as a content-agnostic anchor in a Soft–Hard–Soft curriculum rather than mixing hard and soft losses simultaneously. On ImageNet-1K, HALD reaches $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$0 using only $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$1M soft-label storage and outperforms LPLD by $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$2 (Cui et al., 17 Dec 2025). Taken together, these results suggest that the relative value of hard labels depends on whether the bottleneck is representational ambiguity, student complexity, sparsity of annotator counts, or instability of crop-level soft supervision.

4. Hard-label-only access in black-box attacks, stealing, and auditing

In black-box security settings, “hard label” usually means that the target returns only the predicted class. Several papers show that this is a substantial restriction, but not an effective barrier. In model stealing, DFMS-HL demonstrates that a clone can be trained without victim gradients, softmax outputs, or original training data by alternating a generator, discriminator, and clone model, using the clone’s gradients as a proxy for the victim’s. The paper reports $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$3 million queries for CIFAR-10, $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$4 million for CIFAR-100, and $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$5 clone accuracy on CIFAR-100 with CIFAR-10 proxy data; even with synthetic-only data, the CIFAR-100 clone reaches $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$6 (Sanyal et al., 2022). Black-Box Dissector addresses the same hard-label regime by combining CAM-driven erasing with random-erasing-based self-knowledge distillation, reporting improvements of up to $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$7 over prior hard-label stealing baselines (Wang et al., 2021).

The hard-label-only threat model becomes more demanding when robustness rather than nominal accuracy is the target. DFHL-RS assumes no natural data, no architecture knowledge, and access only to top-1 predictions, yet seeks to steal both clean accuracy and adversarial robustness. Its main mechanism alternates substitute data generation with clone training, and uses High-Entropy Examples rather than uniform-uncertainty targets to probe informative boundary regions. On CIFAR-10, the clone reaches $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$8 clean accuracy and $l_s(x)=\frac{1}{N}\sum_{i=0}^N a_i(x)\in[0,1]^k.$9 robust accuracy against AutoAttack, compared with the target’s $15{,}000$0 clean and $15{,}000$1 robust accuracy, under a default budget of $15{,}000$2M queries (Yuan et al., 2023).

The same access restriction appears in adversarial NLP and interpretability. TextHacker attacks text classifiers and entailment models when only the predicted label is observable, learning word importance from attack history rather than probabilities. On BERT, it reports $15{,}000$3 success with $15{,}000$4 perturbation on AG’s News, $15{,}000$5 success with $15{,}000$6 perturbation on IMDB, and $15{,}000$7 success with $15{,}000$8 perturbation on Yahoo! Answers (Yu et al., 2022). In a different direction, Label Forensics treats each hard label of an undocumented text classifier as an empirical semantic region in embedding space and reconstructs that region with seed sentences plus a semantic neighborhood sampler, achieving average label consistency of around $15{,}000$9 across five black-box classifiers (Du et al., 1 Dec 2025).

These results jointly contradict a common operational assumption that withholding confidences or logits largely neutralizes extraction or auditing. What the papers show instead is that hard labels remove rich inter-class information, but strategic querying, surrogate gradients, perturbation-based search, or semantic sampling can partially reconstruct it (Sanyal et al., 2022, Wang et al., 2021, Yuan et al., 2023, Yu et al., 2022, Du et al., 1 Dec 2025).

5. Hard constraints, explicit criteria, and domain-specific instantiations

A distinct meaning of hard labels arises when the issue is not access restriction but structural admissibility of outputs. In hierarchical multi-label classification, hard constraints express subclass relations such as $60\%/20\%/20\%$0, meaning that any prediction positive for a subclass must also be positive for its superclass. C-HMCNN$60\%/20\%/20\%$1 enforces this by a constraint layer guaranteeing hierarchy-coherent outputs, while CCN$60\%/20\%/20\%$2 generalizes the logic to stratified normal rules with positive and negative literals. The paper proves that C-HMCNN$60\%/20\%/20\%$3 commits no hierarchy violations and reports superior average rankings over state-of-the-art baselines on $60\%/20\%/20\%$4 HMC benchmarks and statistically significant gains on $60\%/20\%/20\%$5 datasets with general logical constraints (Giunchiglia et al., 2021).

In domain adaptation, “hardness” is assigned at the class level rather than the output-access level. HCRPL defines a hard class through under-prediction on the target domain, operationalized by the ratio

$60\%/20\%/20\%$6

where $60\%/20\%/20\%$7 is the source prior class proportion and $60\%/20\%/20\%$8 is the predictive class distribution on target samples. Adaptive Prediction Calibration rescales target predictions by $60\%/20\%/20\%$9, while Temporal-Ensembling and Self-Ensembling stabilize the recalibrated pseudo labels. The reported effect is reduced collapse into dominant classes and stronger performance in both UDA and SSDA, especially when domain shift is large (Zhang et al., 2020).

Some papers introduce an explicit algorithm named HLC. In acoustic non-stationarity assessment, HLC converts the scale-dependent Index of Non-Stationarity into a single binary label by thresholding non-stationary scales within regions and then taking a majority vote across regions. With $40\%$0 and $40\%$1 temporal regions, the paper reports correct HLC labeling of $40\%$2 for Office, $40\%$3 for Volvo, $40\%$4 for Babble, $40\%$5 for Factory, and $40\%$6 for Machine Gun, for an average of $40\%$7. The downstream NANSA model reaches $40\%$8, $40\%$9, and $60\%$0 accuracy on AudioSet, DCASE, and FSD50K, respectively, while reducing processing time from $60\%$1 ms for INS to $60\%$2 ms for NANSA and $60\%$3 ms for NANSA$60\%$4 (Zucatelli et al., 8 Aug 2025).

Beyond machine learning, the literature on labels as certification under costly self-selection studies deterministic label systems as incentive mechanisms. There the principal result is that exact certification is often “inefficiently fine”: pooling a small bottom interval saves first-order signaling costs while losing only higher-order decision value, and under sufficient conditions lower censorship is optimal. The same work gives a finite-classification theorem: under a local wastefulness condition, every optimal admissible signal has finite essential range, with at most $60\%$5 categories (Whitmeyer, 24 Jun 2026). This use is not a machine-learning HLC algorithm, but it extends the broader idea that label granularity itself can be an object of optimization.

6. Recurring themes, misconceptions, and research directions

One recurring misconception is that hard labels are a universally adequate representation of ground truth. The ambiguity studies reject that view directly: even without label noise, collapsing an empirical distribution to a single class can discard essential target information and degrade both accuracy and distributional fidelity (Grossmann et al., 2022). A second misconception is the converse, namely that hard labels are always inferior to soft labels. The annotator-distribution and LVSD results qualify that claim: hard-label delivery can outperform soft-label training when empirical soft targets are sparse, and hard labels can act as a corrective anchor when limited crop supervision induces semantic drift (Gheibi et al., 20 May 2026, Cui et al., 17 Dec 2025).

A third misconception is that hard-label-only access is a strong security defense. The model stealing, robustness stealing, adversarial attack, label-forensics, and gradient inversion papers all show otherwise. Their common result is not that hard labels are easy to exploit, but that the restriction from distributions to top-1 outputs is insufficient on its own to prevent functionality extraction, semantic auditing, or privacy leakage once attackers can generate, perturb, or strategically sample queries (Sanyal et al., 2022, Yuan et al., 2023, Yu et al., 2022, Du et al., 1 Dec 2025, Wang et al., 2024).

Finally, the acronym itself is unstable across fields. In the same broad research ecosystem, HLC can denote hard-label criteria, a binary labeling rule for acoustic signals, the High-quality Lightweight Codec, or the Hard Lefschetz Condition (Zucatelli et al., 8 Aug 2025, He et al., 31 Mar 2026, Hind et al., 22 Jun 2026). A plausible implication is that future work benefits from disambiguating “hard-label criterion” from “hard-label-only access,” “hard logical constraints,” and unrelated acronymic uses. The surveyed literature suggests that the decisive technical question is not whether labels are hard in the abstract, but which information has been discretized, at what stage of the pipeline, and for what objective.