Class-Based Label Replacement
- Class-based label replacement is a process that substitutes, refines, or reconstructs original class annotations using contextual, class-conditioned evidence to address noise and underexploited semantics.
- It spans diverse applications such as multi-label augmentation in remote sensing, adversarial substitution in vertical federated learning, and semantic augmentation in few-shot text classification and graph analysis.
- Empirical results demonstrate significant performance boosts and robustness improvements, with tailored techniques achieving higher mAP, accuracy, and F1 scores across various benchmarks.
Class-based label replacement denotes a family of operations in which the original class annotation is not used verbatim but is instead updated, substituted, reconstructed, or injected through class-conditioned evidence. In the literature, this appears in several distinct forms: recomputing a mixed multi-label vector from propagated pixel-level class maps after CutMix in remote sensing (Burgert et al., 2024), replacing encrypted gradient messages so poisoned samples train toward an attacker-chosen target class in black-boxed vertical federated learning (Liu et al., 2021), embedding class labels into support sentences through a natural-language template in few-shot text classification (Hou et al., 2024), propagating neighboring labels inside a GCN by masking self-loops on label features (Bellei et al., 2021), reconstructing a probabilistic label matrix by inverse tri-NMF (Satoh, 12 Oct 2025), and inferring unseen labels through a class label autoencoder in zero-shot learning (Lin et al., 2018). Related work also shows that some settings are more precisely described as class-informed label promotion or class-aware label cleaning rather than literal replacement (Zhang et al., 2024, Huang et al., 2022).
1. Conceptual scope and operational forms
Across these works, the replaced object is not always the same. In some methods, the target of replacement is the training label itself: the multi-label vector of a CutMix image is recomputed from propagated class-position maps, and the true label of a poisoned sample in VFL is effectively replaced by an attacker-chosen target class through gradient manipulation. In other methods, the replaced object is a representation of class information rather than the final supervisory target: label templates are embedded into support sentences in few-shot text classification, label channels are appended to node features in Label-GCN, and the label matrix becomes the observed variable in NMF-LAB (Burgert et al., 2024, Liu et al., 2021, Hou et al., 2024, Bellei et al., 2021, Satoh, 12 Oct 2025).
The technical motivation is also heterogeneous. In multi-label augmentation, replacement corrects additive and subtractive label noise produced by naive CutMix. In black-boxed VFL, replacement is adversarial and exploits the fact that batch-averaged gradients still encode label structure under homomorphic-encryption-based protocols. In text, graph, and factorization settings, class-based replacement is used to expose semantic or structural information that standard pipelines underuse. This suggests that class-based label replacement is best understood as a class-conditioned operation on supervision, rather than as a single canonical algorithm.
2. Label recomputation after mixed-sample augmentation
In multi-label remote sensing image classification, the central problem is that standard CutMix knows which pixels are cut and pasted, but the image-level multi-label vector does not encode where each class is located. The resulting mixed sample can therefore exhibit subtractive multi-label noise, where a class is erased from the mixed image while its label remains present, or additive multi-label noise, where a new class is pasted in while its label remains absent. The proposed remedy is a label propagation strategy that carries out a second, parallel pairing operation on pixel-level class positional information. For a cutout region with area and binary mask , two images are mixed as
Ordinary CutMix would use
but this direct label mixing is unsuitable for multi-label classification. When reference maps are available, the same pairing is applied to them,
where extracts the set of class labels appearing anywhere in the updated map. If no reference maps exist, class explanation masks are used instead,
The read-out 0 declares class 1 present if the number of activating pixels exceeds 2, and heatmaps are first thresholded with 3 if the explanation method does not produce binary masks. Valid CutMix boxes satisfy
4
and are sampled uniformly until 5 valid rectangles are obtained (Burgert et al., 2024).
The empirical role of replacement is explicit. On DeepGlobe-ML, with box sizes in 6, CutMix w/ LP reaches 82.30 mAP macro, versus 80.49 for CutMix w/o LP and 80.20 for CutOut, starting from a baseline of 78.40. Larger cut regions make LP more valuable because they increase the chance that naive CutMix would erase or add labels incorrectly. Robustness was evaluated under Mask Shift, Dilation/Erosion, Rectify Borders, Border Deformation, Segment Swap, and Class Swap; border deformations barely hurt performance, and only severe dilation/erosion or extensive segment swaps reduce the advantage more substantially. On BigEarthNet-S2, CutMix w/ LP improves over CutMix w/o LP under all box-size settings, with gains of about 2 mAP points in the best cases and best results around 76.7 mAP macro. On TreeSatAI, using Grad-CAM-derived class explanation masks with 7 and 8, the best setting reaches 68.91 mAP macro versus 64.67 without LP, a gain of 4.2 points (Burgert et al., 2024).
3. Target-class substitution as an attack primitive in vertical federated learning
In black-boxed vertical federated learning, class-based label replacement appears as a backdoor-style attack built on top of label inference. The setting assumes HE-protected communication: parties exchange encrypted per-sample intermediate results 9, compute encrypted gradients, send them to a TTP for decryption, and only then obtain batch-averaged gradients. The passive party therefore does not see per-sample gradients directly, but still knows its own local forward and backward computations. For softmax plus cross-entropy, the gradient with respect to logits has the standard sign structure in which the true class is the unique coordinate with negative sign. The paper rewrites the batch-averaged last-layer gradient as
0
where 1 is observed, 2 is unknown and contains the label information, and 3 is locally computable. If
4
then the labels can be fully recovered. The practical attack uses gradient inversion by initializing 5 and minimizing
6
over the guessed labels and latent representations (Liu et al., 2021).
Once labels have been inferred, the attacker replaces the gradient corresponding to the true label with the gradient of a target label 7: 8 Because HE is additively homomorphic, this can be done directly on encrypted values by adding or subtracting the encrypted constant 9 at selected coordinates. The paper also gives a batch/class-level replacement procedure: for each poisoned sample 0, the passive party randomly selects a clean target sample 1, records a pair 2, replaces 3 with 4 in the forward pass, and in backpropagation replaces the encrypted gradient for sample 5 with the gradient of sample 6, amplified by 7: 8 The backdoor objective is
9
The reported attack accuracies for label inference are often over 90%, including 0.977 on CIFAR10 with batch size 128, 0.934 with batch size 512, 0.893 with batch size 2048, and 0.999 on CIFAR100 with batch size 128. Backdoor accuracy can exceed 90% on MNIST, NUS-WIDE, and CIFAR-style settings, and the paper uses 0 as a good balance between backdoor success and utility. The proposed defense, CoAE, maps 1, 2 and optimizes
3
It is reported to outperform DP, gradient sparsification, and MARVELL in the label inference setting, and the paper notes that without entropy loss, 4, the CoAE becomes much less robust because it merely permutes labels and does not sufficiently confuse the attacker (Liu et al., 2021).
4. Label semantics as an input-side replacement signal in few-shot text classification
In few-shot text classification, class-based label replacement takes the form of replacing the usual label-agnostic support encoding with label-templated inputs. The method argues that standard prototypical approaches underuse class labels because labels are often only used at the final classification step, while simple insertion of labels into text can make the input unnatural. The proposed solution is a manually designed template, with the best-performing choice reported as “Overall, the topic of the text is”. For class 5, the label text is written as
6
and BERT encodes the pair
7
Support and query samples are treated differently: support samples concatenate the template with the original sentence and the class label information, whereas query samples do not have their true labels available and are kept in the original form. The episode is represented as
8
The templated encoding is coupled with supervised contrastive learning over the merged episode
9
where positives for 0 are
1
and cosine similarity is used: 2 The per-sample contrastive loss is
3
Prototype construction replaces averaging with an attention mechanism: 4 and the class prototype becomes
5
Query classification uses
6
with prototype-network loss
7
and joint optimization
8
where 9 increases linearly from 0 to 1 during training.
The reported results indicate that label semantics contribute the largest single improvement. On 20News, Amazon, HuffPost, and Reuters, the method outperforms nearly all baselines in both 1-shot and 5-shot settings. Average improvements over the best baseline are +4.7% accuracy and +5.4% F1 in 1-shot, and +2.6% accuracy and +2.9% F1 in 5-shot. On HuffPost, the 1-shot gains are +7.5% accuracy and +8.4% F1. In ablation, label template alone gives +7.22% accuracy in 1-shot and +3.37% in 5-shot; for F1, label template alone gives +8.12% in 1-shot and +3.00% in 5-shot. The full model gives +8.12% accuracy in 1-shot and +3.48% in 5-shot, and +9.26% F1 in 1-shot and +3.15% in 5-shot. Faster convergence than ContrastNet and t-SNE plots with more compact intra-class clusters and clearer inter-class boundaries are also reported (Hou et al., 2024).
5. Structured label propagation and reconstruction in graphs, factorization, and zero-shot learning
A graph-based form of class-based label replacement is implemented by Label-GCN. The method appends one-hot labels to node features so that the input dimension becomes 0, but modifies only the first layer so that self-loops are removed selectively for the label coordinates: 1 while deeper layers remain standard GCN layers,
2
This masks the label of the center node while still allowing labels of neighboring nodes to flow in. The method supports binary and multi-class settings, works in both transductive and inductive regimes, and improves as more support labels are available at inference time. On the Elliptic dataset, it improves illicit-class detection and is reported to be more robust to the “dark market shutdown,” while the paper also notes that illicit nodes are only about 2% of the graph (Bellei et al., 2021).
A matrix-factorization form of replacement is given by NMF-LAB, which inverts the usual roles of features and labels by treating the label matrix as the observation: 3 Here 4 is the label matrix, 5 is the covariate matrix, 6 is a basis matrix, and 7 maps covariates to latent coefficients. With 8, coefficients are normalized by
9
and predicted class probabilities are reconstructed as
0
Columns of 1 are normalized to sum to one, often with 2, so bases often align closely with classes and can become close to a one-hot basis matrix up to permutation. Unlabeled examples can be encoded as uniform distributions, for example 3, which supports semi-supervised learning. The paper reports that NMF-LAB (direct) was often weaker than nonlinear baselines, while NMF-LAB (kernel) was much stronger: on Iris, direct 66.7% and kernel 95.5%; on Penguins, direct 79.1% and kernel 97.9%; on Digits, direct 68.0% and kernel 94.6%. On MNIST, direct NMF-LAB reaches 69.1%, kernel500 91.1%, kernel1000 93.7%, and full kernel 96.0% (Satoh, 12 Oct 2025).
A zero-shot form of replacement is implemented by the Class Label Autoencoder, which uses the class label space as the intermediate latent space. The encoder maps visual features to labels by
4
and the decoder reconstructs features with tied weights,
5
The optimization objective is
6
For seen and unseen classes, the method uses
7
and unseen label scores are computed by
8
Prediction is then
9
CLA fuses multiple semantic embeddings through linear combinations of 0, 1, and 2, and iteratively refines unseen structure. Reported gains over SAE and SP are at least +2% on AwA, +3.7% on CUB, +3.2% on Dogs, and +0.1% on ImNet-2. In multi-semantic settings, examples include 88.8 on AwA with all semantics, 60.3 on CUB with all semantics, and 48.6 on Dogs without attributes; on ImNet-2, Top-1 through Top-5 are 13.6, 19.6, 23.3, 26.0, and 28.3 (Lin et al., 2018).
6. Weak supervision, noisy labels, and the limits of literal replacement
Not every class-based modification of supervision is a literal label replacement. In scribble-based weakly supervised semantic segmentation, Class-driven Scribble Promotion begins from sparse scribbles 3, extracts image-level class labels 4, trains a classification model, and generates dense pseudo-labels
5
With CAMs, class 6 is represented by
7
The segmentation loss combines scribble supervision and pseudo-label supervision,
8
with a partial cross-entropy on scribbled pixels and a smoothed cross-entropy on the pseudo-mask using 9. To correct noisy pseudo-label guidance, the method adds a Localization Rectification Module with
0
and a Distance Entropy Loss
1
leading to
2
The paper explicitly states that this is not a literal pixel-level replacement of scribble labels with class labels but a hybrid scribble-plus-class-guided label promotion method. On ScribbleSup, baseline scribble-only with ResNet101 + DeepLabV3+ gives 66.2 mIoU val and 69.7 test, while the proposed method gives 75.9 mIoU val and 76.0 test. Using only scribble or only pseudo-label gives around 67% mIoU, but using both gives 72.13%; LoRM improves performance from 67.33% to 73.64%; and under scribble-shrink or scribble-drop, methods like AGMM and TEL lose about 10% when scribbles shrink to points, while this method drops by only about 1% (Zhang et al., 2024).
A related but distinct case is class-aware label cleaning under noisy supervision. CPC starts from a noisy-label dataset 3, uses a global two-component GMM over per-sample cross-entropy losses to obtain a first-pass partition 4 and 5, and then introduces class prototypes 6. The class-aware clean score is
7
and the clean/noisy decision rule is
8
Prototype learning uses clean samples from 9, noisy samples from 00, and high-confidence pseudo-clean samples 01, with full objective
02
The method is justified through an EM-style interpretation in which the exact E-step posterior 03 is approximated by the GMM posterior 04. CPC is therefore a class-aware cleaner for partitioning and relabeling confidence, rather than a direct target-label substitution. The paper reports that CPC outperforms both class-agnostic GMM cleaner and naive class-aware GMM cleaner in cleaner AUC, improves final accuracy on CIFAR-10, CIFAR-100, Clothing1M, and WebVision, and that using GMM to warm up prototype learning yields better prototypes than prototype learning without GMM (Huang et al., 2022).
The common misconception is that every use of class information to modify supervision is a literal replacement of one label by another. The literature instead spans exact recomputation from propagated class-position evidence, adversarial target-class substitution, input-side semantic augmentation, neighborhood label propagation, latent probabilistic label reconstruction, class-informed pseudo-label generation, and class-aware clean/noisy repartitioning. A plausible implication is that the unifying property is not the data structure being modified, but the decision to let class-conditioned structure overwrite or refine an original supervisory signal when the original signal is incomplete, noisy, strategically manipulated, or semantically underexploited.