Positive Congruent Training (PCT)

• PCT is a training paradigm that minimizes both error rates and negative flip rates by aligning new model outputs with the reference model’s correct predictions.
• It employs selective positive-congruence losses such as Focal Distillation to focus on samples correctly classified by the old model, thereby reducing regression errors.
• Empirical results on benchmarks like ImageNet and iNaturalist demonstrate significant reductions in negative flip rates, emphasizing PCT’s practical impact on model update stability.

Searching arXiv for recent and foundational papers on Positive Congruent Training and related extensions. Positive Congruent Training (PCT) is a training paradigm for model updates in which the updated model is optimized not only for lower error rate, but also for lower regression on samples that a reference model already handled correctly. Introduced for image classification as a route toward regression-free model updates, PCT addresses the practical problem that two model versions can have similar or even improved aggregate accuracy while still differing substantially on which individual samples they fail. In that setting, the central failure mode is the negative flip: a test sample that was correctly classified by the old model but incorrectly classified by the new model (Yan et al., 2020).

1. Problem setting and conceptual motivation

PCT begins from the observation that reducing inconsistencies between versions of an AI system can be as important in practice as reducing overall error. In deployed systems, downstream components may depend on stable predictions for previously processed data, cached outputs, or post-processing logic. A model update that lowers average error but introduces new mistakes on previously correct cases can therefore be undesirable, even if it appears better under conventional evaluation (Yan et al., 2020).

The original formulation distinguishes three sample-wise outcomes for a test sample $(x_i,y_i)$. If both old and new models are correct, the sample is positive-congruent. If the old model is wrong but the new model is correct, the event is a positive flip. If the old model is correct but the new model is wrong, the event is a negative flip. This vocabulary reframes model updating as a problem of preserving already-safe behavior while still allowing desirable corrections on cases the old model handled poorly.

A recurrent misconception in this setting is that better accuracy automatically implies fewer regressions. The original paper argues the opposite on several grounds. Two models can have nearly identical error rates but disagree sharply on which samples they get wrong; a model with better average accuracy can still have many negative flips relative to the reference model; and simply copying the old model yields zero negative flips but no improvement. PCT therefore treats regression reduction as a distinct objective rather than a side effect of ordinary accuracy optimization (Yan et al., 2020).

2. Formalization, metrics, and objective functions

The basic PCT objective is to minimize both the new model’s error rate and the negative flip rate relative to a reference model. If $\phi^{\mathrm{old}}$ is the reference model and $\phi^{\mathrm{new},w}$ is the updated model, the training problem is formulated as

$$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$

where $\mathcal{L}_{\mathrm{CE}}$ is the standard empirical cross-entropy loss,

$$\mathcal{L}_{\mathrm{CE}}(\phi,w)=\frac{1}{N}\sum_{i=1}^{N} -\log p_w(y_i\mid x_i),$$

and $\mathcal{L}_{\mathrm{PC}}$ is a positive-congruence term that biases the new model toward the old one only in a selective way (Yan et al., 2020).

The central deployment metric is the negative flip rate (NFR),

$$\mathrm{NFR}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left(\hat y_i^{\mathrm{new}}\neq y_i,\ \hat y_i^{\mathrm{old}}=y_i\right),$$

which directly measures sample-wise inconsistency across model versions. Because raw NFR is influenced by the accuracies of the old and new models, the paper also introduces a normalized quantity,

$$\mathrm{NFR}_{\mathrm{rel}}=\frac{\mathrm{NFR}}{(1-\mathrm{ER}_{\mathrm{old}})\cdot \mathrm{ER}_{\mathrm{new}}},$$

to factor out the overlap expected if the two models were independent. The use of $\mathrm{NFR}_{\mathrm{rel}}$ makes it possible to distinguish regression reduction that follows trivially from lower error rate from regression reduction due to a genuinely congruent update strategy (Yan et al., 2020).

The framework is intentionally generic. The positive-congruence term is written in the form

$\phi^{\mathrm{old}}$0

where $\phi^{\mathrm{old}}$1 is a filter or weighting function, $\phi^{\mathrm{old}}$2 is a distance function, and $\phi^{\mathrm{old}}$3 is a target derived from the old model or the ground-truth label. The defining design principle is that $\phi^{\mathrm{old}}$4 should place extra emphasis on samples that the old model classified correctly. This selective emphasis is what differentiates PCT from broad imitation of a reference model (Yan et al., 2020).

3. Methodological instantiations: selective congruence rather than full imitation

The simplest baseline within the framework is a naive positive-congruence loss that adds extra cross-entropy weight on samples correctly classified by the old model:

$\phi^{\mathrm{old}}$5

This is equivalent to reweighting correctly classified old-model samples by a factor of $\phi^{\mathrm{old}}$6. Empirically, however, the paper reports that the naive strategy is hard to tune, can lead to rote memorization if the weight is too high, and does not reduce NFR substantially (Yan et al., 2020).

The main practical method proposed in the original work is Focal Distillation (FD). FD applies a distillation penalty to all samples but assigns larger weight to samples the old model classified correctly:

$\phi^{\mathrm{old}}$7

Its weighting function is

$\phi^{\mathrm{old}}$8

so every sample receives weight $\phi^{\mathrm{old}}$9, while samples correctly classified by the reference model receive an additional $\phi^{\mathrm{new},w}$0. The selective emphasis is the core mechanism: it pushes the new model harder to preserve the old model’s correct decisions while leaving room for improvement on the old model’s mistakes (Yan et al., 2020).

The method admits several informative special cases. Setting $\phi^{\mathrm{new},w}$1 recovers ordinary distillation. Setting $\phi^{\mathrm{new},w}$2 yields distillation only on samples correctly predicted by the old model. The best results are generally obtained with $\phi^{\mathrm{new},w}$3 and $\phi^{\mathrm{new},w}$4 between about $\phi^{\mathrm{new},w}$5 and $\phi^{\mathrm{new},w}$6. The paper also reports that $\phi^{\mathrm{new},w}$7 was ineffective, indicating that the new model still needs to learn the old model’s behavior on all samples if regression reduction is to work well in practice (Yan et al., 2020).

Two distance functions are considered for FD. The first is temperature-scaled KL divergence,

$\phi^{\mathrm{new},w}$8

where $\phi^{\mathrm{new},w}$9 is a temperature parameter. The paper notes that to make this effective for PCT, $$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$0 must be quite large, for example $$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$1, at which point the loss behaves more like direct logit matching. The second is logit matching,

$$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$2

which the paper often finds slightly better for reducing NFR (Yan et al., 2020).

This methodology is defined in opposition to two more familiar update strategies. Ordinary fine-tuning optimizes only the task loss, even when initialized from the old model; it provides no explicit mechanism for preserving correctness on the old model’s positive examples. Ordinary model distillation, by contrast, attempts to mimic the old model broadly, whether the old model is right or wrong. PCT rejects both extremes. It does not merely continue training for accuracy, and it does not treat all teacher predictions as equally worth preserving. Its guiding principle is to maximize congruency only on positive predictions of the reference model (Yan et al., 2020).

4. Reference models, ensembles, and later variants

The reference model functions as the behavioral anchor for update stability. In the backward setting, it is fixed and inherited from the deployed system. In that case, PCT adapts the new model to the old one, but only on the subset of samples already handled correctly. The old model is therefore not treated as universally correct; it is treated as a guide on its reliable subset. This selective use is a defining conceptual distinction between PCT and general imitation-based approaches (Yan et al., 2020).

The original paper also studies a forward setting in which the reference model itself can be designed. It considers an ensemble of old models rather than a single network, with ensemble prediction

$$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$3

The intuition is that independently trained models tend to make different mistakes even when their error rates are similar, and averaging reduces that sample-wise variability. Empirically, the paper finds that ensembles reduce NFR more rapidly than error rate as ensemble size grows, and that increasing ensemble size keeps lowering NFR even after accuracy plateaus. This suggests that NFR can be reduced beyond what would be predicted from average error rate alone (Yan et al., 2020).

A later development, ELODI (“Ensemble Logit Difference Inhibition”), analyzes the role of ensembles in reducing NFR and proposes a way to retain ensemble-like regression reduction at the inference cost of a single model (Zhao et al., 2022). ELODI distills a homogeneous ensemble into a single student and replaces full-logit matching with Logit Difference Inhibition (LDI), which penalizes only a subset of classes with the highest logit values:

$$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$4

The full objective is

$$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$5

with default settings $$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$6, $$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$7, and $$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$8. The paper argues that negative flips are often driven by a few outstanding logits with large differences, so top-$$\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),$$9 inhibition transfers the ensemble’s low-variance behavior without forcing the student to match all classes equally (Zhao et al., 2022).

ELODI also formalizes legacy-model integration for cases in which the deployed old model was not trained with ELODI. In that case, the student is optimized against both the newly constructed ensemble and the existing legacy model:

$\mathcal{L}_{\mathrm{CE}}$0

This later work preserves the core PCT objective—simultaneous reduction of error rate and NFR—while shifting emphasis from selective positive-example distillation to ensemble-induced stabilization of logit geometry (Zhao et al., 2022).

5. Empirical behavior in image classification

The original evaluation of PCT focuses on large-scale image classification, primarily ILSVRC12 / ImageNet and iNaturalist, across several update scenarios: same architecture with different training runs, architecture changes, more training samples per class, more classes, combinations of architecture and data change, fine-tuning settings, and ensemble-based reference models (Yan et al., 2020).

The main metrics are the error rates of old and new models, NFR, relative NFR, and number of parameters for model cost comparisons. On the benchmark scenario of training the same architecture twice on ILSVRC12, the paper reports NFR of about 6.44\% for no treatment, 2.50\% for FD-KL, and 2.35\% for FD-LM. On iNaturalist, the corresponding NFR values are about 7.77\%, 2.83\%, and 2.71\%. The ensemble approach yields the lowest NFR overall, reaching about 1.70\% on ILSVRC12, albeit at much higher computational cost (Yan et al., 2020).

Setting	Method	NFR
ILSVRC12, same architecture trained twice	no treatment	about 6.44%
ILSVRC12, same architecture trained twice	FD-KL	2.50%
ILSVRC12, same architecture trained twice	FD-LM	2.35%
ILSVRC12, same architecture trained twice	ensemble reference	about 1.70%
iNaturalist	no treatment	about 7.77%
iNaturalist	FD-KL	2.83%
iNaturalist	FD-LM	2.71%

Several additional findings sharpen the interpretation of these results. First, fine-tuning from the old model’s weights does not remove the problem: negative flips still occur, and PCT reduces them significantly even in that setting. Second, when updating from ResNet-18 to larger models such as ResNet-50 or DenseNet-161, PCT again reduces regression. Third, the same pattern holds when increasing training data or adding classes. These results support the claim that sample-wise inconsistency is not an artifact of one architecture or one data regime, but a general property of model replacement (Yan et al., 2020).

The paper also tracks NFR during training and reports that NFR follows the general trend of the error rate, while Focal Distillation creates and maintains a lower relative NFR earlier in training and throughout convergence. An uncertainty analysis based on deep ensembles shows that negative-flipped samples tend to be more uncertain on average, but uncertainty does not cleanly separate them from other samples. PCT with focal distillation nevertheless reduces NFR across samples with different uncertainty levels, indicating that regression reduction cannot be reduced to a simple confidence-thresholding story (Yan et al., 2020).

ELODI reports related empirical gains on multiple image classification benchmarks. On the ImageNet update ResNet-18 $\mathcal{L}_{\mathrm{CE}}$1 ResNet-50, the no-treatment baseline has ER 30.24 / 24.66, NFR 4.30, and Rel-NFR 25.00; ELODI with $\mathcal{L}_{\mathrm{CE}}$2 reports ER 30.95 / 23.10, NFR 2.11, and Rel-NFR 13.23. The paper also states a 29\% relative reduction on ImageNet for ResNet-18 $\mathcal{L}_{\mathrm{CE}}$3 ResNet-50 over previous methods, and reports improvements on iNaturalist, AG News, chain-of-update settings, and several architecture changes, while keeping single-model inference at deployment (Zhao et al., 2022).

6. Extensions beyond classification and terminological scope

The positive-congruence principle has been adapted beyond supervised image classification. In Monitored Distillation for Positive Congruent Depth Completion, the setting is depth completion without ground-truth depth, and the key difficulty is that the student must learn from an ensemble of teachers without knowing which teacher is correct at each region (Liu et al., 2022). The method therefore defines a monitor based on photometric reprojection error and sparse-depth consistency. For teacher $\mathcal{L}_{\mathrm{CE}}$4, the weighted residual is

$\mathcal{L}_{\mathrm{CE}}$5

with

$\mathcal{L}_{\mathrm{CE}}$6

and the best teacher is selected per pixel. After taking

$\mathcal{L}_{\mathrm{CE}}$7

the monitor is

$\mathcal{L}_{\mathrm{CE}}$8

The distilled depth loss is then weighted by $\mathcal{L}_{\mathrm{CE}}$9, whereas the standard unsupervised reconstruction and smoothness losses are weighted by $$\mathcal{L}_{\mathrm{CE}}(\phi,w)=\frac{1}{N}\sum_{i=1}^{N} -\log p_w(y_i\mid x_i),$$0. In effect, the student agrees with teachers only where their predictions are positively validated by the observed image and sparse point cloud, and otherwise falls back to unsupervised geometry (Liu et al., 2022).

This depth-completion formulation preserves the core logic of PCT while replacing “teacher correctness” with data-supported teacher compatibility. A plausible implication is that positive congruence is not tied to classification labels as such; it can be interpreted more generally as selective agreement with a prior model only on regions or examples for which that prior model is supported by external evidence. In the depth setting, this evidence is reconstruction consistency rather than a known label (Liu et al., 2022).

The acronym PCT is also overloaded in the broader arXiv literature. It denotes Probabilistic Constraint Training in transformer-based probabilistic logical reasoning (Nafar et al., 2023), Political Consistency Training in reinforcement-learning alignment for reducing covert political bias in LLMs (Phan et al., 21 May 2026), and the PCT theorem in work on Wightman axioms and conformal bootstrap (Maharana, 2021). These usages are terminologically unrelated to Positive Congruent Training, whose defining concern remains regression-free model updates through joint optimization of error rate and sample-wise congruence with a reference model.