---
title: Positive Congruent Training (PCT)
url: https://www.emergentmind.com/topics/positive-congruent-training-pct
type: topic
---

# Positive Congruent Training (PCT)

Searching arXiv for recent and foundational papers on Positive Congruent Training and related extensions.
Positive Congruent Training (PCT) is a training paradigm for model updates in which the updated model is optimized not only for lower error rate, but also for lower regression on samples that a reference model already handled correctly. Introduced for image classification as a route toward regression-free model updates, PCT addresses the practical problem that two model versions can have similar or even improved aggregate accuracy while still differing substantially on which individual samples they fail. In that setting, the central failure mode is the **negative flip**: a test sample that was correctly classified by the old model but incorrectly classified by the new model [2011.09161].

## 1. Problem setting and conceptual motivation

PCT begins from the observation that reducing inconsistencies between versions of an AI system can be as important in practice as reducing overall error. In deployed systems, downstream components may depend on stable predictions for previously processed data, cached outputs, or post-processing logic. A model update that lowers average error but introduces new mistakes on previously correct cases can therefore be undesirable, even if it appears better under conventional evaluation [2011.09161].

The original formulation distinguishes three sample-wise outcomes for a test sample $(x_i,y_i)$. If both old and new models are correct, the sample is **positive-congruent**. If the old model is wrong but the new model is correct, the event is a **positive flip**. If the old model is correct but the new model is wrong, the event is a **negative flip**. This vocabulary reframes model updating as a problem of preserving already-safe behavior while still allowing desirable corrections on cases the old model handled poorly.

A recurrent misconception in this setting is that better accuracy automatically implies fewer regressions. The original paper argues the opposite on several grounds. Two models can have nearly identical error rates but disagree sharply on which samples they get wrong; a model with better average accuracy can still have many negative flips relative to the reference model; and simply copying the old model yields zero negative flips but no improvement. PCT therefore treats regression reduction as a distinct objective rather than a side effect of ordinary accuracy optimization [2011.09161].

## 2. Formalization, metrics, and objective functions

The basic PCT objective is to minimize both the new model’s error rate and the negative flip rate relative to a reference model. If $\phi^{\mathrm{old}}$ is the reference model and $\phi^{\mathrm{new},w}$ is the updated model, the training problem is formulated as
$$
\min_w \mathcal{L}_{\mathrm{CE}}(\phi^{\mathrm{new},w}) + \lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}}),
$$
where $\mathcal{L}_{\mathrm{CE}}$ is the standard empirical cross-entropy loss,
$$
\mathcal{L}_{\mathrm{CE}}(\phi,w)=\frac{1}{N}\sum_{i=1}^{N} -\log p_w(y_i\mid x_i),
$$
and $\mathcal{L}_{\mathrm{PC}}$ is a positive-congruence term that biases the new model toward the old one only in a selective way [2011.09161].

The central deployment metric is the **negative flip rate (NFR)**,
$$
\mathrm{NFR}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left(\hat y_i^{\mathrm{new}}\neq y_i,\ \hat y_i^{\mathrm{old}}=y_i\right),
$$
which directly measures sample-wise inconsistency across model versions. Because raw NFR is influenced by the accuracies of the old and new models, the paper also introduces a normalized quantity,
$$
\mathrm{NFR}_{\mathrm{rel}}=\frac{\mathrm{NFR}}{(1-\mathrm{ER}_{\mathrm{old}})\cdot \mathrm{ER}_{\mathrm{new}}},
$$
to factor out the overlap expected if the two models were independent. The use of $\mathrm{NFR}_{\mathrm{rel}}$ makes it possible to distinguish regression reduction that follows trivially from lower error rate from regression reduction due to a genuinely congruent update strategy [2011.09161].

The framework is intentionally generic. The positive-congruence term is written in the form
$$
\lambda \mathcal{L}_{\mathrm{PC}}(\phi^{\mathrm{new},w};\phi^{\mathrm{old}})
= \mathcal{F}(x_i)\,\mathcal{D}(\phi_w^{\mathrm{new}}(x_i), q(x_i)),
$$
where $\mathcal{F}$ is a filter or weighting function, $\mathcal{D}$ is a distance function, and $q(x_i)$ is a target derived from the old model or the ground-truth label. The defining design principle is that $\mathcal{F}$ should place extra emphasis on samples that the old model classified correctly. This selective emphasis is what differentiates PCT from broad imitation of a reference model [2011.09161].

## 3. Methodological instantiations: selective congruence rather than full imitation

The simplest baseline within the framework is a naive positive-congruence loss that adds extra cross-entropy weight on samples correctly classified by the old model:
$$
\mathcal{L}_{\mathrm{PC}}^{\mathrm{naive}}
=\frac{1}{N}\sum_{i=1}^{N}
-\mathbf{1}\!\left(\hat y^{\mathrm{old}}(x_i)=y_i\right)\log p_w^{\mathrm{new}}(y_i\mid x_i).
$$
This is equivalent to reweighting correctly classified old-model samples by a factor of $1+\lambda$. Empirically, however, the paper reports that the naive strategy is hard to tune, can lead to rote memorization if the weight is too high, and does not reduce NFR substantially [2011.09161].

The main practical method proposed in the original work is **Focal Distillation (FD)**. FD applies a distillation penalty to all samples but assigns larger weight to samples the old model classified correctly:
$$
\mathcal{L}_{\mathrm{PC}}^{\mathrm{Focal}}
=
-\sum_{i=1}^{N}
\left[\alpha+\beta\cdot \mathbf{1}\!\left(\hat y^{\mathrm{old}}(x_i)=y_i\right)\right]
\mathcal{D}(\phi_w^{\mathrm{new}},\phi_w^{\mathrm{old}}).
$$
Its weighting function is
$$
\mathcal{F}=\alpha+\beta\cdot \mathbf{1}\!\left(\hat y^{\mathrm{old}}(x_i)=y_i\right),
$$
so every sample receives weight $\alpha$, while samples correctly classified by the reference model receive an additional $\beta$. The selective emphasis is the core mechanism: it pushes the new model harder to preserve the old model’s correct decisions while leaving room for improvement on the old model’s mistakes [2011.09161].

The method admits several informative special cases. Setting $\alpha=1,\beta=0$ recovers ordinary distillation. Setting $\alpha=0,\beta>0$ yields distillation only on samples correctly predicted by the old model. The best results are generally obtained with $\alpha=1$ and $\beta$ between about $5$ and $10$. The paper also reports that $\alpha=0,\beta=1$ was ineffective, indicating that the new model still needs to learn the old model’s behavior on all samples if regression reduction is to work well in practice [2011.09161].

Two distance functions are considered for FD. The first is temperature-scaled KL divergence,
$$
\mathcal{D}^{\mathrm{KL}}(\phi_w^{\mathrm{new}},\phi_w^{\mathrm{old}})
=
\mathrm{KL}\!\left(
\sigma\!\left(\frac{\phi_w^{\mathrm{new}}(x_i)}{\tau}\right),
\sigma\!\left(\frac{\phi_w^{\mathrm{old}}(x_i)}{\tau}\right)
\right),
$$
where $\tau$ is a temperature parameter. The paper notes that to make this effective for PCT, $\tau$ must be quite large, for example $\tau=100$, at which point the loss behaves more like direct logit matching. The second is **logit matching**,
$$
\mathcal{D}^{\mathrm{LM}}(\phi_w^{\mathrm{new}},\phi_w^{\mathrm{old}})
=
\frac{1}{2}\left\|
\phi_w^{\mathrm{new}}(x_i)-\phi_w^{\mathrm{old}}(x_i)
\right\|_2^2,
$$
which the paper often finds slightly better for reducing NFR [2011.09161].

This methodology is defined in opposition to two more familiar update strategies. Ordinary fine-tuning optimizes only the task loss, even when initialized from the old model; it provides no explicit mechanism for preserving correctness on the old model’s positive examples. Ordinary model distillation, by contrast, attempts to mimic the old model broadly, whether the old model is right or wrong. PCT rejects both extremes. It does not merely continue training for accuracy, and it does not treat all teacher predictions as equally worth preserving. Its guiding principle is to maximize congruency only on **positive predictions** of the reference model [2011.09161].

## 4. Reference models, ensembles, and later variants

The reference model functions as the behavioral anchor for update stability. In the backward setting, it is fixed and inherited from the deployed system. In that case, PCT adapts the new model to the old one, but only on the subset of samples already handled correctly. The old model is therefore not treated as universally correct; it is treated as a guide on its reliable subset. This selective use is a defining conceptual distinction between PCT and general imitation-based approaches [2011.09161].

The original paper also studies a forward setting in which the reference model itself can be designed. It considers an ensemble of old models rather than a single network, with ensemble prediction
$$
p^{\mathrm{ensemble}}(y\mid x_i)=
\sigma\!\left(
\frac{1}{L}\sum_{j=1}^{L}\phi_{w_j}^{\mathrm{old}}(y\mid x_i)
\right).
$$
The intuition is that independently trained models tend to make different mistakes even when their error rates are similar, and averaging reduces that sample-wise variability. Empirically, the paper finds that ensembles reduce NFR more rapidly than error rate as ensemble size grows, and that increasing ensemble size keeps lowering NFR even after accuracy plateaus. This suggests that NFR can be reduced beyond what would be predicted from average error rate alone [2011.09161].

A later development, **ELODI** (“Ensemble Logit Difference Inhibition”), analyzes the role of ensembles in reducing NFR and proposes a way to retain ensemble-like regression reduction at the inference cost of a single model [2205.06265]. ELODI distills a homogeneous ensemble into a single student and replaces full-logit matching with **Logit Difference Inhibition (LDI)**, which penalizes only a subset of classes with the highest logit values:
$$
L_{\mathrm{LDI}}(x)=
\sum_{k\in \mathcal{K}(x)}
\left(\|M_k(x)-M_k^{\mathrm{(ens)}}(x)\|_p\right)^p.
$$
The full objective is
$$
L=(1-\alpha)L_{\mathrm{CE}}+\alpha L_{\mathrm{LDI}},
$$
with default settings $p=2$, $K=10$, and $\alpha=0.8$. The paper argues that negative flips are often driven by a few outstanding logits with large differences, so top-$K$ inhibition transfers the ensemble’s low-variance behavior without forcing the student to match all classes equally [2205.06265].

ELODI also formalizes legacy-model integration for cases in which the deployed old model was not trained with ELODI. In that case, the student is optimized against both the newly constructed ensemble and the existing legacy model:
$$
L_{\mathrm{total}}=
\lambda L_{\mathrm{LDI}}(M^\diamondsuit_{\mathrm{new}};M^{\mathrm{(ens)}}_{\mathrm{new}})
+(1-\lambda)L_{\mathrm{LDI}}(M^\diamondsuit_{\mathrm{new}};M_{\mathrm{old}}).
$$
This later work preserves the core PCT objective—simultaneous reduction of error rate and NFR—while shifting emphasis from selective positive-example distillation to ensemble-induced stabilization of logit geometry [2205.06265].

## 5. Empirical behavior in image classification

The original evaluation of PCT focuses on large-scale image classification, primarily **ILSVRC12 / ImageNet** and **iNaturalist**, across several update scenarios: same architecture with different training runs, architecture changes, more training samples per class, more classes, combinations of architecture and data change, fine-tuning settings, and ensemble-based reference models [2011.09161].

The main metrics are the error rates of old and new models, NFR, relative NFR, and number of parameters for model cost comparisons. On the benchmark scenario of training the same architecture twice on ILSVRC12, the paper reports NFR of about **6.44\%** for no treatment, **2.50\%** for FD-KL, and **2.35\%** for FD-LM. On iNaturalist, the corresponding NFR values are about **7.77\%**, **2.83\%**, and **2.71\%**. The ensemble approach yields the lowest NFR overall, reaching about **1.70\%** on ILSVRC12, albeit at much higher computational cost [2011.09161].

| Setting | Method | NFR |
|---|---:|---:|
| ILSVRC12, same architecture trained twice | no treatment | about 6.44% |
| ILSVRC12, same architecture trained twice | FD-KL | 2.50% |
| ILSVRC12, same architecture trained twice | FD-LM | 2.35% |
| ILSVRC12, same architecture trained twice | ensemble reference | about 1.70% |
| iNaturalist | no treatment | about 7.77% |
| iNaturalist | FD-KL | 2.83% |
| iNaturalist | FD-LM | 2.71% |

Several additional findings sharpen the interpretation of these results. First, fine-tuning from the old model’s weights does not remove the problem: negative flips still occur, and PCT reduces them significantly even in that setting. Second, when updating from ResNet-18 to larger models such as ResNet-50 or DenseNet-161, PCT again reduces regression. Third, the same pattern holds when increasing training data or adding classes. These results support the claim that sample-wise inconsistency is not an artifact of one architecture or one data regime, but a general property of model replacement [2011.09161].

The paper also tracks NFR during training and reports that NFR follows the general trend of the error rate, while Focal Distillation creates and maintains a lower relative NFR earlier in training and throughout convergence. An uncertainty analysis based on deep ensembles shows that negative-flipped samples tend to be more uncertain on average, but uncertainty does not cleanly separate them from other samples. PCT with focal distillation nevertheless reduces NFR across samples with different uncertainty levels, indicating that regression reduction cannot be reduced to a simple confidence-thresholding story [2011.09161].

ELODI reports related empirical gains on multiple image classification benchmarks. On the ImageNet update ResNet-18 $\rightarrow$ ResNet-50, the no-treatment baseline has ER **30.24 / 24.66**, NFR **4.30**, and Rel-NFR **25.00**; ELODI with $K=10$ reports ER **30.95 / 23.10**, NFR **2.11**, and Rel-NFR **13.23**. The paper also states a **29\% relative reduction on ImageNet for ResNet-18 $\rightarrow$ ResNet-50** over previous methods, and reports improvements on iNaturalist, AG News, chain-of-update settings, and several architecture changes, while keeping single-model inference at deployment [2205.06265].

## 6. Extensions beyond classification and terminological scope

The positive-congruence principle has been adapted beyond supervised image classification. In **Monitored Distillation for Positive Congruent Depth Completion**, the setting is depth completion without ground-truth depth, and the key difficulty is that the student must learn from an ensemble of teachers without knowing which teacher is correct at each region [2203.16034]. The method therefore defines a monitor based on photometric reprojection error and sparse-depth consistency. For teacher $i$, the weighted residual is
$$
E_i(x)=\beta_i P_i(x),
$$
with
$$
\beta_i=1-\exp(-\alpha Z_i),
$$
and the best teacher is selected per pixel. After taking
$$
E(x)=\min_i E_i(x),
$$
the monitor is
$$
Q(x)=\exp(-\lambda E(x)).
$$
The distilled depth loss is then weighted by $Q(x)$, whereas the standard unsupervised reconstruction and smoothness losses are weighted by $1-Q(x)$. In effect, the student agrees with teachers only where their predictions are positively validated by the observed image and sparse point cloud, and otherwise falls back to unsupervised geometry [2203.16034].

This depth-completion formulation preserves the core logic of PCT while replacing “teacher correctness” with data-supported teacher compatibility. A plausible implication is that positive congruence is not tied to classification labels as such; it can be interpreted more generally as selective agreement with a prior model only on regions or examples for which that prior model is supported by external evidence. In the depth setting, this evidence is reconstruction consistency rather than a known label [2203.16034].

The acronym **PCT** is also overloaded in the broader arXiv literature. It denotes **Probabilistic Constraint Training** in transformer-based probabilistic logical reasoning [2305.13179], **Political Consistency Training** in reinforcement-learning alignment for reducing covert political bias in LLMs [2605.22771], and the **PCT theorem** in work on Wightman axioms and conformal bootstrap [2102.01888]. These usages are terminologically unrelated to **Positive Congruent Training**, whose defining concern remains regression-free model updates through joint optimization of error rate and sample-wise congruence with a reference model.

Source: https://www.emergentmind.com/topics/positive-congruent-training-pct