Reverse O-Tag Representation Drift

• The paper introduces reverse O-tag representation drift as a bias where the background class O absorbs future PII features during initial training.
• It outlines a diagnostic protocol using prototype drift metrics and adaptive freezing strategies to mitigate the impact on incremental entity learning.
• Empirical results on French ASR data show that unfreezing the O head significantly improves performance over naive incremental training.

Reverse O-tag representation drift is a form of representation bias in incremental named entity recognition (NER) in which the background class $O$ has previously absorbed the surface-pattern features of entities that were unlabeled at initial training time, and those features must later be actively “released” before the new entities can be learned. The term was introduced in the study of NER model extension for noisy spoken-language data, where personally identifiable information (PII) entities such as EMAIL, PHONE, IBAN, and PDL were initially masked as $O$, then added later as explicit labels. In that setting, the dominant failure mode was not standard catastrophic collapse of the old semantic classes, but a background-class ownership of morphological cues that blocked new learning until the $O$ classifier was unfrozen (Zhang et al., 20 Oct 2025).

1. Definition and conceptual scope

Reverse O-tag representation drift is the paper’s term for “a background-class (O) bias that is created before the extension step and then must be actively undone for the model to learn new PII classes” (Zhang et al., 20 Oct 2025). It is “reverse” because it inverts the usual incremental NER failure mode. In ordinary incremental degradation, old entities are often pushed toward $O$ after model extension. Here, by contrast, the initial model has already learned to map future entity types to $O$, because those types were deliberately masked as background during the original training stage.

The core setting is incremental extension of a token classifier that already recognizes semantic entities such as PER, LOC, and ORG. During the baseline phase, tokens that instantiate later-added PII classes are labeled $O$. In noisy spoken-language ASR text, examples include verbalized email and phone patterns such as “john dot doe at gmail dot com” and “zero six eighty-three…”, both initially treated as background. When extension data later relabel those same patterns as EMAIL or PHONE, the model encounters a conflict: the $O$ classifier has already learned that these pattern features belong to background, and that learned ownership obstructs reassignment to the new labels (Zhang et al., 20 Oct 2025).

This phenomenon is therefore not merely a generic shift in embedding geometry. It is a label-policy-induced representational drift centered on the background class. A plausible implication is that reverse O-tag drift is especially likely in staged annotation pipelines, privacy-preserving preprocessing, or domain-adaptation workflows in which future entity types are initially suppressed into $O$.

2. Mechanism: how the background class comes to “own” pattern features

The mechanistic account in the NER extension study rests on a distinction between largely independent semantic and morphological feature mechanisms. Semantic entities such as PER and ORG are described as relying on contextual cues, whereas PII classes are described as relying on pattern-like features such as digit runs, separators, and domain tokens. During the initial phase, because all PII instances are labeled $O$, the $O$ head becomes the sink for those morphological features (Zhang et al., 20 Oct 2025).

At the classifier level, the paper writes the token-level softmax and cross-entropy in the standard form

$O$0

In the initial phase, for a token with phone-like morphology, $O$1 and $O$2. If the $O$3 head already gives such patterns a large logit $O$4, then $O$5, the gradients are small, and the mapping of those features to $O$6 is reinforced. In the extension phase, the labels reverse: now $O$7 and $O$8. If the $O$9 head is frozen, the gradient $O$0 cannot change the $O$1 parameters even when $O$2 remains large. The PHONE head must then “fight uphill” against a frozen, dominant background classifier (Zhang et al., 20 Oct 2025).

The encoder-level account is analogous. Let $O$3 denote the hidden state passed to the classifier and $O$4 the class-specific classifier weights. If $O$5 is large on pattern features and fixed, then learning new PII classes requires the encoder to move $O$6 away from $O$7 dominance while the most salient background decision boundary remains immovable. The paper characterizes this as learning inertia. Unfreezing the $O$8 head changes the gradient flow: for tokens now labeled as PHONE or EMAIL, the $O$9 head can reduce $O$0 on those patterns while the new PII heads increase their own logits, and the encoder can then reshape those features toward the new classes (Zhang et al., 20 Oct 2025).

The result is an explicitly asymmetric plasticity claim. PER, LOC, and ORG do not all need equal freedom during extension. What must remain plastic is the part of the classifier that previously monopolized the feature family needed by the new labels.

3. Experimental setting and empirical signature

The main empirical study used Jean-Baptiste/camembert-ner, a French RoBERTa-based model with RoBERTa-style subword segmentation and a token-level linear classification head. The domain was private French transcribed spoken language with ASR noise, including conversational and non-standard forms. The entity inventory comprised PER, LOC, ORG, PHONE, EMAIL, IBAN, and PDL. Dataset-A-Original was used to train the baseline on PER, LOC, ORG, and $O$1, with all PII tokens masked as $O$2. Dataset-B-Additional was a smaller, PII-rich corpus in which PHONE, EMAIL, IBAN, and PDL were explicitly labeled (Zhang et al., 20 Oct 2025).

The paper compared four extension regimes: baseline training on the original tag set, joint training on the union of both datasets with all seven labels, naive incremental training on Dataset-B-Additional, and incremental training with PER/LOC/ORG frozen while $O$3 and the new PII heads remained trainable. Two ablations were decisive: freezing the entire encoder and training only new entity classifiers produced complete failure to learn new entities, and freezing all old classifier heads including $O$4 while fine-tuning the encoder and new PII heads also produced complete failure (Zhang et al., 20 Oct 2025).

A concise performance snapshot is as follows.

Regimen	Overall F1	Selected class F1
Training on BERT tags (Baseline)	63.11	LOC 62.03, PER 83.63
Joint training	67.76	PHONE 61.54, EMAIL 33.30, IBAN 41.30
Naive incremental	54.96	LOC 42.24, EMAIL 28.39, IBAN 21.25
Incremental + freeze old classifier heads except O	59.63	LOC 60.32, EMAIL 30.12, IBAN 33.30

The paper interprets joint training as evidence of “peaceful coexistence”: original semantic classes remain comparatively stable while new PII classes are added. Under naive incremental learning, however, LOC was uniquely vulnerable. Its F1 dropped to $O$5, from a baseline LOC performance of $O$6, and its covariance drift rose to $O$7, consistent with representation overlap between location-like tokens and pattern-based PII such as postal-code-containing strings. By contrast, allowing $O$8 to adapt during extension raised LOC F1 from $O$9 to $O$0, PER from $O$1 to $O$2, ORG from $O$3 to $O$4, EMAIL from $O$5 to $O$6, and IBAN from $O$7 to $O$8, while PHONE changed from $O$9 to $O$0 (Zhang et al., 20 Oct 2025).

The drift statistics reinforce the interpretation. In Original vs. Naive, the $O$1 class showed covariance drift $O$2 with variance contraction $O$3, indicating substantial restructuring of the background class once PII patterns were removed from it. LOC covariance drift fell from $O$4 in Original vs. Naive to $O$5 in Original vs. Freeze except $O$6, while the $O$7 class itself still exhibited high drift $O$8, consistent with the claim that $O$9 must change in order to release pattern features to the new heads (Zhang et al., 20 Oct 2025).

4. Diagnosis and reversal

The study uses incremental learning as a diagnostic probe rather than treating extension failure as a black-box outcome. Its primary representation metrics are class prototype drift, cosine overlap between class centroids, and output-distribution shift:

$O$0

$O$1

$O$2

In practice, the paper computes drift on token embeddings from the final encoder layer and supplements it with mean drift, covariance drift, and variance change (Zhang et al., 20 Oct 2025).

The recommended detection protocol is operational. First, train the baseline on Dataset-A-Original with PII masked as $O$3. Second, compute per-class prototypes $O$4 and covariance statistics on held-out data. Third, run two short extension probes on Dataset-B-Additional: Probe A freezes old heads including $O$5, while Probe B freezes PER/LOC/ORG but leaves $O$6 and new PII heads trainable. Fourth, recompute $O$7, covariance drift for $O$8 and LOC, and $O$9 for $O$0 versus new PII outputs. The signature of reverse O-tag drift is then defined by a conjunction of symptoms: high $O$1 covariance drift and variance contraction during PII training, failure of Probe A to learn PII, and success of Probe B in learning PII while stabilizing LOC (Zhang et al., 20 Oct 2025).

The reversal procedure is likewise explicit. The mitigation algorithm freezes $O$2, unfreezes $O$3 and the new PII heads, and keeps the encoder at least partially plastic by unfreezing the last $O$4 layers. It further proposes loss weighting that upweights new PII classes and can downweight $O$5 to avoid re-entrenchment, augments training with ASR-style variants such as spoken digits, “dot,” and “at,” and trains until PII F1 plateaus and LOC covariance drift falls below a threshold such as $O$6. A consolidation stage then gradually unfreezes the semantic heads and may apply knowledge distillation from a joint-training teacher on the old entities while excluding $O$7 from KD (Zhang et al., 20 Oct 2025).

The key misconception addressed by these diagnostics is that freezing all old components necessarily protects previous knowledge. In this setting, freezing $O$8 prevented the model from learning the new entities at all.

5. Relation to broader representation-drift research

Reverse O-tag representation drift belongs to a wider family of representation-drift problems, but neighboring literatures target different objects and invariances. In exemplar-free continual representation learning, prototype misalignment is treated as a major cause of forgetting, and Learnable Drift Compensation learns a forward projector $O$9 by minimizing

$O$0

then transports old prototypes into the current space. The same line proposes an explicit token-level NLP adaptation in which the heterogeneous $O$1 tag is modeled with $O$2 sub-prototypes $O$3, and both $O$4-subprototypes and entity prototypes are updated via $O$5 without storing exemplars (Gomez-Villa et al., 2024).

A different continual-learning approach counters negative representation drift by anchoring task representations to global token prototypes from a pre-trained decoder and by introducing neighbor attention with a small mixing ratio $O$6. In the paper’s adaptation to sequence labeling, the same anchoring strategy is proposed as a way to stabilize heterogeneous $O$7-tag representations through masked-language-model-style conditioning and shared token-level global prototypes (Bai et al., 2022).

Projective-geometry work on representation drift makes yet another distinction. There, the concern is that Euclidean and cosine distances overestimate change when representations are only defined up to global scaling, sign, or phase, and the Fubini–Study distance

$O$8

is used to isolate intrinsic ray-level evolution (Tozzi, 1 Feb 2026). This contrast suggests that reverse O-tag representation drift is not primarily a gauge problem. Its salient issue is not sign or phase ambiguity, but a semantically consequential relocation of decision boundaries caused by earlier background labeling.

Attention-side drift analysis provides a further extension. In a separate study of hallucination dynamics, hidden-state cosine drift, attention entropy drift, JS-Drift, and Spearman-Drift are tracked under iterative context injection, with an empirical “attention-locking” threshold near JS-Drift $O$9 and Spearman-Drift $O$0. The same work proposes a sequence-tagging adaptation that monitors O-token attention and triggers soft interventions when $O$1, $O$2, or entropy drift rises sharply, and hard interventions near $O$3 and $O$4 (Wei et al., 22 May 2025). Those thresholds were not derived from the NER extension paper itself, but they provide an auxiliary monitoring framework for background-class instability.

6. Limitations, likely domains, and open questions

The empirical findings on reverse O-tag drift come from a proprietary French ASR dataset and from the addition of strongly pattern-based entities. The paper explicitly notes that cross-language and cross-dataset validation is still needed, that other extension scenarios may behave differently, and that the work is diagnostic rather than algorithmic, leaving optimal schedules and weighting schemes as open research problems (Zhang et al., 20 Oct 2025).

The phenomenon is most likely when future labels are driven by strong surface-form regularities and those regularities were previously collapsed into $O$5. The paper states that it should generalize to additions such as license plates or ticket numbers, while also noting that for semantic additions the effect may be weaker because $O$6 may not “own” the decisive features in the same way (Zhang et al., 20 Oct 2025). A plausible implication is that reverse O-tag drift is especially relevant in staged annotation programs where entity ontologies expand from coarse semantic tags toward morphology-heavy privacy tags.

Several controls are listed as diagnostic recommendations rather than reported results. These include increasing PII examples in the additional dataset, regex-based prelabeling to reduce the initial masking of obvious PII as $O$7, adding separate heads or adapters for new PII classes, and using regularization such as EWC or KD while excluding $O$8 from KD so as not to re-entrench the old background mapping (Zhang et al., 20 Oct 2025). Their status is therefore prospective, not validated within the main experimental comparison.

The central theoretical implication is that the $O$9 tag is not a neutral residual class. Under incremental extension, it can become an active repository of discriminative features. Reverse O-tag representation drift names the situation in which that repository has to be reorganized before new classes can emerge. In the reported NER setting, the practical consequence is clear: background-class plasticity is a prerequisite for successful addition of pattern-based entities, and freezing $O$00 can be more damaging than freezing the older semantic heads themselves (Zhang et al., 20 Oct 2025).