Contrastive Neighborhood Alignment (CNA)

• Contrastive Neighborhood Alignment (CNA) is an unsupervised method that preserves local topology in representation spaces by aligning source-defined neighbors in a target model using a contrastive loss.
• It computes neighbor relations using cosine similarity and employs an InfoNCE-style loss to pull positive neighbors together while pushing non-neighbors apart.
• CNA finds practical applications in manifold learning, model distillation, and legacy model updates, delivering improved accuracy and reduced error rates.

Contrastive Neighborhood Alignment (CNA) is an unsupervised manifold learning approach for preserving local topology across representation spaces. Its central principle is that if two samples are mapped to nearby representations by a source or teacher model, they should remain neighbors when mapped by a target or student model. CNA operationalizes this principle with a contrastive objective that pulls source-defined neighbors together in the target space while pushing other samples apart, thereby preserving local structure without requiring ground-truth labels (Zhu et al., 2022).

1. Definition and conceptual scope

CNA was introduced as a method for maintaining the topology of learned features under representation transfer, dimensionality reduction, and model replacement (Zhu et al., 2022). In the original formulation, a source feature extractor defines local neighborhoods, and a target model is trained to mimic that local structure rather than reproduce exact logits or hidden states. This makes CNA a topology-preserving alignment method rather than a strict output-matching procedure.

The method is explicitly described as an unsupervised learning algorithm. Its objective is not semantic supervision in the label space, but preservation of neighborhood relations in the source representation space. This places CNA at the intersection of manifold learning, representation distillation, and metric-preserving transfer (Zhu et al., 2022).

A useful way to situate CNA is to distinguish it from pairwise similarity matching. CNA does not require globally matching all interpoint distances. Instead, it preserves which points are neighbors. This suggests that CNA prioritizes local geometry and connectivity structure over exact metric reconstruction.

2. Core formulation

For a dataset $\mathcal{D} = \{x_i\}$ and a source feature extractor $f_S(\cdot)$, the nearest neighbor of each sample in the source feature space is defined by cosine similarity as (Zhu et al., 2022): $\aleph(x_i) = x_{i^\star}$ where

$$i^\star = \argmax_{j=1,...,M,\, j\ne i} \frac{f_S(x_i) \cdot f_S(x_j)}{ \|f_S(x_i)\|_2 \cdot \|f_S(x_j)\|_2 }.$$

Given a target feature extractor $f_T(\cdot)$, CNA defines a contrastive loss for one sample as (Zhu et al., 2022): $$\ell_{\mathrm{CNA}}(x_i) = -\log\frac{ \exp\left[ f_T(x_i)\cdot f_T(\aleph(x_i)) / \tau \right] }{ \sum_{j\ne i}^M \exp \left( f_T(x_i)\cdot f_T(x_j) / \tau \right) }.$$

Here, $\tau$ is a temperature hyperparameter, and $f_T(x)$ is assumed $\ell_2$-normalized. The numerator contains the target-space similarity between a sample and its source-defined neighbor; the denominator contrasts that similarity against the rest of the minibatch. The effect is to preserve local neighborhoods while preventing representational collapse (Zhu et al., 2022).

The method also generalizes to top-$K$ neighbors by averaging over a source-defined neighborhood set (Zhu et al., 2022): $f_S(\cdot)$0

When labels are available, CNA can be combined with classification training through (Zhu et al., 2022): $f_S(\cdot)$1

This combined objective reflects an important design property of CNA: its contrastive loss is compatible with cross-entropy, which is part of the reason it can serve as a regularizer during supervised transfer or model update (Zhu et al., 2022).

3. Neighborhood preservation as topology transfer

The defining claim of CNA is that preserving neighborhood relations is sufficient to preserve the local geometry of a representation manifold (Zhu et al., 2022). Rather than minimizing a dense all-pairs distortion objective, CNA imposes a sparse local relational constraint: neighbors in source space should remain neighbors in target space.

In manifold learning terms, this makes CNA a local-topology preservation method. In distillation terms, it transfers structural knowledge rather than exact responses. In deployment terms, it provides a way to update or compress models while maintaining consistency in how nearby inputs are organized in representation space.

This differs from approaches that directly match teacher and student outputs via KL divergence, mean squared error, or feature regression. Standard knowledge distillation aligns outputs. CNA aligns neighborhoods. The latter can be advantageous when exact output matching is undesirable, when teacher errors should not be copied rigidly, or when teacher and student architectures differ substantially (Zhu et al., 2022).

A plausible implication is that CNA preserves inductive biases encoded in source geometry while leaving the target model some flexibility in how it realizes those relations internally. This is consistent with the observation that CNA regularizes structure rather than exact activations (Zhu et al., 2022).

4. Relation to manifold learning and contrastive representation learning

The original paper positions CNA against classical manifold learning methods such as Isomap, LLE, Hessian LLE, and MVU (Zhu et al., 2022). Those methods are described as transductive, often require operating on large affinity graphs, and do not directly yield inductive mappings for unseen points. CNA instead learns an explicit function via a neural network or MLP, trained in minibatches, and therefore supports inductive generalization.

This distinction is central. Traditional manifold learning preserves local structure by directly solving for low-dimensional coordinates of a fixed dataset. CNA preserves local structure by training a parametric mapper. The first yields an embedding of known points; the second yields a reusable model (Zhu et al., 2022).

CNA also sits within a broader contrastive family. Its loss resembles an InfoNCE-style objective, but the positives are not augmentations of the same sample. They are neighbors selected in a source representation space. This makes CNA a topology-guided contrastive method rather than an augmentation-invariance method.

Subsequent work has broadened the neighborhood-based contrastive idea in several directions. “Node Embeddings via Neighbor Embeddings” formulates graph Contrastive Neighbor Embedding as direct optimization of an InfoNCE objective over graph neighbors, where each node is pulled toward its neighbors and away from non-neighbors (Böhm et al., 31 Mar 2025). “Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings” uses controlled nearest neighbor sampling in citation embedding space to align text representations with citation-space neighborhoods (Ostendorff et al., 2022). These later methods are not the same algorithm as CNA, but they reflect the wider adoption of neighborhood-preserving contrastive learning across domains.

5. Principal application scenarios

The original CNA work illustrates three application scenarios: manifold learning, model distillation, and legacy model update (Zhu et al., 2022).

In manifold learning, CNA is used to maintain the local topology of high-dimensional data in a lower-dimensional target space. On synthetic manifolds such as the S-curve, Swiss Roll, and Sphere, it is reported to capture the manifold structure successfully in the low-dimensional space while remaining inductive (Zhu et al., 2022). On MNIST and CIFAR10, CNA is reported to outperform or match transductive methods in local error and 5-NN test accuracy; for example, the paper reports MNIST 5-NN accuracy of 92.3% for CNA versus 91.2% for Isomap (Zhu et al., 2022).

In model distillation, CNA is used to transfer representational topology from a larger source model to a smaller target model. The paper reports that CNA matches or exceeds the performance of Knowledge Distillation, Local Feature Alignment, and Contrastive Representation Distillation, especially when teacher and student differ significantly in architecture (Zhu et al., 2022). Reported CIFAR100 top-1 accuracies include 71.96 for ResNet56 $f_S(\cdot)$2 ResNet20, 71.30 for ResNet110 $f_S(\cdot)$3 ResNet20, and 75.20 for ResNet32x4 $f_S(\cdot)$4 ShufV1 (Zhu et al., 2022).

In legacy model update, CNA is used to reduce behavioral regression when replacing an old model with a new one. The paper reports that CNA significantly reduces Negative Flip Rate, with reductions of 29.9% on CIFAR100 and 14.1% on ImageNet, while slightly improving or preserving accuracy (Zhu et al., 2022). This is particularly relevant in production settings where a new model’s absolute accuracy may improve while still introducing undesirable new errors.

6. Empirical behavior, hyperparameters, and practical interpretation

Several practical observations are reported in the original study. CNA is described as robust to batch size given a reasonable size, robust to temperature choice with best results at $f_S(\cdot)$5, and most effective with neighbor number $f_S(\cdot)$6 in the reported experiments (Zhu et al., 2022). The paper also states that simpler architectures suffice and that minibatch-based training is efficient (Zhu et al., 2022).

These details matter because CNA’s neighbor selection is batch-dependent in the original formulation. The source-space nearest neighbor is selected within a minibatch, so the batch acts as the candidate set for both positives and negatives. This implies that batch composition can influence the local structure that is visible to the objective. The reported robustness to batch size suggests this dependence is manageable in practice, though it remains an implementation consideration (Zhu et al., 2022).

Another practical interpretation concerns why $f_S(\cdot)$7 worked best in the reported setting. A plausible implication is that the cleanest topological signal may come from the most reliable local relation rather than from averaging over larger neighborhoods, which can include weaker or noisier neighbors. The paper reports the empirical result, but does not generalize it as a universal rule (Zhu et al., 2022).

The table below summarizes the three reported deployment settings.

Setting	CNA role	Reported outcome
Manifold learning	Preserve local topology in low-dimensional target space	Captures manifold structure; MNIST 5-NN accuracy 92.3 vs. Isomap 91.2
Model distillation	Transfer structural knowledge from teacher to student	Competitive or better than KD, LFA, and CRD
Legacy model update	Regularize new model toward old model’s local structure	Negative Flip Rate reduced by 29.9% on CIFAR100 and 14.1% on ImageNet

7. Extensions, reinterpretations, and related paradigms

After the introduction of CNA, neighborhood-aware contrastive learning developed along several partially overlapping lines. In graphs, “Graph Soft-Contrastive Learning via Neighborhood Ranking” argues against absolute positive pair specification and instead preserves relative similarity ranking across hop neighborhoods using pairwise and listwise gated ranking InfoNCE losses (Ning et al., 2022). “Single-View Graph Contrastive Learning with Soft Neighborhood Awareness” introduces probabilistic neighbor masking and a normalized Jensen-Shannon divergence estimator, treating neighbors as positives only stochastically across epochs (Sun et al., 2024). These works extend the notion of neighborhood alignment from strict local matching to softer, ranked, or probabilistic neighborhood structure.

In supervised vision, “CoNe: Contrast Your Neighbours for Supervised Image Classification” aligns each sample not just with a class center but with its similar intra-class neighbors, explicitly addressing intra-class variance (Zheng et al., 2023). In scientific document representation, SciNCL aligns text embeddings with neighborhoods in citation-graph embedding space through controlled nearest neighbor sampling and a triplet margin loss (Ostendorff et al., 2022). In graph representation learning, graph CNE applies an InfoNCE objective directly to explicit graph neighbors and contrasts this with random-walk-based methods such as DeepWalk and node2vec (Böhm et al., 31 Mar 2025).

These methods should not be conflated with the original CNA algorithm. Their objectives, modalities, and sampling schemes differ. What they share is the broader principle that representation learning can be improved by explicitly preserving or aligning neighborhood structure rather than relying only on instance identity, augmentations, or global class centers.

A separate theoretical reinterpretation appears in “Your contrastive learning problem is secretly a distribution alignment problem,” which connects contrastive losses to distribution alignment with entropic optimal transport and places generalized contrastive alignment in a broader alignment framework (Chen et al., 27 Feb 2025). This suggests a wider mathematical perspective in which neighborhood alignment can be understood as one instance of structured transport between latent distributions.

8. Distinctions, misconceptions, and significance

A common misconception is that CNA is simply another variant of knowledge distillation. The original method differs from standard distillation because it does not require the student to reproduce teacher outputs or logits. Its emphasis is local topology preservation, not exact behavioral imitation (Zhu et al., 2022).

Another misconception is that CNA is just classical manifold learning with a contrastive loss. The original contribution is specifically inductive manifold learning: the goal is to learn a function that generalizes to unseen data, which contrasts with transductive embeddings such as Isomap or LLE (Zhu et al., 2022).

It is also important to distinguish the original CNA from later uses of the acronym in unrelated contexts. For example, “Contrastive Neuron Attribution” in LLMs uses CNA to denote a forward-pass-only neuron attribution method for identifying refusal-related neurons (Herring et al., 12 May 2026). That work is mechanistic interpretability, not neighborhood alignment.

The significance of Contrastive Neighborhood Alignment lies in establishing a concise and flexible principle: preserve source-space neighborhoods in target-space representations through a contrastive objective. In the original paper, this principle yields an unsupervised method applicable to dimensionality reduction, distillation, and model update, with reported improvements in topology preservation, downstream accuracy, and negative flip control (Zhu et al., 2022). Subsequent literature suggests that this neighborhood-centric contrastive view has become a recurring design pattern across graph learning, document representation, supervised vision, and structured multimodal alignment (Ostendorff et al., 2022, Zheng et al., 2023, Ning et al., 2022, Sun et al., 2024, Böhm et al., 31 Mar 2025).