Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures

Abstract: Do different LLM architectures encode high-level concepts in structurally compatible ways? We systematically characterize a geometric-functional universality dissociation: across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Using contrastive-difference CKA (CKA_Delta), a training-free diagnostic that computes kernel alignment on per-sample contrastive differences, we isolate concept-specific convergence from generic similarity -- achieving significant discrimination where standard CKA cannot. The dissociation replicates across all six concept domains we test (five with p <= 0.017 geometric discrimination and safety as a converging-functional trend, p = 0.08), including two non-instruction concepts (code-vs-NL, reasoning-vs-recall) validated without system prompts; a single 70B--70B pair provides an observational note that universality may strengthen with scale, requiring replication with additional >=70B models. We position CKA_Delta as a practical regime classifier and architectural outlier detector (Gemma: d = 1.08, AUC = 0.79) rather than an absolute transfer-accuracy predictor, providing a training-free diagnostic for cross-architecture concept monitoring.

• The paper introduces contrastive-difference CKA, a diagnostic measure that isolates concept-specific signals for structural alignment in LLMs.
• It leverages prompt-based activation extraction and PCA projection to achieve near-perfect functional transfer despite moderate geometric convergence.
• Empirical evaluations across domains highlight robust universality and prompt efficient transfer, offering practical insights for scalable model deployment.

Concept-Specific Structural Alignment in LLM Architectures: Geometric-Functional Dissociation via Contrastive-Difference CKA

Motivation and Theoretical Framing

The paper investigates the structural compatibility of concept encoding across heterogeneous LLM architectures, specifically probing whether high-level concepts—such as personality traits, safety, and code-reasoning distinctions—are represented in a manner that enables alignment and transfer between models without retraining. The work is positioned amidst recent advances in mechanistic interpretability, building upon the linear representation hypothesis positing that semantic concepts are captured as linear directions within model activation spaces. This question is critical for scalable deployment scenarios, where organizations operate portfolios containing diverse LLMs (e.g., Llama, Qwen, Gemma, Mistral).

A central insight is that raw activation similarity—using standard CKA or related metrics—fails to distinguish concept-specific convergence from generic similarity. The paper introduces contrastive-difference CKA (abbreviated as $CKA$), a kernel alignment measure computed on per-sample contrastive differences, targeting the isolation of concept-specific signal by canceling shared variance and instruction-formatting effects. This diagnostic is training-free, leveraging only prompt-based activation extraction and PCA projection, and it is rigorously justified with formal variance reduction and discriminability analysis.

Methodology: Contrastive-Difference CKA and Functional Transfer

Activation extraction is conducted across eight personality traits and several other domains via contrastive prompt pairs, each pair sharing a neutral question with opposing persona instructions. For each trait, sample-wise residual stream differences are computed, projected onto $k=50$ principal components, and analyzed using a debiased linear CKA. The method is formally shown to yield a provable signal-to-noise ratio advantage, robustly isolating concept-specific contributions irrespective of question set or generic representational overlap.

On the functional side, the paper evaluates universality not just geometrically (via $CKA$), but operationally—by training ridge-regularized logistic classifiers in PCA space for trait polarity and testing transfer to other models, both directly and with learned affine alignment maps. Multiple controls (random-label, cross-trait, random concept) and statistical methods (bootstrap, permutation tests, Mann–Whitney $U$, SVD rank analysis) validate discriminability and universality claims.

Empirical Results: Strong Concept-Specific Signal and Geometric-Functional Dissociation

Contrastive-difference CKA achieves significant discrimination (linear kernel: $d=0.60$, $p=0.002$; RBF: $d=0.67$, $p<0.001$), sharply outperforming traditional CKA, SVCCA, and cosine-based baselines ($d\leq 0.03$). Even random concept pairs yield negligible $CKA$ ($k=50$0, $k=50$1), confirming the specificity of the metric. Universality strength varies by trait (~$k=50$2 across Big Five dimensions). Architectural outliers are reliably flagged (Gemma $k=50$3, AUC $k=50$4), and transfer difficulty is rank-ordered within instruction-level concepts ($k=50$5, $k=50$6, $k=50$7).

Figure 1: CKA heatmap across 8 traits and 6 model pairs; trait universality spans $k=50$8, and architectural outliers (e.g., Gemma) show lower universality.

Despite only moderate geometric convergence ($k=50$9), affine-aligned classifiers achieve near-perfect cross-model functional transfer ($CKA$0 across all 96 trait$CKA$1pair conditions; direct transfer $CKA$2; random-label control $CKA$3). Notably, concepts with lowest $CKA$4 (e.g., neuroticism $CKA$5) transfer as well as those with highest ($CKA$6 for helpfulness), exemplifying geometric-functional dissociation.

Figure 2: Geometric-functional dissociation—affine transfer $CKA$7 for all traits, independent of geometric universality ($CKA$8).

Extended analyses include nine models across five families and additional domains (safety, truthfulness, formality, code vs. NL, reasoning vs. recall). $CKA$9 decreases with architectural diversity, but affine transfer remains $U$0 universally. Observational evidence suggests universality may strengthen with scale ($U$1 for Llama-70B$U$2Qwen-72B), though additional large-scale replications are needed to disentangle scale from dataset overlap.

Generalization Across Concept Types

The geometric-functional dissociation is robustly replicated across six distinct concept domains:

• Personality: Moderate $U$3, perfect affine transfer.
• Safety: Geometric discrimination trends significant ($U$4, $U$5); functional transfer $U$6.
• Truthfulness, Formality: Lowest geometric convergence ($U$7 for formality) yet maintains $U$8 transfer.
• Non-instruction (Code vs. NL, Reasoning vs. Recall): Even stronger same-vs-cross contrast ($U$9–$d=0.60$0), minimal direct transfer, but $d=0.60$1 sharply isolates concept-specific signal.

Encoding-depth diagnostics reveal two transfer regimes: instruction-level concepts trade off surface and deep encoding; non-instruction concepts are uniformly deep-encoded, lacking system-prompt surface markers.

Implications: Theoretical and Practical

The findings nuance the Platonic Representation Hypothesis: architectures exhibit partial geometric convergence with functionally universal concept encoding, recoverable via affine transformations but not universally identical. Concept representations modulate generation behavior (e.g., persona-induced completion shifts, refusal circuits) across architectures, enabling alignment tool transfer and scalable deployment.

Contrastive-difference CKA is positioned as a practical, training-free distributional diagnostic—screening for alignment viability, architectural outliers, and concept encoding depth, before investing in alignment mapping or constructing feature dictionaries (e.g., USAEs, SAE stitching). For instruction-level concepts, $d=0.60$2 reliably predicts whether direct transfer suffices or alignment investment is warranted.

From a deployment perspective, organizations can rapidly triage transfer difficulty, audit model alignment, and optimize architecture choices using relative (not absolute) $d=0.60$3 computed from unlabeled contrastive prompts. The diagnostic is low-resource, requiring $d=0.60$4 minutes per model pair, and does not assume SAE training or labeled data.

Limitations and Future Directions

The paper delineates multiple limitations: concept-domain coverage (six tested, more required), question-set sensitivity (absolute $d=0.60$5 varies, discrimination strength robust), scale observations requiring further replication, non-significant geometric discrimination for safety (relying on converging evidence), and practical challenges for standardization and multimodal extension.

Future work should extend cross-architecture analysis to multimodal encoders, test universality for syntactic and factual concepts, clarify scale effects, and refine diagnostic protocols for cross-lingual and cross-domain deployment.

Conclusion

Contrastive-difference CKA provides a robust, training-free diagnostic for concept-specific structural alignment across LLM architectures, sharply distinguishing concept-level convergence from generic similarity. Empirical analyses reveal a geometric-functional universality dissociation: moderate geometric convergence coexists with near-perfect functional transfer across traits, concepts, scales, and architectural families. The practical implications facilitate portable alignment monitoring, transfer triage, and architectural selection in heterogeneous model portfolios. Theoretically, the results refine the understanding of universality in LLM concept representation, opening avenues for scalable mechanistic interpretability and alignment.