A Mechanistic Investigation of Supervised Fine Tuning
Abstract: The cosine similarity between a LLM's hidden activations before and after Supervised Fine-Tuning (SFT) remains very high. This, at first glance, suggests that SFT leaves the model's activation geometry largely undisturbed. However, projecting both sets of activations through a Sparse Autoencoder (SAE) pretrained on the base model reveals that the underlying sparse latents diverge significantly. We introduce a novel investigative pipeline which utilizes these pretrained SAEs as a high-resolution diagnostic tool to mechanistically investigate the drivers of this representational divergence. Through our analytical pipeline, we discover task-specific and layer-specific distributions of the precise semantic features that are systematically altered during supervised fine-tuning. We additionally identify a layer-wise update profile specific to safety alignment. All code, experimental scripts, and analysis files associated with this work are publicly available at: https://github.com/ruhzi/sae-investigation.
Summary
- The paper demonstrates that supervised fine-tuning induces widespread latent semantic drift in pretrained LLMs, challenging the notion of surgical behavioral modification.
- It shows that while raw cosine similarities remain high, projections via Sparse Autoencoders expose significant reordering of underlying semantic features.
- Task-specific analyses reveal unique profiles, such as safety alignment concentrating feature disruptions in shallow layers, highlighting trade-offs between alignment and collateral modifications.
Mechanistic Insights into SFT-Induced Representation Drift with Sparse Autoencoders
Introduction
This work presents a rigorous analysis of how Supervised Fine-Tuning (SFT) perturbs the internal representations of a pretrained LLM. The primary contribution is a mechanistic pipeline leveraging Sparse Autoencoders (SAEs) as frozen semantic probes to quantify and interpret representational drift that standard comparisons of hidden activations fail to reveal. By investigating four diverse downstream SFT objectives (natural language inference, mathematical reasoning, safety alignment, and tool-calling) on the Gemma 3 1B model family, the study exposes the limitations of raw geometric similarity measures and highlights the semantic volatility introduced by SFT—even under conditions where the model's hidden states appear almost unchanged.
Cosine Similarity Analysis: Surface vs. Latent Representation Drift
Empirical results demonstrate that, across all fine-tuning tasks and model layers, the cosine similarity between the base and SFT model's raw activation vectors remains exceptionally high (always exceeding 0.96, often >0.99), signaling strong alignment in the dense activation geometry. However, when these post-SFT activations are projected into a static SAE dictionary trained on the base model, a dramatic and monotonic degradation in latent cosine similarity emerges, with values plummeting as low as 0.557 in deep layers, especially for tasks like MultiNLI and GSM8K.
Figure 1: Cosine similarity trajectories between pretrained and fine-tuned representations across training steps. Raw hidden activations remain highly aligned throughout SFT, while SAE latent representations diverge substantially, particularly in deeper layers.
This dissociation implies that the SFT process operates in semantically rich but geometrically subtle subspaces: while the overall cloud of hidden activations is nearly preserved, their decomposition into monosemantic features (as indexed by the SAE) is fundamentally reordered. The divergence is most pronounced in deeper transformer layers and is consistent across SAE dictionary widths. This finding directly challenges interpretations based solely on raw feature space metrics and motivates a feature-centric view of SFT's representational impact.
Principal Component and Feature Flip Diagnostics
Subspace analysis via singular value decomposition of the SFT-induced activation change identifies low-rank, highly salient directions of representational drift in early layers, transitioning to more distributed perturbations in deeper layers. Specifically, a single principal component captures the majority of SFT-induced variance in shallow layers (up to 89.3%), while this compressibility sharply attenuates with depth.
Perturbing base hidden states along these dominant drift directions and projecting them through the SAE reveals which sparse semantic features undergo state transitions ("flips") as a consequence of SFT. Quantifying the flip rates of individual SAE features, and semantically classifying them via a taxonomy (Persona, Structure, Code, Reasoning, Safety, Multilingual, Collateral) using an LLM-as-a-judge pipeline, the analysis uncovers several notable patterns:
- Non-local and Collateral Disruption: A substantial portion of features disrupted by SFT bear no direct relevance to the fine-tuning task. For instance, Code-related features are heavily rewritten even in tasks devoid of programming content, and Collateral (semantically incidental or hyper-specific) features frequently dominate flipped counts, especially in shallower layers.
- Task- and Layer-Specific Profiles: Each SFT objective leaves a distinct signature. Safety alignment uniquely concentrates its latent disruption in early layers, with a 3:1 ratio of shallow to deep feature flips, while other objectives manifest more distributed or deep-focused rewrites.
- Feature Cluster Alignment: Outlier drift directions (those inducing anomalously high flip rates) are found to align most strongly with Collateral and Code clusters rather than with domains semantically related to the SFT task.
Implications and Theoretical Impact
These findings collectively refute the hypothesis that SFT operates as a surgical, task-specific intervention in latent space. Instead, SFT introduces widespread, functionally promiscuous representational drift, repurposing or perturbing a broad spectrum of features—including those with no clear connection to the newly instilled behavior. This holds across all tested objectives and SAE granularities.
The identification of safety alignment as a special case—with its representational effects localized in shallow layers and functionally distinct from other fine-tuning regimes—suggests that safety-oriented behaviors may recruit and modify internal mechanisms differently, possibly leveraging or disrupting domains (e.g., refusal and hedging patterns) encoded early in the processing pipeline. This raises foundational questions about how alignment techniques interact with architectural inductive biases and the distribution of semantic competencies across model depth.
From a practical perspective, the ubiquity of non-task-specific feature disruption argues for caution in assuming the "surgicality" of SFT-driven behavioral modification and points to a trade-off between alignment efficacy and semantic collateral damage. This motivates development of more targeted fine-tuning or feature-isolation methods, and signals the inherent limitations of current task-specific SFT approaches for safe and robust model alignment.
Limitations and Future Directions
The present analysis is bounded by the available SAE dictionaries, limited to Gemma 3 1B and the layers covered by GemmaScope 2. Generalization across architectures, scales, and alternative fine-tuning protocols remains hypothetical. The intrinsic feature-mixing that persists even with large (262k) dictionaries highlights the need for further advances in feature disentanglement and cross-model transferability of mechanistic tools.
Beyond immediate mechanistic diagnostics, future work could explore (1) alternative interpretability priors, (2) causal interventions to isolate or preserve task-critical features, and (3) direct use of SAE-based metrics to regularize or monitor fine-tuning for improved alignment and minimization of unintended semantic drift.
Conclusion
SFT induces extensive, often task-unrelated rewriting of a model’s underlying semantic features, frequently masked by the stability of raw activation geometry. Sparse autoencoders expose this latent volatility, demonstrating that geometric preservation in the activation space coexists with substantial functional reorganization at the feature level. Safety alignment exhibits unique, shallow-layer-centric effects, compared to other SFT tasks. These results challenge the notion of SFT as a localized behavioral modification and highlight the necessity of feature-resolved interpretability for alignment and model editing paradigms.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper asks a simple but deep question: when we fine-tune a LLM (teach it new skills with examples), what actually changes inside the model’s “thinking”? On the surface, the model’s internal signals look almost the same before and after training. But when the authors look more closely with a special tool (a Sparse Autoencoder, or SAE), they find that important hidden “features” do change a lot—especially in certain layers of the model and for certain tasks like safety.
Key Questions
The authors focus on five plain-language questions:
- Does supervised fine-tuning (SFT) really change how a model represents information internally, even if the overall signals look similar?
- If yes, which parts (layers) of the model change the most?
- Which specific “features” (like “JSON structure,” “refusal wording,” or “math reasoning steps”) are being turned up or down by fine-tuning?
- Do different tasks (reasoning, math, safety, tool use) reshape different parts of the model?
- Is safety training special in how and where it changes the model?
How They Studied It
Think of a LLM as a big factory with 26 stations (layers). As text goes through, each station transforms it a bit. The “activations” are like the factory’s conveyor belts carrying signals between stations. Fine-tuning tweaks the factory’s dials so the final product (the model’s answers) improves.
Here’s the tricky part: if you compare the conveyor belt signals before and after fine-tuning, they point in almost the same direction (very high cosine similarity, above 0.96). That makes it look like not much changed. But that can be misleading—like saying two songs sound “equally loud” while ignoring that many individual sliders on the equalizer moved.
To see those sliders, the authors use Sparse Autoencoders (SAEs):
- What is an SAE? Imagine a magnifying glass that takes the messy, dense factory signal and explains it using a small number of clear “feature switches.” Each switch is something human-understandable, like “markdown bullets,” “code syntax,” “safety refusal,” or “because/therefore” (reasoning).
- Importantly, these SAEs were trained only on the original (pre-fine-tuning) model and then frozen. That makes them a consistent ruler or yardstick. If the SAE says different switches light up after fine-tuning, that means the model’s inner representation truly changed, not the measuring tool.
The authors:
- Fine-tuned the same base model on four different tasks:
- MultiNLI (logical/semantic reasoning)
- GSM8K (step-by-step math)
- WildJailbreak (safety: refuse harmful requests)
- Tool Calling (produce correct JSON for tools)
- Measured similarity over time in two ways:
- Raw activations (the conveyor belts). These stayed very similar.
- SAE “latent features” (the equalizer sliders). These diverged a lot, especially in deeper layers.
- Found the main directions of change:
- They computed the difference between “after” and “before” activations and used a math method (SVD/PCA) to find the biggest “change directions.” Think of these as the main ways the factory’s process shifted.
- Probed what those changes meant:
- They gently pushed the base model’s activations along each main change direction and ran them through the frozen SAE to see which feature switches flipped from off to on (or on to off).
- They looked up human-readable feature descriptions (via Neuronpedia) and grouped features into seven clusters: Persona, Structure, Code, Reasoning, Safety, Multilingual, and Collateral (random/unrelated trivia).
- They also used another small AI to label many features automatically.
They examined three layers in detail: an early layer (7), a middle layer (13), and a late layer (22), and tested different SAE sizes (dictionaries with 16k, 65k, or 262k features).
Main Findings (What Changed and Why It Matters)
- The “big picture” looks stable, but the details change a lot.
- Raw activations barely move (cosine similarity > 0.96).
- But the SAE features—the meaningful switches—shift a lot, and the shift grows with training steps.
- The deeper the layer, the bigger the hidden change. For example, in a reasoning task, the late layer’s feature similarity dropped to about 0.56 (a big change), even though raw signals stayed highly aligned.
Why this matters: It shows fine-tuning is not just “doing nothing.” It’s more like carefully rewiring small, meaningful circuits inside the model while keeping the overall flow similar.
- Early vs. late layers change differently.
- Early layers: changes are “low-rank,” meaning a single main direction explains most of the shift (a broad, sweeping tweak).
- Late layers: changes are more spread out across many directions (more detailed, multi-part adjustments).
Why this matters: Fine-tuning seems to “set the stage” early on and then make more targeted, complex edits later.
- Safety training is special.
- For safety (WildJailbreak), more changes happen in early layers than in late ones, which is the opposite of other tasks.
- This suggests safety behavior is baked in earlier, influencing how the model processes inputs from the start.
Why this matters: To build safer models, early-layer adjustments may be especially important.
- “Collateral” changes happen—even for unrelated features.
- Features tied to Code are often disrupted even when the task has nothing to do with programming (like math or language reasoning).
- “Collateral” features (random facts or unrelated details) are frequently affected, especially in early layers.
Why this matters: Fine-tuning isn’t perfectly “surgical.” Adjusting the model for one goal can bump other, unrelated parts. This helps explain why models sometimes lose or change abilities after fine-tuning.
- Bigger SAE dictionaries give a clearer picture (but the main story stands).
- With very wide SAEs (262k features), the features are more disentangled and the measurements are cleaner.
- Even then, the core findings remain: deeper layers show large hidden changes; safety shifts early; and collateral/code features get affected across tasks.
What This Means (Implications)
- For model builders: Don’t judge changes by raw activations alone. Use feature-level tools (like SAEs) to see the true rewiring.
- For safety: Consider focusing on earlier layers when training refusal or guardrail behaviors. Early changes seem key.
- For reliability: Expect some “collateral damage” during fine-tuning. Plan tests to check unrelated abilities (like coding) even when you’re training for other skills.
- For interpretability research: Freezing a pre-trained SAE and using it as a yardstick is a powerful way to spot how fine-tuning reshapes the model’s internal concepts.
Limitations and Next Steps
- This study used one model size (Gemma 3, 1B parameters) and a specific interpretability suite (GemmaScope 2). Bigger models or different architectures might behave differently.
- SAEs only covered certain layers and depend on the quality of their feature dictionaries.
Future work could test more models, more layers, and different fine-tuning methods to see how general these patterns are—and to design fine-tuning strategies that are more precise and cause fewer unwanted side effects.
In Short
Fine-tuning often looks gentle on the surface, but under a “feature microscope” (SAE), you see strong, targeted rewrites—especially in deeper layers and especially early for safety. Those rewrites can spill over into unrelated features. Understanding and measuring these hidden changes can help us fine-tune models more safely and reliably.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper; each item is phrased to be directly actionable for future research.
- Generalization across models: Does the latent representational drift observed here hold across other architectures (e.g., Llama, Mistral), pretraining regimes, and larger parameter scales (7B, 27B), not just Gemma 3 1B IT?
- Base vs. IT initialization: How do results change when fine-tuning starts from a non-instruction-tuned base (Gemma 3 1B base) versus the instruction-tuned variant used here?
- Task coverage: Do findings persist across broader task families (summarization, translation, coding, dialog safety variants, retrieval-augmented tasks) beyond the four studied datasets?
- Fine-tuning method dependence: How do representational drift and collateral feature disruptions compare under parameter-efficient methods (LoRA/adapters), partial-layer fine-tuning, frozen-embedding or frozen-output heads, and different optimization schedules (warmup, cosine decay, AdamW)?
- RLHF and post-SFT alignment: Does the early-layer-heavy safety rewrite persist or change when subsequent alignment stages (RLHF, DPO, RLAIF) are applied after SFT?
- Dataset size and composition: How sensitive are drift patterns to training-set size, diversity, class balance, and lexical makeup (e.g., refusal phrasing density vs. harmful content density), especially given the small N (2000 train / 100 test; ToolCalling 50/13)?
- Random seed robustness: Are the layer-wise drift profiles and collateral disruptions stable across multiple random seeds, shuffles, and initializations (beyond seed s=42)?
- Performance linkage: What is the quantitative relationship between latent divergence (SAE cos-sim drop, flip rates) and downstream behavioral metrics (accuracy, refusal rate, formatting fidelity, calibration)?
- Causal validation: Do targeted interventions that activate/deactivate SAE-identified features during inference causally change model outputs in the expected directions (e.g., increase refusals, alter structure), confirming that flips reflect behaviorally relevant units?
- Output-level probing: When PCs are injected into hidden states, do the model’s generated outputs measurably change along the predicted semantic axes, not just SAE latents?
- Measurement artifact vs. genuine drift: To what extent is latent divergence driven by SAE dictionary mismatch post-SFT? Train SAEs on post-SFT checkpoints and compare reconstruction error, feature alignment, and transferability to base dictionaries.
- Reconstruction fidelity tracking: How do SAE reconstruction MSE and sparsity profiles change pre/post SFT across layers, and do these correlate with latent cosine drops?
- Alternative representational metrics: Do results replicate using complementary measures (CKA/CCA, principal angles, mutual information, RSA) rather than cosine similarity alone?
- Perturbation sensitivity: Are flip rates robust to the perturbation magnitude choice (ε_i = σ_i/√N)? Map flip rates vs. ε and test nonlinearity/saturation to ensure JumpReLU thresholds aren’t inducing artifacts.
- Thresholding and SAE variants: How do results vary with different SAE sparsity regimes (L0/L1 settings), activation functions (e.g., ReLU, GELU), training objectives, and transcoders?
- Layer coverage and streams: Extend beyond three probe layers and residual stream only; analyze attention outputs, MLP blocks, embeddings, and logits to map a complete layer/component-wise drift profile.
- Component attribution: Decompose drift PCs into attention-head and MLP-contribution analyses to identify which architectural components implement the observed subspace rewrites.
- Weight-space geometry: What is the rank and structure of parameter updates per layer (e.g., low-rank adapters vs. distributed updates), and how does this relate to activation-drift rank and SAE flips?
- Sign and directionality: Quantify amplification vs. suppression (positive vs. negative alignment) per semantic cluster to determine whether SFT consistently upregulates or downregulates specific features.
- Code-feature disruption mechanism: Why are Code-related features disrupted across non-code tasks? Test whether code-like structural scaffolds (brackets, separators, indentation-like patterns) are reused by the model; run controlled ablations that remove/add such tokens during SFT.
- Collateral damage quantification: Systematically measure trade-offs (capability retention/forgetting) caused by collateral feature flips across a broad evaluation suite outside the SFT objective.
- Time dynamics: Track SVD-based drift structure and flip distributions across the full training trajectory (beyond every 400 samples and final snapshot) to test for saturation, reversals, or phase transitions.
- Token-level heterogeneity: Which token categories (function words, punctuation, special tokens, language-specific scripts) exhibit the highest flip rates, and do these patterns explain early vs. late layer differences?
- Annotation validity: Validate the seven-cluster taxonomy and the GPT-5.4-mini LLM-as-a-judge labels via human expert annotation, inter-annotator agreement, and cross-model consistency; assess misclassification rates and ambiguity.
- Safety generality: Replicate safety findings with multiple safety datasets (e.g., AdvBench, Anthropic HH, red-teaming corpora) and different refusal styles to test whether early-layer concentration is dataset-specific or general.
- Width dependence beyond 262k: Explore even larger or hierarchical dictionaries and structured sparsity to determine whether increased disentanglement further reduces Collateral flips or reveals finer-grained safety/structure features.
- Cross-family SAE transfer: Train SAEs on different model families and test transfer across base/chat/SFT checkpoints to assess whether observed fragility and drift are model- or training-regime-specific.
- Cross-task interference: Study sequential/multi-task SFT to measure how drift directions for one task interfere with or align to those of another, quantifying additive vs. antagonistic subspace updates.
Practical Applications
Immediate Applications
Below is a concise set of deployable use cases that translate the paper’s methods and findings into concrete workflows and tools.
- SAE-driven SFT monitoring and gating — Sectors: software/AI, MLOps
- What: Integrate pretrained SAEs into training loops to track latent cosine similarity and per-layer feature flip rates during fine-tuning. Trigger alerts or early stopping when collateral disruption (e.g., Code or Multilingual features) exceeds thresholds, even if raw activation cos-sim remains high.
- Tools/workflows: Add “latent drift dashboards” to Hugging Face/Lightning training; compute SAE latent cos-sim by layer; SVD over ΔH to surface dominant update directions; flip-rate reports via Neuronpedia-annotated clusters.
- Assumptions/dependencies: High-quality SAEs for the target model and layers; access to activations during training; compute overhead for periodic SAE forward passes.
- Layer-aware adapter placement and rank selection — Sectors: software/AI
- What: Use the paper’s layer-wise drift profile (low-rank early, higher-rank late) to place LoRA/adapters and set per-layer ranks: smaller ranks in early layers for broad contextual shifts; higher ranks in deeper layers for distributed semantics.
- Tools/workflows: Automated “rank by drift” script that reads SVD variance ratios by layer to set LoRA ranks; PEFT integration.
- Assumptions/dependencies: Consistency of drift structure across model families; availability of per-layer SAE probes or an initial audit run.
- Safety SFT with early-layer focus — Sectors: safety, policy, software/AI
- What: For safety alignment, freeze (or lightly adapt) deeper layers and concentrate updates in earlier layers, matching the paper’s finding that safety rewrites occur shallowly. Reduces collateral damage and compute.
- Tools/workflows: Fine-tuning recipe templates for “early-layer-only” safety passes; per-layer unfreeze schedules validated by latent drift checks.
- Assumptions/dependencies: Observed early-layer concentration for safety generalizes beyond Gemma 3 1B; capability retention tests still required.
- Collateral damage regression tests for fine-tuning — Sectors: software/AI, enterprise IT
- What: Add latent-space regression tests targeting unrelated clusters (e.g., Code, Multilingual) when fine-tuning on non-code tasks. Fail the build if drift exceeds budgets to protect existing products (code generation, i18n).
- Tools/workflows: Per-release “feature preservation budget” reports; automated SAE+SVD test harness on held-out probes.
- Assumptions/dependencies: Adequate feature coverage in SAEs and Neuronpedia labels; organizational agreement on drift budgets.
- Dataset curation guided by latent drift — Sectors: software/AI, education
- What: Identify which semantic clusters are unintentionally perturbed and adjust datasets (add counterexamples, balancing prompts, or constraints) to mitigate disruption.
- Tools/workflows: “Drift-to-dataset” loop that surfaces clusters with high flip rates and proposes targeted augmentation or contrastive examples.
- Assumptions/dependencies: Curation impacts are measurable within available compute/time; SAEs capture the relevant subspaces.
- Model change auditing for compliance and trust — Sectors: policy/regulation, enterprise GRC
- What: Produce post-SFT interpretability dossiers showing per-layer drift, dominant PCs, and affected semantic clusters (e.g., Safety, Persona), complementing model cards and audit trails.
- Tools/workflows: Compliance-ready PDFs/JSON reports with latent cos-sim trajectories, outlier PC descriptions, and feature flip summaries.
- Assumptions/dependencies: Regulators/customers accept latent-feature-based evidence; stable taxonomies and annotations.
- Capability “hotspot” triage during incidents — Sectors: platform operations, security
- What: When regressions (e.g., tool-calling failures) appear after a fine-tune, quickly locate the responsible layers/directions by reproducing the pipeline on the affected slices.
- Tools/workflows: On-call playbooks with preconfigured SAE probes and SVD diffing; diffs tied to product telemetry.
- Assumptions/dependencies: Access to affected model checkpoints and representative prompts; SAEs that still transfer post-tune.
- Continuous multi-tenant fine-tune management — Sectors: SaaS/LLM platforms
- What: For fleets serving many customer fine-tunes, enforce per-customer “drift budgets” and block updates that risk cross-tenant capability degradation.
- Tools/workflows: CI/CD gate with latent drift checks, layer-specific quotas, and rollback hooks.
- Assumptions/dependencies: Operational maturity to maintain many SAE profiles; contractual alignment around quality guardrails.
- Research baselining across tasks and layers — Sectors: academia
- What: Adopt the pipeline to standardize comparisons of SFT effects by task and layer; publish reproducible layer-wise drift profiles and PC-feature alignments.
- Tools/workflows: Jupyter/Colab templates bundled with the public repo; cross-lab benchmark datasets for ΔH SVD analyses.
- Assumptions/dependencies: Availability of SAEs for target models; shared taxonomies for labeling.
- Red-team validation for jailbreak defenses — Sectors: security, safety
- What: Use the early-layer safety signature to verify that safety interventions changed the intended subspaces and were not bypassed by later-layer edits.
- Tools/workflows: Before/after reports for defense deployments; flag anomalous patterns (e.g., late-layer-only changes).
- Assumptions/dependencies: Attack/defense datasets are representative; SAE coverage of early layers is strong.
Long-Term Applications
These applications require further research, scaling, or productization, but are natural extensions of the paper’s innovations.
- Latent-regularized “surgical” fine-tuning — Sectors: software/AI, safety
- What: Constrain gradients to avoid known collateral features (e.g., Code) by penalizing projection onto specified SAE directions; or encourage projection to target clusters.
- Tools/products: Optimizers with SAE-based projections/regularizers; “no-go” feature lists; per-layer constraint schedulers.
- Dependencies/assumptions: Robust SAE stability under training; efficient differentiable probes; validated trade-offs between alignment and performance.
- Feature-vector “capability patches” — Sectors: software/AI, model marketplaces
- What: Package top principal drift directions as small edit vectors that can be added to base models to toggle behaviors (e.g., stricter refusal tone) without full fine-tuning.
- Tools/products: “Behavior vector” libraries; patch managers; safety/persona vector catalogs.
- Dependencies/assumptions: Linear editability holds across models/tasks; safety and legal frameworks for distributing edits.
- Dynamic controller for multi-objective training — Sectors: MLOps, enterprise AI
- What: A training controller that monitors latent drift in real time and adaptively changes data sampling, loss weights, or layer freezes to keep cluster-specific drift within budgets.
- Tools/products: “Drift-aware” schedulers integrated with trainers; dashboards with alerts and automated remediation.
- Dependencies/assumptions: Reliable online SAE inference at scale; clear KPIs linking drift budgets to business outcomes.
- Standardized representational audit frameworks — Sectors: policy/regulation, industry consortia
- What: Define norms for “latent drift budgets,” layer-wise reports, and cluster-level impact statements to meet transparency obligations (e.g., AI Act/NIST RMF).
- Tools/products: Audit specifications, test suites, and third-party certification services.
- Dependencies/assumptions: Policy acceptance of latent interpretability evidence; cross-vendor SAE compatibility.
- Security/backdoor screening via latent drift fingerprints — Sectors: security
- What: Detect anomalous early-layer or cluster-specific flip patterns indicative of poisoning/backdoors; establish baseline fingerprints per task.
- Tools/products: “Drift-forensics” scanners; CI policies for data/model ingestion.
- Dependencies/assumptions: Distinctive signatures for attacks vs. benign SFT; curated reference corpora.
- Curriculum and sequencing design for continual learning — Sectors: education, enterprise AI
- What: Plan the order of fine-tunes to minimize interference by aligning task types with layers (e.g., early safety, later reasoning/tooling), informed by measured rank structure.
- Tools/products: Curriculum planners with drift simulators; what-if tools for layer/task scheduling.
- Dependencies/assumptions: Generalization of layer-task patterns to larger, different architectures.
- Cross-model, cross-scale SAE ecosystems — Sectors: open-source AI, vendor platforms
- What: Train and distribute high-resolution SAEs for major model families and scales, enabling broad deployment of latent monitoring and editing.
- Tools/products: “SAE hubs” with versioned dictionaries; APIs; coverage badges.
- Dependencies/assumptions: Funding and community effort for high-quality SAEs; license compatibility.
- RLHF and reward modeling in latent space — Sectors: software/AI, safety
- What: Augment reward models with penalties/bonuses based on activation of Safety/Persona/Structure features to guide behaviors more directly.
- Tools/products: Hybrid token+latent reward functions; latent-feature critics.
- Dependencies/assumptions: Stable mapping from latent features to human preferences; efficiency of latent features at scale.
- Hardware/runtime co-design for layer-targeted updates — Sectors: hardware, systems
- What: Optimize inference/training runtimes to emphasize early-layer updates for safety or allocate compute to deeper layers for reasoning, matching observed drift patterns.
- Tools/products: Layer-aware compilation, caching, and PEFT offloading strategies.
- Dependencies/assumptions: Significant performance gains outweigh complexity; consistency across models.
- User-facing capability transparency — Sectors: consumer apps, enterprise UX
- What: Generate concise “capability change summaries” after updates (e.g., stronger refusals, unchanged code-gen) to build user trust.
- Tools/products: Automated change logs based on cluster-level drift; UI badges.
- Dependencies/assumptions: Users value interpretability signals; messaging avoids revealing sensitive internals.
- Multi-lingual and code integrity guardians — Sectors: localization, developer tools
- What: Persistent policies that ensure non-target clusters (e.g., Multilingual, Code) remain stable across successive fine-tunes; enforce via latent constraints and tests.
- Tools/products: “Integrity guardians” that run during CI/CD; cross-release stability metrics.
- Dependencies/assumptions: Strong correlation between latent stability and downstream task performance.
- Open-weight distribution via latent controls — Sectors: open-source AI, IP/licensing
- What: Share base weights plus controlled latent edit vectors instead of full fine-tuned weights, enabling safer capability distribution.
- Tools/products: Edit-vector repositories; policy wrappers restricting certain latent edits.
- Dependencies/assumptions: Community norms and governance for edit distribution; effectiveness across tasks.
Notes on Feasibility and Dependencies
- Generalization: Results were shown on Gemma 3 1B IT using GemmaScope 2 SAEs; replication on other architectures/scales is advised before standardizing workflows.
- SAE quality and coverage: High-resolution SAEs (e.g., 262k width) improved disentanglement in deeper layers; limited coverage can obscure effects.
- Access constraints: Many applications require access to internal activations (not feasible for closed, API-only models without vendor support).
- Compute overhead: Periodic SAE projections and SVD analyses add cost; sampling strategies and checkpoint cadence can mitigate this.
- Annotation dependence: Taxonomy-based judgments depend on feature descriptions (Neuronpedia) and LLM-as-a-judge stability; auditing for consistency is recommended.
Glossary
Activation geometry: The spatial arrangement of activation vectors in the model’s hidden layers, which is largely preserved during Supervised Fine-Tuning (SFT). "A naive analysis of the activation space suggests that SFT barely perturbs the model's internal geometry."
Cosine similarity: A metric used to determine how similar two vectors are, based on the cosine of the angle between them. "The raw activation cosine similarity between the base and fine-tuned models remains high (> 0.96)."
Latent: Underlying hidden variables or features that represent the internal state or properties of a model. "The underlying sparse latents diverge significantly."
Principal component: A dominant direction or aspect in a set of data, identified using mathematical techniques like Singular Value Decomposition (SVD). "We subsequently compute the cosine similarity between the dominant PCs of ΔH and the individual feature vectors..."
Representational drift: The change or divergence in the internal representations of a model over time or as it is fine-tuned. "We use the resulting sparse latents as a proxy to quantify representational drift."
Sparse Autoencoder (SAE): A type of neural network designed to learn efficient representations (codings) of data, using a sparse structure. "Pretrained SAEs reconstruct dense activation vectors as sparse linear combinations of interpretable feature directions..."
Superposition: A phenomenon in neural networks where neurons represent multiple, independent features within the same vector space, leading to mixed activations. "A hypothesised cause of polysemanticity is superposition, whereby models encode more features than the dimensionality of the layer permits..."
Activation difference matrix: A matrix capturing the differences in model activations pre- and post-training, used for analyzing representational changes. "We isolate the representational shift by computing the activation difference matrix, ΔH = H_{SFT} - H_{base}..."
Interceptor flip rate: A metric quantifying how often specific features switch between active and inactive states during interventions in a model's activation space. "From which we compute the average flip rate to quantify the disruption across different semantic clusters."
Frozen semantic basis: A static reference frame of semantic features, typically used to analyze and track changes in model behavior without being influenced by ongoing training. "Since these SAEs were trained prior to fine-tuning and therefore are strictly invariant to it, they enable post-hoc attribution of representational change..."
SFT: An abbreviation commonly used for Supervised Fine-Tuning, a method to adapt pre-trained LLMs for specific tasks. "Supervised Fine-Tuning (SFT) is the most prevalent immediate post-training intervention used to adapt pretrained LLMs..."
GemmaScope 2: An interpretability suite combining tools like sparse autoencoders and transcoders, designed for analyzing model behaviors and representations. "GemmaScope 2 combines sparse autoencoders and transcoders to allow researchers to examine models' internal representations..."
Safety alignment: A process in model training intended to ensure the model behaves safely, particularly in refusing harmful requests and providing legal disclaimers. "We also discover that safety alignment disproportionately affects earlier transformer layers..."
Singular Value Decomposition (SVD): A mathematical technique used to break down matrices into their component parts, revealing underlying structures. "We then apply Singular Value Decomposition (SVD) to ΔH to extract the principal components (PCs)..."
Sparse linear combination: A method of representing data as a linear mix of sparse vectors, typically used for more interpretable and efficient coding. "Pretrained SAEs reconstruct dense activation vectors as sparse linear combinations..."
Mechanistic interpretability: The study of how internal mechanisms of models work, often aimed at understanding and explaining model behaviors through technical analysis. "A novel mechanistic analysis pipeline that leverages pretrained SAEs..."
Neuronpedia API: An application programming interface used to obtain human-readable interpretations and descriptions of machine learning model features. "We programmatically retrieve precomputed, human-interpretable feature descriptions associated with the most disrupted features via the Neuronpedia API..."
Gemma 3 1B IT: A specific instruction-tuned transformer model variant used in experimentation, having specific dimensions and configurations. "We use Gemma 3 1B IT (google/gemma-3-1b-it), an instruction-tuned transformer with model dimension..."
JumpReLU: A specific activation function used in sparse autoencoders, characterized by its learned threshold for activation. "The SAEs employ a JumpReLU activation function with learned threshold τ..."
Functional taxonomy: A classification system used to categorize different semantic functions or features within a model, often utilized for interpretability. "Through qualitative review of these explanations, we derive a discrete semantic taxonomy..."
LLM-as-a-Judge: A framework using LLMs as automated classifiers or evaluators, often for tasks requiring linguistic judgment. "We then implement an automated LLM-as-a-judge annotation pipeline utilizing GPT-5.4-mini..."
Polysemanticity: A characteristic of neural network neurons where they activate in response to multiple, semantically different contexts. "A central challenge in mechanistic interpretability is the phenomenon of polysemanticity..."
Conditional language modeling objective: The goal or task of predicting occurrences of the language given particular conditions or contexts. "SFT optimises the conditional language modelling objective..."
Chain-of-thought reasoning: A logical process mimicking human thinking patterns, involving step-by-step deduction or inference. "Natural Language Inference & MultiNLI @ Logical Reasoning @ Chain-of-thought reasoning..."
Collections
Sign up for free to add this paper to one or more collections.