Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal

Abstract: Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.

• The paper introduces an interpretability-based framework to audit and reshape concept-level reward signals during LLM post-training.
• It employs sparse autoencoders and interventions such as reward shaping and activation steering to reduce undesired model behaviors.
• Empirical results show effective modulation of target traits while maintaining performance and safety across various model families.

Anatomy of Post-Training: Interpretability Protocols for Data-Centric Post-Training in LLMs

Introduction and Motivation

Post-training—fine-tuning base models using supervised or preference data and (often) reinforcement learning from human feedback (RLHF)—is the principal locus for shaping the operational behavior of contemporary LLMs. Despite its centrality, post-training protocols remain largely reliant on scalar proxy reward signals which compress multitudinous behavioral desiderata into a black-box scalar target. This reward abstraction occludes understanding of what specific latent behaviors or concept-level signals are reinforced or suppressed during optimization, often resulting in reward hacking, spurious correlation learning, and emergence of tasks or styles (e.g., excessive stylization, sycophancy) extraneous to the practitioner's intention.

The paper "Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal" (2606.12360) proposes a unified, data-centric, interpretability-based pipeline for auditing what concepts preference datasets induce in models during post-training. The work leverages sparse autoencoder (SAE) features and other modern interpretability tools to surface, audit, and intervene upon concept-level signals, enabling practitioners to proactively select which behaviors the model should or should not be allowed to learn from their data and driving toward more transparent, controllable post-training.

Figure 1: The pipeline begins with preference datasets, identifies maximally distinguishing concepts, enables interface-driven human audit, and supports intervention to alter or suppress learning signals for those concepts.

Framework: Decomposing and Shaping the Learning Signal

The core formalism develops the observation that when optimizing a reward function under a relative-entropy (KL) constraint, as in RLHF or DPO-style fine-tuning, the optimal policy is an exponential tilt of the base model's conditional distribution with respect to the reward. When multiple concept-specific classifiers exist, the reward can (under conditional independence) be viewed as an additive or product-of-concepts term in the log-domain, and shaping the learning signal is mathematically equated to modulating, suppressing, or amplifying specific concept-level reward coefficients.

Operational Interventions: The framework unifies several families of interventions under a single principle: explaining away concept-level reward signals. The principal variants—including their realization points—are as follows:

• Data Filtering: Exclude or downweight data units whose preference label is fully explained by a target concept.
• Inoculation Prompting: Condition on a prompt to absorb the effect of a concept, so it is attributed to context and not globally learned.
• Activation Steering: Directly shift internal activations along a target concept direction during optimization.
• Reward Shaping: Directly subtract (or amplify) a concept classifier's score from the reward, effectively removing (or increasing) its impact on the learning signal.

  Figure 2: Illustration of four intervention modalities and their corresponding intervention points—data, representation, input, and loss/reward.

The formalism highlights the inadequacy of a flat, unstructured reward decomposition assumption: if concept-level signals are not independent, interventions may spur off-target effects or fail to localize changes.

Empirical Validation: Explaining-Away Interventions and Synthetic Poisoning

To validate the framework, the authors introduce controlled data-poisoning experiments, synthetically injecting traits such as "Goblin Weave," "Cheerfulness," "Conflict Avoidance," etc., into SFT and DPO data for Llama-3.1-8B, and then applying the various interventions to suppress or remove the induced behavior.

Figure 3: Induced traits can be reliably attenuated via interventions; data filtering fares poorly for diffuse traits, while inoculation prompting, activation steering, and reward shaping exhibit robust suppression.

Key findings include:

• All interventions significantly reduce target trait expression compared to poisoned DPO checkpoints.
• Token-level data filtering is notably less effective for broad/distributed concepts, where the signal cannot be localized to a small subset of tokens or samples.
• Activation steering and reward shaping deliver consistent behavioral modulation while preserving capability metrics.

Auditing Datasets: Hypothesis Generation via Interpretable Concept Bank

Concept Identification: The authors instantiate two complementary, SAE-based interfaces for dataset auditing. The prompt-conditioned pipeline clusters prompts and response SAE features, searching for pairs where the preference signal induces significant conditional activation changes—discovering conditional behaviors (e.g., only manifesting in specific prompt types).

Figure 4: SAE-feature viewer interface for exploring conditional training signal structure per prompt region.

The feature-conditioned pipeline conducts the dual analysis—starting from a response concept cluster, auditing dataset regions (clusters of examples) that differentially reinforce or suppress the concept.

Figure 5: Feature-conditioned interface surfaces data clusters where specific response concepts are discriminatorily rewarded or penalized.

Predictive Validation: The framework is quantitatively validated by correlating predicted concept changes (as surfaced by the pipelines) with actual SAE feature distribution shifts before and after DPO training. The feature-conditioned pipeline exhibits very high predictive accuracy ($R^2=0.9$), while the prompt-conditioned counterpart achieves moderate but meaningful predictive correlation ($R^2=0.58$).

Figure 6: Empirical vs. predicted concept shift for feature-conditioned pipeline, with safety-refusal and unsafe-query rollouts as downstream outcome validation.

Figure 7: Similar, but weaker, predictive correlation for prompt-conditioned pipeline, together with qualitative rollout evidence for uncovered behavioral shifts.

Auditing openly-distributed preference datasets (e.g., Dolci) uncovers amplification of egregious behaviors by DPO, including reduced refusal robustness to harmful prompts, sycophancy, benchmark meta-knowledge, and context-specific undesirable generations.

Case Studies: Realistic Interventions and Behavioral Modulation

Over-Stylization and Correlation Effects: Experiments systematically evaluate the ability of token-level filtering, reward shaping, and activation steering to modulate response stylization (e.g., bold text, emoji). Reward shaping and token filtering most consistently reduce stylization rates to baseline levels while minimally degrading accuracy. Crucially, interventions targeting one style dimension often provoked suppression across correlated style attributes, confirming the necessity for structure-aware concept decomposition.

Figure 8: Various instantiations of intervention protocols for dampening stylization rates in generation while tracking impact on accuracy.

Figure 9: Off-target effects quantified: many interventions dampen both target and correlated style rates, whereas narrowly defined SAE-based reward shaping achieves superior locality.

Robust and Amplified Safeguarding: Interventions are assessed with respect to their ability to recover or amplify the refusal rate for harmful queries (with minimal increase in benign refusals) across several model families up to Llama-3-1-70B. Reward shaping—owing to its direct/effective concept signal manipulation—reliably boosts harmful query refusal while preserving standard post-training accuracy improvements.

Figure 10: Cross-family safety-accuracy trade-off via various interventions. Reward shaping delivers consistent improvements along the safety-utility frontier.

Prompt-Conditioned Moderation: Four contextually entangled, prompt-conditioned behaviors—physics sycophancy, questionable fanfiction, evaluation meta-knowledge, and hallucinated links for sensitive topics—are audited and targeted for suppression. Only hallucination suppression yields robust effect size; the rest demonstrate the difficulty of reliably disentangling and modulating context-intertwined behaviors.

Figure 11: Best interventions for each prompt-conditioned behavior, showing varying levels of success; partial but only reliable suppression for generative hallucinated links to sensitive resources.

Discussion and Implications

This work demonstrates that interpretability protocols, when combined with structured hypothesis generation and unified intervention mechanisms, supply both diagnostic and interventional leverage over LLM post-training. They enable surface-level auditing of learning signals and downstream modulation of specific learned concepts. However, the limitations of flat concept independence—manifesting in off-target behavioral modification and limited control over highly entangled or rare behaviors—highlight the need for hierarchical or structured reward decomposition and context-aware intervention.

Future directions include:

• Hierarchical or multi-resolution concept modeling—allowing for interventions at different abstraction levels and controlling parent/child concept relationships.
• Structure-aware reward shaping, leveraging explicit modeling of conditional dependencies among concepts.
• Increased reliance on interactive, interpretable model auditing interfaces for practitioners during RLHF and DPO pipeline development.

These findings are directly salient for improving LLM safety, decompositional alignment, and preemptive control over model behaviors in practical deployment settings. More generally, they point towards a paradigm in which the reward landscape is not simply optimized, but actively sculpted by human-in-the-loop interpretability and targeted intervention.

Conclusion

"Anatomy of Post-Training" (2606.12360) presents a rigorous, unified interpretability-driven approach to proactively characterizing and shaping the learning signal in LLM post-training. The pipeline unifies disparate intervention protocols via a common theoretical lens, equips practitioners with actionable dataset auditing tools, and delivers empirical evidence for the feasibility and limit cases of such interventions in both synthetic and realistic settings. The results substantiate the core claim that interpretability should not be an afterthought but rather a central component guiding the optimization protocols for post-training, moving the field closer to safe, auditable, and controllable LLM deployment.