Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants (2512.15712v1)

Abstract: Interpreting the internal activations of neural networks can produce more faithful explanations of their behavior, but is difficult due to the complex structure of activation space. Existing approaches to scalable interpretability use hand-designed agents that make and test hypotheses about how internal activations relate to external behavior. We propose to instead turn this task into an end-to-end training objective, by training interpretability assistants to accurately predict model behavior from activations through a communication bottleneck. Specifically, an encoder compresses activations to a sparse list of concepts, and a decoder reads this list and answers a natural language question about the model. We show how to pretrain this assistant on large unstructured data, then finetune it to answer questions. The resulting architecture, which we call a Predictive Concept Decoder, enjoys favorable scaling properties: the auto-interp score of the bottleneck concepts improves with data, as does the performance on downstream applications. Specifically, PCDs can detect jailbreaks, secret hints, and implanted latent concepts, and are able to accurately surface latent user attributes.

• The paper introduces Predictive Concept Decoders (PCDs) that extract sparse, learned concepts from LM activations to predict model behavior.
• The encoder employs a top-k sparsity bottleneck with an auxiliary loss to ensure robust, human-auditable explanations with improved precision and recall.
• Downstream applications include jailbreak detection and secret hint revelation, demonstrating practical capabilities in auditing and ensuring model safety.

Authoritative Summary of "Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants"

Introduction

"Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants" (2512.15712) proposes and analyzes the Predictive Concept Decoder (PCD), an end-to-end architecture designed to support scalable, interpretable analysis of LM activations. Departing from prior approaches reliant on hand-crafted agents or architectural bottlenecks with fixed human-derived concepts, PCDs directly optimize for behavior prediction from model activations: an encoder observes internal activations, compresses them through a top-$k$ sparse bottleneck into a small set of learned “concepts,” and a decoder, given only these concepts (and a query), produces natural language answers predictive of the underlying subject model’s behavior.

Figure 1: A PCD processes a prompt leading to a potentially harmful response (e.g., via jailbreaking), compresses activations to sparse concepts, and a decoder, conditioned solely on those, predicts or explains model behavior; concepts can be independently interpreted via downstream pipelines.

This pipeline enforces general-purpose, human-auditable explanations, with concepts encouraged to cover a broad behaviorally-relevant subspace. The encoder is forced to discover “generators” of useful explanations by virtue of being blind to the question, while the decoder’s performance is directly tied to the informativeness and compositional coverage of these abstracted concepts. Downstream, these components enable several interpretability and auditing use cases, including the detection of jailbreaks, latent hint usage, and introspective reasoning over subject model behavior that is not necessarily faithfully verbalized.

Architecture and Training Regime

The PCD instantiation consists of a linear encoder with a learned dictionary, a top-$k$ sparsity mask, and a re-embedding, patched into the residual stream of a decoder LM at a specified layer. The decoder is a LoRA-adapted clone of the subject model; it receives only sparse encoded concepts and a question, and must answer a natural language prompt about some aspect of the model’s behavior. The encoder and decoder are optimized jointly during pretraining on next-token prediction over huge uncurated text data (FineWeb), with no explicit behavior labels required. Then, the encoder is frozen and the decoder is finetuned on question-answering datasets specifically targeting the subject model’s internal state (SynthSys).

Figure 2: The architecture: encoder reads activations from one subject model layer, selects top-$k$ concepts, and decoder receives only these as input, composable with arbitrary question queries at inference.

A central technical challenge arises from the dead concept problem typical in sparse autoencoders: many dictionary directions die off during long training. To counteract this, an auxiliary activation loss is introduced to revive dead concepts and promote coverage in the learned subspace. This regularization yields a highly active and interpretable dictionary, as assessed by automatic interpretability pipelines built on held-out context exemplars.

Encoder Interpretability and Scaling

The encoder’s output must be not only predictive but also interpretable and compositional. Interpretability is assessed both by precision (auto-interpretability scores: how reliably a concept’s description predicts its activation pattern) and recall (coverage over a large suite of user-defined behavioral attributes, e.g., SynthSys’s user attribute labels). PCDs with a well-tuned bottleneck strongly outperform vanilla SAEs, especially at moderate data scales, on these metrics. Moreover, both the interpretability and coverage metrics reliably improve with additional pretraining, but all KL-based objectives (including PCD and KL-SAE variants) exhibit a plateau at large scales, suggesting a need for richer objectives or improved architectures for further gains.

Figure 3: As pretraining progresses, PCD’s predictive loss drops and interpretability metrics rise; without an auxiliary loss, a substantial fraction of concepts die, reducing precision and recall.

Downstream Question-Answering: Faithful Model Auditing via Sparse Bottleneck

Once pretraining is complete, performance is assessed by freezing the encoder and finetuning the decoder for question-answering about model beliefs or latent states using the SynthSys dataset, holding out several attributes for robust generalization. Despite the information bottleneck, the PCD rapidly matches the performance of a directly supervised LatentQA baseline that reads the entire, uncompressed residual stream. Performance clearly tracks with encoder scale: at 36M tokens and above, PCD QA accuracy saturates LatentQA despite using only sparse encodings.

Figure 4: Question-answering accuracy improves with more pretraining and matches LatentQA; ablations removing the bottleneck post hoc reveal that, at sufficient data regime, the bottlenecked encoder transmits most relevant state information.

Case Studies: Jailbreaking, Secret Hint Detection, and Latent Concept Surfacing

PCDs are systematically evaluated on three central interpretability tasks that stress the limits of model self-report and standard prompting: jailbreak detection, secret hint usage, and introspection after concept vector injection.

Jailbreaking:

PCDs display strong awareness of when a model is about to perform a harmful action due to a prompt injection (jailbreak), often detecting latent intent even when direct query of the subject model fails or produces refusals. Importantly, performance improves with training scale and the number of active concepts $k$, though too large $k$ can create OOD decoder issues.

Figure 5: PCDs exhibit awareness of indirect harmful actions at rates matching or surpassing the subject model’s own responses when asked directly.

Figure 6: Jailbreak awareness scales with pretraining tokens and sparseness hyperparameters; intermediate $k$ works best.

Secret Hints:

PCDs, especially when the bottleneck is relaxed at test time, robustly reveal cases where the subject model uses hidden or unreportable hints for answering questions—cases where direct prompting almost always yields denials or confabulation.

Figure 7: PCD and LatentQA reveal the use of top-secret hint usage that cannot be elicited from the subject model via direct querying.

Introspection:

Upon injecting synthetic concept vectors (following [lindsey2025emergent]), PCD decoders are superior to prompt-based methods at verbalizing the latent concept being “thought about” by the model. The encoder, via its dictionary and automatic descriptions, surfaces highly relevant concepts in alignment with these manipulations.

Figure 8: PCDs verbalize injected concepts much more reliably than LatentQA or the raw model, especially under bottleneck; both SAE and PCD dictionaries surface semantically aligned features.

Auditing Decoder Explanations: Human-Readable Encoder Concepts

A salient feature of the PCD approach is the auditability and cross-validation of explanations: when the decoder produces a surprising rationale for (e.g.) model refusal, the corresponding encoder concepts and their automated descriptions provide an orthogonal check. In instances where the model refuses a request and the decoder attributes it to “legal liability” (rather than user safety), the most active encoder concept is often one whose dictionary definition was independently annotated as legal/compliance related.

Figure 9: In probing refusals, encoder concepts align with the decoder’s attribution, highlighting tokens and contexts associated with legal and compliance patterns.

Ablations and Scaling Trends

Additional ablations highlight that as the encoder is trained with more data, the effect of the bottleneck becomes less limiting—information efficiently traverses the sparse interface by emergence. However, all KL-based objectives exhibit eventual plateauing in interpretability metrics (Figure 10), and the effect of LoRA rank is ambiguous (Figure 11). There are slight improvements in interpretability at intermediate $k$ (Figure 12), but extending dictionary size indefinitely degrades human alignment.

Theoretical and Practical Implications

The work advances the notion of “end-to-end interpretability assistants” that rely on verifiable prediction as the scalar for scalable interpretability, with concepts in the bottleneck optimized not for human legibility per se, but for predictive utility in downstream natural language explanations.

Practically, the PCD demonstrates the feasibility of robust, scalable auditability for LMs in high-impact behaviors, including security-sensitive and safety-critical settings (e.g., jailbreak detection, sensitive attribute exposure), with cross-auditable explanations at both the feature and decision level.

Theoretically, the methodology bridges concept bottleneck models and sparse autoencoder literature with LM-driven QA auditing and opens avenues for architecture search (better encoders, richer bottlenecks), more sophisticated objectives, and the use of assistance games as a formal framework for legibility optimization. Notably, the bottleneck can be seen as an “assistance game” in which the encoder is forced to encode subspaces legible to a decoder, which ideally mirrors the demands of human auditors.

Future Directions

Key bottlenecks to further scaling and generalization include the observed plateau with KL-based objectives, the challenge of extending compositional coverage without loss of interpretability, and the design of richer encoder priors or decoders that can leverage multimodal/time-stratified dependencies. The integration with other end-to-end interpretability objectives, compositional/disentangled concept spaces, and expanded multi-layer or cross-model reading architectures, all represent promising research axes.

Conclusion

Predictive Concept Decoders (PCDs) demonstrate that joint end-to-end training of sparse concept bottlenecks and natural language decoders yields interpretable, auditably faithful explanations of LM behavior, robust to obfuscation and manipulation that defeat direct prompting or naive chain-of-thought self-reporting. The architecture’s scalability, modularity, and cross-validated explanations set a paradigm for future interpretability research, with wide-ranging implications for both theoretical understanding of neural representations and practical assurance in high-stakes LM deployments.