PRISM: Recovering Instruction Sets from Language Model Activations

Abstract: As LLMs are deployed as agents, reliable monitoring requires knowing not only what they output, but which instructions are steering their behavior. This is difficult when models infer unintended subgoals, follow contextual cues, or are influenced by prompt injections and hidden objectives. While activation-to-language methods suggest that hidden states can reveal natural-language information, existing approaches are not designed to recover the full set of simultaneous instructions, constraints, prohibitions, and subgoals active in agentic settings. We formalize this problem as instruction set retrieval and introduce PRISM, an activation-conditioned interpreter that decodes hidden states from a frozen target model into a faithful bullet list of active instructions. Unlike prior activation-to-language methods, PRISM is trained to recover instruction sets directly, using judge-guided GRPO to reward covered instructions and penalize unsupported ones. Across benign, constrained, prompt-injection, and hidden-objective settings, PRISM outperforms activation-to-language baselines, especially on security-relevant objectives.

• The paper introduces ISR and PRISM, a novel method that accurately retrieves the complete instruction set governing LLM behavior from internal activations.
• It employs a two-stage training process combining supervised pretraining with judge-guided reinforcement learning to ensure high recall and low hallucination.
• Empirical results demonstrate that PRISM outperforms baselines across benign, adversarial, and covert settings, marking a significant advance in LLM interpretability.

Decoding LLM Internals: Instruction Set Retrieval with PRISM

Introduction and Motivation

The deployment of LLMs as agents in complex environments has created an imperative to monitor and understand not just their outputs, but the objectives, constraints, and instructions actively steering their behavior. Realistic agentic deployments often involve simultaneous, potentially conflicting instructions, some arising from explicit prompts, others inferred or adversarially injected through surface or hidden channels. Prior interpretability approaches—primarily probing, latent-space classifiers, and textual descriptions derived from activations—fall short in exhaustively recovering the full set of instructions currently controlling the model, especially under distribution shifts, prompt injections, or covert objectives. The paper "PRISM: Recovering Instruction Sets from LLM Activations" (2606.09563) addresses this problem by formalizing instruction set retrieval (ISR) and introducing PRISM, an activation-conditioned interpreter trained for faithful extraction of all operative instructions from an LLM’s activations.

Instruction Set Retrieval: Formalization and Significance

ISR is instantiated as recovering, from internal activations, the complete set of instructions, constraints, prohibitions, and subgoals governing model behavior at a given time. This is distinct from traditional probing (predicting scalars or class labels for fixed properties) or activation-to-language methods producing single high-level summaries. The ISR setting is inherently open-domain, combinatorial, and adversarially sensitive; it requires precise set-level recall and minimization of unsupported additions (hallucinations).

The process involves a frozen target model $\mathcal{M}$ generating a response, from which a snapshot of activations $H_\ell$ is extracted. An interpreter model $\phi$ then decodes these activations into a natural-language, bullet-list $\hat{\mathcal{I}}$ representing the model's active instruction set. Semantic matching between predicted and ground-truth instructions is operationalized via a calibrated LLM judge, which rigorously evaluates both coverage (recovery rate) and hallucination (unsupported claims).

PRISM: Methodology and Architecture

PRISM is an activation-conditioned interpreter composed of a learned activation projection and LoRA adapters atop the frozen base weights of the target LLM. The workflow is as follows: for each input, a window of residual-stream activations from a fixed transformer layer is projected into the embedding space, forming a soft token prefix for the reporter, which then decodes the predicted instruction set.

Figure 1: Activation-conditioned instruction set retrieval. A frozen model $\mathcal{M}$ produces activations, which are projected and fed via a soft prefix to the interpreter, yielding a decoded instruction list.

PRISM is trained in two stages:

1. Supervised Pretraining: Teacher-forced sequence-to-sequence training on activation-instruction pairs, minimizing cross-entropy over next-token logits given the activation prefix; this establishes a foundational mapping but does not guarantee set-level faithfulness or control for paraphrasing and overlapping constraints.
2. Judge-Guided Reinforcement Learning: Optimization follows a Group Relative Policy Optimization (GRPO) objective. For each input, multiple candidate reports are sampled, semantically scored by a calibrated LLM judge for both recall and hallucination, and the policy is updated to maximize coverage and penalize unsupported additions, with explicit KL constraint against supervised outputs.

This design ensures deployment efficiency (minimal additional parameters) and robustness against overfitting to surface cues or reliance on model self-reports.

Training Data and Evaluation Paradigms

The paper constructs a training corpus by aggregating UltraChat, IF-Multi-Constraints, and IFEval datasets, each contributing diverse instruction distributions (open-domain, constraint-stacked, and verifiable formatting-prompts). The evaluation protocol is strictly out-of-distribution, utilizing multiple targeted, security-relevant held-out suites: Benign (BN), Behavioral Constraints (BC), Adversarial Prompts (AP; surface task + prompt injection), and Hidden Objectives (HO; covert goals and explicit deception scaffolding).

Figure 2: Overview of instruction-recovery data resources. Training leverages open/constraint-rich corpora; held-out evaluation balances benign, constrained, adversarial, and covert-objective cases.

Ground-truth instruction lists are automatically generated from prompts via system queries and filtered and audited for correctness. Prediction evaluation is performed with a judge LLM calibrated against human-labeled annotation (agreement $\kappa>0.8$ for instruction recovery, $\kappa>0.7$ for hallucination detection), providing per-instruction semantic fidelity metrics independent of phrasing.

Empirical Results

PRISM achieves the highest instruction-recovery rates and lowest hallucination rates across benign, constraint-heavy, adversary-injected, and covert-objective settings:

• Aggregate Reward (R): 0.736 for PRISM, outperforming the strongest activation-to-language baseline (Activation Oracles, R=0.532).
• Coverage (Cvg): 0.745 (PRISM) vs. 0.536 (best baseline).
• Hallucination: PRISM achieves a mean hallucination rate of 0.014; lower than all other baselines—critical for auditor trustworthiness.
• Security Sensitivity: On security-relevant splits, PRISM achieves adversarial-instruction recovery of 65.2% (AP), 82.8% (HO), both substantially higher than the next best model.

Judge-guided RL (relative to an SFT-only version) yields nontrivial improvements in both reward and coverage, especially on distributions with synthetic covert and adversarial instructions.

Qualitative analysis demonstrates that PRISM alone can disentangle multiplexed task directives, injected instructions, hidden objectives, and associated prohibitions, correctly surfacing both overt and latent constraints without conflation or hallucination—a property baselines lack.

Practical and Theoretical Implications

Practical Impact: PRISM enables robust activation-based monitoring and auditing of LLM agents, surfacing the full operative instruction set including covert or injected subgoals even when these are absent from visible input, or when the model's response superficially aligns with only part of its directive set. This is essential for compliance, security, and safety auditing in high-stakes deployments, where reliance on prompt text or model outputs is insufficient. PRISM’s low-overhead architecture further facilitates feasible integration into existing LLM deployments with minimal inference cost.

Theoretical Impact: The results demonstrate that transformer activations encode structured, compositionally recoverable instruction sets far beyond single-task or attribute-classifier paradigms, reinforcing the need for set-level semantic training and RL-augmented evaluation. The success of the judge-guided GRPO approach suggests that well-calibrated semantic feedback is superior to string-level or token-level objectives in open-ended interpretability tasks. Furthermore, the clear performance gradients as a function of training regime and reward design provide guidance for designing future activation-to-language systems for high-recall semantic recovery.

Relation to Broader Activation-Based Interpretability

PRISM advances the activation-to-language paradigm by targeting exhaustive set-level extraction, rather than question-answering (LatentQA), concept bottlenecking (Predictive Concept Decoders), or scalar probing. It addresses critical failure modes of previous activation-to-language systems, such as inadvertent collapse of multi-instruction prompts or failure to separate overlapping subgoals and constraints. Additionally, it subsumes the goals of adversarial monitoring and hidden-objective auditing, providing unified mechanisms for both transparency and red-team operations.

Limitations and Future Directions

PRISM’s effectiveness is currently established on single-model, single-layer, fixed-window scenarios. Generalization to larger, multi-modal, or real-time agentic deployments, as well as longer contexts or temporally distributed instructions, remains an open area. Cross-family or cross-layer transferability, as well as application to instructable models with more diverse behaviors and oversight incentives, is a critical next step. The challenge of scaling robust annotation and judge calibration persists for even more open-ended or ambiguous instruction landscapes.

Possible extensions include:

• Application to multimodal transformer architectures and tool-augmented agent traces
• Investigation of activation segmentation and hierarchical instruction-set decompositions
• Integration with differential privacy and secure auditing protocols for safe deployment in user-facing or regulated environments
• Exploration of joint end-to-end ISR+control pipelines for instruction-set-aware safe control and behavioral correction

Conclusion

PRISM establishes a robust, interpretable methodology for retrieving the complete set of operative instructions from transformer activations. By encoding ISR as a set-level, semantically calibrated RL objective, it advances the transparency and auditability of black-box LLM agents, demonstrates superior performance to prior activation-to-language baselines—particularly on security-critical and hidden-objective scenarios—and lays the groundwork for systematic white-box behavioral auditing in future generative AI systems.