Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations

Abstract: LLMs are increasingly deployed in high-stakes settings, yet they frequently violate contextual privacy by disclosing private information in situations where humans would exercise discretion. This raises a fundamental question: do LLMs internally encode contextual privacy norms, and if so, why do violations persist? We present the first systematic study of contextual privacy as a structured latent representation in LLMs, grounded in contextual integrity (CI) theory. Probing multiple models, we find that the three norm-determining CI parameters (information type, recipient, and transmission principle) are encoded as linearly separable and functionally independent directions in activation space. Despite this internal structure, models still leak private information in practice, revealing a clear gap between concept representation and model behavior. To bridge this gap, we introduce CI-parametric steering, which independently intervenes along each CI dimension. This structured control reduces privacy violations more effectively and predictably than monolithic steering. Our results demonstrate that contextual privacy failures arise from misalignment between representation and behavior rather than missing awareness, and that leveraging the compositional structure of CI enables more reliable contextual privacy control, shedding light on potential improvement of contextual privacy understanding in LLMs.

• The paper demonstrates that LLMs encode contextual privacy norms as robust, multi-dimensional signals while often failing to enforce them in output, revealing a key awareness gap.
• A multi-level probing framework reveals that steering along three independent CI axes reduces leakage from 42.5% to near 0.5% on synthetic tests compared to monolithic methods.
• CI-parametric steering offers an actionable, inference-time mechanism to align LLM outputs with privacy expectations without requiring model retraining.

Probing and Steering Contextual Privacy Norms in LLMs

Motivation and Problem Formulation

The paper "Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in LLM Representations" (2604.00209) investigates whether LLMs intrinsically encode contextual privacy norms as structured representations, specifically within the framework of Contextual Integrity (CI). CI theory posits that privacy is defined by context-appropriate information flows determined by multiple parameters (information type, recipient, transmission principle, sender, subject), rather than static notions of sensitivity or exclusion.

This study addresses a core question: do LLMs represent contextual privacy norms internally, and if so, why does behavior frequently contradict these norms in high-stakes scenarios—even when functional downstream compliance is required, e.g., refusal to disclose confidential information?

Probing CI-Structured Representations in LLMs

A multi-level probing framework is introduced to separately analyze (i) the existence of linearly separable privacy signals, (ii) the translation of these signals into behavior, and (iii) the compositional alignment of these signals with the principal CI norm parameters.

Concept-Level Probing: Linear Separability and Multi-Axiality

Probing experiments on both synthetic and benchmarked datasets (e.g., CONFAIDE Tier 2) reveal that CI-encoded privacy norms manifest as robust, linearly separable signals in deep layers of LLM activation space. However, these representations are inherently multi-dimensional—unlike previously studied attributes such as honesty or truthfulness, which are often captured by a single steering vector.

Performance gains are observed only when at least three principal axes are considered; a one-dimensional PCA projection fails to capture the privacy-relevant variance and yields near-chance AUROC, while three principal axes yield robust separation.

Figure 1: PCA achieves highest AUROC for CI privacy signals at $k=3$ principal components; single-component methods underestimate signal complexity.

Behavioral-Level Probing: The Privacy Awareness Gap

Despite strong internal linear probes (often exceeding 0.90 AUROC in deeper layers), LLMs' actual generative behavior exhibits a significant privacy awareness gap: models leak private information in up to 42.5% of role-play scenarios designed to test norm compliance, with lower but still substantial leakage on community benchmarks. This demonstrates a dissociation between internal awareness and behavioral compliance—LLMs "know" CI norms in their representation space but do not adhere to them reliably in output.

CI-Parametric Probing: Subspace Selectivity

Analysis decomposes the privacy subspace along the key CI norm-determining parameters (information type, recipient, transmission principle). Subspace selectivity testing (using LDA and cross-projection) establishes that each CI parameter is encoded in a functionally independent axis, with strong diagonal dominance in cross-decoding accuracy matrices and low off-diagonal generalizability.

Figure 2: Cross-decoding matrices confirm that each privacy norm parameter occupies an independent subspace.

CI-Parametric Steering: Methodology

To exploit this structured, multi-dimensional encoding, the authors introduce CI-parametric steering. This method intervenes at inference time by perturbing the hidden-state representation along each CI axis independently (as defined by PCA or supervised probes from synthetic, parameter-isolating examples):

$$h'_l = h_l + \alpha \sum_{p \in \mathcal{P}} \mathbf{v}^{(p)}_l,$$

where $p$ indexes over the relevant norm parameters and $\alpha$ controls intervention strength. The three axes are specifically (i) information type, (ii) recipient, and (iii) transmission principle; including additional parameters (e.g., sender, data subject) dilutes and sometimes inverses the steering benefit.

This decomposition enables independent ablation of norm facets and more precise adaptation to the violation landscape of target datasets, outperforming single-direction "monolithic" baselines.

Empirical Evaluation and Analysis

Strong Performance Gains

CI-parametric steering demonstrates clear superiority to monolithic approaches (additive steering, LoRRA, representation tuning), both on the synthetic testbed and under benchmarked cross-dataset transfer.

• On synthetic data, leakage is reduced from 42.5% to 0.5% (PPI = 98.8%).
• On CONFAIDE Tier 3, additive steering increases leakage for some models (up to 47.4%), while CI-parametric steering brings leakage down to near zero for Llama-3.1 and substantial relative reductions for all architectures.
• On PrivaCI-Bench, the only method to consistently eliminate leakage is CI-parametric steering.

Robust cross-domain transfer is achieved using steering directions constructed only from synthetic probing data, indicating the parameter-space axes encode domain-invariant privacy structure.

Figure 3: Pareto frontiers demonstrate that CI-parametric steering dominates NCR–leakage tradeoffs across operating points.

Steering Stability and Utility Cost

A sweep over steering strength $\alpha$ shows that CI-parametric steering is less sensitive to hyperparameter tuning and consistently reduces leakage, whereas monolithic methods are brittle—sometimes exacerbating leakage for modest values of $\alpha$.

Figure 4: Leakage as a function of $\alpha$; monolithic steering is unstable, CI-parametric steering is robust.

However, applying three independent perturbations increases the risk of utility degradation (measurable by response coherence and output relevance). Empirical analysis indicates this trade-off can be managed by further parameter-specific tuning or adaptive strength adjustment, without loss of compositional effectiveness.

CI-Parameter Ablation

Ablation results confirm none of the individual parameters is sufficient for suppression across all distribution shifts: steering along only one CI axis often increases leakage or is ineffectual. Only simultaneous intervention on all three produces the robust behavioral change required for broad privacy compliance.

Theoretical and Practical Implications

• Theoretical: The presence of an awareness gap decouples representation learning from control, suggesting that alignment problems in LLMs are not always rooted in absent or impoverished conceptual encoding, but in failures of action-level policy translation. The decomposition of CI signals into independent axes supports the hypothesis that complex behavioral norms can be factorized in LLMs' latent geometries.
• Practical: The CI-parametric steering mechanism offers an inference-time control interface for privacy (and likely other behaviors governed by contextualizable norms) without model retraining or prompt engineering, compatible with a variety of transformer architectures. Its transferability provides a promising route for domain-robust, regulation-aligned deployment of LLM systems.

Future Directions

The findings invite several extensions:

• Evaluation under adversarial or dynamically adaptive attacks to test the durability of steering interventions against evasive tactics.
• Generalization to more ambiguous, overlapping, or evolving social contexts and expansion to legal, regulatory, or cross-ontology privacy standards.
• Integration with multi-agent or hierarchical reasoning for improved end-to-end compliance in conversational or dialogic agents.
• Exploration of steering compositionality for other structured, multi-criteria behavioral attributes.

Conclusion

This work demonstrates that contextual privacy in LLMs is encoded as a multi-dimensional, CI-aligned subspace, not as a single concept vector. Robust and predictable behavioral alignment requires decomposing and independently steering along the principal norm parameters of CI. Such structured interventions realize substantial, domain-general reductions in privacy leakage and expose critical limitations in monolithic steering approaches. This framework forms a foundation for direct, conceptual norm interventions in future AI safety and alignment research.