PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Abstract: AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.

• The paper presents an automated pipeline that synthesizes diverse, privacy-sensitive scenarios to capture and benchmark leakage in LLM agent activities.
• It employs a robust human annotation framework using detailed leak/omit labeling, achieving improved inter-annotator agreement (Cohen’s κ up to 0.71).
• Annotation-conditioned reward modeling in RL training significantly reduces leakage, enabling smaller models to approach frontier performance in contextual privacy.

Contextual Privacy Alignment in LLM Agents: An Expert Analysis of "PrivacyAlign"

Introduction and Motivation

"PrivacyAlign: Contextual Privacy Alignment for LLM Agents" (2606.21710) addresses the pivotal issue of privacy decision-making in agentic AI, where LLM agents autonomously act upon multi-modal context, tool results, persistent memories, and user instructions. Traditional privacy mitigation strategies focus on memorization and training data leakage or rely on string-matching and LLM-judge proxies for privacy evaluation, which fail to capture the deeply contextual and normative aspects of privacy rooted in human judgment. The paper asserts that authentic privacy alignment mandates human-centric supervision and evaluation, as privacy violations are inherently defined by societal norms, contextual relations, and human expectations.

Synthetic Scenario Generation Pipeline

The authors introduce a fully automated synthetic scenario generation pipeline to establish a high-quality, diverse benchmark for privacy-sensitive agent actions. The pipeline starts from randomly sampled user identities (first names, demographic attributes), synthesizes user profiles and agentic stories, generates tool-call trajectories and memory stores, and mines cross-model response pairs exhibiting actual privacy leakage.

Figure 1: PrivacyAlign synthetic data generation pipeline, illustrating scenario generation and selection targeting agent leakage events.

Scenario synthesis utilizes Qwen3.5-397B-A17B, gpt-oss-120b, and Nemotron-3-Super-120B-A12B, with each model running independently and serving as both generator and judge during cross-model pair mining. Filtering is done using sensibility checks, scenario-quality judges, leakage judges, and embedding-based diversity pruning, with hard caps to prevent domain collapse (especially into healthcare). The pipeline thus yields agentic scenarios with tool-use and persistent memory, where current LLMs are empirically documented to leak sensitive information.

Human Annotation Framework

PrivacyAlign centers human annotation, collecting 3,516 detailed annotations from 599 unique annotators across 1,350 scenario pairs. The annotation interface enforces rigorous quality protocols, requiring per-response leak/omit labeling, pairwise preference selection, and written rationales contextualized to each scenario.

Figure 2: Complete annotation page, with scenario context, dual candidate responses, leak/omit checkboxes, preference selector, comparative analysis, and explanation box.

The per-response leak and omit checkboxes are foundational in transcending simplistic content-matching approaches, with annotators marking exactly what they deem invasive or insufficient.

Figure 3: Granular leak/omit checkboxes and robust preference selector for capturing nuanced privacy judgments.

Rationales are mandated to be specific and scenario-grounded, promoting interpretability and auditability.

Figure 4: Free-text justification interface enforcing normative, context-aware privacy reasoning.

Quality control procedures include LLM-based audits for rationale specificity and manual review to reject low-quality or contradictory annotations, ensuring inter-annotator reliability (Cohen’s $\kappa = 0.606$ for preference, $0.558$ for leaks).

Annotation-Conditioned Reward Modeling and Evaluation

Central to the methodology is annotation-conditioned reward modeling. The paper studies two approaches: (1) pairing the scoring judge with in-context human annotations for the same prompt, and (2) trained generative reward models fit to the annotation corpus, but not context-conditioned at inference.

The annotation-conditioned judge significantly outperforms both annotation-free LLM judges and trained reward models in inter-judge agreement and alignment with audited gold labels. Cohen’s $\kappa$ for leaks and omits increases from $0.47 \to 0.71$ and $0.25 \to 0.44$, respectively, when conditioning on annotations. Agreement with gold labels rises from $0.45 \to 0.59$ for leaks and $0.29 \to 0.47$ for omits as additional annotation context is exposed, matching or exceeding Prolific annotator majority baseline.

RL Training Results and Model Performance

RL training using annotation-conditioned rewards is conducted on small open-weight LLMs (Qwen3-4B, Qwen3-8B, Nemotron-3-Nano-4B). The evaluation on PrivacyAlign, PrivacyLens, and CIMemories demonstrates:

• Frontier LLMs (GPT-5.5, Gemini 3.1 Pro) retain leak rates of $14.5\% - 30.3\%$ even with privacy-enhanced prompting, underscoring the persistence of contextual privacy failures.
• PrivacyAlign-trained models via annotation-conditioned reward achieve substantial reduction in leakage, with Nemotron-3-Nano-4B + RL reaching $32.6\%$ clean rate under naive prompting—approaching frontier model performance despite much smaller scale. The clean rate rises (non-leaking and non-omitting responses) across all open-weight families.
• Training with string-matching reward (CI-RL) or trained generative reward models consistently underperforms the annotation-conditioned regime, with the latter trading lower leakage for higher omission, an undesirable privacy/helpfulness imbalance.

Strong generalization to external benchmarks is observed: the annotation-conditioned Nemotron-3-Nano-4B model achieves $38.3\%$ leak rate on PrivacyLens (lowest among all models) and improves violation/completeness metrics on CIMemories, despite benchmark labels being model-defined.

Critique of Prior Supervision Paradigms

The authors rigorously document structural weaknesses in existing privacy supervision methods:

• CI-RL’s substring-matching reward suffers from contradictory supervision, fragile paraphrase vulnerability, and reliance on synthetic, model-generated labels.
• PrivacyBench, CIMemories, and PrivAct inherit automated LLM-judge biases and lack authentic human grounding.
• Model-only benchmarks flag legitimate user-requested content as sensitive if the proxy fails to interpret social context (e.g., PrivacyLens).
• Pluralistic privacy norms, annotator heterogeneity, and reliance on LLM judge evaluation, even with annotation conditioning, remain as inherent limitations.

Practical and Theoretical Implications

The practical implications are immediate:

• Model training with human-in-the-loop annotation-conditioned rewards raises contextual privacy and utility without sacrificing task completion.
• Human annotations directly remediate inter-judge disagreement, closing the gap between model proxy evaluation and true normative disclosure judgments.
• The released PrivacyAlign corpus enables reproducible, cross-model privacy benchmarking, validation, and future dataset expansion.

Theoretical implications extend to alignment and reward modeling in RLHF: the research strengthens the case for in-context conditioning on human explanations and scenario-specific reasoning, beyond averaged preference aggregation or rubric-driven reward signals. The paradigm supports pluralistic alignment and distributional preference learning, anticipating broader adaptation to non-verifiable or highly contextual generative domains.

Future Directions

The methodology is currently limited to small open-weight models due to computational constraints. Scaling annotation-conditioned RL to frontier-scale LLMs, diversifying annotator pools, and incorporating authentic user traces (with robust privacy guarantees) will be necessary advances. Cross-cultural privacy norm generalization, model robustness to distribution shift and adversarial scenarios, and automated evaluation meta-calibration remain open challenges. The annotation-conditioned reward framework is transferrable to other normative agentic decision domains (e.g., fairness, safety, utility trade-offs).

Conclusion

"PrivacyAlign" constitutes a rigorous approach to contextual privacy alignment for LLM agents rooted in human annotation. By deploying both synthetic scenario generation and annotation-conditioned reward modeling, the paper demonstrates quantifiable advances in agentic privacy judgment, bridging the gap between current LLM proxies and genuine human expectations. The released corpus and methods enable ongoing progress in privacy-aware RLHF, agent evaluation, and pluralistic social alignment, while delimiting the boundaries and risks of human-in-the-loop supervision in highly contextual tasks.