PICACO: Alignment for Pluralistic LLM Values

• PICACO is a methodology that aligns large language model outputs with pluralistic human values using in-context optimization.
• It leverages a variational information-theoretic objective and an EM-like optimization loop to balance conflicting values without any parameter fine-tuning.
• Experimental results across diverse LLMs show improved value conformity and robustness compared to traditional in-context alignment approaches.

PICACO (Pluralistic In-Context Alignment via COrrelation Optimization) is a methodology for pluralistic value alignment of LLMs through in-context optimization, enabling the simultaneous elicitation of multiple, potentially conflicting, human values in LLM outputs—without any parameter fine-tuning. PICACO operates within the In-Context Alignment (ICA) paradigm, automatically constructing a single “meta-instruction” prompt that maximizes total correlation between a selected set of values and the model’s generated outputs. By leveraging a variational information-theoretic objective, PICACO mitigates the instruction bottleneck and demonstrably improves value conformity and balance compared to existing ICA approaches across both black-box and open-source LLMs (Jiang et al., 22 Jul 2025).

1. Background: In-Context Alignment and the Instruction Bottleneck

In-Context Alignment (ICA) refers to methods for aligning LLM behavior with desired value criteria by augmenting the model’s prompt with instructions or demonstrations, rather than modifying model parameters via post-training (such as Supervised Fine-Tuning or Reinforcement Learning from Human Feedback). ICA offers flexibility and low cost, addressing the high resource demands of traditional fine-tuning methodologies.

A core challenge for ICA is the pluralistic nature of human values, which are often mutually conflicting (e.g., stimulation vs. tradition). Standard ICA approaches typically concatenate multiple value specifications into a single instruction. LLMs, being agnostic to prompt semantics and value tensions, tend to prioritize some values while neglecting others, resulting in incomplete or biased alignment—a phenomenon termed the “Instruction Bottleneck.” The inability to systematically encode and realize multiple values manifests as model outputs that inadequately represent or balance the intended value set (Jiang et al., 22 Jul 2025).

2. Methodology: Total Correlation Optimization in PICACO

PICACO addresses the instruction bottleneck by formulating value alignment as a conditional total correlation maximization problem. The central construct is the “meta-instruction” $e$—a natural language prompt optimized to elicit the full set of target values $\bm v = (v_1, ..., v_K)$ in outputs $\bm y$ from queries $\bm x$.

Total Correlation (TC) is defined as

$$TC(X_1, ..., X_m) = D_{KL}(P(X_1, ..., X_m) \Vert \prod_{i=1}^m P(X_i)) = \sum_{i=1}^m H(X_i) - H(X_1, ..., X_m)$$

where $H(\cdot)$ is entropy and $D_{KL}$ denotes Kullback-Leibler divergence. In the context of in-context prompt optimization, the conditional total correlation objective is

$$TC_e(\bm v, \bm y \mid \bm x) = \sum_{k=1}^K I_e(v_k; \bm y \mid \bm x) - I_e(\bm v; \bm y \mid \bm x) \tag{1}$$

where $I_e$ is mutual information under the LLM's distribution with meta-instruction $e$. The approach maximizes individual value-signal while minimizing redundant copying or superficial prompt adherence.

Direct computation being intractable, PICACO employs variational lower bounds (Barber-Agakov and CLUB) using two auxiliary evaluators:

• $\bm v = (v_1, ..., v_K)$0: a “value evaluator” estimating the extent to which $\bm v = (v_1, ..., v_K)$1 exhibits value $\bm v = (v_1, ..., v_K)$2,
• $\bm v = (v_1, ..., v_K)$3: a “redundancy evaluator” quantifying leakage of irrelevant details from exemplar $\bm v = (v_1, ..., v_K)$4 into $\bm v = (v_1, ..., v_K)$5.

The variational optimization objective is:

$\bm v = (v_1, ..., v_K)$6

where $\bm v = (v_1, ..., v_K)$7 adjusts the emphasis on value conformity versus redundancy reduction.

3. Optimization Procedure

PICACO implements an EM-like Variational Information Maximization (VIM) loop over a small labeled dataset $\bm v = (v_1, ..., v_K)$8. The procedure alternates between:

• Response Enhancement (E-step): With current meta-instruction $\bm v = (v_1, ..., v_K)$9, sample $\bm y$0 responses per prompt; score candidate outputs by $\bm y$1, retaining the top $\bm y$2. Additionally sample $\bm y$3 “noisy” responses to estimate background redundancy.
• Instruction Refinement (M-step): Using the collected responses and their scores, search for a new meta-instruction $\bm y$4 maximizing the empirical approximation of (2):

$\bm y$5

The updated $\bm y$6 is produced by prompting a strong LLM (e.g., GPT-4o) with a meta-optimization template and selecting the optimal candidate. Algorithm 1 in the source paper encapsulates the workflow (Jiang et al., 22 Jul 2025).

4. Theoretical Foundations and Information-Theoretic Justification

Maximizing conditional total correlation in PICACO encourages strong joint presence of all target values in model output, while penalizing trivial or literal copying of prompt instructions. High $\bm y$7 implies that $\bm y$8 is informative about each $\bm y$9, and low $\bm x$0 suppresses spurious correlations introduced by surface-level prompt features.

The difference between these terms quantifies evidence that the response meaningfully integrates the pluralistic value set, rather than reflecting prompt syntax or artifacts. This information-theoretic regularization specifically targets the instruction bottleneck characteristic of naïve multi-value prompting, providing a principled framework for pluralistic value realization in LLMs (Jiang et al., 22 Jul 2025).

5. Experimental Design and Baselines

PICACO was evaluated on five pluralistic value compositions:

1. Helpful (Coherence, Complexity, Verbosity, Helpfulness)
2. Harmless (Non-Toxicity, Fairness, Information Safety, Responsible Uses)
3. HH Balance (all 8 values above)
4. Confucianism (Benevolence, Conformity, Tradition, Security; Schwartz Theory)
5. Modern Liberalism (Universalism, Self-direction, Hedonism, Stimulation; Schwartz Theory)

Each value set was assessed with 800 prompts drawn from ten benchmarks (e.g., Just-Eval-Instruct, BeaverTails, NYT-Dilemma). Tested LLMs included GPT-3.5-Turbo, Gemini-1.5-Flash, and LLaMA-3.1-8B-Instruct. Variational evaluators comprised a GPT-4o-mini score (1–5 scale) for value conformity and a cosine similarity-based redundancy evaluator.

Experimental hyperparameters:

• $\bm x$1 prompts,
• $\bm x$2 aligned outputs,
• $\bm x$3 noisy outputs,
• $\bm x$4 optimization steps,
• $\bm x$5 for HH tasks, $\bm x$6 for Schwartz tasks.

Baselines encompassed naïve prompting, instruction plus demonstration, dual-process prompts, community modular LMs, iterative instruction search (OPRO), and CICL-style correction (Jiang et al., 22 Jul 2025).

6. Results, Comparisons, and Case Analysis

Across all five value compositions and all three LLMs, PICACO consistently achieved best or second-best overall conformity as measured by GPT-4o-2024-08-06 (scale 1–5), with a relevant penalty for Schwartz values. Notably:

• PICACO led on Confucianism, Modern Liberalism, and HH Balance for GPT-3.5-Turbo and LLaMA-3.1-8B-Instruct.
• On strictly “HH” tasks, performance was comparable or superior to previous state-of-the-art ICA methods.
• On Gemini-1.5-Flash, similar relative improvement trends were observed (see Table 1 and Table A.1 of the source).

PICACO exhibited milder score degradation with an increasing number of values (2 to 8), maintaining better balance (lower coefficient of variation) compared to Modular Pluralism or Q+IF. Ablation studies confirmed insensitivity to optimizer choice (GPT-4o or target LLM), and increased robustness to jailbreak attacks. Case studies highlighted PICACO’s ability to elicit nuanced, non-tokenistic integration of value tensions—such as reconciling Tradition and Hedonism—unlike baseline methods, which often produced superficial or biased compliance (Jiang et al., 22 Jul 2025).

Value Composition	LLM Target	PICACO Rank
Confucianism	GPT-3.5, LLaMA	Best
Modern Liberalism	GPT-3.5, LLaMA	Best
HH Balance (8)	GPT-3.5, LLaMA	Best
Harmless (4)	Varied	Best or Second
Gemini Flash (all)	Gemini-1.5-Flash	Matches trends

7. Limitations and Future Research Directions

Limitations highlighted in (Jiang et al., 22 Jul 2025) include:

• Scope of Values: The study focuses on two taxonomies: Schwartz Theory and HH taxonomy. Broader or alternative moral-cultural frameworks remain untested.
• Scope of Models: Only three LLMs evaluated; generalization to reasoning-rich models (e.g., DeepSeek) is undemonstrated.
• Evaluator Bias: Use of LLM-based scorers (GPT-4o-2024-08-06) as evaluative oracles may introduce assessment bias or guardrail effects.

Suggested future directions comprise testing on additional value theories, incorporating human-in-the-loop evaluation schemes, and extending PICACO to fine-grained or dynamic ICA strategies (Jiang et al., 22 Jul 2025).