What are human values, and how do we align AI to them? (2404.10636v2)

Abstract: There is an emerging consensus that we need to align AI systems with human values (Gabriel, 2020; Ji et al., 2024), but it remains unclear how to apply this to LLMs in practice. We split the problem of "aligning to human values" into three parts: first, eliciting values from people; second, reconciling those values into an alignment target for training ML models; and third, actually training the model. In this paper, we focus on the first two parts, and ask the question: what are "good" ways to synthesize diverse human inputs about values into a target for aligning LLMs? To answer this question, we first define a set of 6 criteria that we believe must be satisfied for an alignment target to shape model behavior in accordance with human values. We then propose a process for eliciting and reconciling values called Moral Graph Elicitation (MGE), which uses a LLM to interview participants about their values in particular contexts; our approach is inspired by the philosophy of values advanced by Taylor (1977), Chang (2004), and others. We trial MGE with a representative sample of 500 Americans, on 3 intentionally divisive prompts (e.g. advice about abortion). Our results demonstrate that MGE is promising for improving model alignment across all 6 criteria. For example, almost all participants (89.1%) felt well represented by the process, and (89%) thought the final moral graph was fair, even if their value wasn't voted as the wisest. Our process often results in "expert" values (e.g. values from women who have solicited abortion advice) rising to the top of the moral graph, without defining who is considered an expert in advance.

Aligning AI with Human Values through Moral Graph Elicitation

The paper "What are human values, and how do we align AI to them?" addresses a critical concern in AI alignment: ensuring AI systems operate in harmony with human values. This work departs from traditional alignment methodologies by proposing a sophisticated process known as "Moral Graph Elicitation (MGE)" and constructing a "moral graph" to synthesize diverse human inputs into an alignment target.

Key Contributions

The authors divide the problem of aligning AI with human values into three stages: eliciting values from individuals, reconciling these values into a coherent alignment target, and training models to optimize this target. This paper focuses on the first two stages, proposing a novel method for value elicitation and reconciliation.

1. Criteria for Alignment Targets: The paper sets forth six rigorous criteria essential for an alignment target: it must be fine-grained, generalizable, scalable, robust, legitimate, and auditable. These criteria ensure that an alignment target can effectively shape model behavior in alignment with human values.
2. Moral Graph and Values Cards: MGE introduces the concept of the "moral graph", composed of values cards and a network of relationships between values, informed by context and user input. Moral graphs map how values cohere and evolve over time, offering a dynamic and context-aware framework for capturing human moral reasoning.
3. Innovative Value Elicitation Process: The authors describe using a LLM to extract values from participants. This process refines raw human extracts into "values cards" by focusing on attentional policies, providing a structured way to articulate values beyond preferences and ideologies.
4. Empirical Evaluation: Through a case paper with 500 American participants, the authors demonstrate the process's feasibility and efficacy in creating a moral graph that meets their criteria. Notably, they report high participant satisfaction, with 89.1% feeling well-represented, and a strong alignment of expert opinions emerging organically.

Methodological Insights

Values are elicited by guiding participants through dynamic dialogues facilitated by LLMs, refining their responses into actionable "values cards." The moral graph forms by analyzing how participants rate values in context, capturing transitions that reflect evolving wisdom and moral learning.

The paper differentiates its approach from traditional methods like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI (CAI). Unlike RLHF, the moral graph is structured for auditability and robustness, protecting it against manipulation. Whereas CAI relies on high-level principles, the moral graph’s context-rich associations between values offer a nuanced mechanism that is scalable and capable of integrating diverse human inputs.

Implications and Future Directions

The implications of this research extend deeply into both theoretical and practical domains:

• Theoretical Contribution: The structured articulation of values into non-instrumental choice criteria offers a fresh perspective for defining moral reasoning within AI systems. It aligns model behavior with a broad representation of human moral insights, potentially transforming how AI systems are perceived and deployed.
• Practical Application: Practically, the MGE process demonstrates how AI systems can be trained to reflect nuanced human values, offering a pathway for AI to operate autonomously yet responsibly. As AI systems gain autonomy in decision-making across various sectors, models trained via moral graphs could mitigate risks associated with narrow AI objectives and unintended consequences.
• Scalable Alignment Mechanisms: By adopting a framework that integrates human deliberation and moral learning, this approach posits a scalable model for evolving AI alignment, accommodating growing model complexity and diverse cultural contexts.

The authors call for further exploration into scalable methods for constructing larger moral graphs and developing models that dynamically adjust alignment targets in real-world deployments. Future work must also explore the potential for biases within LLM-driven elicitation processes and refine methods to ensure cultural and contextual applicability.

In conclusion, this paper delivers a multifaceted framework, combining philosophical insight with technical innovation, to address an enduring challenge in AI safety. It sets the stage for a new era of AI systems that can genuinely embody human values, thus fostering trust and ensuring sustainable integration into society.