A Roadmap to Pluralistic Alignment (2402.05070v3)

Abstract: With increased power and prevalence of AI systems, it is ever more critical that AI systems are designed to serve all, i.e., people with diverse values and perspectives. However, aligning models to serve pluralistic human values remains an open research question. In this piece, we propose a roadmap to pluralistic alignment, specifically using LLMs as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems: 1) Overton pluralistic models that present a spectrum of reasonable responses; 2) Steerably pluralistic models that can steer to reflect certain perspectives; and 3) Distributionally pluralistic models that are well-calibrated to a given population in distribution. We also formalize and discuss three possible classes of pluralistic benchmarks: 1) Multi-objective benchmarks, 2) Trade-off steerable benchmarks, which incentivize models to steer to arbitrary trade-offs, and 3) Jury-pluralistic benchmarks which explicitly model diverse human ratings. We use this framework to argue that current alignment techniques may be fundamentally limited for pluralistic AI; indeed, we highlight empirical evidence, both from our own experiments and from other work, that standard alignment procedures might reduce distributional pluralism in models, motivating the need for further research on pluralistic alignment.

• The paper introduces pluralistic alignment strategies by outlining Overton, steerable, and distributional approaches to capture diverse human values.
• The paper demonstrates that standard alignment methods like RLHF can narrow response diversity, highlighting the need for more varied benchmarks.
• The paper advocates for refined evaluations and pluralistic frameworks to develop AI models that accurately reflect a spectrum of human perspectives.

The Concept of Pluralism in AI Systems

Introduction to Pluralism

Assessing AI systems' capability to align with human values has taken center stage in current research, given the complex and diverse nature of human perspectives and values. Attempts to tailor AI responses to fit an averaged preference often overlooks the inherent human diversity, presenting an urgent need for AI models that accommodate a broad spectrum of values – in other words, pluralistic systems. Establishing a pluralistic AI system involves presenting an array of reasonable responses, adjusting outputs to reflect specific perspectives, and accurately representing the diversity in a given population.

Defining Pluralism in AI

Efforts to formalize pluralism in AI models have proposed three main approaches: Overton pluralism that encompasses a range of reasonable responses, Steerable pluralism which allows for the reflection of particular attributes or perspectives, and Distributional pluralism that calibrates model outputs according to a given population's distribution. Moreover, aligning incident benchmarks with pluralism can be approached through multi-objective benchmarks, benches that measure the model's flexibility in steering among various objectives, and benchmarks that model a diverse range of human ratings.

The empirical evidence suggests that existing AI alignment strategies may inadvertently diminish distributional pluralism. Models trained on standard alignment procedures like Reinforcement Learning from Human Feedback (RLHF) tend to concentrate on a less varied set of answers, deviating from the more distributed nature of human responses.

The Relationship Between Alignment Techniques and Pluralism

Current alignment practices such as RLHF, where models are optimized to maximize human preferences derived from limited data sets, often ignore the nuances of human variance. However, certain current alignment techniques indicate capabilities for Overton pluralism, to a degree that human preferences allow. Although LLMs exhibit a form of steerable pluralism, further assessment is needed to evaluate this property comprehensively. Notably, the methodologies to ascertain pluralistic benchmarks and the resulting degree of pluralism in LLMs, necessitate deeper investigation.

Future Research Directions

The roadmap towards pluralistic AI emphasizes the necessity for more refined evaluations and the development of pluralistic frameworks that provide a comprehensive understanding of a model's characteristics. It also foregrounds the importance of engagement in normative discussions regarding what and whom we align AI systems to and what bounds of customization are considered acceptable. Additional research is needed to explore alignment techniques capable of producing more pluralistically-aligned models effectively.

In essence, this paper stands as a pivotal effort in charting the course for the creation and measurement of AI systems that genuinely resonate with, and respect, the diverse values and perspectives that shape human societies.