On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods

Abstract: Preference elicitation frameworks feature heavily in the research on participatory ethical AI tools and provide a viable mechanism to enquire and incorporate the moral values of various stakeholders. As part of the elicitation process, surveys about moral preferences, opinions, and judgments are typically administered only once to each participant. This methodological practice is reasonable if participants' responses are stable over time such that, all other relevant factors being held constant, their responses today will be the same as their responses to the same questions at a later time. However, we do not know how often that is the case. It is possible that participants' true moral preferences change, are subject to temporary moods or whims, or are influenced by environmental factors we don't track. If participants' moral responses are unstable in such ways, it would raise important methodological and theoretical issues for how participants' true moral preferences, opinions, and judgments can be ascertained. We address this possibility here by asking the same survey participants the same moral questions about which patient should receive a kidney when only one is available ten times in ten different sessions over two weeks, varying only presentation order across sessions. We measured how often participants gave different responses to simple (Study One) and more complicated (Study Two) repeated scenarios. On average, the fraction of times participants changed their responses to controversial scenarios was around 10-18% across studies, and this instability is observed to have positive associations with response time and decision-making difficulty. We discuss the implications of these results for the efficacy of moral preference elicitation, highlighting the role of response instability in causing value misalignment between stakeholders and AI tools trained on their moral judgments.

• The paper demonstrates significant instability (~10–18%) in moral judgments for controversial cases, questioning the reliability of single-session surveys.
• It reveals that longer response times and increased decision difficulty are linked to greater fluctuations in moral preference elicitation.
• The study emphasizes the need for repeated measures and robust, multi-session data collection to ensure ethical AI alignment with stable stakeholder values.

On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods

The paper "On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods" by Kyle Boerstler et al. addresses a fundamental challenge in the development of participatory ethical AI tools, particularly regarding the stability of moral preferences over time. Researchers in AI and moral psychology often employ preference elicitation frameworks to capture the moral values of various stakeholders, aiming to incorporate these values into AI systems. This study critically examines the assumption of response stability underlying these frameworks and investigates whether this assumption consistently holds true.

Study Objectives and Design

The researchers conducted two human subject experiments, investigating the consistency of participants' moral preferences regarding kidney allocation scenarios over multiple sessions. Two studies were designed: Study One with 30 participants and Study Two with 82 participants, both using pairwise comparisons of fictional patients needing a kidney transplant. The participants' moral preferences were elicited through repeated scenarios across ten sessions spanning two weeks.

Key Findings and Results

The response stability was evaluated at the individual participant level and analyzed across controversial and uncontroversial scenarios. For controversial scenarios, the observed instability was approximately 10-18% ($\pm$ 14-15%), indicating that participants changed their responses between 10-18% of the time. This instability was positively associated with prolonged response times and perceived decision difficulty, suggesting that moral decisions perceived as difficult or complex are more likely to fluctuate over time.

However, responses to uncontroversial scenarios exhibited higher stability and agreement across participants, confirming that situations with clear moral choices yield more consistent judgments. Additionally, the study found evidence that response instability could be linked to small differences in the priority scores assigned by participants to the compared profiles, implying that closer moral dilemmas inherently cause more variability in decision-making.

Implications for AI Development and Ethical Considerations

The findings present several practical and theoretical implications for AI development:

1. Accuracy in Moral Preference Elicitation: The observed instability highlights a significant challenge in accurately capturing moral preferences through single-session surveys. For applications requiring high certainty and ethical alignment, such as resource allocation and autonomous vehicle programming, reliance on unstable preferences can lead to ethical misalignment between stakeholders and AI systems.
2. Model Robustness and Training: The stability of decision models across sessions is crucial for developing robust and reliable AI systems. The study’s model-based evaluations revealed that participants' decision-making models changed across different sessions, emphasizing the need for multi-session data collection to enhance model accuracy and reliability.
3. Need for Repeated Measures: For critical AI applications, the study suggests that repeated measurement and cross-validation of stakeholder preferences are essential. This approach helps account for potential variability due to temporary moods, decision difficulty, and external factors, thereby ensuring that the AI systems are trained on more stable and representative moral judgments.
4. Aggregation Methods and Indecision: While aggregating responses may smooth individual variability, it risks diluting minority opinions, which can be crucial in diverse populations. Furthermore, the study’s findings on the association between response instability and decision difficulty underline the importance of modeling indecision and close-call scenarios in AI training frameworks.

Future Directions

The study opens several avenues for future research:

• Extended Demographic and Contextual Analysis:

Broadening the participant population and exploring different moral scenarios can help generalize the findings and understand context-specific stability patterns.

• Mechanisms of Instability:

Further research into the cognitive and contextual mechanisms driving instability can inform the development of more robust elicitation frameworks and AI alignment techniques.

• Enhanced Survey Designs:

Developing methods to handle and model response instability, such as incorporating redundancy in moral queries or eliciting detailed explanations for choices, could improve the fidelity of moral preference surveys.

Conclusion

The paper by Boerstler et al. provides a comprehensive analysis of the stability of moral preferences and its implications for ethical AI development. By uncovering substantial instability in participants' moral judgments, the study challenges current computational elicitation methods and underscores the need for enhancements in survey design and AI training protocols. These findings are critical for ensuring that AI systems align with genuine and stable moral values of stakeholders, thereby fostering trust and ethical consistency in AI applications.