ChatGPT's advice drives moral judgments with or without justification (2501.01897v1)

Abstract: Why do users follow moral advice from chatbots? A chatbot is not an authoritative moral advisor, but it can generate seemingly plausible arguments. Users do not follow reasoned more readily than unreasoned advice, though, we find in an experiment. However, this is also true if we attribute advice to a moral advisor, not a chatbot. Hence, it seems that advice offers users a cheap way to escape from a moral dilemma. This is a concern that chatbots do not raise, but they exacerbate it as they make advice easily accessible. We conclude that it takes ethical in addition to digital literacy to harness users against moral advice from chatbots.

• The paper reveals that ChatGPT's advice significantly influences users' moral judgments regardless of whether the advice is justified.
• It employs an online experiment comparing ChatGPT and human advisors to show that users rationalize AI advice even when its moral authority is questioned.
• The study highlights the need to improve digital and ethical literacy to mitigate undue reliance on AI-driven moral guidance.

Analysis of ChatGPT's Influence on Moral Judgments

The paper "ChatGPT's advice drives moral judgments with or without justification" explores the influential role of AI chatbots, specifically OpenAI's ChatGPT, in shaping users' moral judgments. The central question addressed by Krügel, Ostermaier, and Uhl (2023) is why users are inclined to follow moral advice from chatbots, entities that inherently lack human moral reasoning and convictions.

Experimental Approach

The authors implemented an online experiment using the trolley dilemma to evaluate the impact of chatbot advice, both with and without justification. Participants were exposed to advice either favoring or disfavoring the sacrifice of one life to save five. The paper's design compared the influence of ChatGPT to that of a human moral advisor. The experiment involved eight different conditions, assessing the variance in participants' judgments when advice came with an argument and when it did not, and whether it was attributed to ChatGPT or a human advisor.

Key Findings

1. Consistent Influence Regardless of Justification: The paper found that the moral advice given by ChatGPT significantly influences users' judgments, irrespective of whether the advice is reasoned or unreasoned. This finding remained consistent whether advice was attributed to ChatGPT or a human advisor.
2. Psychological Mechanism: The research identified an ex-post rationalization mechanism influencing how users assess the chatbot's advice. Participants rated the moral authority of ChatGPT lower than that of a human advisor, yet rated the plausibility of its advice higher. This suggests that users follow advice to alleviate the cognitive burden of moral dilemmas and rationalize their choices post-decision, attributing higher plausibility to advice after following it.
3. Implications for Digital and Ethical Literacy: The research underscores the necessity for both digital and ethical literacy in mitigating the susceptibility of users to potentially random and arbitrary moral advice from AI-powered advisors. While chatbots can provide arguments to support advice, the paper reveals that users often accept advice as a relief from moral quandaries rather than an informed decision-making process.

Theoretical Implications

The findings highlight that the propensity to follow moral advice does not necessarily arise from the argument quality or the authoritative perception of the advisor, be it a human or an AI. Instead, the task's nature, particularly the complexity of moral dilemmas, heavily influences this tendency. Users demonstrate a substantial reliance on advice, suggesting a tendency to defer personal judgment in challenging moral situations due to the ease offered by external validation.

Practical Implications and Future Directions

The paper's results burden developers of AI systems with the ethical responsibility of preventing undue manipulation through AI advice. Although one immediate solution could be training chatbots to abstain from providing moral guidance, this approach may not be feasible in all contexts given that moral dilemmas are pervasive in everyday decision-making.

The authors advocate for enhancing users' digital and ethical literacy to foster a more critical and responsible usage of AI. A successful integration of these literacies would empower users to challenge rather than unreservedly accept AI-derived moral advice.

Conclusion

This paper offers significant insights into the psychological dynamics of advice-taking in moral contexts mediated by AI technologies. The research defines a paradigm wherein advice's mere availability, irrespective of its source, can influence decision-making. This raises important ethical considerations for future AI development and underscores the need for cultivating robust digital and ethical literacy that empowers users to navigate the complex landscape of AI-enabled interactions responsibly.