The illusion of artificial inclusion (2401.08572v3)

Abstract: Human participants play a central role in the development of modern AI technology, in psychological science, and in user research. Recent advances in generative AI have attracted growing interest to the possibility of replacing human participants in these domains with AI surrogates. We survey several such "substitution proposals" to better understand the arguments for and against substituting human participants with modern generative AI. Our scoping review indicates that the recent wave of these proposals is motivated by goals such as reducing the costs of research and development work and increasing the diversity of collected data. However, these proposals ignore and ultimately conflict with foundational values of work with human participants: representation, inclusion, and understanding. This paper critically examines the principles and goals underlying human participation to help chart out paths for future work that truly centers and empowers participants.

• The paper demonstrates that substituting human participants with LLMs fails due to inaccuracies, hallucinations, and outdated value lock-in.
• It reveals that current LLM technology struggles to simulate non-verbal cues and human intersubjectivity essential for authentic behavioral research.
• The study calls for robust inclusion frameworks and continual human engagement to ensure ethical and valid research outcomes.

Analyzing "The Illusion of Artificial Inclusion"

The paper, "The Illusion of Artificial Inclusion," engages in a critical examination of the burgeoning concept of substituting human participants in research and development with LLMs and other generative AI systems. This analysis is pertinent within fields such as AI development, human-computer interaction (HCI), behavioral science, and psychology, where the practical and ethical dimensions of participant substitution are under scrutiny.

Substitution Proposals and Stated Motivations

The paper provides a scoping review of substitution proposals, identifying four primary motivations: increased efficiency, reduced costs, enhanced demographic diversity, and participant protection from harm. Notably, the review encompasses a range of academic articles, technical reports, and commercial products, revealing both enthusiasm and concern about substitution within the scientific community.

Technical and Practical Considerations

The paper underscores significant limitations in current LLM technology that undermine the purported benefits of substitution:

1. Inaccuracy and Hallucinations: Despite advancements, LLMs are not ready to replicate human cognition accurately due to inherent tendencies to hallucinate or produce incorrect information. This problem compromises claimed increases in speed and cost efficiency.
2. Value Lock-In: LLMs, trained on data reflective of past societal norms, cannot adapt to evolving cultural attitudes, limiting their ability to generate representative and current insights.
3. Marginalized Perspectives: LLMs often fail to accurately reflect the diverse and intersecting identities found in human populations, challenging the notion of improved diversity.
4. Interference with Non-Linguistic Measures: The capability of LLMs to simulate non-verbal cognitive indicators, used extensively in behavioral research, remains inadequate.

Intrinsic Challenges to Substitution

Beyond practical issues, the paper identifies intrinsic conflicts with core values in research and development:

• Representation and Inclusion: The paper argues that substituting human input with AI undermines the participatory ethos of tech development by marginalizing the actual presence and influence of human participants. Participatory approaches, which traditionally emphasize reflection of community perspectives and redistribution of power, are inherently compromised by AI surrogates.
• Intersubjectivity and Understanding: In psychological research, the paper highlights the importance of human-to-human intersubjectivity—understanding that arises from shared experiences. This intersubjective relationship cannot be replicated by AI, thereby limiting the depth and validity of insights from substitution.

Implications and Future Directions

The findings provoke reconsideration of the ethical and operational intersection of AI and human-centered research. The authors advocate for more robust representation and inclusion frameworks and caution against overlooking the essence of intersubjective understanding in research. They call for mechanisms to ensure continual participant influence and for frameworks to critically evaluate the validity of LLM-produced data.

In future developments, balancing the benefits of technological enhancement against the foundational values of participant representation and understanding will be vital. Innovative frameworks and methodologies need to be developed to ensure AI's integration into research and development remains ethical and substantively productive.

The authors urge ongoing dialogue and re-evaluation of the role of generative AI in research, posing critical questions around authenticity and empowerment that are central to navigating these complex issues in the evolving landscape of AI technology.