Generalizability of inconsistency effects to other passage types

Determine whether the observed effects of Large Language Model output inconsistencies on users’ perceived AI capacity and comprehension generalize beyond the factual, single-paragraph answers with supporting details used in the experiment to other passage types, such as logical or mathematical reasoning texts.

Background

The paper evaluated how presenting one, two, or three AI-generated passages—constructed to exhibit Main Answer Inconsistency and Paraphrasing Inconsistency—affects users’ perceived AI capacity and comprehension in an information-seeking task. Passages were long-form answers to the question about which European country has the most Nobel laureates in science, organized as a main answer followed by supporting details.

Findings showed that inconsistencies decreased perceived AI capacity while increasing comprehension, especially with two passages. The authors note uncertainty about whether these results would hold for other types of passages that may impose different cognitive demands (e.g., logical or mathematical reasoning content), prompting a need to assess generalizability across diverse text genres.