Dice Question Streamline Icon: https://streamlinehq.com

Effect of LLM transmission chains on text difficulty

Determine whether iterated transmission chains of large language models preserve, increase, or reduce text difficulty, as quantified by the Gunning–Fog index, when each agent sequentially receives the previous agent’s generated text and produces a new output.

Information Square Streamline Icon: https://streamlinehq.com

Background

The paper studies how cultural information changes when passed through iterated chains of LLMs, focusing on properties such as toxicity, positivity, difficulty, and length. Accessibility of generated text is a key consideration for societal impact.

While toxicity and positivity have been examined in single-turn settings, the authors note that the evolution of text difficulty through multi-turn LLM interactions had not been established. They operationalize difficulty using the Gunning–Fog index, which estimates the years of formal education required to understand a text, and propose to assess how difficulty behaves under repeated transmissions.

References

However, whether text difficulty is preserved, increased, or reduced over transmission chains is currently unknown.

When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2407.04503 - Perez et al., 5 Jul 2024) in Section 3.2, Metrics (Text properties: Difficulty)