Bias introduced by chunkization in long-document summarization

Ascertain whether the chunkization approach used to summarize long documents—splitting the document into smaller segments, summarizing each with a large language model such as GPT-3.5-turbo or GPT-4, and aggregating the summaries—introduces systematic bias relative to single-pass summaries generated by a large language model with a sufficiently large context window, including potential inflation of combined summary length.

Background

To handle documents exceeding model context limits, practitioners often summarize in chunks and aggregate results. The paper flags a potential methodological concern: aggregated chunk summaries might differ in systematic ways from a one-shot summary produced when the entire document fits into the context window.

The authors note a specific plausible bias—longer combined summary length from multiple passes—but emphasize that whether such bias occurs has not been determined and warrants investigation.