Disentangling context-length effects from theory-of-mind demands in CharToM-QA

Determine whether the primary source of difficulty in answering questions in the CharToM-QA benchmark arises from processing long input contexts (novel-length passages exceeding 2,000 words) or from the theory-of-mind reasoning requirements of the questions themselves.

Background

CharToM-QA employs classic novels to study theory-of-mind understanding across characters’ perspectives, but the benchmark’s contexts are lengthy (over 2,000 words). This introduces a potential confound between long-context processing and ToM reasoning.

The paper notes that it is unclear which factor primarily drives the challenge in CharToM-QA, motivating their development of shorter-context datasets to better isolate ToM-related capabilities. Establishing the source of difficulty would help attribute model performance deficits appropriately.

References

However, their questions are based on lengthy novels (over 2K words), making it unclear whether the challenge stems from context length or ToM understanding.

— Are LLMs Smarter Than Chimpanzees? An Evaluation on Perspective Taking and Knowledge State Estimation  (2601.12410 - Yang et al., 18 Jan 2026) in Section 2, Related Works — Theory of Mind (ToM) in LLMs

The interaction between length and reasoning depth remains uncertain. Predicted scores are lower at high reasoning depth across the plotted length range, consistent with RQ1. However, the difference between length contrasts ($-0.66$\,pp) has an interval spanning zero ($p_{\mathrm{adj}=1.8\times10{-1}$). The logit interaction is likewise inconclusive.

— Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning  (2610.22067 - Qin et al., 15 Oct 2026) in Section 5.3, “RQ2: Length-dependent Difficulty”