Think Before you Write: QA-Guided Reasoning for Character Descriptions in Books

Abstract: Character description generation is an important capability for narrative-focused applications such as summarization, story analysis, and character-driven simulations. However, generating accurate character descriptions from long-form narratives (e.g., novels) is challenging: models must track evolving attributes (e.g., relationships and events), integrate evidence scattered across the text, and infer implicit details. Despite the success of reasoning-enabled LLMs on many benchmarks, we find that for character description generation their performance improves when built-in reasoning is disabled (i.e., an empty reasoning trace). Motivated by this, we propose a training framework that decouples reasoning from generation. Our approach, which can be applied on top of long-context LLMs or chunk-based methods, consists of a reasoning model that produces a structured QA reasoning trace and a generation model that conditions on this trace to produce the final character description. Experiments on two datasets (BookWorm and CroSS) show that QA-guided reasoning improves faithfulness, informativeness, and grounding over strong long-context baselines.

• The paper demonstrates that decoupling reasoning from generation via structured QA traces enhances the fidelity and informativeness of character descriptions.
• It introduces a two-stage pipeline that leverages GRPO to optimize QA pair extraction and integrate them into a coherent, evidence-based narrative output.
• Empirical results on Book Worm and CroSS show robust improvements in entity coverage and grounding metrics across various long-context strategies.

QA-Guided Decoupled Reasoning for Character Description Generation in Long Narratives

Problem Setting and Motivation

Accurately generating character descriptions from book-length narratives poses substantial challenges for LLMs due to entity evolution, implicit detail, and highly dispersed evidence. Existing chunk-based and retrieval-augmented methods struggle with integrating salient information across long contexts, while even advanced long-context LLM architectures exhibit failures related to position bias and underuse of distant evidence. Notably, this paper empirically demonstrates that enabling built-in reasoning traces—such as chain-of-thought prompting—actually degrades the faithfulness of LLM-generated character descriptions in this domain.

Methodological Contributions

The core advance consists of a two-stage, modular framework wherein explicit reasoning is separated from surface realization. The pipeline consists of: (i) a dedicated reasoning model that outputs a structured trace of targeted QA pairs (covering roles, relationships, personalities, events, etc.) about the character, derived chunk-wise; and (ii) a generation model tasked solely with producing a coherent character description, conditioned both on the source narrative and the intermediate reasoning trace (injected between "thinking" tokens). This architecture is agnostic to both backbone LLM and long-context management strategy (retrieval, hierarchical, incremental, or direct).

The QA reasoning traces are operationalized as tuples $(q_j, e_j, a_j, t_j)$, where $q_j$ is a question, $e_j$ a short supporting explanation, $a_j$ a concise answer, and $t_j$ a typed category (role, relationship, personality, event, other). Traces are generated per chunk and concatenated.

Training the reasoner leverages GRPO, a sample-efficient RL algorithm, using F1 between generated and reference QA pairs derived from gold character descriptions as reward, thus circumventing the challenge of reward definition in the open-ended generation space. The generator is trained either in zero-shot or SFT mode to utilize the structured traces.

Experimental Results and Analysis

Evaluation spans two datasets: Book Worm and CroSS, containing books and their human-authored character descriptions. Metrics include PRISMA (LLM-as-judge factuality), entailment-based NLI grounding, QA F1 (reference question answerability), entity-mention F1, and Rouge-L for surface overlap.

Key empirical findings:

• Built-in reasoning traces harm faithfulness: Across long-context processing strategies, using the model's default reasoning traces leads to systematically lower PRISMA, NLI, and QA F1 scores relative to omitting reasoning, indicating verbosity and unsupported detail introduction.
• QA-guided reasoning improves grounding and informativeness: Conditioning generation on QA-guided traces robustly increases evidence-sensitive metrics. For example, in the Book Worm dataset under the BM25-32k retrieval regime, guided QA raises entity-mention F1 from 34.06 to 36.23 and NLI F1 from 58.58 to 59.96, with corresponding improvements in QA F1 and PRISMA.
• Benefits extend across context policies and model scales: Gains are found in both chunk-based and long-context LLM settings, and application of guided QA narrows the gap to larger proprietary models (e.g., GPT-4.1-mini), outperforming them in entity coverage despite using smaller open models.
• Trace quality is pivotal: Ablation and training analyses reveal that trace faithfulness and coverage—improved via GRPO optimization—directly translate into stronger downstream descriptions. The oracle traces (from gold descriptions) offer much headroom, highlighting limitations in current automatic reasoning generation.
• Chunk-level QA pair supervision is more effective than topic- or plan-driven alternatives: Simpler chunk-based approaches outperform multi-step topic or plan extraction methods for reasoning trace construction, both in per-trace F1 and in supporting final description metrics.
• Incremental integration settings are challenging: Integrating reasoning traces in strictly incremental description updating shows limited gains, likely due to compounding partial evidence and reconciliation difficulties.

Theoretical and Practical Implications

This work establishes that for non-verifiable, open-ended long-form text generation, effective reasoning must be decoupled from the surface generation process. Structured, targeted QA traces serve as robust planning scaffolds, enabling improved evidence aggregation and implicit detail inference in extreme long-context settings, addressing core weaknesses of both chunk-based and full-context LLM approaches.

Practically, the framework can be integrated atop any long-context or retrieval-based LLM system, with modular training. No gold-standard intermediate traces are required, only downstream access to human-written descriptions for trace supervision.

Theoretically, these findings reinforce the limits of end-to-end monolithic decoding for interpretive or summarization tasks where evidence is non-local and unstated inference is required. Decoupling aligns with human narrative analysis practices and connects to recent advances in externally supervised reasoning and chain-of-thought planning.

Future Directions

Promising avenues include:

• Expanding beyond QA-style traces to richer or more abstract intermediate reasoning formats.
• Exploring alternative RL objectives (e.g., PPO or DPO) and reward aggregation procedures suited to generative tasks with multi-fact output spaces.
• Developing more discriminating, character-centric evaluation metrics, as current automatic metrics, despite their utility, undermeasure nuanced narrative coherence, implicit trait inference, or global character modeling.
• Extending weak supervision strategies for trace extraction to leverage plot summaries, character sheets, or crowd-sourced knowledge bases, thus reducing dependence on gold-standard descriptions.

Conclusion

Decoupling reasoning from generation via structured QA-guided traces significantly enhances the fidelity, informativeness, and grounding of character descriptions in long-form narrative texts. This modular approach overcomes limitations of monolithic LLM reasoning, provides a transparent and optimizable planning scaffold, and improves transferability across datasets and context management paradigms. The results substantiate a robust alternative to current reasoning-augmented LLM pipelines for character-centric narrative understanding and suggest broader applicability for other long-context, open-ended text generation tasks (2604.11435).