---
title: 'NovelHopQA: Multi-Hop Long-Context QA Benchmark'
url: https://www.emergentmind.com/topics/novelhopqa
type: topic
---

# NovelHopQA: Multi-Hop Long-Context QA Benchmark

NovelHopQA is a diagnostic benchmark designed to evaluate and analyze multi-hop reasoning capabilities in long-context question answering (QA) systems, specifically within full-length narrative contexts spanning 64,000–128,000 tokens. Unlike prior benchmarks that separately assess long-context comprehension or multi-hop reasoning in synthetic or short Wikipedia snippets, NovelHopQA systematically varies both context length and reasoning hop depth (1–4 hops) within authentic, contiguous storylines. Its construction, evaluation, and subsequent experimental analysis offer controlled insights into the limitations of current large language models (LLMs), highlighting characteristic failure modes and informing the development of retrieval-augmented and hop-aware QA methodologies [2506.02000].

## 1. Conceptual Foundation and Motivation

NovelHopQA addresses the intersection of two previously isolated axes in QA research: processing extended textual context and performing multi-hop sequential inference. Existing datasets such as HotpotQA and WikiHop focus on two-hop reasoning within short documents, while others like NarrativeQA or LV-Eval test long-form comprehension but restrict themselves to single-hop or summary-style questions. NovelHopQA was developed in response to the absence of benchmarks probing both extended context lengths (≥64k tokens) and increased reasoning depth (1–4 hops) within coherent narratives, using public-domain novels as the source texts [2506.02000]. The motivation is to diagnose accuracy degradation as both context window and number of reasoning steps grow, and to expose systematic failure modes in state-of-the-art models.

## 2. Dataset Construction and Instances

The dataset comprises 4,000 QA instances, stratified into approximately 1,000 examples per hop level, sampled from 64k, 96k, and 128k-token excerpts of 83 English public-domain novels (Project Gutenberg). The construction pipeline consists of anchor-keyword selection (min. 50 occurrences per anchor), paragraph pool creation (≥30 words per paragraph), and guided multi-hop chain building. For each example, hop chains H = {h₁, h₂, ..., h_H} are constructed by selecting paragraphs via keyword-driven filtering and semantic clue expansion:

- For the initial hop, the system selects a paragraph containing a designated anchor keyword and generates an answerable question.
- For subsequent hops, context is expanded via semantically novel keywords and additional paragraphs, mandating answer integration across all selected evidence.

The following algorithmic sketch details the iterative chain construction:

```python
for each novel in corpus:
  anchors ← select_anchors(novel)  # ≥5 high-frequency keywords
  pool    ← split_into_paragraphs(novel)
  for target_hop_count H in {1..4}:
    C₀ ← ∅
    for h in 1..H:
      if h == 1:
        k₁ ← anchors[1]
        p₁ ← f_select({k₁}, pool)
        C₁ ← p₁
        (Q₁,A₁) ← GenQA(C₁)
      else:
        kₕ ← f_extract_keyword(C_{h-1})
        pₕ ← f_select({k₁,kₕ}, pool)
        Cₕ ← C_{h-1} ∥ pₕ
        (Qₕ,Aₕ) ← GenQA(Cₕ)
      remove pₕ from pool
```
Subsequent oracle-filtering ensures that only questions answerable using the available evidence are retained, validated by human annotators for both alignment and required reasoning depth [2506.02000].

## 3. Evaluation and Metrics

Seven state-of-the-art LLMs were assessed, including OpenAI o1, multiple GPT-4​o variants, LLaMA-3.3-70B-Instruct, and Google Gemini 2.5 Pro/Flash models. Standard chain-of-thought prompting was employed. Two evaluation regimes were applied:

- Full-context inference (golden context).
- Retrieval-augmented generation (RAG): novels are chunked into 350-token segments; top-k (k=7) chunks retrieved by bge-large-en embeddings and FAISS, yielding ∼2.5k tokens concatenated for model input.

Metrics used:

- Exact Match (EM): $EM = 1[â = a^*]$
- Token-level F1: $F1 = 2 \cdot \text{precision} \cdot \text{recall} / (\text{precision} + \text{recall})$

## 4. Empirical Results and Failure Modes

### Accuracy Trends

The relationship between hop depth and context length yields consistent degradation in model performance:

| Context (tokens) | H=1      | H=2         | H=3        | H=4        |
|------------------|----------|-------------|------------|------------|
| 64k              | 86.4%    | 79.5%       | 77.1%      | 73.8%      |
| 96k              | 83.5%    | 76.5%       | 74.7%      | 71.9%      |
| 128k             | 81.7%    | 73.5%       | 72.0%      | 68.9%      |

Top-performing models (e.g., Gemini 2.5 Pro, o1) achieve EM >95% on golden context but fall to ∼60% in RAG, while mid-tier models show more pronounced drops. The average EM decreases by 12.6 points from H=1 to H=4 in 64k contexts; RAG-based inference yields additional drops of 25–35 points, underscoring the brittleness of current retrieval techniques.

### Characteristic Failure Modes

Four dominant error classes were documented, each illustrated via concrete narrative examples:

1. **Missing Final-Hop Integration**: Sequential clues are processed correctly up to H−1 but final-hop evidence is ignored.
2. **Entity Confusion/Coreference Errors**: Model misattributes actions or facts to incorrect entities, usually in later hops.
3. **Incomplete Evidence Combination**: Only a subset of required facts is reported, typically omitting the newest evidence.
4. **Contextual Drift**: Model returns to irrelevant details from earlier hops rather than maintaining focus on the necessary chain.

## 5. Methodological Innovations and Adaptation

NovelHopQA motivates hop-aware retrieval and QA pipeline strategies, such as iterative expand-and-retrieve or multi-hop dense retrievers with posterior regularization. Momentum Posterior Regularization (MoPo) [2502.20399] is of particular relevance:

- **MoPo framework**: Uses query-focus summaries for each hop rather than raw answers, and distills posterior retrieval knowledge into the prior via a momentum-moving average. This stabilizes learning by keeping posterior parameters φ close to the evolving prior θ during training.
- **Adaptation protocol**: NovelHopQA instances are serialized into “PostSum” chains, enabling training of dual-encoder retrievers (E5/Contriever) before inference proceeds with iterative summary-guided retrieval, discarding the posterior retriever at inference.

MoPo yielded state-of-the-art retrieval and QA results on HotpotQA, with R@2 = 94.8% and EM@2 = 63.0%, outperforming fixed/dynamic two-stage posterior regularization and naive concatenation [2502.20399]. In principle, these methods transfer directly to NovelHopQA via backward summary generation, plug-and-play encoder initialization, and single-stage momentum distillation.

## 6. Comparative Methodologies: Hop-Agnostic and Graph-Based Approaches

Unified frameworks such as Iterative Document Reranking (IDRQA) [2009.07465] demonstrate hop-agnostic, adaptive pipelines valid for HotpotQA and extensible in theory to NovelHopQA:

- **Retrieval and Question Expansion**: Subsequent hops employ a Question Updater to refine query by concatenating extracted clue spans, enabling multi-hop retrieval without pre-specifying the number of steps.
- **Graph-Based Reranking**: Multi-document interaction is achieved via a graph of entity nodes, with co-occurrence adjacency and contextual features aggregated in a Graph Attention Network, facilitating bridge reasoning.
- **Filtering and Adaptive Stopping**: Paragraph-level scores enable dynamic filtering, and termination is controlled by a no-answer probability prediction.

Table of performance (as reported):

| Dataset           | EM         | F1         |
|-------------------|------------|------------|
| SQuAD Open        | 56.6%      | 56.6%      |
| HotpotQA (joint)  | 36.0%      | 63.9%      |

Ablation studies confirm graph-based reranking and iterative retrieval as necessary for strong multi-hop performance. Limitations include reduced robustness to numerically or structurally complex questions, and support-fact prediction is still decoupled from span selection, suggesting further joint modeling is required in NovelHopQA-like domains [2009.07465].

## 7. Consequences, Recommendations, and Open Problems

The empirical and methodological analyses of NovelHopQA lead to several key insights:

- Scaling context windows alone does not mitigate multi-hop degradation; accuracy declines systematically as hop count increases, irrespective of absolute context size.
- Standard RAG protocols cannot reliably recover multi-hop evidence or narrative order, especially above 64k tokens; more sophisticated hop-aware retrieval and iterative summary generation are recommended.
- Persistent failure modes indicate gaps in both evidence combination across hops and coreference tracking, implying the need for chain-of-thought-aligned architectures.
- Future research directions include scalable models that maintain coherent inference chains over ≥100k tokens, retrieval-conditioned text generation that guarantees all hops are covered, and benchmarks diversified to non-narrative technical domains.

NovelHopQA, with its open-source code, data, templates, and extensive validation, establishes a reproducible standard for diagnosing, benchmarking, and ultimately improving long-context multi-hop question answering in realistic narrative settings [2506.02000].

Source: https://www.emergentmind.com/topics/novelhopqa