KnowMT-Bench: Multi-Turn LFQA Benchmark

• KnowMT-Bench is a benchmark that evaluates multi-turn long-form question answering in knowledge-intensive domains, emphasizing factual accuracy and coherent synthesis.
• The benchmark transforms single-turn LFQA pairs into dynamic dialogues through progressive context building, ensuring semantic alignment and realistic consultation simulation.
• Evaluation metrics such as factual capability, hallucination reliability, and information delivery efficiency are measured via a two-stage automated pipeline.

KnowMT-Bench is a benchmark for Multi-Turn Long-Form Question Answering (MT-LFQA) in knowledge-intensive domains, introduced to evaluate how LLMs perform when factual long-form answering must be sustained across dialogue turns rather than in a single isolated exchange. It is described as the first-ever benchmark designed to systematically evaluate MT-LFQA across medicine, finance, and law, with a dynamic setting in which models generate their own dialogue histories from logically progressive question sequences and are then evaluated on the final-turn answer for both factual capability and information delivery efficiency (Chen et al., 26 Sep 2025). The benchmark is motivated by the observation that real-world consultations in medicine, finance, and law are inherently multi-turn, whereas prior single-turn LFQA benchmarks and existing dialogue benchmarks do not jointly test paragraph-level synthesis, factuality, and the effects of self-generated conversational context.

1. Scope, motivation, and relation to prior benchmarks

KnowMT-Bench targets a gap between two established evaluation traditions. On one side are single-turn LFQA benchmarks such as K-QA, FinTextQA, and cLegal-QA, which provide one question–answer pair and therefore do not capture the complications introduced by dialogue history. On the other side are conversational QA and dialogue benchmarks such as QuAC, CoQA, MT-Bench, BotChat, and MT-Eval, which assess short extractive replies or broader conversational capabilities such as alignment and fairness rather than knowledge-intensive factuality (Chen et al., 26 Sep 2025).

The benchmark is designed around the premise that MT-LFQA requires two properties simultaneously: high factual accuracy and coherent, concise delivery under potentially noisy multi-turn context. In the benchmark framing, single-turn QA and MT-LFQA differ not merely in the number of turns but in the failure modes induced by conditioning on prior exchanges: redundancy, contextual distraction, and hallucinations. KnowMT-Bench therefore evaluates paragraph-level synthesis of multiple facts rather than short-form extraction.

A common misconception is that multi-turn evaluation can be approximated by existing dialogue benchmarks that already include several turns. KnowMT-Bench rejects that equivalence. The benchmark’s emphasis is not generic conversation quality but factual long-form answering in knowledge-intensive settings. This suggests that performance on alignment- or style-oriented dialogue benchmarks is not, by itself, an adequate proxy for performance in consultation-like factual interactions.

2. Dataset composition and benchmark construction

KnowMT-Bench is built from 801 high-quality LFQA pairs spanning three domains: finance (579), law (278), and medicine (209); 33% of instances overlap finance and legal (Chen et al., 26 Sep 2025). Each answer $g_i$ is grounded in an authoritative evidence set $\mathcal{E}_i$, such as official websites or expert-curated documents. The benchmark therefore begins with evidence-grounded single-turn LFQA instances and then transforms them into multi-turn dialogue settings.

Dialogue lengths are sampled according to ShareGPT usage, with 2–5 turns and proportions 37.5%, 37.5%, 15.0%, 10.1%. For each base question $q_j$, a human-reviewed LLM prompt expansion generates preceding questions $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$ according to three principles:

1. Progressive Context Building
2. Intent Preservation
3. No Answer Leakage

The final question is fixed as $q^{(d)}_{N_d}=q_j$, which ensures semantic alignment with the original single-turn QA. This construction preserves the original target information need while embedding it in a dialogue trajectory that resembles a progressive consultation.

Single-turn answers were refined by graduate annotators, and multi-turn expansions were templated and manually checked. The resulting benchmark is not based on handcrafted dialogue histories; instead, it is constructed to let models populate the history themselves during evaluation. A plausible implication is that the benchmark is intended to capture compounding conversational error, rather than merely measuring robustness to a fixed prompt template.

3. Dynamic evaluation setting

KnowMT-Bench uses a dynamic evaluation setting in which the model generates the dialogue history turn by turn. For dialogue $d$, the model produces earlier answers as

$$a^{(d)}_t = \mathcal{M}(q^{(d)}_1,a^{(d)}_1,\dots,q^{(d)}_{t}),\quad t=1,\dots,N_d-1$$

and the final-turn answer as

$$a^{(d)}_{N_d}=\mathcal{M}(H^{(d)}_{N_d},\,q^{(d)}_{N_d}).$$

This setup is intended to simulate real-world interactive sessions without handcrafted histories (Chen et al., 26 Sep 2025). The crucial design choice is that the benchmark does not inject gold or externally curated previous answers; the model must live with its own earlier outputs. That makes the final-turn answer a function not only of the question sequence but also of accumulated self-generated context.

The significance of this design lies in what it isolates. If performance drops in multi-turn settings, the degradation can arise from the model’s own conversational artifacts rather than from any externally imposed history. This differs from static multi-turn evaluation, where a benchmark designer provides fixed context and the model only answers the last question. KnowMT-Bench therefore operationalizes a more realistic notion of conversational factuality.

A frequent assumption is that longer dialogue alone explains any observed degradation. The later history-replacement experiments complicate that interpretation: the benchmark evidence indicates that self-generated history noise, rather than length per se, is the primary driver of factual decline.

4. Evaluation pipeline and metrics

KnowMT-Bench evaluates the final-turn answer through a two-stage automated pipeline. First, the ground-truth answer $g_j$ is decomposed into an atomic fact set $\mathcal{F}_j$, and the generated answer $\mathcal{E}_i$0 is decomposed into a statement set $\mathcal{E}_i$1. Second, the benchmark applies symmetric NLI judgments: correctness evaluates each $\mathcal{E}_i$2 against $\mathcal{E}_i$3, and completeness evaluates each $\mathcal{E}_i$4 against $\mathcal{E}_i$5, with labels in $\mathcal{E}_i$6 (Chen et al., 26 Sep 2025).

The benchmark reports three classes of metrics: factual capability, hallucination reliability, and information delivery efficiency.

For factual capability, the reported metrics are:

$\mathcal{E}_i$7

$\mathcal{E}_i$8

$\mathcal{E}_i$9

For hallucination (reliability), the metrics are:

$q_j$0

$q_j$1

$q_j$2

For information delivery efficiency, the metrics are:

$q_j$3

$q_j$4

$q_j$5

with terms having zero denominators smoothed via dataset maxima.

The benchmark also reports human validation of the automated pipeline on 100 sampled answers. For atomic fact decomposition, the decomposer achieved SMAPE=18.1% and omission rate=5.9%. For the NLI evaluator, agreement with human majority yielded F1=83.6% and Cohen’s $q_j$6–$q_j$7 (Chen et al., 26 Sep 2025). These numbers do not imply human-level semantic judgment, but they do indicate that the automated evaluator was explicitly checked against human annotation rather than adopted without validation.

5. Empirical findings

Across 12 LLMs—including Gemini-2.5-Pro, GPT-4o, Llama-3, Qwen-2.5 family, and DeepSeek—KnowMT-Bench reports a consistent deterioration from single-turn to multi-turn evaluation. The average changes are $q_j$8, $q_j$9, $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$0, and $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$1 (Chen et al., 26 Sep 2025). In the benchmark’s interpretation, multi-turn context degrades factual capability and also reduces information efficiency because models become more verbose as dialogue length increases.

The results distinguish two effects. First, with respect to efficiency, analysis by dialogue length shows that $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$2 and $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$3 degrade monotonically with more turns, indicating a verbosity effect. Second, with respect to factuality, $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$4 and $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$5 show no clear trend with dialogue length alone. A history-quality replacement experiment sharpens that distinction: conditioning weaker models on GPT-4o-generated histories yields +15–20% higher $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$6 and lower $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$7, which the paper presents as confirmation that self-generated history noise, rather than sheer length, drives factual degradation.

The benchmark also reports that larger models (GPT-4o) form a performance frontier, while Chain-of-Thought–optimized variants show limited gains in factuality. This is important because it constrains a common expectation that general reasoning optimization will automatically transfer to robust conversational factuality. In KnowMT-Bench, such gains appear limited relative to the benchmark’s factual metrics.

Two visual summaries are described. In the Factuality Scatter (Figure 4a), multi-turn settings shift to top-left with decreased $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$8 and increased $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$9. In the Efficiency Scatter (Figure 4b), multi-turn settings shift to top-right with increased $q^{(d)}_{N_d}=q_j$0 and increased $q^{(d)}_{N_d}=q_j$1. Together, these patterns characterize multi-turn degradation as both less factual and less efficient.

6. Mitigation strategies, interpretation, and future directions

KnowMT-Bench evaluates several mitigation strategies. For domain-specific fine-tuning, HuatuoGPT in medicine shows +4.6% in $q^{(d)}_{N_d}=q_j$2 single-turn and +5.4% multi-turn, whereas Fin-R1 in finance yields modest/no consistent gains (Chen et al., 26 Sep 2025). The reported pattern suggests that domain adaptation can help, but its effectiveness is uneven across settings and models.

The most substantial mitigation comes from retrieval-augmented generation (RAG). The reported setup uses Qwen3-Embedding-0.6B + FAISS IndexFlatL2 as retriever with 15 candidates, Qwen3-Reranker-0.6B as reranker, and a final context of top-5 chunks built from 512-token chunks with 128 overlap. Four multi-turn retrieval strategies are tested:

Strategy	Description
Base	last question
Last	full history at final turn
Rounds	each turn retrieval on current query
All	history at every turn

Among these, Rounds is reported as the best strategy. For Qwen-2.5-7B, comparing Rounds with no RAG yields:

Setting	$q^{(d)}_{N_d}=q_j$3	$q^{(d)}_{N_d}=q_j$4	$q^{(d)}_{N_d}=q_j$5
Single-turn no RAG	19.7%	3.6%	366
Single-turn RAG-Rounds	43.2%	2.1%	216
Multi-turn no RAG	17.2%	3.8%	547
Multi-turn RAG-Rounds	43.1%	2.1%	216

The paper states that RAG-Rounds even surpasses single-turn RAG baselines, fully reversing factual decline. Within the benchmark’s scope, this makes per-round retrieval the clearest demonstrated intervention for MT-LFQA robustness.

A lighter-weight intervention, prompt-based noise filtering, uses a system prompt instructing the model to “focus on internal knowledge, ignore irrelevant context.” This improves $q^{(d)}_{N_d}=q_j$6 (+9%) and $q^{(d)}_{N_d}=q_j$7 (–19%) but raises $q^{(d)}_{N_d}=q_j$8 (+2%), indicating a trade-off and the need for more nuanced interventions. The benchmark therefore does not present prompt filtering as a complete solution.

The benchmark’s main conclusions are that multi-turn dialogue context significantly degrades factual capability via self-generated noise, increases verbosity, and causes single-turn LFQA benchmarks to overestimate real-world performance in knowledge-intensive applications (Chen et al., 26 Sep 2025). The reported future directions are to incorporate retrieval at every conversational turn, develop noise-aware prompt engineering or context-filtering modules, expand the benchmark to domains such as science and engineering and to multilingual settings, and investigate advanced context selection and summarization techniques as orthogonal noise-reduction strategies. A plausible implication is that KnowMT-Bench positions MT-LFQA evaluation as a joint problem of retrieval, context management, and long-form factual synthesis rather than as a straightforward extension of single-turn QA.