---
title: 'KnowMT-Bench: Multi-Turn LFQA Benchmark'
url: https://www.emergentmind.com/topics/knowmt-bench
type: topic
---

# KnowMT-Bench: Multi-Turn LFQA Benchmark

KnowMT-Bench is a benchmark for **Multi-Turn Long-Form Question Answering (MT-LFQA)** in knowledge-intensive domains, introduced to evaluate how large language models perform when factual long-form answering must be sustained across dialogue turns rather than in a single isolated exchange. It is described as the **first-ever** benchmark designed to systematically evaluate MT-LFQA across medicine, finance, and law, with a dynamic setting in which models generate their own dialogue histories from logically progressive question sequences and are then evaluated on the **final-turn** answer for both factual capability and information delivery efficiency [2509.21856]. The benchmark is motivated by the observation that real-world consultations in medicine, finance, and law are inherently multi-turn, whereas prior single-turn LFQA benchmarks and existing dialogue benchmarks do not jointly test paragraph-level synthesis, factuality, and the effects of self-generated conversational context.

## 1. Scope, motivation, and relation to prior benchmarks

KnowMT-Bench targets a gap between two established evaluation traditions. On one side are single-turn LFQA benchmarks such as **K-QA**, **FinTextQA**, and **cLegal-QA**, which provide one question–answer pair and therefore do not capture the complications introduced by dialogue history. On the other side are conversational QA and dialogue benchmarks such as **QuAC**, **CoQA**, **MT-Bench**, **BotChat**, and **MT-Eval**, which assess short extractive replies or broader conversational capabilities such as alignment and fairness rather than knowledge-intensive factuality [2509.21856].

The benchmark is designed around the premise that MT-LFQA requires two properties simultaneously: **high factual accuracy** and **coherent, concise delivery** under potentially noisy multi-turn context. In the benchmark framing, single-turn QA and MT-LFQA differ not merely in the number of turns but in the failure modes induced by conditioning on prior exchanges: redundancy, contextual distraction, and hallucinations. KnowMT-Bench therefore evaluates paragraph-level synthesis of multiple facts rather than short-form extraction.

A common misconception is that multi-turn evaluation can be approximated by existing dialogue benchmarks that already include several turns. KnowMT-Bench rejects that equivalence. The benchmark’s emphasis is not generic conversation quality but factual long-form answering in knowledge-intensive settings. This suggests that performance on alignment- or style-oriented dialogue benchmarks is not, by itself, an adequate proxy for performance in consultation-like factual interactions.

## 2. Dataset composition and benchmark construction

KnowMT-Bench is built from **801 high-quality LFQA pairs** spanning three domains: **finance (579)**, **law (278)**, and **medicine (209)**; **33% of instances overlap finance and legal** [2509.21856]. Each answer $g_i$ is grounded in an authoritative evidence set $\mathcal{E}_i$, such as official websites or expert-curated documents. The benchmark therefore begins with evidence-grounded single-turn LFQA instances and then transforms them into multi-turn dialogue settings.

Dialogue lengths are sampled according to **ShareGPT usage**, with **2–5 turns** and proportions **37.5%, 37.5%, 15.0%, 10.1%**. For each base question $q_j$, a human-reviewed LLM prompt expansion generates preceding questions $q^{(d)}_1,\dots,q^{(d)}_{N_d-1}$ according to three principles:

1. **Progressive Context Building**
2. **Intent Preservation**
3. **No Answer Leakage**

The final question is fixed as $q^{(d)}_{N_d}=q_j$, which ensures semantic alignment with the original single-turn QA. This construction preserves the original target information need while embedding it in a dialogue trajectory that resembles a progressive consultation.

Single-turn answers were refined by graduate annotators, and multi-turn expansions were templated and manually checked. The resulting benchmark is not based on handcrafted dialogue histories; instead, it is constructed to let models populate the history themselves during evaluation. A plausible implication is that the benchmark is intended to capture compounding conversational error, rather than merely measuring robustness to a fixed prompt template.

## 3. Dynamic evaluation setting

KnowMT-Bench uses a **dynamic evaluation setting** in which the model generates the dialogue history turn by turn. For dialogue $d$, the model produces earlier answers as

$$
a^{(d)}_t = \mathcal{M}(q^{(d)}_1,a^{(d)}_1,\dots,q^{(d)}_{t}),\quad t=1,\dots,N_d-1
$$

and the final-turn answer as

$$
a^{(d)}_{N_d}=\mathcal{M}(H^{(d)}_{N_d},\,q^{(d)}_{N_d}).
$$

This setup is intended to simulate real-world interactive sessions without handcrafted histories [2509.21856]. The crucial design choice is that the benchmark does not inject gold or externally curated previous answers; the model must live with its own earlier outputs. That makes the final-turn answer a function not only of the question sequence but also of accumulated self-generated context.

The significance of this design lies in what it isolates. If performance drops in multi-turn settings, the degradation can arise from the model’s own conversational artifacts rather than from any externally imposed history. This differs from static multi-turn evaluation, where a benchmark designer provides fixed context and the model only answers the last question. KnowMT-Bench therefore operationalizes a more realistic notion of conversational factuality.

A frequent assumption is that longer dialogue alone explains any observed degradation. The later history-replacement experiments complicate that interpretation: the benchmark evidence indicates that **self-generated history noise**, rather than length per se, is the primary driver of factual decline.

## 4. Evaluation pipeline and metrics

KnowMT-Bench evaluates the final-turn answer through a **two-stage automated pipeline**. First, the ground-truth answer $g_j$ is decomposed into an atomic fact set $\mathcal{F}_j$, and the generated answer $a_j$ is decomposed into a statement set $\mathcal{S}_j$. Second, the benchmark applies **symmetric NLI judgments**: correctness evaluates each $s\in\mathcal{S}_j$ against $g_j$, and completeness evaluates each $f\in\mathcal{F}_j$ against $a_j$, with labels in $\{\text{Entailment}, \text{Contradiction}, \text{Neutral}\}$ [2509.21856].

The benchmark reports three classes of metrics: factual capability, hallucination reliability, and information delivery efficiency.

For **factual capability**, the reported metrics are:

$$
R_f \;=\;\frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{|\mathcal{F}_j^+|}{|\mathcal{F}_j|}
$$

$$
P_f \;=\;\frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{|\mathcal{S}_j^+|}{|\mathcal{S}_j|}
$$

$$
S_f \;=\;\frac{2\,P_f\,R_f}{P_f+R_f}
$$

For **hallucination (reliability)**, the metrics are:

$$
R_m \;=\;\frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{|\mathcal{F}_j^-|}{|\mathcal{F}_j|}
$$

$$
P_{fc} \;=\;\frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{|\mathcal{S}_j^-|}{|\mathcal{S}_j|}
$$

$$
S_h \;=\;\frac{2\,P_{fc}\,R_m}{P_{fc}+R_m}
$$

For **information delivery efficiency**, the metrics are:

$$
D_f = \frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{T(a_j)}{|\mathcal{F}_j^+|}
$$

$$
D_h = \frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{T(a_j)}{|\mathcal{F}_j^-|}
$$

$$
D_R = \frac{1}{|\mathcal{D}|}\sum_{j\in\mathcal{D}} \frac{T(a_j)}{|\mathcal{F}_j^+|/|\mathcal{F}_j|}
$$

with terms having zero denominators smoothed via dataset maxima.

The benchmark also reports human validation of the automated pipeline on **100 sampled answers**. For atomic fact decomposition, the decomposer achieved **SMAPE=18.1%** and **omission rate=5.9%**. For the NLI evaluator, agreement with human majority yielded **F1=83.6%** and **Cohen’s $\kappa \approx 0.60$–$0.63$** [2509.21856]. These numbers do not imply human-level semantic judgment, but they do indicate that the automated evaluator was explicitly checked against human annotation rather than adopted without validation.

## 5. Empirical findings

Across **12 LLMs**—including **Gemini-2.5-Pro, GPT-4o, Llama-3, Qwen-2.5 family, and DeepSeek**—KnowMT-Bench reports a consistent deterioration from single-turn to multi-turn evaluation. The average changes are **$\Delta S_f\approx -12\%$**, **$\Delta S_h\approx +9\%$**, **$\Delta D_f\approx +30\%$**, and **$\Delta D_h\approx +5\%$** [2509.21856]. In the benchmark’s interpretation, multi-turn context degrades factual capability and also reduces information efficiency because models become more verbose as dialogue length increases.

The results distinguish two effects. First, with respect to **efficiency**, analysis by dialogue length shows that **$D_f$** and **$D_h$** degrade monotonically with more turns, indicating a verbosity effect. Second, with respect to **factuality**, **$S_f$** and **$S_h$** show no clear trend with dialogue length alone. A history-quality replacement experiment sharpens that distinction: conditioning weaker models on **GPT-4o-generated histories** yields **+15–20% higher $S_f$** and lower **$S_h$**, which the paper presents as confirmation that **self-generated history noise**, rather than sheer length, drives factual degradation.

The benchmark also reports that **larger models (GPT-4o) form a performance frontier**, while **Chain-of-Thought–optimized variants show limited gains in factuality**. This is important because it constrains a common expectation that general reasoning optimization will automatically transfer to robust conversational factuality. In KnowMT-Bench, such gains appear limited relative to the benchmark’s factual metrics.

Two visual summaries are described. In the **Factuality Scatter (Figure 4a)**, multi-turn settings shift **to top-left** with **decreased $S_f$** and **increased $S_h$**. In the **Efficiency Scatter (Figure 4b)**, multi-turn settings shift **to top-right** with **increased $D_f$** and **increased $D_h$**. Together, these patterns characterize multi-turn degradation as both less factual and less efficient.

## 6. Mitigation strategies, interpretation, and future directions

KnowMT-Bench evaluates several mitigation strategies. For **domain-specific fine-tuning**, **HuatuoGPT** in medicine shows **+4.6% in $S_f$ single-turn** and **+5.4% multi-turn**, whereas **Fin-R1** in finance yields **modest/no consistent gains** [2509.21856]. The reported pattern suggests that domain adaptation can help, but its effectiveness is uneven across settings and models.

The most substantial mitigation comes from **retrieval-augmented generation (RAG)**. The reported setup uses **Qwen3-Embedding-0.6B + FAISS IndexFlatL2** as retriever with **15 candidates**, **Qwen3-Reranker-0.6B** as reranker, and a final context of **top-5 chunks** built from **512-token chunks with 128 overlap**. Four multi-turn retrieval strategies are tested:

| Strategy | Description |
|---|---|
| Base | last question |
| Last | full history at final turn |
| Rounds | each turn retrieval on current query |
| All | history at every turn |

Among these, **Rounds** is reported as the best strategy. For **Qwen-2.5-7B**, comparing **Rounds** with **no RAG** yields:

| Setting | $S_f$ | $S_h$ | $D_f$ |
|---|---:|---:|---:|
| Single-turn no RAG | 19.7% | 3.6% | 366 |
| Single-turn RAG-Rounds | 43.2% | 2.1% | 216 |
| Multi-turn no RAG | 17.2% | 3.8% | 547 |
| Multi-turn RAG-Rounds | 43.1% | 2.1% | 216 |

The paper states that **RAG-Rounds even surpasses single-turn RAG baselines, fully reversing factual decline**. Within the benchmark’s scope, this makes per-round retrieval the clearest demonstrated intervention for MT-LFQA robustness.

A lighter-weight intervention, **prompt-based noise filtering**, uses a system prompt instructing the model to **“focus on internal knowledge, ignore irrelevant context.”** This improves **$S_f$ (+9%)** and **$D_f$ (–19%)** but raises **$S_h$ (+2%)**, indicating a trade-off and the need for more nuanced interventions. The benchmark therefore does not present prompt filtering as a complete solution.

The benchmark’s main conclusions are that multi-turn dialogue context significantly degrades factual capability via self-generated noise, increases verbosity, and causes single-turn LFQA benchmarks to overestimate real-world performance in knowledge-intensive applications [2509.21856]. The reported future directions are to incorporate retrieval at every conversational turn, develop noise-aware prompt engineering or context-filtering modules, expand the benchmark to domains such as **science** and **engineering** and to **multilingual settings**, and investigate advanced context selection and summarization techniques as orthogonal noise-reduction strategies. A plausible implication is that KnowMT-Bench positions MT-LFQA evaluation as a joint problem of retrieval, context management, and long-form factual synthesis rather than as a straightforward extension of single-turn QA.

Source: https://www.emergentmind.com/topics/knowmt-bench