MT-bench: Multi-Turn AI Evaluation

• MT-bench Benchmarks are a suite of high-fidelity evaluation protocols that assess multi-turn, context-sensitive, and multimodal capabilities in modern AI systems.
• They employ diverse methods such as pairwise comparisons, numeric scoring, and checklist evaluations to provide actionable insights into model performance and potential vulnerabilities.
• Extended variants like MT-Bench-101 and MT-Video-Bench enable fine-grained diagnostics for LLM reasoning, multimodal interactions, and time-series analysis, highlighting ongoing challenges in dialogue robustness.

MT-bench Benchmarks provide a family of targeted, high-fidelity evaluation protocols, question sets, and metrics designed to rigorously assess modern AI systems—most prominently LLMs, RL agents, and multimodal models—on multi-task, multi-turn, and context-sensitive capabilities. While the term “MT-bench” has been used in several disciplinary areas, its foundational usage is as a suite for LLM evaluation via curated dialog tasks, with subsequent adaptations for robotics, time-series reasoning, and multimodal dialogue. The following sections enumerate the principal instantiations, experimental methodologies, key findings, recent vulnerabilities, and ongoing directions in the development and application of MT-bench variants.

1. Origins: MT-bench for LLM Multi-turn Evaluation

The original MT-bench was introduced as an LLM-centric benchmark to assess open-ended dialogue, reasoning, coding, math, and knowledge retention across 80 carefully designed two-turn user–assistant questions (Zheng et al., 2023). Each set contains a first-turn user prompt and a follow-up, spanning eight capability categories: Writing, Roleplay, Information Extraction, Reasoning, Math, Coding, Knowledge I (STEM), and Knowledge II (Humanities/Social Science). Unlike single-turn benchmarks (e.g., MMLU, BBH), MT-bench explicitly targets the evolving context demands of conversational AI.

Evaluation involves both human expert raters—providing approximately 3,000 annotated votes across the set—and LLM “judges”, most commonly strong generalist models such as GPT-4. Judging methodologies include both pairwise (A/B) comparison with bias mitigation and single-turn numeric ratings. Agreement between LLM judges and human experts on the benchmark is high: for example, GPT-4’s judgments agree with human votes at 85% (second turn, non-tied) (Zheng et al., 2023).

MT-bench data, response outputs, and annotation scripts are released at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge, enabling reproducible evaluation and leaderboard tracking.

2. Design Extensions: Fine-grained and Multi-modal MT-bench

Subsequent work extended MT-bench along three axes: dialogue depth, multimodal context, and cross-modal reasoning.

2.1 MT-Bench-101: Hierarchical Multi-turn Dialogue Benchmark

MT-Bench-101 (Bai et al., 2024) addresses the coarse granularity of original multi-turn benchmarks by providing a three-tier hierarchical ability taxonomy—Perceptivity, Adaptability, Interactivity—covering 13 tasks over 1,388 dialogues and 4,208 turns. Tasks include context memory, anaphora resolution, topic shift, content rephrasing, mathematical reasoning, and proactive interaction. Aggregation uses a minimum-turn metric, penalizing dialogues for single defective responses: $s_{\rm dialogue} = \min_{i \in {\rm turns}} s_i,$ where $s_i$ is the judge score per turn. GPT-4 alignment with human annotation exceeds 87%.

Task-level diagnostic reveals weaknesses in reasoning and proactive questioning, and provides actionable error profiles: e.g., poor handling of topic shifts or premature responses in Separate Input. Chat-specific SFT or RLHF methods do not strongly improve multi-turn abilities over foundation model variants.

2.2 MT-Video-Bench: Multi-turn, Multimodal Dialogue

MT-Video-Bench (Pan et al., 20 Oct 2025) introduces multi-turn, video-grounded dialogue evaluation for multimodal LLMs (MLLMs). The benchmark consists of 987 dialogues (5,805 QA pairs) from 135 source videos across five domains, operationalizing six core competencies: three in perceptual fidelity (Object Reference, Memory Recall, Content Summary) and three in interactive abilities (Answer Refusal, Topic Shift, Proactive Interaction).

Scoring leverages multi-turn accuracy over checklist evaluations: $$\mathrm{ACC} = \frac{\sum_{i=1}^{N}\sum_{j=1}^{Q_i} \mathbb{I}[\hat{c}_{ij}=c_{ij}]}{\sum_{i=1}^{N}Q_i},$$ where $Q_i$ is the number of items for the $i$th QA pair. Top closed-source MLLMs (Gemini 2.5 Pro) peak at 68.45% overall, with interactivity subtasks substantially lagging (PI: 55.12%).

2.3 Multimodal Time Series Benchmarks

A specialized instantiation of MTBench (as “MTBench” (Chen et al., 21 Mar 2025)) evaluates LLMs on temporally aligned multimodal data (financial news+stock series, weather reports+temperature series), providing four task types: time-series forecasting, semantic trend analysis, technical indicator prediction, and news-driven question answering. The testbed reveals LLMs’ difficulty in handling long-range temporal dependencies and cross-modal causality, with best results in short-horizon, text-assisted settings.

3. Protocols, Annotation, and Judge Methodologies

For LLM benchmarks, the corpus is either human-authored or LLM-constructed, with task/review pipelines designed for high coverage and ambiguity avoidance (Zheng et al., 2023, Bai et al., 2024, Pan et al., 20 Oct 2025). Evaluation protocols vary:

• Pairwise Comparison: GPT-4 or similar model judges are presented with anonymized model outputs for each turn; swap-and-tie logic counteracts position bias.
• Single-turn Scoring: Numeric scores (1–10) per turn are averaged or minimum-pooled.
• Checklists: For complex or multimodal tasks, auto-generated itemized checklists measure correctness, relevance, and specific capabilities.

Human expert validation is included as a gold standard; LLM–human agreement levels meet or surpass inter-human agreement norms in all studies (Zheng et al., 2023, Bai et al., 2024).

4. Quantitative Findings and Model Insights

The following table summarizes representative quantitative results from two key MT-bench variants:

Benchmark	Top Model	Metric	Score	Notable Gap
MT-bench (Zheng et al., 2023)	GPT-4	Agreement	85%	GPT-4 vs human (second turn)
MT-Bench-101 (Bai et al., 2024)	GPT-4	Min-Turn	8.86	Reasoning and questioning are bottlenecks
MT-Video-Bench (Pan et al., 20 Oct 2025)	Gemini 2.5 Pro	Accuracy	68.45%	PI lowest per-task; cross-scene drops ~15 pp
MTBench (Chen et al., 21 Mar 2025) (TS+Text)	GPT-4o	Finance QC	9.8% MAE impr.	Text improves short-term, lags in long-term

Performance patterns reiterate the challenges: interactivity and deep multi-turn reasoning remain weak across models and modalities; model scaling alone is insufficient for core dialogue advances.

5. Vunerabilities: Template Attacks and Benchmark Robustness

Recent evidence exposes critical vulnerabilities in the automatic LLM evaluation pipeline. MT-Bench’s reliance on a fixed prompt template and a single-model judge makes it susceptible to structured “null model” cheating (Zheng et al., 2024). Adversaries can prepend answer content with tokens that hijack the template parsing (“[[…]] [[rating]]”), causing the judge to emit inflated scores independent of genuine answer quality. Random-search optimized adversarial prefixes, tuned on public instructions, transfer directly to private MT-Bench evaluation, achieving scores of 9.55 (versus SOTA 8.96) (Zheng et al., 2024).

Defenses such as template paraphrasing and perplexity filters have proven inadequate. Suggested hardening approaches include randomized multi-template prompting, judge ensembling, adversarial judge fine-tuning, human-in-the-loop audits, and structural consistency verification.

6. Significance, Impact, and Future Directions

MT-bench benchmarks serve as stress-tests of LLM and multimodal model capabilities where context, interactivity, and robustness are key. Their fine-grained diagnostic structure informs model development (e.g., RLHF, DPO, chat-specific SFT), and their adoption in leaderboards and open repositories drives community standardization. Nonetheless, the demonstrated gaming potential of current evaluation frameworks necessitates immediate innovation in anti-cheating mechanisms and dynamic, adversarially resistant protocols.

Future MT-bench extensions are anticipated along modality axes (audio, image, further structured data), adversarial user simulation, and domain generalization. Harmonization across MT-bench, MT-Image-Bench, and MT-Text-Bench is underway to support unified, multimodal, multi-turn dialogue diagnostics (Pan et al., 20 Oct 2025). Continued model advances will require corresponding evolution in both the breadth and security of benchmark instrumentation.