---
title: 'MMLBD-C: Correcting MMLongBenchDoc'
url: https://www.emergentmind.com/topics/mmlbd-c
type: topic
---

# MMLBD-C: Correcting MMLongBenchDoc

Searching arXiv for the exact paper and benchmark context.
arXiv search query: 2602.15257 MMLBD-C MMLongBenchDoc
MMLBD-C is a manually corrected, quality-filtered version of MMLongBenchDoc, introduced as “MMLBD-C: correcting MMLongBenchDoc” in "How to Train Your Long-Context Visual Document Model" [2602.15257]. It is a corrected derivative rather than a new benchmark from scratch, preserving the same overall task domain—long-context document VQA over visual documents—while addressing annotation and evaluation defects in the original benchmark. Its stated purpose is to provide a more reliable benchmark for long-context visual document understanding, especially long-document VQA, by reducing erroneous and low quality examples [2602.15257].

## 1. Definition and benchmark role

MMLBD-C starts from MMLongBenchDoc and changes example content only where needed: keep as-is, correct question, correct answer, or remove the example [2602.15257]. The benchmark therefore remains in the same benchmark family and targets the same ability, but with higher annotation quality. The paper evaluates on both MMLongBenchDoc and MMLBD-C and includes both in the aggregate Visual-LC Avg (VA), defined over MMLongBenchDoc, MMLBD-C, MMLongBench, DUDE, and SlideVQA [2602.15257].

The benchmark is positioned as especially relevant to long-document visual question answering. The authors state that MMLBD-C is used as a tiebreaker in their aggregate evaluation because it is “the most relevant benchmark to our work” [2602.15257]. They also report that MMLBD-C scores correlate highly with MMLongBenchDoc scores, though they are generally higher, which the benchmark description associates with cleaning away bad or unfair examples rather than with a redesign of the task [2602.15257].

This suggests that MMLBD-C functions simultaneously as a correction layer and as a model-selection instrument. A plausible implication is that it is intended not to replace historical reporting on MMLongBenchDoc, but to provide a cleaner measurement point within the same evaluation lineage.

## 2. Motivation: annotation pathologies in MMLongBenchDoc

MMLBD-C was introduced in response to quality problems identified in MMLongBenchDoc [2602.15257]. The paper describes several classes of benchmark flaws that can distort model evaluation.

First, it identifies incorrect question-document pairing. One cited example is the question “List all the PM health effects that increse by more than 35% in India and Thailand.” paired with an unrelated document about digital marketing [2602.15257]. Such mismatches invalidate the retrieval and reasoning demands of the task.

Second, it identifies underspecified or ambiguous questions. The example “List all the sections that discuss about the experiment setup?” was paired with the answer `['Section 4.1', 'Section 4.2', 'Section 4.3', 'Appendix A']`, but the authors note that it is hard to argue that the Methodology section does not discuss the experiment setup [2602.15257]. This makes the evaluation sensitive to annotation choices rather than underlying model competence.

Third, it identifies typos that change answerability. The example “How do Amazon recognize least cost?” should read “lease cost” [2602.15257]. Because “least cost” is itself plausible in context, such errors can induce spurious failures.

Fourth, it identifies incorrect answers. The paper gives an example in which “How many percentage respondents in this survey access to internet more than two times per month?” was labeled unanswerable even though the answer is explicitly derivable from the document [2602.15257].

Fifth, it identifies overly narrow answer acceptance for unanswerable questions. For “Not answerable” questions, equivalent responses such as “None,” “0,” and “No one” should sometimes also be accepted [2602.15257]. This is not a change in task definition, but a change in answer normalization and acceptance criteria.

The benchmark motivation is thus explicitly about benchmark reliability. If question-document pairs, wording, or answers are wrong or misleading, then reported scores become noisy or unfair measures of actual long-document understanding [2602.15257]. The release of MMLBD-C is framed as a response to this evaluation noise.

## 3. Correction pipeline and editorial interventions

The construction of MMLBD-C combines automatic inconsistency flagging with human review [2602.15257]. The authors state that they “apply a version of the recursive pipeline … adapted to find inconsistencies between the source, question and answer” [2602.15257]. The role of this adapted pipeline is to flag suspicious benchmark items rather than to replace human judgment.

A total of 342 examples were flagged for review and then manually reviewed [2602.15257]. For each flagged example, the paper reports four possible actions: leave as is, modify the question, modify the answer, or remove from the benchmark [2602.15257]. The correction criteria are therefore source-document consistency, question adequacy, answer correctness, and recoverability.

The benchmark modifications are summarized below.

| Quantity | Count |
|---|---:|
| Original MMLongBenchDoc examples | 1091 |
| Flagged for review | 342 |
| Modified | 251 |
| Removed | 16 |

The paper explicitly states that it modifies “251 out of 1091” examples for errors, incorrect grammar or misleading/underspecified questions and removes 16 [2602.15257]. It also states, with respect to document mismatches, “We remove 9 of 10 affected questions and convert the last to ‘Not answerable’” [2602.15257]. The corrected fraction is \(251 / 1091 \approx 23.0\%\), the removed fraction is \(16 / 1091 \approx 1.47\%\), the total affected fraction is \(267 / 1091 \approx 24.5\%\), and the flagged fraction is \(342 / 1091 \approx 31.4\%\) [2602.15257].

The paper clearly indicates that the flagged examples were manually reviewed, but it does not specify the number of annotators, whether the annotators were authors or external crowdworkers, the full annotation guidelines, the adjudication procedure, or inter-annotator agreement [2602.15257]. The safe factual conclusion is therefore limited: human manual review was involved, but the annotator protocol is not fully specified.

The classes of correction are explicitly described as incorrect question-document pairing, underspecified or misleading wording, typos or incorrect grammar, incorrect answers, and expanded acceptable answer forms for “Not answerable” items [2602.15257]. This suggests that MMLBD-C is a curation and repair effort rather than a redistribution or rebalancing effort.

## 4. Dataset continuity, scale, and task composition

MMLBD-C preserves the overall task domain of MMLongBenchDoc: long-document visual question answering on visual documents or PDF pages [2602.15257]. The paper does not report a redesign of task categories or question-type composition, and it presents MMLBD-C as a correction layer rather than a rebalanced benchmark [2602.15257]. Accordingly, the task/domain composition appears largely unchanged, with individual examples changed or removed.

The core dataset statistics are inherited from the correction process. MMLongBenchDoc contains 1091 examples, 16 are removed, and the implied final size of MMLBD-C is 1075 examples [2602.15257]. The paper does not explicitly print “1075,” but that value follows directly from the reported counts.

The paper also does not mention any new train/dev/test split or altered evaluation split for MMLBD-C [2602.15257]. It appears to be a corrected evaluation set corresponding to MMLongBenchDoc’s evaluation use. Likewise, the paper does not provide a standalone table for MMLBD-C category counts, average document length, or modality breakdown [2602.15257]. What is stated is that the benchmark remains a visual long-context document QA benchmark.

There is, however, a slight change in answer format handling. The benchmark broadens acceptable answers for some “Not answerable” items by accepting equivalents such as “None,” “0,” and “No one” where appropriate [2602.15257]. This is an evaluation-format change at the answer normalization level rather than a task redesign.

## 5. Evaluation protocol and long-context settings

The primary metric for MMLBD-C is F1, specifically the benchmark’s `overall_f1` [2602.15257]. The paper does not provide a new formula for this F1 and treats it as inherited from benchmark tooling. MMLBD-C is therefore scored with F1 rather than exact match [2602.15257].

The evaluation setup includes several practical long-context modifications. In contrast to the default VLM Eval Kit settings, the paper increases the maximum number of pages from 120 to 336 for MMLongBenchDoc and MMLBD-C and sets the maximum resolution to \(1024 \times 1024\) to ensure long examples fit in context while preserving fine details [2602.15257]. These settings are central to MMLBD-C evaluation because they determine whether long document inputs fit within the model’s usable context.

A further intervention is the use of page indices. The paper prepends page labels in the input context in the form:

```text
Page 1: <image>
Page 2: <image>
Page 3: <image>
...
```

[2602.15257]

The paper reports that adding page indices during both training and evaluation improves MMLBD-C by +2.8 in one configuration, and emphasizes that including page indices during training is necessary to benefit during evaluation [2602.15257]. Within the paper’s ablations, this is one of the strongest benchmark-specific interventions.

MMLBD-C also appears in two higher-level aggregate metrics. Visual-LC Avg (VA) averages over MMLongBenchDoc, MMLBD-C, MMLongBench, DUDE, and SlideVQA, while LC Avg (LCA) averages over visual-LC benchmarks plus HELMET and LongBench v2 [2602.15257]. The paper states that benchmark scores are normalized by benchmark-specific maximums before averaging because the component benchmarks have different ranges and distributions [2602.15257].

## 6. Empirical behavior and benchmark significance

The paper states that MMLBD-C scores correlate highly with MMLongBenchDoc scores, but are generally higher [2602.15257]. The stated interpretation is that the corrected benchmark is less unfair or noisy, not that it is necessarily less challenging in an underlying cognitive sense. This distinction is important because a higher score on MMLBD-C does not imply lower task relevance; it may instead reflect lower annotation error.

In the paper’s best-checkpoint table, several concrete MMLBD-C scores are reported [2602.15257]. These include Qwen3 VL 235B A22B at 56.2, LongPO Short Stage at 56.4, Qwen3 VL 32B at 53.8, Qwen3 VL Plain Distillation Short Stage at 57.3, Mistral Plain Distillation at 47.4, and Mistral 3.1 Small (24B) at 41.4 [2602.15257]. The strongest reported score in that table is Qwen3 VL Plain Distillation Short Stage at 57.3 [2602.15257].

The paper uses MMLBD-C to distinguish among training strategies. For example, it states: “For best MMLBD-C performance specifically, plain distillation with SFT is more effective” [2602.15257]. This differs from the broader Visual-LC Avg, where LongPO is often strongest [2602.15257]. In the reported comparison, LongPO Short Stage reaches 56.4 on MMLBD-C, while Qwen3 VL Plain Distillation Short Stage reaches 57.3 [2602.15257].

The paper also uses MMLBD-C to support the claim that training on context lengths that match evaluation context lengths outperforms training on longer contexts [2602.15257]. Representative values are Mistral Short Stage at 45.0 versus Mistral Short + Long at 43.3, Qwen3VL Short Stage at 57.3 versus Qwen3VL Short + Long at 57.0, and LongPO Short Stage at 56.4 versus LongPO Short + Long at 54.0 [2602.15257]. Within this empirical frame, MMLBD-C serves as a benchmark sensitive to context-length alignment and training objective choice.

The paper explicitly argues that MMLBD-C is a more reliable measure of long-context visual document understanding than the raw MMLongBenchDoc benchmark because it was created to reduce erroneous and low quality examples, it is manually corrected, its annotations are released for public inspection, and it is used as a tiebreaker in evaluation [2602.15257]. A plausible implication is that MMLBD-C is intended as a benchmark-quality intervention as much as a benchmarking artifact.

## 7. Limitations, interpretation, and research use

MMLBD-C is presented as an improved benchmark, but the paper also leaves several limits explicit or implicit [2602.15257]. The manual correction protocol is not fully specified, since annotator counts and agreement statistics are not reported. The benchmark also remains highly correlated with MMLongBenchDoc, so it is an improvement in quality rather than a complete redefinition of the task [2602.15257]. In addition, the broader benchmark ecosystem used in the paper may under-represent extreme contexts, since most benchmarks are under 128K tokens and thus do not fully validate performance at the model’s maximum context such as 344K [2602.15257].

For practical use, the paper supports a specific evaluation recipe for long-document VQA on MMLBD-C: use F1 (`overall_f1`), increase maximum pages to 336, set image resolution up to \(1024 \times 1024\), and consider adding page indices in both training and evaluation [2602.15257]. It also recommends matching training context length to the benchmark’s effective context range rather than simply training on the longest possible contexts [2602.15257].

From an encyclopedia perspective, MMLBD-C is best understood as a curated correction of MMLongBenchDoc that preserves task identity while attempting to reduce annotation-induced evaluation noise [2602.15257]. Its importance lies less in introducing a new task than in making an existing and highly relevant long-document visual question answering benchmark more trustworthy. In that sense, MMLBD-C occupies a specific place in the methodological literature on multimodal evaluation: it is a benchmark-quality intervention embedded inside a larger study of long-context vision-language model training [2602.15257].

Source: https://www.emergentmind.com/topics/mmlbd-c