MMLBD-C: Correcting MMLongBenchDoc
- MMLBD-C is a corrected derivative of MMLongBenchDoc designed to enhance benchmark reliability for long-context visual document VQA.
- It employs a hybrid pipeline that flags inconsistencies automatically and uses manual review to modify questions, answers, or remove flawed examples.
- Empirical evaluations show that MMLBD-C yields higher F1 scores and more trustworthy results, making it a key tool for model selection.
Searching arXiv for the exact paper and benchmark context. arXiv search query: (Veselka, 16 Feb 2026) MMLBD-C MMLongBenchDoc MMLBD-C is a manually corrected, quality-filtered version of MMLongBenchDoc, introduced as “MMLBD-C: correcting MMLongBenchDoc” in "How to Train Your Long-Context Visual Document Model" (Veselka, 16 Feb 2026). It is a corrected derivative rather than a new benchmark from scratch, preserving the same overall task domain—long-context document VQA over visual documents—while addressing annotation and evaluation defects in the original benchmark. Its stated purpose is to provide a more reliable benchmark for long-context visual document understanding, especially long-document VQA, by reducing erroneous and low quality examples (Veselka, 16 Feb 2026).
1. Definition and benchmark role
MMLBD-C starts from MMLongBenchDoc and changes example content only where needed: keep as-is, correct question, correct answer, or remove the example (Veselka, 16 Feb 2026). The benchmark therefore remains in the same benchmark family and targets the same ability, but with higher annotation quality. The paper evaluates on both MMLongBenchDoc and MMLBD-C and includes both in the aggregate Visual-LC Avg (VA), defined over MMLongBenchDoc, MMLBD-C, MMLongBench, DUDE, and SlideVQA (Veselka, 16 Feb 2026).
The benchmark is positioned as especially relevant to long-document visual question answering. The authors state that MMLBD-C is used as a tiebreaker in their aggregate evaluation because it is “the most relevant benchmark to our work” (Veselka, 16 Feb 2026). They also report that MMLBD-C scores correlate highly with MMLongBenchDoc scores, though they are generally higher, which the benchmark description associates with cleaning away bad or unfair examples rather than with a redesign of the task (Veselka, 16 Feb 2026).
This suggests that MMLBD-C functions simultaneously as a correction layer and as a model-selection instrument. A plausible implication is that it is intended not to replace historical reporting on MMLongBenchDoc, but to provide a cleaner measurement point within the same evaluation lineage.
2. Motivation: annotation pathologies in MMLongBenchDoc
MMLBD-C was introduced in response to quality problems identified in MMLongBenchDoc (Veselka, 16 Feb 2026). The paper describes several classes of benchmark flaws that can distort model evaluation.
First, it identifies incorrect question-document pairing. One cited example is the question “List all the PM health effects that increse by more than 35% in India and Thailand.” paired with an unrelated document about digital marketing (Veselka, 16 Feb 2026). Such mismatches invalidate the retrieval and reasoning demands of the task.
Second, it identifies underspecified or ambiguous questions. The example “List all the sections that discuss about the experiment setup?” was paired with the answer ['Section 4.1', 'Section 4.2', 'Section 4.3', 'Appendix A'], but the authors note that it is hard to argue that the Methodology section does not discuss the experiment setup (Veselka, 16 Feb 2026). This makes the evaluation sensitive to annotation choices rather than underlying model competence.
Third, it identifies typos that change answerability. The example “How do Amazon recognize least cost?” should read “lease cost” (Veselka, 16 Feb 2026). Because “least cost” is itself plausible in context, such errors can induce spurious failures.
Fourth, it identifies incorrect answers. The paper gives an example in which “How many percentage respondents in this survey access to internet more than two times per month?” was labeled unanswerable even though the answer is explicitly derivable from the document (Veselka, 16 Feb 2026).
Fifth, it identifies overly narrow answer acceptance for unanswerable questions. For “Not answerable” questions, equivalent responses such as “None,” “0,” and “No one” should sometimes also be accepted (Veselka, 16 Feb 2026). This is not a change in task definition, but a change in answer normalization and acceptance criteria.
The benchmark motivation is thus explicitly about benchmark reliability. If question-document pairs, wording, or answers are wrong or misleading, then reported scores become noisy or unfair measures of actual long-document understanding (Veselka, 16 Feb 2026). The release of MMLBD-C is framed as a response to this evaluation noise.
3. Correction pipeline and editorial interventions
The construction of MMLBD-C combines automatic inconsistency flagging with human review (Veselka, 16 Feb 2026). The authors state that they “apply a version of the recursive pipeline … adapted to find inconsistencies between the source, question and answer” (Veselka, 16 Feb 2026). The role of this adapted pipeline is to flag suspicious benchmark items rather than to replace human judgment.
A total of 342 examples were flagged for review and then manually reviewed (Veselka, 16 Feb 2026). For each flagged example, the paper reports four possible actions: leave as is, modify the question, modify the answer, or remove from the benchmark (Veselka, 16 Feb 2026). The correction criteria are therefore source-document consistency, question adequacy, answer correctness, and recoverability.
The benchmark modifications are summarized below.
| Quantity | Count |
|---|---|
| Original MMLongBenchDoc examples | 1091 |
| Flagged for review | 342 |
| Modified | 251 |
| Removed | 16 |
The paper explicitly states that it modifies “251 out of 1091” examples for errors, incorrect grammar or misleading/underspecified questions and removes 16 (Veselka, 16 Feb 2026). It also states, with respect to document mismatches, “We remove 9 of 10 affected questions and convert the last to ‘Not answerable’” (Veselka, 16 Feb 2026). The corrected fraction is , the removed fraction is , the total affected fraction is , and the flagged fraction is (Veselka, 16 Feb 2026).
The paper clearly indicates that the flagged examples were manually reviewed, but it does not specify the number of annotators, whether the annotators were authors or external crowdworkers, the full annotation guidelines, the adjudication procedure, or inter-annotator agreement (Veselka, 16 Feb 2026). The safe factual conclusion is therefore limited: human manual review was involved, but the annotator protocol is not fully specified.
The classes of correction are explicitly described as incorrect question-document pairing, underspecified or misleading wording, typos or incorrect grammar, incorrect answers, and expanded acceptable answer forms for “Not answerable” items (Veselka, 16 Feb 2026). This suggests that MMLBD-C is a curation and repair effort rather than a redistribution or rebalancing effort.
4. Dataset continuity, scale, and task composition
MMLBD-C preserves the overall task domain of MMLongBenchDoc: long-document visual question answering on visual documents or PDF pages (Veselka, 16 Feb 2026). The paper does not report a redesign of task categories or question-type composition, and it presents MMLBD-C as a correction layer rather than a rebalanced benchmark (Veselka, 16 Feb 2026). Accordingly, the task/domain composition appears largely unchanged, with individual examples changed or removed.
The core dataset statistics are inherited from the correction process. MMLongBenchDoc contains 1091 examples, 16 are removed, and the implied final size of MMLBD-C is 1075 examples (Veselka, 16 Feb 2026). The paper does not explicitly print “1075,” but that value follows directly from the reported counts.
The paper also does not mention any new train/dev/test split or altered evaluation split for MMLBD-C (Veselka, 16 Feb 2026). It appears to be a corrected evaluation set corresponding to MMLongBenchDoc’s evaluation use. Likewise, the paper does not provide a standalone table for MMLBD-C category counts, average document length, or modality breakdown (Veselka, 16 Feb 2026). What is stated is that the benchmark remains a visual long-context document QA benchmark.
There is, however, a slight change in answer format handling. The benchmark broadens acceptable answers for some “Not answerable” items by accepting equivalents such as “None,” “0,” and “No one” where appropriate (Veselka, 16 Feb 2026). This is an evaluation-format change at the answer normalization level rather than a task redesign.
5. Evaluation protocol and long-context settings
The primary metric for MMLBD-C is F1, specifically the benchmark’s overall_f1 (Veselka, 16 Feb 2026). The paper does not provide a new formula for this F1 and treats it as inherited from benchmark tooling. MMLBD-C is therefore scored with F1 rather than exact match (Veselka, 16 Feb 2026).
The evaluation setup includes several practical long-context modifications. In contrast to the default VLM Eval Kit settings, the paper increases the maximum number of pages from 120 to 336 for MMLongBenchDoc and MMLBD-C and sets the maximum resolution to to ensure long examples fit in context while preserving fine details (Veselka, 16 Feb 2026). These settings are central to MMLBD-C evaluation because they determine whether long document inputs fit within the model’s usable context.
A further intervention is the use of page indices. The paper prepends page labels in the input context in the form:
1 2 3 4 |
Page 1: <image> Page 2: <image> Page 3: <image> ... |
The paper reports that adding page indices during both training and evaluation improves MMLBD-C by +2.8 in one configuration, and emphasizes that including page indices during training is necessary to benefit during evaluation (Veselka, 16 Feb 2026). Within the paper’s ablations, this is one of the strongest benchmark-specific interventions.
MMLBD-C also appears in two higher-level aggregate metrics. Visual-LC Avg (VA) averages over MMLongBenchDoc, MMLBD-C, MMLongBench, DUDE, and SlideVQA, while LC Avg (LCA) averages over visual-LC benchmarks plus HELMET and LongBench v2 (Veselka, 16 Feb 2026). The paper states that benchmark scores are normalized by benchmark-specific maximums before averaging because the component benchmarks have different ranges and distributions (Veselka, 16 Feb 2026).
6. Empirical behavior and benchmark significance
The paper states that MMLBD-C scores correlate highly with MMLongBenchDoc scores, but are generally higher (Veselka, 16 Feb 2026). The stated interpretation is that the corrected benchmark is less unfair or noisy, not that it is necessarily less challenging in an underlying cognitive sense. This distinction is important because a higher score on MMLBD-C does not imply lower task relevance; it may instead reflect lower annotation error.
In the paper’s best-checkpoint table, several concrete MMLBD-C scores are reported (Veselka, 16 Feb 2026). These include Qwen3 VL 235B A22B at 56.2, LongPO Short Stage at 56.4, Qwen3 VL 32B at 53.8, Qwen3 VL Plain Distillation Short Stage at 57.3, Mistral Plain Distillation at 47.4, and Mistral 3.1 Small (24B) at 41.4 (Veselka, 16 Feb 2026). The strongest reported score in that table is Qwen3 VL Plain Distillation Short Stage at 57.3 (Veselka, 16 Feb 2026).
The paper uses MMLBD-C to distinguish among training strategies. For example, it states: “For best MMLBD-C performance specifically, plain distillation with SFT is more effective” (Veselka, 16 Feb 2026). This differs from the broader Visual-LC Avg, where LongPO is often strongest (Veselka, 16 Feb 2026). In the reported comparison, LongPO Short Stage reaches 56.4 on MMLBD-C, while Qwen3 VL Plain Distillation Short Stage reaches 57.3 (Veselka, 16 Feb 2026).
The paper also uses MMLBD-C to support the claim that training on context lengths that match evaluation context lengths outperforms training on longer contexts (Veselka, 16 Feb 2026). Representative values are Mistral Short Stage at 45.0 versus Mistral Short + Long at 43.3, Qwen3VL Short Stage at 57.3 versus Qwen3VL Short + Long at 57.0, and LongPO Short Stage at 56.4 versus LongPO Short + Long at 54.0 (Veselka, 16 Feb 2026). Within this empirical frame, MMLBD-C serves as a benchmark sensitive to context-length alignment and training objective choice.
The paper explicitly argues that MMLBD-C is a more reliable measure of long-context visual document understanding than the raw MMLongBenchDoc benchmark because it was created to reduce erroneous and low quality examples, it is manually corrected, its annotations are released for public inspection, and it is used as a tiebreaker in evaluation (Veselka, 16 Feb 2026). A plausible implication is that MMLBD-C is intended as a benchmark-quality intervention as much as a benchmarking artifact.
7. Limitations, interpretation, and research use
MMLBD-C is presented as an improved benchmark, but the paper also leaves several limits explicit or implicit (Veselka, 16 Feb 2026). The manual correction protocol is not fully specified, since annotator counts and agreement statistics are not reported. The benchmark also remains highly correlated with MMLongBenchDoc, so it is an improvement in quality rather than a complete redefinition of the task (Veselka, 16 Feb 2026). In addition, the broader benchmark ecosystem used in the paper may under-represent extreme contexts, since most benchmarks are under 128K tokens and thus do not fully validate performance at the model’s maximum context such as 344K (Veselka, 16 Feb 2026).
For practical use, the paper supports a specific evaluation recipe for long-document VQA on MMLBD-C: use F1 (overall_f1), increase maximum pages to 336, set image resolution up to , and consider adding page indices in both training and evaluation (Veselka, 16 Feb 2026). It also recommends matching training context length to the benchmark’s effective context range rather than simply training on the longest possible contexts (Veselka, 16 Feb 2026).
From an encyclopedia perspective, MMLBD-C is best understood as a curated correction of MMLongBenchDoc that preserves task identity while attempting to reduce annotation-induced evaluation noise (Veselka, 16 Feb 2026). Its importance lies less in introducing a new task than in making an existing and highly relevant long-document visual question answering benchmark more trustworthy. In that sense, MMLBD-C occupies a specific place in the methodological literature on multimodal evaluation: it is a benchmark-quality intervention embedded inside a larger study of long-context vision-LLM training (Veselka, 16 Feb 2026).