MFMDBench: Multilingual Financial Misinformation Benchmark
- MFMDBench is a multilingual benchmark that unifies diverse financial fact-checking tasks across English, Chinese, Greek, and Bengali.
- It employs standardized instruction templates for claim verification, evidence matching, and detection of manipulated social-media content.
- Evaluation using accuracy and Macro-F1 metrics reveals insights into numerical reasoning challenges and label polarity reversal issues.
Searching arXiv for the specified paper to ground the article in the source record. MFMDBench is a multilingual evaluation benchmark for Multilingual Financial Misinformation Detection (MFMD), introduced in the paper "MFMDQwen: Multilingual Financial Misinformation Detection Based on LLM" (Liu et al., 20 Apr 2026). It was designed to provide a unified testbed for evaluating LLMs on financial misinformation tasks across English, Chinese, Greek, and Bengali. Within the associated framework, MFMDBench functions specifically as the held-out evaluation component, distinct from the supervised fine-tuning dataset MFMD4Instruction, and is intended to support rigorous cross-lingual comparison on heterogeneous financial fact-checking and misinformation detection problems (Liu et al., 20 Apr 2026).
1. Origin and motivation
MFMDBench emerged from two stated needs in the automated fact-checking community. The first was that financial misinformation can distort markets and mislead investors, while prior benchmarks and datasets such as FinFact, FinDVer, and RFC-Bench had largely remained monolingual and English-centric. The second was that financial claims often depend on subtle numerical changes, sentiment shifts, or causal inversions, and that these difficulties become more acute across languages with different grammar and discourse conventions (Liu et al., 20 Apr 2026).
The benchmark was therefore conceived as the first truly multilingual evaluation suite spanning English, Chinese, Greek, and Bengali and unifying diverse financial-misinformation tasks into a single benchmark (Liu et al., 20 Apr 2026). In functional terms, MFMDBench is not a training corpus but an evaluation suite. This design choice matters methodologically because it separates multilingual capability assessment from the supervised data used to adapt a model.
A plausible implication is that MFMDBench addresses not only language coverage but also task heterogeneity. The included tasks range from claim verification with evidence to manipulated social-media post detection and reference-free evaluation settings. This suggests that benchmark performance should be interpreted as a composite measure of multilingual generalization, label-space adaptation, and robustness to different evidence structures, rather than as a single-task fact-checking score.
2. Constituent datasets and task composition
MFMDBench draws material from nine established datasets, each covering one or more of the four target languages (Liu et al., 20 Apr 2026). In English, it includes FinDVer and RFC-Bench; in Chinese, CHEF and MDFEND; in Bengali, BanMANI; and across all four languages, Global4Languages.
The benchmark’s source datasets and label structures are summarized below.
| Source dataset | Language coverage | Label/task format |
|---|---|---|
| FinDVer | English | entailed / refuted |
| RFC-Bench | English | counterfactual paragraph pairs |
| CHEF | Chinese | Supported / Refuted / NotEnoughInfo |
| MDFEND | Chinese | Real / Fake |
| BanMANI | Bengali | MANI / NO_MANI |
| Global4Languages | English, Chinese, Greek, Bengali | True / False |
The source descriptions are specific. FinDVer contains 700 long-document claim-verification instances, originally balanced as 350 "entailed" and 350 "refuted." RFC-Bench contains 1,826 counterfactual paragraph pairs manipulated via GPT-4.1. CHEF contains 1,188 evidence-grounded claims labeled Supported, Refuted, or NotEnoughInfo. MDFEND contains 1,321 Weibo posts labeled Real or Fake. BanMANI contains 101 social-media manipulation examples labeled MANI or NO_MANI. Global4Languages contains 144 aligned claim-context pairs in each language labeled True or False (Liu et al., 20 Apr 2026).
The benchmark integrates these tasks into one multilingual evaluation environment. This unification is significant because the underlying tasks are not label-isomorphic: some are binary, some ternary, and some require explicit evidence conditioning while others center on post manipulation or counterfactual variation. This suggests that model evaluation on MFMDBench probes not only language transfer but also the ability to follow task-specific labeling instructions under a shared benchmarking framework.
3. Data formatting, benchmark split, and class structure
Each raw example was converted into an instruction-tuning format within MFMD4Instruction using lightweight prompt templates that specify the task and expected output labels (Liu et al., 20 Apr 2026). The benchmark itself uses the same general templating logic for model evaluation. Representative templates include:
- Global4Languages: "Task Description: Determine whether the claim is ‘True’ or ‘False’. Claim: [Claim]"
- FinDVer: "Task Description: Assess the truthfulness of the given statement … Output the entailment label (‘entailed’ or ‘refuted’). Claim: [Claim] Relevant Financial Report: [Document]"
- CHEF: "Task Description: Label each claim based on the evidence provided … Choose one of Supported, Refuted, Not enough information. Claim: [Claim] Evidence: [Evidence]"
- BanMANI: "Task Description: Determine whether the social media post is manipulated … output
MANI’ orNO_MANI’. Original News: [Original] Social Media Post: [Post]" (Liu et al., 20 Apr 2026)
After merging and formatting, the MFMDBench evaluation portion comprises 2,432 held-out examples (Liu et al., 20 Apr 2026). The paper reports the following source-level distribution.
| Language | Benchmark components | Held-out examples |
|---|---|---|
| English | FinDVer, Global-EN, RFC | ≈ 1,821 |
| Chinese | CHEF, MDFEND, Global-ZH | ≈ 532 |
| Bengali | BanMANI, Global-BN | 50 |
| Greek | Global-GR | 29 |
The detailed breakdown is: FinDVer 140, Global-EN 29, RFC 1,652, CHEF 238, MDFEND 265, Global-ZH 29, BanMANI 21, Global-BN 29, and Global-GR 29 (Liu et al., 20 Apr 2026).
Because MFMDBench is purely an evaluation benchmark, there is no further train/validation split within these 2,432 examples (Liu et al., 20 Apr 2026). By contrast, MFMD4Instruction, which contains 4,837 examples, serves as supervised fine-tuning data. This division between benchmark and fine-tuning corpus is central to the framework: benchmark instances are held out for evaluation only.
Class balance varies by source dataset. Most binary tasks remain roughly 50/50 or reflect their original imbalances; Global4Languages originally contains 23 True and 121 False. CHEF preserves its three-way Supported/Refuted/NotEnoughInfo distribution, and BanMANI retains a near-even MANI/NO_MANI split (Liu et al., 20 Apr 2026). A common misconception would be to treat MFMDBench as a uniformly balanced multilingual benchmark; the paper does not support that interpretation. Instead, the benchmark inherits class structure and domain unevenness from its constituent datasets.
4. Evaluation protocol and metrics
MFMDBench measures both overall correctness and class-balanced performance. The primary reported metrics are Accuracy and Macro-F1 (Liu et al., 20 Apr 2026). The paper explicitly defines the underlying submetrics:
For tasks with more than two labels, such as CHEF, the F1 score is macro-averaged equally across classes, while Accuracy remains the proportion of total correct predictions over all examples (Liu et al., 20 Apr 2026). This is methodologically important because several included datasets differ substantially in label distributions, making Macro-F1 a more informative measure of class-balanced behavior than Accuracy alone.
The evaluation setup used greedy decoding under the same prompting templates as the instruction-formatted tasks (Liu et al., 20 Apr 2026). The evaluated models included reasoning-focused Qwen3 variants (8B-R, 14B-R, 32B-R), non-reasoning Qwen3 variants (8B, 14B, 32B), Qwen2.5-72B-Instruct, Llama3.1-8B-Instruct, Llama3.3-70B-Instruct, FMDLlama, and MFMDQwen (Liu et al., 20 Apr 2026).
MFMDQwen itself is based on Qwen3-8B and was supervised fine-tuned on the 4,837-example MFMD4Instruction dataset for two epochs with a learning rate of , a 0.1 warmup ratio, batch size 128, and DeepSpeed ZeRO-3 with CPU offloading. Inputs could span up to 24k tokens, outputs up to 8k, and training ran on four NVIDIA L40 GPUs (Liu et al., 20 Apr 2026). Although these settings pertain to the model rather than to MFMDBench directly, they determine how the benchmark’s reported top results were obtained.
5. Empirical performance and comparative results
Table 2 of the paper reports Accuracy and F1 for each model across all nine datasets, and MFMDQwen is reported to outperform all open-source baselines, achieving the highest F1 on six of the nine tasks and an overall average F1 of 0.818 (Liu et al., 20 Apr 2026). The benchmark therefore serves as the empirical basis for the paper’s claim that multilingual supervised fine-tuning can materially improve financial misinformation detection across languages.
The paper highlights several task-specific results. On Global-GR, MFMDQwen attains F1 = 0.758. On Global-BN, it reaches F1 = 0.803. On CHEF, it reaches F1 = 0.843 versus the next best approximately 0.320. On MDFEND, it reaches F1 = 0.913 versus the next best approximately 0.689. On RFC, it reaches F1 = 0.954 versus the next best approximately 0.689. On Bengali manipulation, it reaches F1 = 0.952 (Liu et al., 20 Apr 2026).
The paper also reports that the best non-fine-tuned Qwen3-8B-R model achieves an average F1 of 0.631, FMDLlama reaches 0.293, and Llama3.3-70B manages 0.663 (Liu et al., 20 Apr 2026). These comparisons indicate that scale alone does not guarantee superior performance on the benchmark, and that domain-specific multilingual supervised fine-tuning can dominate larger general-purpose models in this setting.
The reported results also support a more granular interpretation. The gains are most pronounced on Chinese datasets, particularly CHEF and MDFEND, whereas the performance gap on English benchmarks narrows (Liu et al., 20 Apr 2026). This suggests that English financial fact-checking ability is already partially embedded in strong general-purpose LLMs, while multilingual and noisy social-media conditions reward task-specific adaptation more strongly.
6. Diagnostic findings, limitations, and research implications
MFMDBench is presented not only as a leaderboard instrument but also as a diagnostic resource. Error analyses reported in the paper identify recurring weaknesses in label polarity reversal, numerical reasoning, table grounding, partial evidence matching, and subtle semantic shifts (Liu et al., 20 Apr 2026). These categories are operationally significant.
Label polarity reversal refers to cases where models default to "False" absent strong evidence. Numerical reasoning concerns failures on percentage and arithmetic claims despite explicit figures. Table grounding refers to mis-extraction of rows or columns from semi-structured financial tables. Partial evidence matching describes cases where models accept partially correct claims while overlooking decisive details. Subtle semantic shifts refer to lexical modifications in manipulated posts that remain undetected (Liu et al., 20 Apr 2026).
These findings clarify what benchmark success does and does not establish. Strong average Macro-F1 on MFMDBench does not imply complete reliability on numerically intensive or semi-structured financial reasoning. A plausible implication is that benchmark performance should be supplemented by targeted stress tests if the intended deployment environment involves tables, arithmetic transformations, or adversarial paraphrastic manipulation.
The benchmark also has explicit limitations. It relies on existing datasets, which leads to uneven class distributions and uneven domain coverage; it has no separate validation split within the benchmark; and it excludes languages beyond English, Chinese, Greek, and Bengali (Liu et al., 20 Apr 2026). In addition, the paper notes that performance in low-resource languages such as Bengali and Greek saturates quickly, an effect attributed to the small evaluation sets and simpler binary tasks (Liu et al., 20 Apr 2026). This is relevant when interpreting high scores on those subsets: the results are informative, but the paper itself cautions against overgeneralization.
Future work, as stated by the authors, aims to expand both MFMD4Instruction and MFMDBench to more languages, additional financial subtasks and modalities, and to explore larger base models in multilingual fine-tuning (Liu et al., 20 Apr 2026). This suggests that MFMDBench should be understood as an initial multilingual benchmark rather than a final or exhaustive standard for financial misinformation evaluation.
7. Position within multilingual financial misinformation research
Within the MFMDQwen framework, MFMDBench occupies a specific role: it is the benchmark that operationalizes the evaluation of multilingual financial misinformation detection across multiple tasks and languages (Liu et al., 20 Apr 2026). Its novelty lies in combining multilingual scope with task diversity and in pairing that benchmark with an instruction-format supervised dataset, MFMD4Instruction.
Relative to prior resources named in the paper—FinFact, FinDVer, and RFC-Bench—the benchmark addresses the limitation that earlier datasets had remained largely monolingual and English-centric (Liu et al., 20 Apr 2026). It also differs from single-task datasets by treating financial misinformation detection as a family of related tasks rather than as a single binary classification problem. This broader framing is important for researchers comparing architectures, prompting strategies, or fine-tuning regimens across heterogeneous financial NLP settings.
A plausible implication is that MFMDBench can serve as a bridge benchmark between fact-checking, misinformation detection, multilingual instruction tuning, and financially grounded reasoning. At the same time, the benchmark’s inherited heterogeneity means that aggregate scores should be interpreted alongside per-dataset behavior. In that sense, MFMDBench is both a consolidated multilingual benchmark and a structured collection of task-specific probes, each exposing different failure modes of contemporary LLMs (Liu et al., 20 Apr 2026).