MMA-Bench: Belief Dynamics Benchmark

• MMA-Bench is a multimodal benchmark that evaluates belief dynamics in long-horizon dialogues by controlling speaker reliability, temporal drift, and text–vision contradictions.
• It uses a structured 10-session narrative with staged phases to test memory-augmented agents' ability to update beliefs and appropriately abstain when evidence is insufficient.
• The benchmark diagnoses targeted failure modes such as authority bias, over-interpretation, and the Visual Placebo Effect by comparing text and vision modes.

MMA-Bench is a programmatically generated benchmark for multimodal belief dynamics under controlled conflict, introduced in the context of the Multimodal Memory Agent (MMA). It is a long-horizon dialogue benchmark rather than a simple question-answer collection: a 10-session dialog stream spanning roughly 6 months, with controlled speaker reliability and structured text–vision contradictions, built to test whether a memory-augmented agent can track which speakers are reliable, update beliefs over time, arbitrate between text and visual evidence, and abstain when evidence is insufficient instead of confidently hallucinating (Lu et al., 18 Feb 2026).

1. Research motivation and benchmark scope

MMA-Bench was created from the claim that existing long-context and memory benchmarks are mostly too shallow for realistic multimodal agents. In the benchmark’s framing, prior evaluations may test long-context retrieval, verification, or long-term dialog memory, but generally do not jointly control source reliability priors, temporal drift, and structured text–vision contradictions while also scoring epistemic prudence. MMA-Bench is therefore presented as a multi-modal benchmark designed to evaluate belief dynamics and cognitive robustness, and more specifically as a framework that operationalizes belief dynamics under multimodal conflict and controlled reliability priors (Lu et al., 18 Feb 2026).

Its scope is deliberately diagnostic. The benchmark is designed to expose failure modes that arise when retrieved memories are stale, low-credibility, conflicting, or visually misleading. In that sense, it evaluates not only retrieval or reasoning accuracy, but also whether an agent can behave prudently under uncertainty. A central design premise is that multimodal agents should not be rewarded for answering in every case; they should also know when to reserve judgment.

The benchmark is also the evaluation counterpart to MMA itself. The associated agent assigns each retrieved memory item a dynamic reliability score by combining source credibility, temporal decay, and conflict-aware network consensus, and MMA-Bench is constructed to test whether those signals matter under conflict-heavy multimodal conditions. This suggests a close coupling between benchmark design and system design: the benchmark targets precisely the reasoning pathologies the agent is meant to mitigate.

2. Narrative structure and controlled generation process

MMA-Bench is built as a controlled synthetic social narrative. The dialog stream contains two speakers with asymmetric priors: User A, who is historically reliable, and User B, who is unreliable. The narrative unfolds over four explicitly staged phases (Lu et al., 18 Feb 2026):

Phase	Sessions	Function
Calibration	S1–S4	Establishes reliability priors through verifiable events
Adversarial Noise	S5–S7	Injects chit-chat and distractor entities
The Trap	S8	Introduces the central multimodal contradiction
Resolution	S9–S10	Resolves the conflict or leaves it unknowable

The Calibration phase is meant to teach the agent that User A is generally trustworthy and User B is not. The Adversarial Noise phase then stresses attention and retrieval by adding distractors similar to target facts. The central benchmark event occurs in The Trap, where User B makes a claim that is supported by visual evidence but contradicts User A. The Resolution phase either settles the contradiction or preserves uncertainty, allowing the benchmark to test both belief revision and abstention.

Three controlled variables are emphasized throughout the construction. First, speaker reliability is explicitly encoded, so the benchmark can test whether an agent uses source credibility rather than salience. Second, temporal dynamics matter because the conversation spans 10 sessions over roughly 6 months, making stale-versus-current evidence consequential. Third, the benchmark is organized around paired text–vision evidence and structured text–vision contradictions. This is operationalized through a comparison between Text Mode, which uses oracle captions, and Vision Mode, which uses raw images. The Text/Vision split isolates whether a model is following text alone or genuinely integrating vision.

3. Logic types and targeted failure modes

MMA-Bench formalizes multimodal contradiction through a four-type logic matrix (Lu et al., 18 Feb 2026).

Type	Benchmark condition	Target capability
A	visuals support the reliable User A	baseline consistency
B	visuals support the unreliable User B	overcome authority bias
C	visuals are vague	reject over-interpretation
D	no valid evidence exists	absolute abstention

These types correspond to distinct failure modes. Type B targets authority bias or reliability inversion failure: the model must resist defaulting to the historically reliable speaker when the less reliable speaker is actually supported by visual evidence. Type C targets over-interpretation of ambiguous visuals: the model should avoid reading too much into vague or noisy images. Type D targets failure to abstain in unknowable cases: when no valid evidence exists, the correct behavior is to reserve judgment.

The benchmark also uses these settings to reveal what the paper terms the Visual Placebo Effect. This denotes the case where the mere presence of visual input creates an illusion of evidence, causing agents to become more confident even when the image is ambiguous or unhelpful. In the benchmark’s interpretation, visual input can increase confidence without increasing epistemic warrant.

A further target is belief inertia or poor belief revision. MMA-Bench includes a 3-step probe to measure whether the model can update its belief after reflection, or whether it simply preserves its initial answer. This moves the benchmark away from static correctness and toward a dynamic analysis of cognitive behavior.

4. Evaluation protocol and diagnostic metrics

MMA-Bench uses a hierarchical evaluation protocol with three layers. Layer 1 measures fundamental capabilities through standard QA tests: fact retrieval, logic reasoning, source analysis, and adversarial distraction accuracy. Layer 2 is the benchmark’s core 3-step probe, which evaluates the model’s belief state across three steps and checks whether it updates after reflection. Layer 3 introduces cognitive-dynamics metrics designed to explain why the model behaves as it does, rather than only whether it was correct (Lu et al., 18 Feb 2026).

The reported cognitive-dynamics metrics are Modality Signal Alignment (MSA), Relative Reasoning Uncertainty, Self-Correction Rate (SCR), and False Confession Rate (FCR). Relative Reasoning Uncertainty is defined as

$$\Delta H_{rel} = \frac{2(H_{text} - H_{vis})}{(H_{text} + H_{vis})}$$

with positive values indicating greater certainty in the visual stream. The self-correction and false-confession measures are

$$SCR = \frac{\mathrm{Count}(\mathrm{Step\ 1} = \mathrm{Wrong} \land \mathrm{Step\ 3} = \mathrm{Right})}{\mathrm{Count}(\mathrm{Step\ 1} = \mathrm{Wrong})}$$

and

$$FCR = \frac{\mathrm{Count}(\mathrm{Step\ 1} = \mathrm{Right} \land \mathrm{Step\ 3} = \mathrm{Wrong})}{\mathrm{Count}(\mathrm{Step\ 1} = \mathrm{Right})}.$$

These metrics are intended to distinguish stable correction from unstable or sycophantic revision.

A key feature is CoRe (Confidence-and-Reserve) scoring, an abstention-aware scoring rule. In deterministic cases, namely Types A and B, the model should answer the correct label. In indeterminate cases, namely Types C and D, the model should ideally choose Unknown or abstain; wrongly answering in those cases is penalized. This makes MMA-Bench unusual among multimodal benchmarks: it does not reward an “always answer” strategy, but explicitly rewards prudent refusal when the evidence does not support a determinate conclusion.

5. Reported empirical results

The principal reported result is that MMA substantially outperforms the MIRIX baseline on the hardest reliability-inversion setting in Vision mode. On Type B, where the image supports the unreliable speaker, MMA achieves 41.18% Type-B accuracy in Vision mode, while the MIRIX baseline collapses to 0.0% under the same protocol. This is the benchmark’s canonical case of breaking authority bias by following the visual evidence rather than historical speaker reliability alone (Lu et al., 18 Feb 2026).

The main results table reported for MMA-Bench gives the following values:

• MIRIX baseline, Text mode: Core Acc. 30.94%, Verdict Acc. 47.78%, CoRe 0.37, Type B Acc. 0.00%
• MIRIX baseline, Vision mode: Core Acc. 32.67%, Verdict Acc. 46.67%, CoRe 0.35, Type B Acc. 0.00%
• MMA, Text mode: Core Acc. 13.15%, Verdict Acc. 56.67%, CoRe 0.28, Type B Acc. 23.53%
• MMA, Vision mode: Core Acc. 13.55%, Verdict Acc. 42.22%, CoRe -0.16, Type B Acc. 41.18%

These results indicate that the benchmark is not reducible to a single scalar accuracy. The MMA system improves the targeted inversion case, but its aggregate scores expose nontrivial trade-offs across verdict quality, abstention, and multimodal conflict resolution.

MMA-Bench is also the setting in which the paper most explicitly characterizes the Visual Placebo Effect. For GPT-4.1-mini, the reported Type D CoRe score drops from 0.85 in Text mode to 0.23 in Vision mode, which is interpreted as visual input creating an illusion of information sufficiency. For MMA, the issue remains present but is described as partially buffered by the confidence module. The benchmark table reports MMA’s Type D score in Vision mode as -0.38, whereas the baseline remains at 1.00 largely because it defaults to Unknown due to retrieval blindness rather than actual calibration. This distinction is central to the benchmark’s interpretation: apparently strong abstention can be artifactual rather than prudent.

6. Diagnostic significance, ablations, and interpretive boundaries

MMA-Bench is significant less as a broad-coverage perception benchmark than as a targeted diagnostic for multimodal memory agents. It probes whether an agent can combine reliability tracking, temporal reasoning, multimodal conflict resolution, and selective abstention in a single long-horizon setting. The benchmark’s main contribution is therefore methodological: it turns belief dynamics under multimodal conflict into a controlled evaluation problem (Lu et al., 18 Feb 2026).

The reported ablations reinforce this diagnostic purpose. Without Source ($S$, variant tc), the agent loses agency and becomes paralyzed in deterministic cases. Without Consensus ($C_{\text{con}}$, variant st), the agent may improve some sparse retrieval settings, but becomes much more vulnerable to the Visual Placebo Effect in MMA-Bench. Without Time ($T$, variant cs), the agent struggles with temporal stability and can collapse to 0% accuracy in some vision-mode deterministic settings. These ablations map directly onto the benchmark’s three control dimensions—source reliability, temporal change, and multimodal contradiction—and show that the benchmark is intended to diagnose which cognitive ingredient fails, not merely whether the final answer is wrong.

Several misconceptions are explicitly ruled out by the benchmark design. MMA-Bench is not random multimodal QA; it is a controlled synthetic social narrative with delayed conflict and reliability history. It is not a benchmark in which vision is simply an auxiliary input; the Text-versus-Vision comparison is fundamental to isolating visual influence. It is also not an evaluation in which abstention can be interpreted naively: a system may appear prudent because it is calibrated, or because it fails to retrieve and defaults to Unknown. The benchmark is designed to separate those cases.

A plausible implication is that MMA-Bench defines a narrower but more behaviorally precise problem than general multimodal QA or long-context retrieval benchmarks. By making controlled reliability priors, temporal drift, and text–vision contradiction first-class variables, it provides a framework for studying when multimodal evidence genuinely improves belief formation and when it merely induces overconfident error.