MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA

Abstract: Iterative retrieval-reasoning agents have recently shown promise for multimodal long-document question answering. However, most existing systems maintain a single growing context that mixes retrieval traces, observations, and intermediate reasoning. As interactions accumulate, key evidence becomes scattered and diluted, making multi-hop reasoning noisy. We propose MARDoc, a Memory-Aware Refinement Agent framework that decouples long-document QA into three specialized agents: an Explorer for multi-granularity multimodal retrieval, a Refiner for distilling interaction traces into structured evidence and reasoning memories, and a Reflector for checking evidence sufficiency and providing targeted feedback. Across iterations, the agents rely on a dynamically updated structured memory rather than a full accumulated interaction history. This design reduces context noise while preserving answer-critical facts and their logical dependencies. Experiments on MMLongBench-Doc and DocBench show that MARDoc achieves strong results, outperforming same-backbone baselines and demonstrating the effectiveness of structured memory for agentic document QA.

• The paper introduces a modular agent framework with structured memory components to robustly perform multimodal long-document QA.
• It decouples retrieval, reasoning, and validation to maintain evidence integrity, achieving a 7.8% accuracy gain over prior methods.
• Experimental results demonstrate consistent multi-hop reasoning and minimal accuracy degradation across complex, multi-page evidence.

MARDoc: A Memory-Aware Refinement Agent Architecture for Multimodal Long Document QA

Introduction and Motivation

Multimodal long-document question answering (DocQA) represents a central challenge for LLM-based systems tasked with complex document intelligence, requiring robust, multi-hop cross-modality reasoning over sparse, often widely distributed evidence. Previous approaches—encompassing monolithic multimodal LLMs, retrieval-augmented generation (RAG) variants, and iterative, agent-driven systems—tend to append evidence and intermediate traces into ever-growing contexts. This unstructured accumulation leads to severe context dilution: essential facts become difficult to retrieve, logical dependencies between pieces of evidence are lost, and error-checking or recovery mechanisms lack transparency or grounding. Furthermore, quality assurance for agentic reasoning is hindered by the absence of structured, inspectable memory.

The "MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA" (2606.05749) directly targets these limitations by proposing an agentic architecture featuring a dynamically updated structured memory and a decoupled agent workflow, explicitly separating retrieval, distillation, and reasoning critique.

Architectural Overview

MARDoc operationalizes document QA as an iterative Explore–Refine–Reflect process mediated via structured memory:

• The Explorer retrieves multi-granularity, multimodal evidence informed by a structured document outline, executing targeted tool calls and producing a trace of tool-use and preliminary conclusions.
• The Refiner ingests the Explorer’s traces and distills them into two memory components: (i) Evidence Memory—encapsulating answer-critical, source-grounded facts (text/visual/tabular); and (ii) Reasoning Memory—encoding logical chains linking evidence to intermediate and final conclusions.
• The Reflector validates the memory, assesses sufficiency and faithfulness with respect to the query, and generates concrete feedback or, if appropriate, outputs a final answer.

Across iterations, revised memory replaces previous iterations, and only signal—not full interaction history—propagates, fundamentally de-noising context and supporting multi-hop reasoning.

Figure 1: MARDoc decouples retrieval, reasoning, and validation into three agents interfacing via structured memory, in contrast to the monolithic context paradigm.

Document Parsing and Tooling

Effective evidence orchestration in MARDoc is contingent on fine-grained document representation and flexible toolsets. MinerU2.5 is leveraged to produce detailed, semantically aware outlines, capturing hierarchical relationships, spatial layout, and visual attributes. This, in concert with Qwen3-VL-235B-A22B-Instruct’s capability for concise element descriptions, maximizes retrievability of multimodal units.

The Explorer’s tool suite is explicitly designed for multi-granularity access—spanning passage, section, layout object, and metadata retrieval—thus supporting both coarse and fine localization across text, structure, charts, tables, and figures.

Structured Memory and Agent Roles

Refiner

The Refiner’s compacted memories are the key structural innovation. Evidence Memory records only de-duplicated, directly relevant facts—each tagged with their modality and provenance. Reasoning Memory forms explicit chains, with each step annotated by source dependencies, enabling subsequent agents to both ascertain logical correctness and isolate the origins of potential errors or omissions.

Reflector

By disentangling correctness assessment (Reflector) from evidence retrieval (Explorer/Refiner), MARDoc can perform iterative sufficiency checks, inject heuristic guidance, and robustly signal unanswerability. The Reflector’s explicit instruction output directly shapes the Explorer’s next retrieval focus, dynamically steering attention to missing facts or logical inconsistencies.

Experimental Results

Evaluated on MMLongBench-Doc and DocBench, MARDoc (backbones: Qwen3-VL-30B/8B) consistently surpasses both MLLM-based and RAG-based systems and outperforms all open-source agentic baselines with the same backbones. It reaches or exceeds closed-source state-of-the-art when backbone capacity is increased. Notably, MARDoc achieves a 7.8% absolute accuracy gain over prior agentic systems at manageable computation cost increases.

On MMLongBench-Doc, MARDoc’s structured memory design maintains robust multi-hop reasoning as the number of evidence pages increases, with markedly less accuracy degradation compared to context-stream paradigms.

Figure 2: Accuracy on MMLongBench-Doc as a function of evidence page count; MARDoc exhibits minimal decline in accuracy as evidence dispersion increases, reflecting effective cross-page reasoning.

Component Analysis and Ablations

Ablation reveals the necessity of both Evidence and Reasoning Memory: removing Reasoning Memory ($M_R$) degrades multi-hop performance, while removing Evidence Memory ($M_E$) weakens faithfulness and increases hallucinations. Replacing both with unstructured summarization collapses overall effectiveness, reaffirming that structural grounding—rather than mere context compression—is essential.

Reflector iterations (with $K=3$ as optimal) consistently improve F1 on hard (multi-hop, unanswerable) queries, but excessive iterations can reintroduce noise. Removing Reflector instructions causes a statistically significant drop in both single- and multi-hop categories.

Figure 3: Performance as a function of Reflector iteration count $K$, showing optimal reasoning at moderate $K$ and highlighting the trade-off between additional evidence integration and context re-noising.

Ablations on parsing/tooling frameworks further show MARDoc’s parsing design (i.e., MinerU2.5-based outlines) is critical for robust retrieval, especially on layout-centric and visually complex sources.

Figure 4: Ablation study by evidence type, with MARDoc outperforming context-stream systems on pure text, layout, and tabular information via fine-grained semantic parsing and tool augmentation.

Computational Cost and Practicality

MARDoc introduces a modest increase in inference tokens and latency (22.5k extra tokens, 15.9s on Qwen3-30B per instance), justified by substantial accuracy gains, especially on the most challenging multi-page, multi-modal reasoning tasks.

Implications and Future Directions

MARDoc demonstrates that explicit modularization of retrieval, reasoning, and validation, mediated by inspectable structured memory, yields significant empirical advances in the DocQA domain. This architecture sets a precedent for integrating more advanced training-based optimizations atop prompt-engineered agents and for extending structured memory approaches to other open-ended, high-noise reasoning settings (e.g., scientific literature, organizational reports).

Generalization across LLM types and backbone sizes, system-level latency management (e.g., smart early stopping or partial reflection), and fully end-to-end differentiable instantiations of agent components represent promising directions. Moreover, the abstraction and transferability of Evidence/Reasoning Memories could facilitate more interpretable and trustworthy LLM-based agents beyond the DocQA task.

Conclusion

MARDoc establishes a coherent, theoretically grounded, and practically effective agentic paradigm for multimodal long-document QA, combining fine-grained document structuring, modular agent collaboration, and memory-aware context pruning. Strong performance across multiple benchmarks validates the efficacy of structured memory, and the framework suggests a path toward scalable, robust, and interpretable agentification for advanced document intelligence applications.

---

Reference: "MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA" (2606.05749)