TQA-to-MMQA Framework for Multimodal QA

Updated 14 July 2026

TQA-to-MMQA Framework is a multimodal QA paradigm that transforms text-only queries into questions requiring both rewritten text and generated images.
It employs distinct transformation patterns and modular agent architectures to coordinate evidence retrieval, cross-modal integration, and reasoning over heterogeneous data.
The framework uses rubric-based evaluation to assess information consistency and integration, addressing challenges like cascading errors and evidence scalability.

The term TQA-to-MMQA Framework is used in recent literature for a cluster of methods that extend question answering from text-only or table-centered settings into genuinely multimodal reasoning over combinations of text, tables, charts, and images. In the narrowest sense, it denotes the transformation of a text-only QA pair $Q_s$ into a multimodal QA pair $Q_m=(T,I)$ whose answer depends on both rewritten text and a generated image. In a broader systems sense, it also covers the transition from single-table or multi-table QA toward MMQA-style reasoning over heterogeneous evidence, including routing, retrieval, evidence selection, agent coordination, and rubric-based evaluation (Wang et al., 29 Sep 2025, Soliman et al., 19 Jan 2026, Zhao et al., 2024, Talmor et al., 2021).

1. Conceptual scope and emergence

The framework emerged from a common diagnosis across several lines of work: classical TQA assumes that the relevant evidence is already localized in a table, whereas realistic analytical QA often requires deciding where the answer resides and how different evidence sources should be combined. In MultiModalQA, the table becomes an anchor but not the sole source of evidence; a question may require identifying an entity through text or image evidence and then using the table for comparison or aggregation (Talmor et al., 2021). In CT2C-QA, the setting is even closer to webpage analysis: the model must reason over Chinese text, tables, and charts, with the answer potentially appearing in one modality, multiple modalities, or not existing at all (Zhao et al., 2024).

This broader interpretation changes the role of TQA. Instead of being a terminal task, TQA becomes one component in a larger pipeline that may involve modality routing, cross-source linking, symbolic computation, or answer arbitration. MAMMQA makes this point explicitly by criticizing monolithic reasoning strategies that flatten tables and images into a single generalized prompt and instead decomposing multimodal QA into modality-specific extraction, cross-modal synthesis, and aggregation (Rajput et al., 27 May 2025).

A recurrent misconception in this area is that multimodalization simply means attaching an image or an additional document to an existing question. The literature is more restrictive. Q-Mirror states that the goal is not merely “add an image,” but to create multimodal questions in which both modalities are indispensable and single-modality shortcuts are avoided (Wang et al., 29 Sep 2025). MultiModalQA similarly distinguishes actual cross-modal reasoning from modality routing alone, showing that a multi-hop model is necessary because many questions cannot be answered by selecting a single modality (Talmor et al., 2021).

2. Canonical transformation patterns

Recent work describes several distinct but related transformation patterns under the TQA-to-MMQA umbrella.

Formulation	Input and output	Key mechanism
Q-Mirror	$Q_s \rightarrow Q_m=(T,I)$	Remove visualizable spans, insert placeholder, generate image, enforce multimodal dependence
MultiModalQA	Single-modality primitives $\rightarrow$ multimodal questions	Typed composition with `Compose`, `Intersect`, `Compare`
CT2C-QA / AED	Webpage with text, tables, charts $\rightarrow$ answer and modality	Allocation, expert reasoning, decision
CORE-T	Open-book pooled tables $\rightarrow S(q)\subseteq \mathcal{T}$	Dense retrieval, coherent subset selection, additive restoration

In Q-Mirror, the formal task is to transform an original TQA $Q_s$ into an MMQA $Q_m=(T,I)$ , where $T$ is the transformed text question and $I$ is the associated generated image (Wang et al., 29 Sep 2025). A valid transformed item must satisfy three conditions: Information Consistency, Cross-Modal Integration, and Standalone Quality. The operational pipeline is: detect visualizable spans, remove them from the question, insert an image placeholder such as [IMAGE_1], generate a detailed image description, render the image with a text-to-image model, assemble the MMQA, and then evaluate it against the rubric. The design goal is that the rewritten question becomes visually dependent.

MultiModalQA uses a different construction paradigm. It starts from single-modality primitives—TableQ, TextQ, ImageQ, and ImageListQ—and composes them with three typed operators: Compose, Intersect, and Compare (Talmor et al., 2021). This turns direct table references into multimodal subquestions while preserving the table as the computational substrate. The resulting grammar,

$Q_m=(T,I)$ 0

provides a controlled route from TQA to MMQA by keeping row/entity identity explicit.

CT2C-QA frames the transition at the webpage level. Instead of converting an existing table question, it assumes a page containing Markdownized text, tagged tables, and chart placeholders, then asks questions whose answer source may be text, table, chart, overlapping modality pairs, or absent (Zhao et al., 2024). The corresponding AED system routes the question by estimating probabilities such as $Q_m=(T,I)$ 1, $Q_m=(T,I)$ 2, and $Q_m=(T,I)$ 3, activating experts when the probability exceeds $Q_m=(T,I)$ 4.

CORE-T occupies an intermediate position. It is not a full MMQA system, but it explicitly presents itself as relevant to a transition from classical table QA toward multi-table QA and eventually MMQA-style reasoning over large evidence collections (Soliman et al., 19 Jan 2026). Its core object is the selected evidence subset

$Q_m=(T,I)$ 5

chosen from a pooled corpus of tables without access to db_id or gold foreign keys. This shifts the problem from “find the best table” to “find a coherent, joinable subset,” which is a precursor to multimodal evidence curation.

3. Evidence representation, localization, and retrieval backbones

A TQA-to-MMQA framework depends on how evidence is represented before reasoning begins. Several table-centric systems provide reusable abstractions.

TableReasoner replaces raw table serialization with a semantically enriched schema containing table description, column descriptions, sampled examples, and type-dependent statistics. Its key claim is that schema complexity scales as

$Q_m=(T,I)$ 6

rather than the $Q_m=(T,I)$ 7 cost of full table verbalization, where $Q_m=(T,I)$ 8 is the number of rows and $Q_m=(T,I)$ 9 the number of columns (Xiong et al., 10 Jul 2025). It then derives a focused table schema by schema linking, query refinement, and LCS-based entity disambiguation, including candidate retrieval when overlap exceeds $Q_s \rightarrow Q_m=(T,I)$ 0. This decomposition—global schema, focused schema, program generation, iterative reflection—is table-native, but it is also a plausible backbone for multimodal evidence abstraction.

TableZoomer makes the same architectural move with stronger emphasis on localization efficiency (Xiong et al., 1 Sep 2025). It constructs a global schema once per table, performs query-aware column selection and entity linking, and compresses the reasoning context aggressively. On DataBench, the ablation shows a progression from baseline PoT at 67.82 to 74.33 with schema representation, 84.67 with column selection, 86.40 with entity linking, and 87.16 with full ReAct integration. It also reports that after column selection the retained column ratio is 5.2% on DataBench, 19.8% on TableBench, and 22.6% on WikiTableQA, indicating that localization rather than raw model scale is the dominant factor for large tables.

CORE-T generalizes localization to open-book multi-table settings (Soliman et al., 19 Jan 2026). Each table is serialized into a 5-row Markdown snapshot plus an LLM-generated purpose description, embedded for dense retrieval, and indexed with FAISS. Offline it also constructs a lightweight compatibility cache storing $Q_s \rightarrow Q_m=(T,I)$ 1 and best_join_columns. At inference time, dense retrieval returns top- $Q_s \rightarrow Q_m=(T,I)$ 2 candidates, a single LLM call selects a coherent subset, and an additive adjustment restores highly compatible omitted tables when

$Q_s \rightarrow Q_m=(T,I)$ 3

This is a retrieval-and-subset-selection pattern rather than a full MMQA solution, but it is directly transferable to multimodal evidence graphs.

TQA-Bench highlights the scaling problem that motivates these abstractions (Qiu et al., 2024). It evaluates multi-table QA with serialized relational instances across context buckets from 8K to 64K tokens and shows that performance drops as context grows, especially on aggregation and complex calculation. The benchmark therefore supports a core systems lesson: a TQA-to-MMQA framework cannot rely on naïve “serialize everything and prompt once” strategies when the evidence space becomes relational or multimodal.

4. Agentic and modular reasoning architectures

A second major thread in TQA-to-MMQA research concerns control flow: how planning, execution, retrieval, and answer synthesis are distributed across agents.

Orchestra addresses single-table TQA with smaller open-weight models by splitting reasoning into a Logic agent, Query agent, and Decision agent, plus Monte Carlo repetition and majority aggregation (Jiang et al., 6 Jan 2026). The logic agent decides what evidence is needed next, the query agent turns the instruction into SQL and/or Python, and the decision agent answers from a cleaned reasoning trace. With Qwen2.5-14B, Orchestra reaches 72.1% on WikiTQ, compared with a prior best 75.3% achieved with GPT-4. The architectural point is not the specific table tools, but the decomposition of a complex QA workflow into narrower agent-local tasks.

AED in CT2C-QA uses a related but explicitly multimodal structure: an Allocating Agent, modality-specific Expert Agents for text, tables, and charts, and a Decision Agent for final answer selection (Zhao et al., 2024). The routing stage estimates probabilities over text, specific tables, and specific charts, and activates experts when relevance exceeds 0.1. This makes source uncertainty a first-class problem rather than assuming that the relevant table is known in advance.

MAMMQA pushes the decomposition further by separating modality-specific insight extraction, cross-modality synthesis, and aggregation into a static and synchronous multi-agent design (Rajput et al., 27 May 2025). In the tri-modal case the active structure expands to seven agents: three modality experts, three synthesis agents anchored on text, table, and image respectively, and one aggregator. The aggregator is deliberately blind to raw inputs and, in the best setting, even to the original question. The paper reports that withholding the question from the aggregator improves performance, for example from 82.06% to 87.61% at 7B scale on ManyModalQA. This suggests that final adjudication may benefit from being constrained to evidence summaries rather than allowed to rely on linguistic priors.

MACT offers a complementary control pattern for complex TQA: a planning agent, a coding agent, a tool set $Q_s \rightarrow Q_m=(T,I)$ 4, and a memory trace updated as

$Q_s \rightarrow Q_m=(T,I)$ 5

(Zhou et al., 2024). The planner generates actions such as Retrieval, Calculation, Search, Read, Ask, and Finish; the coder turns selected actions into Python; execution results are combined with planner-estimated observations. This is not multimodal, but it provides a reusable online planning template for MMQA: planner $Q_s \rightarrow Q_m=(T,I)$ 6 specialized executor/tool $Q_s \rightarrow Q_m=(T,I)$ 7 observation $Q_s \rightarrow Q_m=(T,I)$ 8 memory update $Q_s \rightarrow Q_m=(T,I)$ 9 replanning.

5. Evaluation rubrics, datasets, and benchmark regimes

Evaluation is central in TQA-to-MMQA work because multimodal transformation introduces failure modes that are not visible in final-answer accuracy alone.

Q-Mirror provides the most explicit rubric (Wang et al., 29 Sep 2025). Its quality framework has three principles—Information Consistency (IC), Cross-Modal Integration (CM), and Standalone Quality (QT)—with nine named metrics in the summary table and 11 fine-grained judgments in the human protocol. The final aggregate is

$\rightarrow$ 0

with fixed weights

$\rightarrow$ 1

The acceptance threshold is

$\rightarrow$ 2

Using this rubric, Q-Mirror raises average scores from 78.90 to 85.22 and pass rates from 72\% to 95\% on Q-Mirror-(Expert+Grad), while the best judge alignment scores come from Grok-4: 62.80, GPT-4.1: 62.53, and O3: 61.25. The same study reports Krippendorff’s Alpha $\rightarrow$ 3 before consensus resolution, supporting the rubric’s reliability.

MultiModalQA supplies a different benchmark regime: 29,918 questions generated from Wikipedia tables, linked text, and images, with 35.7% requiring cross-modality reasoning (Talmor et al., 2021). It also provides a typed decomposition space and reports that ImplicitDecomp reaches 51.7 F1 on multimodal questions, compared with 38.2 for AutoRouting and 90.1 for human performance. This benchmark therefore tests whether a system can perform actual cross-modal multi-hop reasoning rather than merely pick the right modality.

CT2C-QA evaluates webpage-style multimodal QA over 200 webpages and 9,981 QA pairs, using KM and CLKM rather than exact match because models may answer in English despite Chinese input (Zhao et al., 2024). End-to-end AED reaches 33.9 KM / 34.3 CLKM, compared with 2.0 / 2.4 for the adapted MultiModalQA baseline and 94.9 / 94.9 for human performance. The gap is especially large on charts, which remain the hardest modality.

A recurrent benchmarking lesson across these datasets is that evaluation must distinguish at least four phenomena: semantic fidelity to the source task, genuine multimodal dependence, intrinsic quality of each modality-specific artifact, and answer-source uncertainty. This is one reason why rubric-driven systems such as Q-Mirror place as much emphasis on evaluation as on generation.

6. Limitations, misconceptions, and open directions

Several limitations recur across the literature. The first is factual grounding of non-text modalities. Q-Mirror identifies Semantical Clarity (SC) as the hardest dimension; even the best one-shot generator has SC = 27.67, and the closed-loop agent only raises it to 54.13, indicating that scientific images may look plausible while being scientifically wrong (Wang et al., 29 Sep 2025). Table-centric backbones do not solve this problem; they mainly improve evidence localization and symbolic execution.

The second is routing and cascading error. AED’s serial design means that mistakes by the Allocating Agent propagate to table or chart selection and then to final answer choice (Zhao et al., 2024). MAMMQA faces a related brittleness: because the pipeline is staged and the aggregator is blind to raw inputs, upstream extraction failures are difficult to recover from (Rajput et al., 27 May 2025). MACT shows the same pattern in table reasoning, where planner errors and coding-agent errors account for a large fraction of failures (Zhou et al., 2024).

The third is evidence selection under scale. CORE-T shows that dense retrieval alone gives high recall but many distractors, while coherent subset selection improves table-selection F1 by up to 22.7 points and retrieves up to 42% fewer tables (Soliman et al., 19 Jan 2026). TQA-Bench likewise shows that long serialized contexts degrade performance most sharply on aggregation and correlation (Qiu et al., 2024). This suggests that full MMQA will require explicit compatibility modeling, not just bigger context windows.

The fourth is cost and system complexity. Orchestra is substantially more expensive than CoT and more expensive than ReAcTable on TableBench, with 37.3 seconds, 23.7 API requests, and 179,752 input tokens per question under the reported setup (Jiang et al., 6 Jan 2026). MAMMQA explicitly acknowledges high inference latency, memory usage, and monetary cost due to separate experts and synthesis agents (Rajput et al., 27 May 2025). A plausible implication is that future TQA-to-MMQA systems will need adaptive routing so that multimodal orchestration is invoked only when evidence ambiguity justifies it.

A final misconception is that a TQA-to-MMQA framework is simply a larger multimodal model. The literature points in a different direction. The strongest designs are modular: schema or metadata construction, query-aware localization, tool-grounded execution, expert routing, synthesis, and rubric-based verification. What remains unresolved is how to generalize these modules from tables and generated images to richer multimodal evidence with OCR noise, chart semantics, document layout, and soft cross-modal links. This suggests that the next stage of the framework will likely center on multimodal evidence schemas, cross-modal compatibility graphs, and verification loops that can detect not only wrong answers, but wrong modality bindings and wrong intermediate grounding.