Caption-Assisted Reasoning Framework

• Caption-assisted reasoning frameworks are systems that convert visual data into structured textual captions, decoupling perception from reasoning.
• They employ roles such as auxiliary evidence, trainable interfaces, and regularizers to enhance accuracy and modularity in multimodal tasks.
• Empirical studies indicate significant performance gains in areas like geometry and visual QA, yet emphasize the critical impact of caption quality.

Caption-assisted reasoning framework denotes a class of multimodal systems in which an image or video is converted into an intermediate textual representation—a caption, structured description, evidence list, or morphological report—and that representation is then used to support, steer, regularize, or verify downstream reasoning. In recent work, the caption may function as auxiliary inference-time evidence, a trainable interface between a vision module and a frozen text-only reasoner, a policy-optimization scaffold, or a verified perceptual substrate for self-training (Gou et al., 5 Jun 2025, Gkountouras et al., 30 Sep 2025, Li et al., 3 May 2026, Tu et al., 26 Sep 2025, Sharma et al., 20 Jun 2026). Earlier precursors already treated captions as semantic objects to be probed or enriched rather than mere outputs, including VQA-based fact-plausibility representations for image-caption ranking and knowledge-augmented caption generation with ConceptNet-derived lexical biasing (Lin et al., 2016, Huang et al., 2020).

1. Definition and lineage

A caption-assisted reasoning framework is characterized by an explicit separation between perception and reasoning. Instead of requiring one model to jointly parse raw visual input and solve the task end-to-end, the framework inserts a caption-like textual layer that can be inspected, optimized, or controlled. In RACRO, this appears as perceptual decoupling: a visual extractor $M_V$ produces a query-conditioned caption $\mathbf{c}$, and a stronger text-only reasoner $M_R$ solves the task from textualized visual information (Gou et al., 5 Jun 2025). In AC-RL, the caption is described as an “interface contract” between a trainable vision-language captioner and a frozen text-only reasoner, with clarification behavior used as supervision for what the interface should contain (Gkountouras et al., 30 Sep 2025). In CapGeo, the same pattern is specialized to geometry, where a figure is converted into a formal, information-dense textual description before answer generation (Li et al., 10 Oct 2025).

The lineage is broader than recent multimodal reasoning. An early precursor used VQA as a feature extraction module for both images and captions, with each feature dimension corresponding to the plausibility of a question-answer fact for the image or caption; image-caption matching then became a problem of semantic consistency under probing (Lin et al., 2016). Another precursor improved captioning by combining word-attention-guided visual grounding with external knowledge from ConceptNet, already framing captions as outputs that should reflect both visible evidence and latent semantic implications (Huang et al., 2020). Recent frameworks extend this from caption generation to multimodal decision making, where the caption is a first-class computational object rather than a byproduct.

2. Main architectural patterns

Across current work, the caption serves several technically distinct roles. A useful synthesis is that caption-assisted reasoning is not one architecture but a family of architectures organized by how the caption enters the computation.

Caption role	Representative frameworks	Defining mechanism
Inference-time auxiliary evidence	GEASS, CapGeo, SeePhys pipeline	Caption fused with answer generation, optionally together with the original image
Trainable interface	AC-RL, RACRO, PRCO	Captioner optimized by downstream task success or solver utility
Training-time regularizer or verifier	CapPO, perception-verified self-training, Visionary-R1	Caption-conditioned consistency, caption verification, or caption reward
Structured reasoning target	CineCap, CPJ, Agri-CPJ	Caption itself is a grounded reasoning trace or audit trail

In decoupled systems, the core pattern is:

$A = \text{MLLM}(Q, I, C),$

or, in caption-only variants, reasoning proceeds from $Q$ and $C$ alone (Li et al., 10 Oct 2025). The caption thus acts as a semantic bridge that can replace redundant visual tokens with structured text, a design exploited in scientific reasoning on SeePhys and in geometric reasoning on MathVerse (Liang et al., 7 Sep 2025, Li et al., 10 Oct 2025).

Other frameworks retain the image but regulate caption influence rather than committing fully to decoupling. GEASS runs a clean image-question path and a caption-conditioned path in parallel, then performs residual logit interpolation:

$$Z_{\text{final}}^{(t)} = Z_{\text{clean}}^{(t)} + \alpha_{\text{eff}}^{(t)}\left(Z_{\text{cap}}^{(t)} - Z_{\text{clean}}^{(t)}\right),$$

where $\alpha_{\text{eff}}^{(t)}$ depends on clean-path confidence, entropy reduction, and agreement structure (Li et al., 3 May 2026). PRCO splits a single policy into an Observer that writes a question-conditioned evidence caption and a Solver that answers from that caption, using role-specific rewards while sharing parameters (Miao et al., 30 Mar 2026). CapPO does not use captions at inference time at all; instead, it regularizes the image-conditioned policy against a caption-conditioned policy during RL, treating captions as a semantic anchor for perceptual consistency (Tu et al., 26 Sep 2025).

3. Why captions help, and why they can hurt

The strongest positive argument for caption assistance is that caption generation and task-conditioned answering are not the same perceptual process. GEASS argues that answering is query-driven and often locally focused, whereas a generic prompt such as “Describe this image in detail” induces a broader, saliency-weighted scan that can surface objects or scene information the question-conditioned path overlooks (Li et al., 3 May 2026). AC-RL makes a closely related claim from the opposite direction: current VLM captions are usually optimized for human readers, but downstream reasoners in visual mathematical tasks need exact transcriptions, values, geometric relations, and diagram structure; failures therefore arise from an interface mismatch rather than from reasoning weakness alone (Gkountouras et al., 30 Sep 2025). In sparse scientific diagrams, the same logic appears as a compression advantage: a short structured caption can state the few semantically decisive relations directly, avoiding the burden of operating over many irrelevant visual tokens (Liang et al., 7 Sep 2025).

The negative case is equally central. Recent work rejects the assumption that captions are uniformly beneficial. On HallusionBench with Qwen2.5-VL-3B+, unconditional self-caption injection drops performance from 61.19 to 51.31, nearly 10 points (Li et al., 3 May 2026). The reason is not simply occasional factual error. Captions alter the model’s reasoning trajectory, lexical choices, and phrase structure; the paper terms this the anchoring effect, and quantifies it by sharply increased n-gram overlap between the embedded caption and the final answer. GEASS further shows that caption errors are asymmetric: omissions vastly outnumber fabrications, but fabricated object mentions are far more damaging per instance. On 500 COCO validation images, InternVL2-8B exhibits an omission-to-fabrication ratio of 20.1:1, yet on 100 paired instances a fabricated object mention shifts the prediction toward the wrong answer by $\Delta p = 0.64$ on average and flips the binary decision 87% of the time, whereas omissions produce $\Delta p = 0.13$ and an 11% flip rate (Li et al., 3 May 2026). The resulting design principle is that caption usefulness is per-query, not per-corpus.

A related misconception is that caption absence should be interpreted as evidence of object absence. GEASS explicitly argues the opposite: positive assertions in a caption are relatively trustworthy, whereas silences are weak evidence. This asymmetry underlies disagreement-aware steering, because a disagreeing caption is more likely to contain a damaging fabrication than a genuinely corrective override (Li et al., 3 May 2026). The same basic concern appears in Agri-CPJ and CPJ, where captions are explicitly forbidden from naming the crop or disease in order to prevent label leakage from contaminating downstream diagnosis (Zhang et al., 31 Dec 2025, Zhang et al., 26 Apr 2026).

4. Learning and control mechanisms

Current frameworks differ most sharply in how they optimize or regulate the caption channel. One family performs inference-time control without training. GEASS is the canonical example: it is training-free, plug-and-play, and operates at the logit level. Caption influence is opened only when the clean path is uncertain, weighted only when the caption reduces entropy, and penalized when clean and caption paths disagree (Li et al., 3 May 2026).

A second family optimizes the captioner directly using downstream task behavior. AC-RL treats clarification requests as implicit supervision and uses a three-tier reward:

$\mathbf{c}$0

with $\mathbf{c}$1 in experiments (Gkountouras et al., 30 Sep 2025). This makes direct single-pass solvability the deployment objective while still learning from training-time interaction. RACRO uses an even more explicit reasoning loop: the captioner samples multiple query-conditioned captions, a frozen text reasoner attempts the task from each caption, and the caption policy is updated by GRPO-style reward optimization based on whether the downstream reasoner answers correctly (Gou et al., 5 Jun 2025). PRCO similarly makes caption utility a reward signal, but with a dual-role formulation in which the Observer is rewarded by Solver success conditioned on its caption, subject to answer-leakage suppression (Miao et al., 30 Mar 2026).

A third family uses the caption as a regularizer or verifier during reasoning training. Visionary-R1 enforces a caption $\mathbf{c}$2 reason $\mathbf{c}$3 answer format, rewards not only answer correctness and output format but also whether the generated caption alone is sufficient for answering, and uses cosine-annealed KL to stabilize multimodal RL (Xia et al., 20 May 2025). CapPO regularizes image-conditioned reasoning against caption-conditioned reasoning by minimizing a caption-based KL term and further uses KL-weighted advantages so that perceptually inconsistent positive trajectories are downweighted and perceptually inconsistent negative ones are punished more strongly (Tu et al., 26 Sep 2025). Perception-verified self-training goes further by explicitly decomposing outputs into [Cap–Reas–Concl], then using PerceptEval—an OCR-based and FG-CLIP-based caption verifier—to partition samples into easy, medium, and hard subsets before curriculum-based self-training (Sharma et al., 20 Jun 2026).

A fourth family treats the caption itself as a structured reasoning trace and optimizes it at the semantic-unit level. CineCap decomposes cinematographic descriptions into atomic statements with spatial and temporal anchors, then uses GRPO with separate rewards for comprehensiveness and accuracy plus a gated coverage reward that activates only when factual accuracy exceeds a threshold (Mao et al., 23 Jun 2026). CPJ and Agri-CPJ use iterative caption refinement with multi-dimensional quality gating before any answer is produced, then generate two complementary candidate answers and apply rubric-based LLM judging for selection (Zhang et al., 31 Dec 2025, Zhang et al., 26 Apr 2026).

5. Empirical performance across domains

Empirical gains are strongest in domains where the perceptual bottleneck is sharper than the reasoning bottleneck. In geometry, CapGeo reports exceptionally large improvements: Qwen2.5-VL-72B rises from 8.6% in vision-only mode to 59.0% with caption assistance, while Claude-Opus-4 rises from 44.8% to 73.0% (Li et al., 10 Oct 2025). CapGeo-Bench then shows that caption quality is not incidental: GPT-o3 is the strongest captioner on elements, relations, and numerical relations, and the paper reports a high positive correlation between its three-dimensional caption score and downstream reasoning performance. The same geometry-focused perspective appears in synthetic data generation: RLVR-refined geometric captions improve both in-domain geometric reasoning and, according to the paper, out-of-distribution reasoning in statistics, arithmetic, algebraic, and numerical tasks (Xin et al., 18 Sep 2025).

In visual mathematical reasoning, the caption as interface view is strongly supported. AC-RL improves the average single-pass accuracy of the Qwen-3B captioner plus frozen reasoner from 39.0 to 43.4, a gain of +4.4 points across seven benchmarks, and analysis suggests clarification requests would be reduced by up to 39% (Gkountouras et al., 30 Sep 2025). The SeePhys technical report shows that structured captioning, image reintegration, format optimization, and critical review form an effective test-time pipeline: on SeePhys-mini, the best direct multimodal baseline reaches 58.0%, whereas the best caption-assisted pipeline reaches 66.0%, and the method also achieved 1st place in the ICML 2025 SeePhys challenge (Liang et al., 7 Sep 2025).

In RL-trained VLM reasoning, caption-first strategies consistently outperform answer-only reward optimization. Visionary-R1, trained from 272.6K CoT-free visual QA pairs using only RL, achieves 69.4 on MathVista, 24.7 on MathVision, 66.5 on MMStar, and 84.1 on MMBench, exceeding both QA-only SFT and vanilla GRPO and surpassing GPT-4o, Claude 3.5 Sonnet, and Gemini-1.5-Pro on several benchmarks listed in the paper (Xia et al., 20 May 2025). CapPO improves average math-related score on Qwen2.5-VL-7B from 44.8 to 50.8 and average general reasoning score from 59.5 to 61.9, while reducing MathVista perception-error share from 51.3% to 45.8% (Tu et al., 26 Sep 2025). PRCO, which explicitly couples caption utility and solver success, improves average accuracy by more than 7 points over the base model across multiple scales and reports a 39.2% reduction in perception errors on WeMath (Miao et al., 30 Mar 2026).

The same pattern extends beyond math and science. On CDDMBench, CPJ reports that GPT-5-Nano with GPT-5-mini captions improves disease classification by +22.7 percentage points and QA score by +19.5 over the no-caption baseline, while providing a readable evidence trail (Zhang et al., 31 Dec 2025). Agri-CPJ reproduces the same headline gains and further reports 77.84% on AgMMU-MCQs without modifying the core caption-first pipeline (Zhang et al., 26 Apr 2026). In video understanding, CineCap—where the caption itself is the reasoning output—raises Qwen3-VL-8B from 41.16 F1 to 73.57 F1 on CineCap Bench, with the largest gains on camera movement, shot size, depth of field, and composition (Mao et al., 23 Jun 2026).

6. Limitations, misconceptions, and open problems

Three misconceptions recur in the literature. The first is that captions are intrinsically beneficial. GEASS directly falsifies this with nearly 10-point collapse under unconditional self-caption injection and argues that query-conditional trust, not blanket use, is the correct principle (Li et al., 3 May 2026). The second is that human-readable caption quality is the right optimization target. AC-RL explicitly states the opposite: informative captions are reasoner-specific and should be optimized for machine usefulness and direct solvability rather than human-style naturalness (Gkountouras et al., 30 Sep 2025). The third is that captions can simply replace images. Some systems indeed show caption-only reasoning rivaling or exceeding image-plus-caption input, especially in geometry (Li et al., 10 Oct 2025), but other frameworks require image reintegration or restore image access after caption-first warmup because textualization is lossy (Liang et al., 7 Sep 2025, Miao et al., 30 Mar 2026).

The main technical limitation is that captions are compressed and fallible representations. CapPO notes that stronger captioners help more, but even weaker synthetic captions still act as noisy anchors rather than perfect surrogates (Tu et al., 26 Sep 2025). PRCO explicitly states that short evidence captions are inherently lossy for fine spatial structure, which is why solver image access after warmup remains beneficial (Miao et al., 30 Mar 2026). CapGeo shows that numerical grounding remains the hardest captioning dimension: even GPT-o3 attains only 26.0 on Numerical Score in CapGeo-Bench (Li et al., 10 Oct 2025). Perception-verified self-training acknowledges that caption verification can still fail when captions omit crucial details or are only partially informative (Sharma et al., 20 Jun 2026). Agri-CPJ likewise remains vulnerable to early-stage symptom ambiguity, co-infection cases, and judge calibration issues despite its strong audit trail (Zhang et al., 26 Apr 2026).

Current research therefore converges on a more specific view of caption assistance. A caption-assisted reasoning framework is most effective when the caption is treated neither as a generic description nor as a trusted oracle, but as a controllable, inspectable, and task-conditioned interface whose value depends on grounding fidelity, query relevance, and the way it interacts with the downstream reasoning policy. This suggests continued work on object-level verification, better caption quality estimation, question-conditioned caption synthesis, more precise control of disagreement between image and text paths, and richer interfaces that preserve perceptual structure without sacrificing the modular advantages of textual reasoning (Li et al., 3 May 2026, Tu et al., 26 Sep 2025, Sharma et al., 20 Jun 2026).