Adaptive-Clarification RL

• Adaptive-Clarification RL is a reinforcement learning framework that treats clarification as an implicit supervision signal to bridge gaps between a vision-language captioner and a fixed reasoner.
• The method decouples caption generation and reasoning, using a tiered reward system that differentiates direct success from clarification-dependent outcomes while ensuring single-pass deployment.
• Empirical evaluations on visual mathematical benchmarks demonstrate significant accuracy gains (up to +10.6 points) and more selective clarification requests, enhancing downstream performance.

Adaptive-Clarification Reinforcement Learning (AC-RL) is a reinforcement-learning framework in which clarification is treated as a learning signal rather than merely a fallback interaction. Its most explicit formulation trains a vision-language captioner to produce captions optimized for a fixed downstream reasoner by using clarification requests as implicit supervision: success without clarification is preferred, success after clarification receives partial credit, and failure receives zero reward (Gkountouras et al., 30 Sep 2025). Closely related work applies clarification-aware rewards and policies to abstention with post-refusal explanation, conversational question answering, task-initiative communication, and interactive query disambiguation, which suggests a broader clarification-centric reinforcement-learning pattern across modular and interactive systems (Zhai et al., 18 Apr 2026, Mukherjee et al., 8 Jun 2025, Khalid et al., 2023, Hu et al., 2020).

1. Interface alignment as the central problem

In the formulation introduced in "Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces" (Gkountouras et al., 30 Sep 2025), AC-RL addresses a specific failure mode in modular vision-language systems: a captioner trained to produce “good captions” for humans is not necessarily producing the right information for a downstream reasoner. The architecture is decoupled. A trainable vision-language captioner $\mathcal{V}_\theta$ maps an image $I$ and question $Q$ to text, while a frozen text-only reasoner $\mathcal{R}$ attempts to solve the problem from that text. This decoupling allows reuse of strong text-only reasoning models, but it creates an interface mismatch because captions optimized for generic visual description or human readability may omit the precise details a particular reasoner needs.

The framework therefore shifts emphasis from general perception quality to interface alignment. The reasoner’s failure is framed not primarily as a reasoning deficit, but as a failure of the captioner to expose a crucial number, spatial relation, structure, or text snippet. A standard supervised remedy would require “ideal captions” tailored to each reasoner, but the formulation argues that this is infeasible because different reasoners need different kinds of detail, those preferences are implicit and task-specific, and the right caption format emerges only through interaction. AC-RL thus optimizes not universal caption quality but reasoner-specific downstream solvability.

This problem statement also clarifies why clarification matters. If a reasoner asks a follow-up question, the request itself reveals an information gap in the initial caption. AC-RL interprets that gap as supervision. The captioner is then trained to front-load the missing information so that future problems can be solved in a single pass.

2. Formal mechanism, reward shaping, and optimization

The full AC-RL training setup contains three components: a trainable captioner that generates the initial description $c_0$, a frozen reasoner $\mathcal{R}$, and a frozen clarification captioner $\pi_{\text{ref}}$ that is used only if clarification is requested (Gkountouras et al., 30 Sep 2025). For each image-question pair $(I,Q)$, the captioner first samples

$c_0 \sim \pi_\theta(\cdot \mid I,Q).$

The reasoner then either answers directly or requests clarification $q_1$. If clarification is requested, the frozen reference captioner produces a follow-up visual description

$I$0

The final answer $I$1 is produced from either $I$2 or $I$3. Only the initial caption generation is trainable; the reasoner and the clarification response are treated as fixed, $I$4-independent parts of the environment.

At inference, the system is deliberately simplified to a single-pass regime. The captioner generates only $I$5, and the reasoner must answer without clarification. This design is integral rather than incidental: interaction is used during training in order to eliminate dependence on interaction at deployment.

The reward is tiered rather than binary:

$I$6

with $I$7. The penalty for needing clarification is therefore $I$8. The stated purpose of this shaping is twofold: it densifies the signal, because many episodes that would have been zero reward under binary RL now receive partial reward, and it pushes toward single-pass sufficiency, because clarification-dependent success is rewarded less than direct success.

The optimization objective is KL-regularized:

$I$9

and the appendix gives a clipped surrogate objective

$Q$0

with

$Q$1

Optimization is performed with Beta-Normalization Policy Optimization (BNPO), using group size $Q$2 and $Q$3 iterations. Gradients flow only through the initial caption tokens $Q$4, not through clarification responses.

The appendix further proves that the three-tier reward preserves REINFORCE unbiasedness under the assumption that post-action randomness is $Q$5-independent. The theorem states

$Q$6

and the estimator

$Q$7

satisfies

$Q$8

Formally, the clarification machinery is therefore incorporated as part of the environment rather than as a differentiable subsystem.

3. Empirical profile in visual mathematical reasoning

The primary AC-RL evaluation spans seven visual mathematical reasoning benchmarks: MathVista, MathVision, MathVerse, MMMU, WeMath, DynaMath, and LogicVista (Gkountouras et al., 30 Sep 2025). The captioner is instantiated with InternVL3-2B and Qwen2.5-VL-3B, while the frozen reasoner is DeepSeek-R1-Qwen-32B. The headline result is an average-accuracy improvement over pretrained plus reasoner baselines: InternVL-2B improves from 39.3 to 42.6, a +3.3 point gain, and Qwen-3B improves from 39.0 to 43.4, a +4.4 point gain. For Qwen-3B, especially large gains are reported on DynaMath (+10.6), LogicVista (+5.8), and MathVerse (+5.2). For InternVL-2B, the largest gains are on DynaMath (+8.0), MathVerse (+7.9), and MathVision (+2.0).

The main ablation with Qwen2.5-VL-3B shows a progression from VLM-only at 34.64, to Decoupled (No RL) at 39.00, to Binary-Reward RL at 40.27, to AC-RL at 43.39. In that summary, AC-RL yields about +4.4 over the decoupled baseline, about +3.1 over binary-reward RL, and about +8.7 over standalone VLM. The largest benchmark-specific gain over Binary-Reward RL is on DynaMath, where performance moves from 17.56 to 24.75, a reported +7.2.

Mechanistic analyses focus on clarification frequency and self-sufficiency of the initial caption. During clarification-enabled evaluation, clarification request rates fall from 40.69% to 28.95% on MathVision and from 49.57% to 30.28% on MathVerse-MINI, corresponding to reductions of 29% and 39%. The clarification gap—defined by comparing clarification-enabled versus single-pass performance—also shrinks. On MathVision, AC-RL yields 37.66 versus 36.71, a gap of +0.95, whereas Binary-Reward yields 37.20 versus 34.31, a gap of +2.89. On MathVerse-MINI, AC-RL yields 34.26 versus 36.80, a gap of -2.54, while Binary-Reward yields 35.15 versus 31.09, a gap of +4.06. The negative gap on MathVerse is explicitly noted as suggesting that extra clarification can introduce noise.

A denied-clarification analysis further differentiates suppression of requests from selective use of requests. On MathVision, AC-RL makes 880 requests; single-pass accuracy is 36.71, denied-clarification accuracy is 22.50, and the drop is 14.21. Binary-Reward makes 1237 requests; single-pass accuracy is 34.31, denied-clarification accuracy is 22.88, and the drop is 11.43. On MathVerse-MINI, AC-RL makes 276 requests with a drop of 3.47, while Binary-Reward makes 496 requests with a drop of 2.66. The interpretation supplied in the study is that AC-RL asks less often, but when it asks, those requests are more selective and more important. The appendix also reports that AC-RL maintains greater generation diversity than standard binary RL, with approximate convergence values of about 0.31 uniform-reward batches for AC-RL versus 0.42 for standard RL.

4. Related clarification-aware reinforcement-learning formulations

A closely related formulation appears in "Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL" (Zhai et al., 18 Apr 2026). There the central distinction is between answerable queries and unanswerable queries that are semantically clear but cannot be reliably resolved from the given information. The method uses GRPO with a composite reward. For answerable queries, correct answers are rewarded and boxed “I don’t know” is penalized. For unanswerable queries, the reward is explicitly two-stage: 0.3 for boxed “I don’t know,” plus an additional 0.7 when a verifier judges the post-refusal clarification to be semantically aligned with a reference clarification. On Abstain-Test, the 3B Abstain-R1 model improves over its Qwen2.5 3B Instruct base from 9.4 to 68.1 on U-Ref and from 0.6 to 55.1 on U-Clar, while conditional answer accuracy on answered answerable examples rises from 60.1 to 71.9. The formulation can therefore be read as a clarification-aware policy in which abstention is not the terminal behavior but the trigger for explanation of missing information.

"Learning to Clarify by Reinforcement Learning Through Reward-Weighted Fine-Tuning" (Mukherjee et al., 8 Jun 2025) formulates clarification in conversational QA as an RL problem over trajectories $Q$9, where actions are clarifying questions and observations are user answers. The paper’s practical contribution is an offline RL lower bound that reduces optimization to reward-weighted supervised fine-tuning:

$\mathcal{R}$0

After expansion, the objective becomes reward-weighted likelihood maximization over the question sequence, and a standardized variant uses per-context reward normalization. The method is evaluated on OpenBookQA, ARC, SciQA, MMLU, CoSQL, and MathDial, in both thinking mode and standard mode, with Llama-3.1-8B-Instruct as the backbone. Its stated contrast is with SFT and DPO-style methods that threshold or binarize reward rather than directly optimize it.

Earlier dialogue work studies clarification as interactive strategy rather than modular supervision. "Investigating Reinforcement Learning for Communication Strategies in a Task-Initiative Setting" (Khalid et al., 2023) uses the Colors in Context referential communication task and optimizes a director policy with deep Q-learning under a reward that trades dialogue success against verbosity. The system must balance initial presentation against subsequent follow-up when a clarifying matcher asks questions. With an always-select matcher, DQN reaches 95.5% success and reward 0.891, versus 94.7% and 0.880 for Direct and 97.8% and 0.874 for Extended. With a clarifying matcher, DQN and Direct are nearly identical at 95.9% success and reward 0.901 versus 95.8% and 0.899. In that study, clarification ability largely erodes the advantage of longer initial descriptions.

"Interactive Question Clarification in Dialogue via Reinforcement Learning" (Hu et al., 2020) addresses ambiguous questions in a closed-domain customer-service setting by suggesting intent phrases rather than generating free-form clarifying questions. Clarification is formulated as a collection partitioning problem over candidate intents, and the policy is trained with a deep policy network plus Monte Carlo Tree Search using a reward that combines coverage and information gain:

$\mathcal{R}$1

with default $\mathcal{R}$2. On 40k ambiguous questions, RL (ours) achieves Recall@6 of 57.22%, compared with 51.53% for Supervised, 32.01% for Greedy, and 29.83% for RL (no state transition). In online evaluation, RL (ours) yields THA 14.20% and CTR 66.36%, compared with THA 15.40% for Top-K intents and THA 14.51%, CTR 62.61% for RL (recall).

5. Conceptual distinctions and recurrent misconceptions

AC-RL is not equivalent to making outputs longer, more verbose, or generically more descriptive. In the visual formulation, it is explicitly “not just captioning better or longer.” No explicit “ideal caption” labels are required; the objective is not human readability but downstream solvability by a fixed reasoner; the supervision signal is implicit and interaction-based; clarification requests are exploited as training data rather than treated as an inconvenient failure mode; and the model is trained for single-pass deployment even though interaction is used during training (Gkountouras et al., 30 Sep 2025). Compared with ordinary captioning, the learned behavior is reasoner-aware description rather than generic description. Compared with binary-reward RL, the method distinguishes direct success from success after clarification.

A related misconception is that reduction in clarification frequency necessarily means a model has learned to suppress questions indiscriminately. The denied-clarification analysis in the vision-language setting does not support that reading: clarification is requested less often, but the requests that remain are more selective and more important (Gkountouras et al., 30 Sep 2025). This suggests a distinction between request frequency and request utility.

The dialogue literature adds a second caution. If clarification is already available and effective, the marginal benefit of elaborate initial presentation can become small. In the referential communication study, DQN departs only modestly from hand-crafted strategies, and with a clarifying matcher the learned policy is essentially a variation of the direct policy (Khalid et al., 2023). A plausible implication is that the value of AC-RL depends strongly on where the communication bottleneck lies: initial under-specification, follow-up behavior, or the coordination between the two.

The abstention setting raises a third distinction. Clarification after refusal is not reducible to generic refusal. In Abstain-R1, removing the clarification reward barely changes refusal but lowers clarification quality, while removing unanswerable data causes refusal to collapse (Zhai et al., 18 Apr 2026). The reward therefore separates the act of abstaining from the semantic content of explaining what is missing.

6. Scope, limitations, and terminological boundaries

The current AC-RL literature is heterogeneous in assumptions and supervision sources. The visual formulation depends on a fixed downstream reasoner and a frozen clarification captioner, and its objective is explicitly reasoner-specific rather than universal (Gkountouras et al., 30 Sep 2025). Abstain-R1 relies on data labeled into answerable and unanswerable subsets, a strict output format with boxed “I don’t know,” and a verifier that judges semantic alignment of clarification against a reference (Zhai et al., 18 Apr 2026). Reward-weighted clarification learning in conversational QA is offline RL on logged data, and the method itself notes that offline RL depends heavily on logged data quality, that the lower bound is loose, and that the experimental setup uses fixed three-turn conversations with scripted user behavior rather than a fully open-ended human-in-the-loop environment (Mukherjee et al., 8 Jun 2025). The task-initiative communication study learns in simulation with a user model rather than live interaction, and it stresses that a human-subject study would likely need very large samples to detect differences reliably because learned and hand-crafted strategies are so similar (Khalid et al., 2023). The ambiguous-query clarification system is closed-domain, relies on a predefined intent inventory and manually constructed labels, and focuses on label selection rather than free-form clarification generation (Hu et al., 2020).

The literature also imposes different notions of “clarification.” In the vision-language case, clarification is a signal about missing visual information; in calibrated abstention, it is an explanation of why a query is unanswerable; in conversational QA, it is a sequence of information-seeking questions; in task-initiative dialogue, it is part of grounding and repair; and in customer-service disambiguation, it is label suggestion for intent refinement. This suggests that AC-RL is better understood as a design pattern centered on clarification-aware reward and policy shaping than as a single algorithmic template.

A terminological caveat is necessary because the acronym “AC-RL” is also used outside clarification research. "Online Algorithms and Policies Using Adaptive and Machine Learning Approaches" uses AC-RL to denote an outer-loop Reinforcement Learning policy combined with inner-loop Adaptive Control for nonlinear dynamic systems under parametric uncertainty, including a quadrotor landing task on a moving platform (Annaswamy et al., 2021). That usage concerns adaptive control and real-time robustness rather than clarification. In the clarification literature, by contrast, the defining issue is how interaction reveals missing information and how that signal is converted into better initial outputs, better abstention behavior, or better follow-up questioning.