Multi-Answer Reinforcement Learning

• Multi-answer reinforcement learning is a paradigm that extends traditional RL by optimizing collections of answers, advisor recommendations, or sequential refinements instead of a single output.
• It incorporates diverse formulations such as advisor aggregation, abstention-aware decision making, and multi-attempt interaction to handle ambiguity and uncertainty.
• Key applications include question answering, planning with structured outputs, and distributional reasoning, resulting in improved coverage, diversity, and calibration.

to=arxiv_search.search 东臣 天天爱彩票app 亚洲男人天堂 հարցադրանք: "multi-answer reinforcement learning GRPO-MA multi-advisor RL abstention RL question answering 2025 2026" limit: 10 to=arxiv_search.search สล็อตโjson_string to=arxiv_search.search 彩票开号 ／久久 ագրументы 利盛json {"query":"multi-answer reinforcement learning GRPO-MA multi-advisor reinforcement learning", "max_results": 10} arxiv_search.search(query="multi-answer reinforcement learning GRPO-MA multi-advisor reinforcement learning", max_results=10) Multi-answer reinforcement learning denotes a family of reinforcement-learning formulations in which optimization is not confined to a single terminal answer. The literature suggests that the common pattern is set-valued, abstention-aware, multi-source, or multi-stage decision making: a policy may aggregate multiple advisors, emit multiple candidate answers in one forward pass, decide to abstain, refine answers over repeated attempts, or jointly generate coupled outputs such as plans, citations, reasoning traces, and final answers. Across these settings, the central technical move is to replace the standard single-completion objective with a reward structure that scores collections of answers, answer-confidence pairs, advisor-conditioned actions, or sequential revisions rather than a lone response (Laroche et al., 2017, Subramanian et al., 2023, Godin et al., 2019, Puri et al., 25 Mar 2026).

1. Scope of the paradigm

The literature suggests that “multi-answer” RL is not a single standardized formalism but a cluster of related problem settings. In one line of work, the multiple objects are advisors: each learner produces local action-values, and an aggregator selects the environment action from the collection of local recommendations. In another, the policy explicitly generates a set of candidate answers $A=\{a_1,\ldots,a_K\}$ and is rewarded for set-level recovery of the valid answer set $\mathcal{Y}^*(x)$. In a third, the environment allows repeated attempts, so the policy is optimized over a sequence of answers conditioned on explicit failure feedback. A further extension treats the output as a structured bundle—plan, evidence declaration, reasoning trace, and answer—whose components are jointly rewarded (Laroche et al., 2017, Puri et al., 25 Mar 2026, Chung et al., 4 Mar 2025, Liu et al., 1 Feb 2026).

These formulations differ in surface form but share a technical departure from standard single-answer RL. The usual objective is written as

$$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$

often with binary correctness reward. Multi-answer formulations replace $y$ by a set, sequence, or composite object. For set-valued generation, one representative formulation is

$$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$

and its calibrated variant subtracts a Multi-Brier-style term over confidences $Q=\{q_1,\dots,q_K\}$. For multi-attempt interaction, the policy conditions on the full dialogue history and maximizes a sum of rewards over turns. For advisor aggregation, the global action is selected from the summed local values $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$ (Puri et al., 25 Mar 2026, Chung et al., 4 Mar 2025, Laroche et al., 2017).

A plausible implication is that the field is unified less by a particular optimizer than by a change in the object of control: the learned policy must represent ambiguity, disagreement, partial observability, or iterative correction explicitly rather than collapsing them into a single best guess.

2. Multi-advisor formulations and aggregated control

A foundational formulation appears in “Multi-Advisor Reinforcement Learning” (Laroche et al., 2017). There, a single-agent MDP is decomposed into $n$ advisors, each with local state $x_j=\phi_j(x)$, local reward $R_j(x_j,a)$, and local action-values $\mathcal{Y}^*(x)$0. The aggregator uses the linear sum

$\mathcal{Y}^*(x)$1

followed by greedy action selection. The paper shows that the local planning semantics are decisive. Under egocentric planning, each advisor evaluates its own greedy future control,

$\mathcal{Y}^*(x)$2

which induces a $\mathcal{Y}^*(x)$3 versus $\mathcal{Y}^*(x)$4 mismatch and can create attractors when advisors disagree. Under agnostic planning, each advisor evaluates future random actions, avoiding this attractor mechanism but becoming overly cautious in danger zones. The proposed empathic planning instead evaluates the aggregator’s future action,

$\mathcal{Y}^*(x)$5

and the paper proves that if all advisors are defined on the full state space, empathic planning converges to the global optimal policy (Laroche et al., 2017).

The multi-agent extension in “Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning” (Subramanian et al., 2023) generalizes this advisor perspective to stochastic games. Each agent $\mathcal{Y}^*(x)$6 has its own advisor set $\mathcal{Y}^*(x)$7, and the method uses a two-level architecture. The low-level table estimates the value of the agent’s own actions against the other agents’ joint action,

$\mathcal{Y}^*(x)$8

while the high-level table directly estimates the return of following each advisor from the current state onward,

$\mathcal{Y}^*(x)$9

Action selection is itself two-level. With probability $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$0, the agent follows advisor guidance; otherwise it follows its low-level policy, with $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$1 linearly decayed to zero. When advisors are consulted, the method evaluates all advisors at the current state and computes a value of vote for each action,

$$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$2

The paper interprets this as a transition from early crowd vote behavior in rarely visited states to a later wisdom of individual effect in frequently visited states. It also proves that, under infinite visitation, Robbins–Monro learning rates, and a global-optimum/saddle-point assumption on each stage game, the low-level $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$3-values converge to Nash $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$4-values. Empirically, across seven experiments in Pommerman, Pursuit, and MPE, the method is reported to outperform DQN, DQfD, CHAT, ADMIRAL-DM, and TLQL, while learning to combine complementary expertise and ignore bad advice (Subramanian et al., 2023).

Taken together, these advisor-based formulations treat “multiple answers” as multiple action-value recommendations. Their central problem is not answer diversity but disagreement management and calibration of local advice to the eventual control rule.

3. Abstention, candidate sets, and calibrated uncertainty

In question answering over knowledge graphs, multi-answer RL frequently takes the form of abstention-aware decision making. “Learning When Not to Answer: A Ternary Reward Structure for Reinforcement Learning based Question Answering” (Godin et al., 2019) argues that binary rewards and ranking-style metrics such as Hits@$$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$5 and MRR are mismatched to practical QA because they do not distinguish incorrect answers from abstention. The proposed terminal reward is ternary: $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$6 with $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$7 and $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$8. Abstention is implemented by adding a synthetic “no answer” action from every entity to a special node $$\max_\theta \; \mathbb{E}_{(x, y^*) \sim D, \; y \sim \pi_\theta(\cdot \mid x)} \big[ R(y, y^*) \big],$$9. The evaluation protocol replaces Hits@$y$0/MRR with Answer Rate, Precision, and QA Score

$y$1

On FB15k-237, standard RL precision is reported as $y$2 and no-answer RL precision as $y$3, while answer rate drops from $y$4 to $y$5 and QA Score rises from $y$6 to $y$7. On Alexa69k-378, precision rises from $y$8 to $y$9, answer rate falls from $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$0 to $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$1, and QA Score rises from $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$2 to $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$3. The best reported results combine DFS-based supervised bootstrapping with ternary-reward RL, reaching QA Scores of $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$4 on FB15k-237 and $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$5 on Alexa69k-378 (Godin et al., 2019).

A more explicit set-valued formulation appears in “Reaching Beyond the Mode: RL for Distributional Reasoning in LLMs” (Puri et al., 25 Mar 2026). The paper argues that standard RLHF and RLVR are mode-seeking and induce train-test mismatch when inference-time procedures such as best-of-$$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$6, beam search, or repeated sampling are used to recover non-modal answers. The policy is therefore trained to emit a structured set of $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$7 distinct candidates $$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$8 in a single forward pass. The set-level reward is

$$R_{\text{RLVR}^{\text{multi}}}(A, \mathcal{Y}^*)=\sum_{i=1}^{K}\mathbf{1}[a_i \in \mathcal{Y}^*],$$9

and the calibrated extension introduces confidences $Q=\{q_1,\dots,q_K\}$0 with

$Q=\{q_1,\dots,q_K\}$1

$Q=\{q_1,\dots,q_K\}$2

The reported empirical pattern is that multi-answer training improves coverage, diversity, and efficiency across DDXPlus, HotPotQA-Modified, and MBPP. On DDXPlus, RLVR Single has coverage $Q=\{q_1,\dots,q_K\}$3, diversity $Q=\{q_1,\dots,q_K\}$4, efficiency $Q=\{q_1,\dots,q_K\}$5, and top-1 $Q=\{q_1,\dots,q_K\}$6, whereas RLVR Multi has coverage $Q=\{q_1,\dots,q_K\}$7, diversity $Q=\{q_1,\dots,q_K\}$8, efficiency $Q=\{q_1,\dots,q_K\}$9, and top-1 $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$0. On MBPP, RLVR Multi improves top-1 from $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$1 to $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$2 relative to RLVR Single and reduces efficiency from $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$3 to $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$4. The paper also reports that simply prompting single-answer models to give multiple answers does not reproduce the benefits of multi-answer training, and that top-1 accuracy can still favor single-answer methods when the task truly has one correct answer (Puri et al., 25 Mar 2026).

These two lines of work define a central axis of multi-answer RL: the policy must decide either which subset of plausible answers to return or whether any answer should be returned at all, and the reward must score coverage, confidence, and abstention behavior directly.

4. Multi-attempt interaction and iterative refinement

“Learning from Failures in Multi-Attempt Reinforcement Learning” (Chung et al., 4 Mar 2025) reframes the problem as sequential answer refinement. A question is posed, the model answers, and if the answer is wrong and attempts remain, the environment returns feedback such as “Your previous answer is wrong. Try again,” allowing another answer. Formally, the prompt at turn $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$5 is produced by

$Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$6

and the multi-turn objective is

$Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$7

with $Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$8. In the multi-attempt setting, the transition mechanism terminates if the answer is correct or attempts are exhausted; otherwise it returns a prompt indicating the remaining attempts and requesting another attempt.

A notable design choice is to randomize the attempt budget during training via

$Q_\Sigma(x,a)=\sum_j w_j Q_j(x_j,a)$9

with $n$0 in the reported setup. The reward function is $n$1 if the model answers correctly in any attempt, $n$2 if the answer is in the correct format but wrong, and $n$3 otherwise, with no discount by number of attempts used. The paper interprets this as encouraging early exploration and later refinement rather than punishing search itself (Chung et al., 4 Mar 2025).

The experiments use Qwen 2.5 Math 1.5B, PPO, and 8K math questions, for $n$4 episodes and $n$5M total training samples. Average accuracy across five math benchmarks is reported as $n$6 for the multi-attempt model versus $n$7 for the baseline. The most prominent finding is the gain from one attempt to two attempts: the multi-attempt model improves from $n$8 to $n$9, whereas the single-turn baseline improves only from $x_j=\phi_j(x)$0 to $x_j=\phi_j(x)$1. Another reported result is an increase from $x_j=\phi_j(x)$2 to $x_j=\phi_j(x)$3 as evaluation attempts rise from $x_j=\phi_j(x)$4 to $x_j=\phi_j(x)$5 (Chung et al., 4 Mar 2025).

This sequential formulation differs from set-valued generation in that later answers are conditioned on explicit failure signals. A plausible implication is that multi-answer RL can be organized either as parallel candidate generation or as interactive revision under budget, with distinct search dynamics in each case.

5. Structured multi-output reasoning, faithfulness, and evidence selection

Recent work extends multi-answer RL from answer sets to structured answer bundles whose internal components are themselves objects of optimization. “CRAFT: Calibrated Reasoning with Answer-Faithful Traces via Reinforcement Learning for Multi-Hop Question Answering” (Liu et al., 1 Feb 2026) treats the output as

$x_j=\phi_j(x)$6

where $x_j=\phi_j(x)$7 is a plan, $x_j=\phi_j(x)$8 a citation set, $x_j=\phi_j(x)$9 a reasoning trace, and $R_j(x_j,a)$0 the final answer. Training uses GRPO with group-normalized rewards and a decomposed reward

$R_j(x_j,a)$1

Deterministic rewards enforce format compliance, citation F1, and answer correctness, while the judge-based faithfulness term

$R_j(x_j,a)$2

checks plan-to-reason consistency, citation-to-reason consistency, reason-to-answer consistency, and grounding. The paper defines an auditability spectrum from CRAFT$R_j(x_j,a)$3 with <plan> + <gold_docs> + <reason> + <answer> down to CRAFT$R_j(x_j,a)$4 with <answer> only. On MuSiQue, HotpotQA, and 2WikiMHQA, the full CRAFT$R_j(x_j,a)$5 7B model is reported to achieve $R_j(x_j,a)$6 EM / $R_j(x_j,a)$7 F1 / $R_j(x_j,a)$8 Faithfulness on MuSiQue, $R_j(x_j,a)$9 EM / $\mathcal{Y}^*(x)$00 F1 / $\mathcal{Y}^*(x)$01 Faithfulness on HotpotQA, and $\mathcal{Y}^*(x)$02 EM / $\mathcal{Y}^*(x)$03 F1 / $\mathcal{Y}^*(x)$04 Faithfulness on 2WikiMHQA. The paper also reports that removing judge reward reduces faithfulness by about $\mathcal{Y}^*(x)$05 to $\mathcal{Y}^*(x)$06 points on MuSiQue across variants (Liu et al., 1 Feb 2026).

“TACO: Think-Answer Consistency for Optimized Long-Chain Reasoning and Efficient Data Learning via Reinforcement Learning in LVLMs” (Kan et al., 27 May 2025) pushes this logic into multimodal reasoning. Built on GRPO, TACO introduces Think-Answer Consistency,

$\mathcal{Y}^*(x)$07

so that the answer is rewarded only insofar as it is consistent with the generated reasoning. For REC, the consistency term is an IoU over the thought bounding box, answer bounding box, and ground-truth box; for VQA, it is supplied by an external supervisor model. TACO supplements this with Rollback Resample Strategy, Adaptive Difficulty Sampling, and Test-Time-Resolution-Scaling. The reported ablation sequence on REC/LISA shows performance rising from a baseline of $\mathcal{Y}^*(x)$08 to $\mathcal{Y}^*(x)$09 with TAC, then to $\mathcal{Y}^*(x)$10 with RRS, $\mathcal{Y}^*(x)$11 with ADS, and $\mathcal{Y}^*(x)$12 with TTRS (Kan et al., 27 May 2025).

An adjacent formulation appears in “Context-Picker: Dynamic context selection using multi-stage reinforcement learning” (Zhu et al., 16 Dec 2025). Here the policy outputs a rationale $\mathcal{Y}^*(x)$13 and a subset of passage identifiers $\mathcal{Y}^*(x)$14, and RL is applied to minimal sufficient subset selection rather than answer string generation. The reward combines coverage of a gold evidence set, redundancy penalties, and format validity, and training proceeds in a two-stage schedule: a recall-oriented stage with looser redundancy margin followed by a precision-oriented stage with a stricter margin. Reported Judge Acc results include LoCoMo $\mathcal{Y}^*(x)$15 for RAG versus $\mathcal{Y}^*(x)$16 for Stage I and $\mathcal{Y}^*(x)$17 for Stage II, HotpotQA $\mathcal{Y}^*(x)$18 versus $\mathcal{Y}^*(x)$19 and $\mathcal{Y}^*(x)$20, and MuSiQue $\mathcal{Y}^*(x)$21 versus $\mathcal{Y}^*(x)$22 and $\mathcal{Y}^*(x)$23 (Zhu et al., 16 Dec 2025).

These works indicate that multi-answer RL can extend beyond multiple final strings to the joint optimization of several answer-bearing components. This suggests a broader interpretation in which the “answer” is a structured artifact whose internal consistency, grounding, and sufficiency are all reward-bearing.

6. Optimization stability, variance reduction, and recurrent limitations

A major optimization question is how to stabilize RL when the reward is sparse and the output contains both reasoning and answer tokens. “GRPO-MA: Multi-Answer Generation in GRPO for Stable and Efficient Chain-of-Thought Training” (Wang et al., 29 Sep 2025) identifies three problems in standard GRPO: gradient coupling between thoughts and answers, sparse reward signals caused by limited parallel sampling, and unstable advantage estimation. The proposed solution samples $\mathcal{Y}^*(x)$24 thoughts and $\mathcal{Y}^*(x)$25 answers per thought. The value of thought $\mathcal{Y}^*(x)$26 is

$\mathcal{Y}^*(x)$27

with thought advantage normalized across the $\mathcal{Y}^*(x)$28 thoughts and answer advantage normalized across all $\mathcal{Y}^*(x)$29 answers. The paper proves that

$\mathcal{Y}^*(x)$30

and derives a delta-method approximation showing that the variance of thought advantage decreases as the number of answers per thought increases. Empirically, GRPO-MA T4A4 improves over GRPO T4A1 on Math from pass@10 $\mathcal{Y}^*(x)$31 and pass@32 $\mathcal{Y}^*(x)$32 to pass@10 $\mathcal{Y}^*(x)$33 and pass@32 $\mathcal{Y}^*(x)$34, and on simulator-based manipulation from seen/unseen $\mathcal{Y}^*(x)$35 to $\mathcal{Y}^*(x)$36. Gradient Spike Score and NoZeroRate analyses are reported to confirm lower gradient spikes and denser positive rewards under multi-answer sampling (Wang et al., 29 Sep 2025).

Across the broader literature, several recurrent limitations are explicit. Advisor aggregation can create overestimation and attractors under egocentric planning, especially at high $\mathcal{Y}^*(x)$37 (Laroche et al., 2017). Set-valued generation can improve coverage and diversity while reducing top-1 accuracy on tasks that genuinely have one correct answer, and calibration can remain difficult on hard single-answer settings because the model may learn a prior to distribute confidence mass to sum to $\mathcal{Y}^*(x)$38 (Puri et al., 25 Mar 2026). In faithful reasoning, answer correctness does not imply faithful intermediate reasoning, and the ability to learn faithfulness exhibits a capacity threshold across model scales (Liu et al., 1 Feb 2026). In multi-attempt RL, the feedback is still coarse and the gains in base single-attempt accuracy are modest even when refinement improves substantially (Chung et al., 4 Mar 2025).

A common misconception is that multi-answer behavior can be recovered simply by prompting a single-answer model to output more candidates or by applying inference-time search to a mode-seeking policy. The reported evidence does not support that claim: trained multi-answer models outperform prompted single-answer baselines on coverage, diversity, and efficiency, and multi-answer GRPO variants reduce variance in a way that repeated independent sampling does not directly replicate (Puri et al., 25 Mar 2026, Wang et al., 29 Sep 2025).

The field therefore presents a consistent technical message. When tasks involve ambiguity, disagreement, abstention, iterative correction, or compositional reasoning, RL objectives defined over a single terminal answer are often behaviorally misaligned. Multi-answer reinforcement learning replaces that scalar endpoint with rewards over advisor collections, answer sets, confidence assignments, revision sequences, or structured reasoning artifacts, thereby making diversity, calibration, faithfulness, and refinement explicit optimization targets rather than incidental by-products.