---
title: Multi-Round Scalar Reward Prompting
url: https://www.emergentmind.com/topics/multi-round-scalar-reward-prompting
type: topic
---

# Multi-Round Scalar Reward Prompting

Multi-round scalar reward prompting refers to an inference or training-time protocol wherein a language model (LM) or a prompting agent engages in a multi-turn interaction, iteratively generating responses or prompts, and receiving scalar (numerical) rewards after each attempt or sequence. This feedback mechanism is used to enable learning or improvement—either via explicit reinforcement learning (RL) or through context-driven adaptation—such that subsequent prompts or generations increasingly maximize a predefined objective, typically response correctness, coherence, generation quality, or reasoning accuracy. Recent research advances notably encompass (1) in-context RL paradigms for static, fixed LMs and (2) agented prompting schemes with trainable policies and explicit RL objectives, each leveraging scalar rewards for optimizing complex prompting strategies [2511.01016, 2506.06303].

## 1. Formal Foundation and Problem Definition

In multi-round scalar reward prompting, each episode consists of a sequence of rounds, indexed by $t$. At round $t$, the system state $S_t$ encodes the interaction so far; the agent (which may be a trainable LM or a fixed LM) selects an action $A_t$ (a response, sub-prompt, or sequence), and an external process or environment returns a scalar reward $R_{t+1}$ reflecting the quality or utility of the response.

- In "Prompt-R1," the process is formulated as a Markov decision process (MDP) with composite state representation $H_t = [(a_1^\text{prompt}, r_1^\text{prompt}), ..., (a_t^\text{prompt}, r_t^\text{prompt})]$ and an agent internal state $F_t = S_t(F_{t-1}, a_t^{\text{think}}, a_t^\text{prompt}, r_t^\text{prompt})$. Actions comprise reasoning decisions and prompt generations, with transitions mediated by large-scale LLM responses [2511.01016].
- In "Reward Is Enough," the state $S_t$ comprises the core task specification, all past attempts, and associated rewards; actions $A_t$ correspond to new candidate outputs; rewards $r_{t+1}$ are externally supplied scalars [2506.06303].

The joint objective across both settings is to maximize the cumulative scalar reward, either explicitly via policy optimization (end-to-end RL) or implicitly through context-driven selection.

## 2. Reward Mechanisms and Design

Central to the effectiveness of multi-round scalar reward prompting is the reward signal. The reward can be structured to reflect properties such as answer correctness, response format compliance, or fine-grained stepwise progress:

- "Prompt-R1" uses a dual-constrained scalar reward composed of a format reward $R_\text{fmt}(\tau)$, which enforces well-formed reasoning and completion, and an answer correctness reward $R_\text{ans}(\tau)$, based on maximal token-level F1-score against references. Rewards are combined via a gated mechanism to ensure format satisfaction is prioritized before correctness [2511.01016].

$$
R(\tau) = 
\begin{cases}
-\epsilon + R_\text{fmt}(\tau) + R_\text{ans}(\tau), & \text{if } R_\text{fmt}(\tau) = \epsilon \\
-\epsilon + R_\text{fmt}(\tau), & \text{otherwise}
\end{cases}
$$

- In "Reward Is Enough," rewards can be episode-level binary/categorical values (e.g., success/failure), model-generated judgment scores, or dense signals from interactive environments. For instance, arithmetic step rewards in Game of 24 are scored as $\{0, 1, 3\}$ by a GPT-4.1 judge, creative writing rewards are GPT-assigned coherence scores (1–10), and ScienceWorld rewards are subgoal completions, all made explicit in the prompt as "Reward: $\cdot$" tokens [2506.06303].

Proper reward design ensures that the prompting loop meaningfully drives the agent (or LM) toward improved reasoning and generation.

## 3. Multi-Round Prompting Protocols

The multi-round protocol entails iterative generation, feedback, and prompt accumulation:

- In the RL-agent paradigm ("Prompt-R1"), a small-scale LLM agent (e.g., Qwen3-4B with LoRA adapters) observes the full prompt history, generates a reasoning trace and prompt, receives a structured response from a large-scale LLM (e.g., GPT-4o-mini or GPT-OSS-20B), and the loop continues until maximal turns or early termination. The final output is judged and the trajectory is used for policy update [2511.01016].

- In the in-context RL (ICRL) paradigm ("Reward Is Enough"), a frozen LLM receives, at each round, a prompt concatenating the static task description, all past attempts and rewards, and a meta-instruction (e.g., "Try again and improve your answer"). The model generates a new attempt, is given the numerical reward, and the process repeats for $K$ rounds (typically up to the context window limit) [2506.06303].

The guiding principle is a monotonic context expansion, treating the prompt as an explicit experience buffer for the model.

## 4. Learning and Optimization

Learning mechanisms in multi-round scalar reward prompting bifurcate into explicit RL (with parameter updates) and implicit in-context learning (fixed parameters):

- "Prompt-R1" employs Group-Relative Policy Optimization (GRPO), a novel PPO variant that normalizes trajectory-level rewards within disjoint groups ($M=16$). The policy $\pi$ is updated by minimizing:

$$
L_\text{GRPO} = -\frac{1}{M}\sum_{i=1}^{M}\left[ \sum_{t=1}^{T_i} \hat A_i \log \pi(a_{i,t}|H_{i,t-1}, q) \right] + \zeta\, D_{KL}(\pi_\text{old} \| \pi_\text{new})
$$

where advantage $\hat A_i$ is standardized per-group, and $\zeta$ modulates policy divergence [2511.01016].

- In the ICRL prompting setting, improvement arises not from parameter updates but from the LLM’s emergent mechanism to attend and adapt to past high-reward attempts; the model uses scalar rewards as in-context signals for policy improvement, without gradient-based updates [2506.06303].

A striking empirical finding is that numerical scalar rewards—even when noisy or LLM-derived—enable robust improvement, consistent with the reward-hypothesis of RL [2506.06303].

## 5. Architectural and Implementation Considerations

Key implementation choices impact the feasibility and effectiveness of multi-round scalar reward prompting:

| Component            | Prompt-R1 [2511.01016]                  | ICRL Prompting [2506.06303]   |
|----------------------|------------------------------------------|-------------------------------|
| Agent                | Qwen3-4B (LoRA adapters)                | Frozen pretrained LLM (e.g., GPT-4.1) |
| Environment          | GPT-4o-mini, GPT-OSS-20B                | Task context + buffer         |
| Reward Integration   | Dual-constrained (format, correctness)  | Scalar numerical, per round   |
| Update Mechanism     | Explicit RL (GRPO, PPO-like, 3 epochs)  | In-context RL, no updates     |
| Prompt Structure     | (Think, Prompt, Response, Reward) tuples| [Task description; Attempts + Rewards; Meta-instruction] |
| Buffer Size          | Up to 2,048 tokens, T=8 rounds          | Up to LLM context window, K=50 rounds |

Distinctive practices include capping format rewards, structured prompt composition, and the option for locally deployed LLMs (for "zero-cost" cycles).

## 6. Empirical Performance and Evaluation

Empirical benchmarks demonstrate the effectiveness of multi-round scalar reward prompting on a variety of tasks spanning multi-hop QA, math reasoning, creative writing, science-based interactive environments, and summarization:

- "Prompt-R1" consistently yields large gains over SFT, CoT, and state-of-the-art automatic prompting optimization methods,
  - Multi-hop QA F1: $17.8 \to 54.4\%$ (HotpotQA, +8.1)
  - GSM8K math EM/F1: $97.7\%$ (vs $92.97\%$ CoT, +4.7)
  - Creative writing SSim: $22.1\%$ (vs $12.2\%$, +9.9)
  - Out-of-distribution F1 improvement: $+4.55$ points [2511.01016].
- ICRL prompting ("Reward Is Enough") achieves:
  - Game of 24 (ICRL Preset): $90\%$ success (vs $49\%$ Best-of-N)
  - Creative writing: $86\%$ win rate vs. Self-Refine, $93\%$ vs. Best-of-N
  - ScienceWorld (mean return): $88 \pm 0.7$ (vs $83 \pm 0.9$ Self-Refine), robust to LLM-generated scalar rewards [2506.06303].

Performance degrades substantially if rewards are omitted, the episode buffer is truncated, or rewards are replaced with natural-language critiques, confirming the critical role of explicit scalar feedback.

## 7. Practical Recommendations and Theoretical Implications

Best practices for multi-round scalar reward prompting emphasize minimal yet explicit encoding of numerical feedback, full retention of all responses and rewards (learning from failure), buffer expansion to context limits, and the use of concise meta-instructions to direct exploration vs exploitation.

A principal theoretical implication is the demonstration that scalar reward signals suffices for forward-pass meta-RL in large LMs; numerical rewards are parsimoniously interpreted, and even self-generated reward signals can induce non-trivial policy improvement in fixed-parameter LMs [2506.06303]. The strong gains obtained by scalar-reward-driven multi-round prompting further substantiate the RL reward-hypothesis and highlight the utility of plug-and-play prompting agents for complex reasoning in large-scale LM inference pipelines [2511.01016, 2506.06303].

Source: https://www.emergentmind.com/topics/multi-round-scalar-reward-prompting