---
title: 'Retformer: Retrospective Prompt Optimization'
url: https://www.emergentmind.com/topics/retformer
type: topic
---

# Retformer: Retrospective Prompt Optimization

Searching arXiv for papers relevant to “Retformer” / “Retroformer” to ground the article in the literature.
Retformer, also written in the source paper as Retroformer, is a framework for retrospective large language agents in which prompt refinement is treated as a learnable policy rather than a fixed prompting heuristic. It introduces a smaller retrospective model that reads trajectories, rewards, and the user prompt, then rewrites the frozen actor model’s prompt for the next attempt. The central claim is that environment feedback can be converted into language-level prompt updates and optimized with policy gradient, specifically PPO, without fine-tuning the actor LLM itself [2308.02151].

## 1. Definition and problem setting

Retformer is designed for objective-oriented multi-step tasks in which a language agent interacts with an external environment and receives task-dependent rewards. In the formulation given by the paper, the action agent is an LLM-based policy
\[
\mathcal{M}_{\xi_l} : \mathcal{X} \rightarrow \mathcal{A},
\]
where \(\mathcal{X}\) is the prompt/input space, \(\mathcal{A}\) is the action space, and \(\xi_l\) denotes randomness in autoregressive sampling [2308.02151]. The environment is represented as
\[
(\mathcal{T}_{\xi_o}, \mathcal{R}),
\]
with \(\mathcal{T}_{\xi_o}: \mathcal{S} \times \mathcal{A} \to \mathcal{S}\) as the stochastic transition function and \(\mathcal{R}: \mathcal{S} \to \mathbb{R}\) as the reward function. The episode return is
\[
G_{cum} = \sum_{t=0}^{T} R(s_t).
\]

The paper frames a core limitation of existing language agents as the absence of principled optimization with environment-specific rewards. It distinguishes Retformer from systems that rely on zero-shot prompting, manually designed prompts, or verbal self-feedback alone. In this account, verbal reflection without gradient-based optimization can fail at credit assignment, fail to identify the true root cause of failure, or merely restate prior actions rather than improving future behavior [2308.02151].

A central practical motivation is that directly fine-tuning the actor may be impossible because the actor LLM can be cloud-hosted, inaccessible, too large to fine-tune, or not intended to be updated. Retformer addresses this by leaving the actor frozen and training only the retrospective component. This suggests a modular architecture aimed at settings where the principal source of adaptability is prompt transformation rather than parameter updates to the action model.

## 2. Core architecture

Retformer uses a two-model architecture consisting of a frozen actor model \(M_a\) and a learned retrospective model \(M_r\) [2308.02151]. The actor is the model that interacts with the environment and produces actions or thoughts; the retrospective model is a separate LLM that diagnoses previous failures and proposes prompt refinements for the next trial.

The retrospective mapping is defined as
\[
\Gamma_{\xi_r, \Theta} : \left[\mathcal{S}_i,\mathcal{A}_i,\mathcal{R}_i,\mathcal{X}^u_i\right]_{i=1}^t \rightarrow \mathcal{X},
\]
where \(\Gamma_{\xi_r,\Theta}\) is the prompt updater, \(\Theta\) are its learnable parameters, \(\xi_r\) is the randomness in retrospective generation, and the output is a refined prompt \(x\) [2308.02151]. This operator is the central mechanism by which past states, actions, rewards, and user prompts are transformed into new textual guidance for future execution.

The actor model is described as a frozen cloud LLM, such as GPT-3 or GPT-4, using a ReAct-style prompt. Its action at time \(t\) is written as
\[
a_{k,i,t} = M_a\left([s_{k,i,\tau}, a_{k,i,\tau}, r_{k,i,\tau}]_{\tau=1}^{t-1}, s_{k,i,t}\right).
\]
The retrospective model is a local model, for example LongChat/Llama-7B, whose output is
\[
y_{k,i} = M_r\left([s_{k,i,\tau}, a_{k,i,\tau}, r_{k,i,\tau}]_{\tau=1}^{T}, G_{k,i}\right),
\]
and this reflection response is appended to the actor prompt for the next trial [2308.02151].

The paper’s conceptual move is to treat the frozen actor LLM as part of the environment for purposes of optimizing \(\Theta\). This reduces the learning problem to RL over the retrospective model alone. A plausible implication is that the method is especially suited to systems where the actor is operationally fixed but the prompting interface remains controllable.

## 3. Retrospective prompt refinement and memory

Retformer’s main novelty is retrospective prompt refinement: after each failed attempt, the agent uses a learned model to convert task trajectories and rewards into a better prompt for the next attempt [2308.02151]. The process is described as follows. The actor executes actions in the environment, the environment returns an episode return \(G_{k,i}\), the retrospective model is prompted with the full trajectory and reward, it generates a reflection \(y_{k,i}\), and that reflection is appended to the next prompt for the actor.

The paper emphasizes that the reward signal is not used merely as a scalar endpoint for evaluation. Instead, it is converted into textual memory and planning guidance. The retrospective model is intended to identify what went wrong, what action sequence should be changed, and what new plan to follow. In this sense, Retformer makes reflection itself an action-like object subject to optimization.

Three memory notions are explicitly distinguished [2308.02151]. Short-term memory is the current episode trajectory \(\tau_i\). Long-term memory consists of past reflection responses that are appended to the prompt so that future trials can reuse lessons learned. The replay buffer stores triplets
\[
D_{\mathrm{RL} = \{(x_{k,i}, y_{k,i}, G_{k,i})\}
\]
containing retrospective prompts, reflection responses, and episode returns.

This architecture places textual memory at the center of agent improvement. The paper also reports an emergent formatting behavior in which the fine-tuned retrospective model tends to produce more structured reflections, separate “reflection” and “new plan” sections, remove irrelevant repetition, and focus on actionable errors [2308.02151]. Since the paper states that this behavior is not explicitly enforced, it should be understood as an empirical property of the trained system rather than a formal design guarantee.

## 4. Reinforcement learning objective and PPO optimization

The optimization target for the retrospective model is given as
\[
\arg\max_\Theta \mathbb{E}_{\xi_l,\xi_o,\xi_r}\left[\sum_{t=1}^{T} R(s_t)\right]
\]
subject to environment dynamics
\[
s_{t+1} = \mathcal{T}_{\xi_o}\left( s_t, \mathcal{L}_{\xi_l} \circ \Gamma_{\xi_r,\Theta}\left([s_i,a_i,r_i,x_i^u]_{i=1}^{t}\right) \right), \quad \forall t \in \{1,\dots,T-1\}.
\]
In prose, the retrospective model transforms history into a refined prompt, the frozen actor acts using that prompt, the environment transitions, rewards are received, and the retrospective model is trained to maximize future returns [2308.02151].

A defining element of the framework is the reward assigned to reflections. The paper defines the rating of a reflection prompt-response pair as
\[
r(x_{k,i}, y_{k,i}) \triangleq G_{k,i+1} - G_{k,i}.
\]
A positive value means the reflection helped the actor perform better on the next trial; zero or negative values indicate that it was unhelpful or harmful [2308.02151]. This is the mechanism by which downstream performance improvement becomes a learning signal for prompt generation.

The PPO-style objective is presented in RLHF-like form, with a learned reward model score and a KL penalty relative to a reference model. The paper states that the intended structure is standard RLHF: maximize reward model score \(r_\theta(x,y)\) while penalizing deviation from a frozen reference model, with \(\beta\) controlling the KL term [2308.02151]. The training logic proceeds in three stages: offline data collection, reward model learning using accepted and rejected reflection pairs, and PPO fine-tuning of the retrospective policy.

The appendix algorithm, described as “Retroformer with Policy Gradient Optimization,” initializes the retrospective model with LongChat/Llama-7B, collects offline rollouts, samples two reflection responses for unsuccessful tasks, rolls out the next episode using each reflection, computes the rating \(G_{k,i+1} - G_{k,i}\), labels the better response as accepted and the worse as rejected, trains a reward model from those comparisons, and fine-tunes the retrospective model with PPO [2308.02151]. This makes Retformer an instance of RLHF-style optimization in which the preference signal is derived from environment outcomes rather than human judgments.

## 5. Experimental environments and empirical findings

The paper evaluates Retformer in three text-based environments: HotPotQA, AlfWorld, and WebShop [2308.02151]. These benchmarks differ substantially in reward structure and task demands, which is important for interpreting the reported gains.

| Environment | Actions | Reward |
|---|---|---|
| HotPotQA | `Search[entity]`, `Lookup[keyword]`, `Finish[answer]` | F1 score between generated answer and ground truth |
| AlfWorld | `goto[location]`, `take[obj]`, `open[obj]`, `close[obj]`, `toggle[obj]`, `clean[obj]`, `heat[obj]`, `cool[obj]` | Binary success/failure |
| WebShop | `Search[Query]`, `Choose[Button]` | Task-specific terminal reward using heuristics and text matching |

The actor model in the main experiments is GPT-3 or GPT-4, frozen with temperature \(0\) and top-\(p\) \(1\). The retrospective model is LongChat-7B-16K initialized from Llama-7B and fine-tuned with LoRA. Reported collection sizes are 3,383 reflection samples from 3,000 training tasks and 3 trials for HotPotQA, 523 reflection samples for AlfWorld, and 267 reflection samples for WebShop. Reported training hyperparameters include epochs \(= 4\), batch size \(= 8\), learning rate \(= 1.4 \times 10^{-5}\), LoRA rank \(r=1\) or \(r=4\), and PPO epochs \(= 4\) [2308.02151].

The main empirical findings are task-dependent. On HotPotQA, Retformer shows strong improvement over trials, reaching around 54% success rate with 4 retries and outperforming Reflexion and ReAct. The paper states that the gains are especially strong in early retries, which it interprets as evidence that the learned retrospective model helps the agent improve faster [2308.02151]. On AlfWorld, the method can solve tasks within 3 retries and achieves up to 100% success in some settings, outperforming frozen baselines and SAC. On WebShop, improvement exists but is smaller, with around 4% gain with 4 retries.

These results support a differentiated interpretation. The paper explicitly suggests that the method works best when rewards are informative and environment structure is stable, while highly exploratory settings such as WebShop appear less amenable to verbal-feedback refinement alone [2308.02151]. This suggests that Retformer’s effectiveness depends not only on the quality of the actor and retrospective model but also on whether the reward signal can support useful retrospective credit assignment.

## 6. Relation to prior agent paradigms, novelty, and limitations

Retformer is positioned against three main alternatives: ReAct, Reflexion, and SAC [2308.02151]. ReAct provides a frozen LLM agent with reasoning and acting but no environment-learning. Reflexion uses self-reflection and verbal reinforcement learning but does not optimize reflection generation with gradient-based learning from environment reward. SAC is included as an online RL baseline that uses continuous embeddings as action representations.

The claimed novelty is therefore not merely the use of reflections, but the use of task-specific reward improvement to learn which reflections are useful. The paper’s summary is explicit on this point: reflections are treated as actions in their own right, their success is measured by whether they improve the next episode return, and the retrospective model is optimized accordingly [2308.02151]. In that sense, Retformer combines verbal reflection with reward-based policy optimization over the reflection process itself.

The paper also identifies several limitations. WebShop improvements are modest, which it presents as evidence that the method may struggle in highly exploratory tasks. Because the actor remains frozen, overall capability remains bounded by the base LLM. The method depends on collecting enough rollouts to estimate which reflections are better, and it does not provide a universal guarantee that reflections always improve future return. The paper further implies that very sparse or noisy rewards may make learning the retrospective model harder [2308.02151].

A broader implication, stated directly in the paper, is that the method is plug-in and architecture-agnostic: any system with a frozen actor and textual memory or reflection could potentially use a learned retrospective module, and the same idea could be applied to memory modules, summarizers, or prompt planners [2308.02151]. This suggests a general design pattern in which prompt-updating policies are separated from action-generation policies and trained from downstream reward.

## 7. Significance

Retformer formalizes prompt refinement as an RL problem over a retrospective language model rather than as a static prompting recipe. Its central contribution is to define a retrospective prompt-updater \(\Gamma_{\xi_r,\Theta}\) that transforms trajectories and rewards into improved prompts for a frozen actor LLM, and to train that updater with PPO using reward signals derived from downstream episode-return improvement [2308.02151].

Within the paper’s own framing, the significance lies in three linked claims. First, environment rewards can supervise reflection generation rather than merely evaluate final performance. Second, prompt refinement can be optimized without modifying the actor LLM. Third, a smaller local retrospective model can improve a larger frozen cloud LLM agent across multiple environments [2308.02151]. Taken together, these claims position Retformer as an early attempt to move language-agent adaptation from heuristic verbal self-improvement toward policy-gradient optimization over textual feedback.

The paper does not present the framework as a universal solution for all agentic settings, and the reported results are uneven across benchmarks. Nevertheless, the formulation establishes a concrete research direction: instead of hand-designing feedback prompts, learn a feedback policy from environment-specific returns. That principle is the defining feature of Retformer as introduced in “Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization” [2308.02151].

Source: https://www.emergentmind.com/topics/retformer