---
title: 'REvolve: Optimization in Checkpointing, RL, and LLM'
url: https://www.emergentmind.com/topics/revolve
type: topic
---

# REvolve: Optimization in Checkpointing, RL, and LLM

REvolve, with the related capitalizations **Revolve** and **REVOLVE**, denotes several distinct research artifacts rather than a single unified method. In numerical optimization and scientific computing, **Revolve** is the classical checkpointing scheduler for reverse/adjoint computations under fixed-memory constraints, and later work extends it to multistage time-stepping and high-level Python interfaces [2106.13879]. In contemporary LLM and RL research, **REvolve** refers to a framework for reward evolution with large language models using human feedback [2406.01309], whereas **REVOLVE** denotes a trajectory-aware textual optimization method that tracks how responses evolve across iterations in LLM systems [2412.03092]. The shared name therefore spans at least three technically unrelated lines of work.

| Variant | Research object | Domain |
|---|---|---|
| Revolve | Optimal checkpointing scheduler | Adjoint computation, PDE/ODE solvers |
| REvolve | Reward evolution with LLMs using human feedback | Reinforcement learning, autonomous driving |
| REVOLVE | Response-evolution-based textual optimizer | LLM system optimization |

## 1. Classical Revolve in adjoint checkpointing

In the checkpointing literature, Revolve addresses the memory bottleneck in reverse-mode and discrete-adjoint computations. For a time-stepping discretization
\[
u_{n+1}=\Phi(u_n), \qquad n=0,\dots,N-1,
\]
with terminal objective \(\psi(u_N)\), the discrete adjoint satisfies
\[
\lambda_n=\left(\frac{d\Phi}{du}(u_n)\right)^T\lambda_{n+1}, \qquad
\lambda_N=\left(\frac{d\psi}{du}(u_N)\right)^T.
\]
The reverse sweep proceeds backward in time, but each adjoint step requires information from the corresponding forward step. Storing all forward states makes the reverse sweep straightforward but causes memory usage to scale linearly with both state size and number of time steps; checkpointing instead stores selected snapshots and regenerates missing states by recomputation [2106.13879].

Under the standard offline model, Revolve assumes that the total number of time steps is known in advance, all forward steps have equal cost, checkpoint store/restore costs are negligible relative to arithmetic cost, and each checkpoint stores only one solution state. In that setting, the optimization objective is the number of recomputations. The conventional problem is denoted \(Prob_{\text{conventional}}(m,s)\): given \(m\) time steps and \(s\) allowable checkpoints, minimize recomputations. If \(t\) is the unique integer such that
\[
\binom{s+t-1}{t-1}<m\le \binom{s+t}{t},
\]
then the minimal number of recomputations is
\[
p(m,s)=tm-\binom{s+t}{t-1}.
\]
The data attributes this offline optimality result to Griewank and Walther [2106.13879].

Operationally, Revolve is a scheduler rather than a PDE solver. In the software skeleton reproduced for inversion codes, a Revolve object is created as `Revolve(steps, snaps)`, and successive calls emit actions such as `advance`, `takeshot`, `restore`, `firsturn`, `youturn`, and eventually `terminate`. The application must interpret these actions on its own state representation: `advance` propagates the forward model, `takeshot` saves a checkpoint, `restore` reloads a checkpoint, `firsturn` starts adjoint mode and initializes objective-related quantities, and `youturn` performs subsequent adjoint steps [1802.02474].

This notion of optimality is narrow but precise. Revolve is optimal for minimizing recomputed forward timesteps under the classical serial/offline model; it is not, by itself, a claim about wall-clock optimality in implementations where memory traffic, language-runtime overheads, or parallel efficiency become significant [1802.02474].

## 2. Multistage-aware extensions and the CAMS generalization

The 2021 multistage checkpointing work identifies a structural limitation in classical Revolve: the abstraction of a time step as an atomic unit fails for multistage integrators such as explicit Runge–Kutta methods, whose discrete adjoints require internal stage values \(U_i\), not only the step states \(u_n\). For an \(\ell\)-stage explicit Runge–Kutta method,
\[
U_i=u_n+h_n\sum_{j=1}^{i-1} a_{ij} f(U_j),\qquad
u_{n+1}=u_n+h_n\sum_{i=1}^{\ell} b_i f(U_i),
\]
and the reverse of a single macro-step requires all stage values \(U_i\). Classical Revolve effectively assumes that these can always be recreated by restoring \(u_n\) and rerunning the forward step immediately before the adjoint step, which is optimal only when checkpoint contents are restricted to step states [2106.13879].

The paper therefore formulates two extensions. The first is a **modified Revolve** problem, \(Prob_{\text{modified}}(m,s)\), in which a checkpoint stores the solution and the stage values at a time step. The key construction shifts each classical checkpoint one step later: if Revolve would checkpoint \(u_i\), the modified scheme checkpoints the data produced at the end of step \(i\to i+1\), namely \(u_{i+1}\) and the associated stage values. Because each reverse step except the last would otherwise need one forward recomputation to regenerate stages, this shift saves exactly \(m-1\) recomputations in total. The resulting optimal count is
\[
\tilde p(m,s)=(t-1)m-\binom{s+t}{t-1}+1,
\]
with the same \(t\) satisfying
\[
\binom{s+t-1}{t-1}<m\le \binom{s+t}{t}.
\]
The paper stresses that this extension remains exact only under the stronger checkpoint-content assumption that each checkpoint always contains the full state-plus-stages object [2106.13879].

The second and more general formulation is the truly optimal multistage problem, \(Prob_{\text{multistage}}(m,s)\), in which memory is measured in **checkpointing units**, each unit storing one solution vector or one stage vector, assumed to have the same size. A checkpoint may then contain any combination of such units at one time step. This yields the dynamic-programming algorithm **CAMS**, with **CAMS-SA** for stiffly accurate schemes and **CAMS-GEN** for general multistage schemes. The paper gives explicit recurrences for the subproblems \(P_{\mathrm{IS}}(m,s)\) and, in the general case, \(P_{\mathrm{SV}}(m,s)\), and reconstructs schedules from tabulated parent pointers [2106.13879].

A central theoretical result is dominance over both earlier formulations:
\[
P(m,s)\le P_{\textnormal{R}}(m,s),\qquad
P(m,s)\le P_{\textnormal{MR}}(m,s).
\]
An important qualitative distinction follows. Classical Revolve has a lower bound of \(m-1\) recomputations because each reverse step except the last must rerun the corresponding forward step to recreate stage values. CAMS can achieve zero recomputations if sufficient memory exists to store enough stage information. Empirically, the paper reports that, for multistage adjoint computations, the proposed algorithms can deliver up to about \(2\times\) speedup over classical Revolve, and it gives a concrete example with \(10\) time steps and \(6\) checkpointing units in which classical Revolve needs \(12\) recomputations, CAMS-GEN needs \(8\), and CAMS-SA needs \(6\) [2106.13879].

The same study also emphasizes a software-engineering distinction. Classical Revolve is a centralized controller with hidden mutable state, whereas CAMS is presented as an offline schedule oracle queried by explicit arguments such as `lastcheckpointstep` and `lastcheckpointtype`. Because the answer depends only on those arguments and precomputed DP tables, the API is idempotent. The algorithms were adopted by the PETSc `TSAdjoint` library, integrated via `TSTrajectorySet()` and `TSTrajectoryGet()`, and demonstrated on a large-scale PDE-constrained optimization problem for the Gray–Scott reaction–diffusion system on the Cori supercomputer at NERSC using \(32\) KNL nodes and \(2048\) MPI processes [2106.13879].

## 3. pyRevolve and high-level software integration

The paper “High-level python abstractions for optimal checkpointing in inversion problems” does not propose a new checkpointing theory; its contribution is a software layer, **pyRevolve**, that makes classical Revolve usable from modern Python-based PDE and inversion environments, particularly the Devito DSL and code-generation stack [1802.02474].

Its motivating application is seismic inversion and imaging, including RTM and FWI. The least-squares objective is written as
\[
\Phi_s(\mathbf{m})=\frac{1}{2}\left\lVert\mathbf{d}_{syn} - \mathbf{d}_{obs}\right\rVert_2^2,
\]
with forward PDE
\[
\mathbf{A}(\mathbf{m}) \mathbf{u} = \mathbf{P}_s^T \mathbf{q},
\]
adjoint equation
\[
\mathbf{A}^T(\mathbf{m}) \mathbf{v} = \mathbf{P}_r^T \delta\mathbf{d},
\]
and gradient
\[
\nabla\Phi_s(\mathbf{m})=\sum_{t =1}^{n_t}\mathbf{u}[\mathbf{t}] \mathbf{v}_{tt}[\mathbf{t}] =\mathbf{J}^T\delta\mathbf{d}.
\]
Because the gradient evaluation requires the forward field \(\mathbf{u}\) and the adjoint field \(\mathbf{v}\) at matching time levels, storing the entire wavefield history can be prohibitive; the paper notes that, in its experiment, a \(230^3\) grid over \(1615\) timesteps required about \(80\) GB just to store the full forward wavefield [1802.02474].

The pyRevolve architecture is layered as **Revolve** \(\rightarrow\) **cRevolve** \(\rightarrow\) **pyRevolve** \(\rightarrow\) **Devito**. Two abstract application interfaces are central. A `Checkpoint` object provides `size`, `save(ptr)`, and `load(ptr)`, encapsulating how application-specific state is serialized into raw storage. An `Operator` object exposes an `apply()` method for forward or reverse computation. A `Revolver` object is then constructed from a `Checkpoint`, a forward `Operator`, a reverse `Operator`, and optionally the number of checkpoints and timesteps; it allocates a `Storage` object sized as \(n_{\text{ckp}} \times \text{checkpoint.size}\) [1802.02474].

At runtime, `revolver.apply_forward()` executes the forward sweep in schedule-determined chunks, invoking `fwd.apply(t_start, t_end)` and issuing `cp.save(ptr)` automatically. During `revolver.apply_reverse()`, pyRevolve follows the schedule to `cp.load(ptr)`, call `rev.apply(...)`, and, when necessary, recompute forward segments by invoking `fwd.apply(...)` before the reverse segment. The design objective is to hide the intricate glue logic between scheduling, save/restore, forward recomputation, and adjoint execution [1802.02474].

The evaluation shows both correctness and practical limitations. Checkpointed gradients matched a reference implementation that stored the full forward history and passed a Taylor-based gradient test with \(\epsilon_0=O(h)\) and \(\epsilon_1=O(h^2)\) for \(h\in[10^{-1},10^{-4}]\). At the same time, measured runtime did not follow the ideal theoretical predictions from classical Revolve. The paper attributes the discrepancy to NumPy deep-copy costs for checkpoints, Python-to-C call overhead, serial checkpointing granularity, and reduced work per `Operator` call that can hurt OpenMP load balancing. It identifies online checkpointing, multistage or hierarchical checkpointing, and richer parallel interfaces as future needs [1802.02474].

## 4. REvolve as reward evolution with LLMs using human feedback

The 2024 paper “REvolve: Reward Evolution with Large Language Models using Human Feedback” defines **REvolve** as an evolutionary framework for reward design in reinforcement learning. Its target problem is the reward design problem in domains where desirable behavior is tacit, subjective, multi-objective, and hard to formalize directly. The paper emphasizes autonomous driving as its central example and relates the difficulty of specifying “good” behavior to Polanyi’s paradox: humans may recognize good behavior without being able to express it completely as equations [2406.01309].

The method searches over **reward programs**, not directly over policies. It begins with a natural-language task description and a set of available environment variables, prompts GPT-4 to generate an initial population of Python reward functions of the form
```python
def compute_reward(...) -> Tuple[float, Dict[str, float]]:
    ...
    return total_reward, reward_components
```
and trains one RL policy per reward candidate. Each individual is stored as
\[
(R,\pi,\sigma,\lambda),
\]
where \(R\) is the reward function, \(\pi\) the trained policy, \(\sigma\) a scalar fitness score, and \(\lambda\) natural-language human feedback. Individuals are assigned to one of \(I\) sub-populations or islands [2406.01309].

Human evaluation operates on rollout videos of 30–40 seconds. Evaluators provide pairwise preferences, with ties allowed, and qualitative natural-language feedback through checkbox selections. Pairwise comparisons are aggregated with an Elo rating system. For two individuals \(A\) and \(B\),
\[
E_A = \frac{1}{1 + 10^{(\sigma_B - \sigma_A) / 400}},\qquad
E_B = \frac{1}{1 + 10^{(\sigma_A - \sigma_B) / 400}},
\]
and
\[
\sigma_A = \sigma_A + K \cdot (f(A,B) - E_A),
\]
with \(K=32\), \(f(X,Y)=1\) for a win, \(0\) for a loss, \(0.5\) for a tie, and all individuals initialized at \(1500\) Elo. GPT-4 then acts as mutation or crossover operator: mutation edits a single parent reward, while crossover combines components from two parents. A new individual survives in island \(P\) if \(\sigma \ge \sigma^P\), where \(\sigma^P\) is the average fitness of that island. The main hyperparameters reported are \(N=5\) generations, \(K=16\) individuals per generation, \(I=13\) islands, \(p_m=0.5\), and \(p_c=0.5\) [2406.01309].

The autonomous-driving experiments use AirSim, an urban neighborhood with static and dynamic obstacles. The policy receives egocentric visual input and sensor data including yaw, pitch, \(V_x\), \(V_y\), speed, and steering; stacked visual observations have size \(100\times256\times12\). The car must drive safely for \(1000\) consecutive time steps. The action space combines steering in \([-32^\circ,32^\circ]\) with \(2^\circ\) increments and throttle \(\{0,1\}\), yielding \(|A|=66\). The RL backbone is Clipped Double Deep Q-Learning with Dueling Architecture and Prioritized Experience Replay, using learning rate \(0.00025\), Adam, \(\gamma=0.99\), batch size \(32\), replay buffer size \(50000\), and \(5\times10^5\) training steps per candidate reward. The reported human-feedback setup uses \(10\) human evaluators per generation, each assessing \(20\) pairs of videos, and training \(16\) policies per generation on \(16\) NVIDIA A100 GPUs, with each policy taking about \(50\) hours to train [2406.01309].

On the manually designed fitness scale, the best final scores reported are Human Driving \(0.90\), REvolve \(0.84\), REvolve Auto \(0.81\), Eureka \(0.72\), Eureka Auto \(0.70\), and Expert Designed \(0.65\). Under human Elo comparisons, the ranking is Human Driving \(1595\), REvolve \(1534\), REvolve Auto \(1518\), Eureka \(1453\), and Eureka Auto \(1401\). The best REvolve reward also transfers better than the expert-designed reward to two new AirSim environments, although performance drops in the more difficult high-traffic setting. The paper frames the method in three challenging settings—autonomous driving, humanoid locomotion, and dexterous manipulation—but the detailed experimental account provided here focuses on autonomous driving [2406.01309].

## 5. REVOLVE as trajectory-aware textual optimization

The 2024 preprint “REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization” uses the acronym **REVOLVE** for a distinct optimization method for LLM-based systems. Its target is textual optimization: prompts, intermediate solutions, code, and more generally text-mediated system components whose “parameters” and feedback are both linguistic rather than differentiable numerical variables [2412.03092].

The paper positions TextGrad-like methods as analogs of first-order optimization because they rely only on the current textual critique. REVOLVE adds a signal derived from how responses change over time. If \(p_t\) is the textual variable at iteration \(t\), \(r(p_t)\) the generated response, and \(\mathcal{L}(r(p_t))\) the language-based loss or critique, the paper writes the TextGrad-style derivative as
\[
\nabla \mathcal{L}(r(p_t))=\frac{\tilde{\partial}\mathcal{L}(r(p_t))}{\tilde{\partial}p_t},
\]
introduces a similarity function
\[
\mathcal{S}(r(p_t),r(p_{t-1})),
\]
and defines the REVOLVE update signal heuristically as
\[
\textnormal{REVOLVE}\Big(\mathcal{L}(r(p_t))\Big)=
\frac{\tilde{\partial}\mathcal{L}(r(p_t))+\mathcal{S}(r(p_t),r(p_{t-1}))}{\tilde{\partial}p_t}.
\]
A concrete proposal is
\[
\mathcal{S}(r(p_t),r(p_{t-1}))=
\frac{\Vert\mathcal{L}(r(p_t))-\mathcal{L}(r(p_{t-1}))\Vert}{\Vert p_t-p_{t-1}\Vert}.
\]
Under a small-step assumption, the method is then interpreted as a first-order textual gradient plus a second-order textual curvature term. The paper is explicit that this is an analogy rather than a true Hessian computation: REVOLVE simulates second-order effects and does not require real second-order derivatives from the LLM [2412.03092].

In implementation terms, the optimizer retains history across iterations, including prior variables, responses, and critiques, and injects this history into prompts as `<PAST_ITERATIONS>`. The optimizer is instructed to analyze how responses evolve, avoid overly similar repetitions, and also avoid abrupt incoherent jumps. This makes REVOLVE a history-aware extension of a TextGrad-style loop rather than a replacement for forward evaluation or language-loss generation [2412.03092].

The experiments cover three settings. For prompt optimization, the paper uses BBH Object Counting and GSM8K with batch size \(3\), \(12\) optimization iterations, and exact-match evaluation. For solution optimization, it uses GPQA Diamond, MMLU-Machine Learning, and MMLU-College Physics with \(3\) iterations and majority voting across iterations. For code optimization, it uses LeetCode Hard with up to \(5\) iterations and completion rate on hidden test cases, averaged over \(5\) seeds. Global generation settings include temperature \(0\) or \(1\times10^{-6}\) for Llama 3.1 8B Instruct, max new tokens \(2000\), and top-p \(0.99\) [2412.03092].

The abstract reports headline gains of \(7.8\%\) in prompt optimization, \(20.72\%\) in solution refinement, and \(29.17\%\) in code optimization. The detailed tables localize these gains. In prompt optimization on Object Counting, Llama 3.1 8B Instruct improves from TextGrad \(77.0\) to REVOLVE \(83.0\), a \(7.8\%\) relative gain. In solution optimization on MMLU-Machine Learning, Llama 3.1 8B Instruct improves from TextGrad \(47.3\) to REVOLVE \(57.1\), a \(20.72\%\) relative gain. In code optimization on LeetCode Hard, Llama 3.1 8B Instruct improves from TextGrad \(0.24\) to REVOLVE \(0.31\), a \(29.17\%\) relative gain. The paper also reports fewer total iterations to convergence in several settings, producing total runtime reductions despite higher per-iteration cost; for example, on Object Counting, total runtime drops from \(276.450\) s for TextGrad to \(137.821\) s for REVOLVE [2412.03092].

## 6. Disambiguation, scope, and recurrent methodological themes

A recurring misconception is to treat **Revolve**, **REvolve**, and **REVOLVE** as variants of one research program. The data instead describe three separate families of methods. In scientific computing, Revolve is a checkpointing scheduler for adjoint computations, and its multistage extensions alter what is stored and when it is restored [2106.13879]. In RL, REvolve is an island-model genetic-programming framework that evolves executable reward functions from LLM proposals and human judgments [2406.01309]. In LLM optimization, REVOLVE is a history-aware update rule for prompts, solutions, or code that uses response evolution as an additional optimization signal [2412.03092].

The strongest commonality is at the level of optimization philosophy rather than technical mechanism. Each method formalizes a constrained search problem in which the directly desired object is hard to obtain: minimal recomputation under bounded memory for checkpointing, human-aligned reward functions for RL, or stable improvement of textual variables under non-numeric feedback. This suggests that the shared name is associated less with a shared algorithmic core than with the idea of iterative refinement under indirect or delayed information.

The meaning of “optimality” also differs sharply across these lines of work. For classical Revolve and CAMS, optimality refers to recomputation count under a specific storage-and-cost model, not necessarily end-to-end wall-clock time; both pyRevolve and the multistage paper explicitly note that deep-copy overheads, checkpoint movement, DRAM bandwidth, and related costs can materially change empirical performance [1802.02474]. In REvolve for reward design, the main bottleneck is policy training cost rather than schedule generation, and the system relies on human preference judgments as its fitness signal [2406.01309]. In REVOLVE for textual optimization, the second-order language is heuristic, and the paper explicitly rejects interpreting the method as literal Hessian computation over text [2412.03092].

Taken together, the term **REvolve** therefore functions as a homograph in recent technical literature. Its most established use remains the checkpointing scheduler and its descendants in adjoint scientific computing, but current arXiv usage also includes unrelated LLM-era systems for reward evolution and response-evolution-based textual optimization [2106.13879].

Source: https://www.emergentmind.com/topics/revolve