RhymeRL: Dual RL Systems for Poetry and LLM Optimization

• RhymeRL is a term that refers to two distinct reinforcement learning frameworks: one for neural poetry revision under rhyme constraints and another for accelerating LLM RL pipelines.
• In neural poetry, it employs a generate-and-revise architecture with PPO to iteratively edit drafts so that quatrains meet target rhyme schemes without explicit token-level supervision.
• In LLM systems, RhymeRL leverages historical rollout reuse via HistoSpec and HistoPipe to enhance GPU utilization and streamline workload scheduling for efficient training.

RhymeRL is a name used for two distinct reinforcement-learning systems in arXiv literature. In neural poetry, it denotes a generate-and-revise framework that models poem revision as a Markov decision process and optimizes a revision policy with Proximal Policy Optimization so that generated quatrains satisfy a target rhyming scheme without explicit supervision about which words create the rhymes (Zugarini et al., 2021). In LLM reinforcement learning, it denotes a systems framework for accelerating disaggregated rollout–reward–train pipelines by exploiting similarity across adjacent training epochs through historical speculative decoding and distribution-aware rollout scheduling, while reporting no degradation in training accuracy and no modification of the RL paradigm (He et al., 26 Aug 2025). The shared nomenclature suggests a common emphasis on iterative reuse of prior text or prior trajectories, although the two systems operate at very different levels of abstraction.

1. Nomenclature and research scope

A common source of confusion is that “RhymeRL” does not refer to a single method family. The 2021 work addresses symbolic and neural revision of poems under rhyme constraints, whereas the 2025 work addresses systems-level acceleration of RL training for LLMs.

Usage	Domain	Core mechanism
RhymeRL in "Generate and Revise: Reinforcement Learning in Neural Poetry" (Zugarini et al., 2021)	Neural poetry generation	Generate a draft, then revise words with PPO to match a target rhyme scheme
RhymeRL in "History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL" (He et al., 26 Aug 2025)	LLM RL systems	Reuse historical rollouts via HistoSpec and HistoPipe

The earlier usage is centered on text quality under formal poetic constraints. The later usage is centered on throughput, GPU utilization, and rollout efficiency in production-scale RL. The overlap is therefore nominal rather than architectural: one is an RL policy over edit positions in a poem, the other is an RL infrastructure for accelerating rollout generation and balancing rollout workloads.

2. Markov decision process and model design in neural poetry

In the neural-poetry formulation, each state $S_t$ is a tuple $(o_t, a, r)$, where $o_t=(w_1,\ldots,w_N)$ is the current $N$-word draft of the poem, $a$ is the author ID, and $r$ is the target rhyme-scheme. The action space consists of selecting an index $i\in\{1,\ldots,N\}$ corresponding to the word position to edit, together with an implicit “do-nothing” action at terminal. After the agent chooses $i$, a prompter module samples a replacement word $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$, and the environment returns the updated state in which $w_i$ is replaced by $(o_t, a, r)$0. Because the prompter sampling is stochastic, the transition dynamics are probabilistic (Zugarini et al., 2021).

The policy network, called the “Detector,” receives the full poem $(o_t, a, r)$1 encoded by a character-aware bi-LSTM, producing per-word vectors $(o_t, a, r)$2. A learned attention mechanism, $(o_t, a, r)$3, produces a fixed-size context vector $(o_t, a, r)$4 that fuses the author embedding $(o_t, a, r)$5, the rhyme embedding $(o_t, a, r)$6, and the poem representation. A one-hidden-layer MLP of size $(o_t, a, r)$7 then outputs a softmax over the $(o_t, a, r)$8 positions,

$(o_t, a, r)$9

The prompter network is context2vec-style and bidirectional around the chosen position $o_t=(w_1,\ldots,w_N)$0, omitting $o_t=(w_1,\ldots,w_N)$1 itself. Left and right LSTMs produce $o_t=(w_1,\ldots,w_N)$2, which are concatenated with $o_t=(w_1,\ldots,w_N)$3 and $o_t=(w_1,\ldots,w_N)$4, then linearly projected and passed through a softmax over $o_t=(w_1,\ldots,w_N)$5. The generator network is a conditional sequence-to-sequence LLM that produces the initial draft $o_t=(w_1,\ldots,w_N)$6. Its encoder is a char-aware bi-LSTM over any given prefix verses; its decoder is an LSTM over the previous word together with $o_t=(w_1,\ldots,w_N)$7 and $o_t=(w_1,\ldots,w_N)$8, followed by a GRU with attention over the encoder, and final softmax sampling with nucleus sampling at top-$o_t=(w_1,\ldots,w_N)$9 (Zugarini et al., 2021).

This architecture decomposes drafting and revision into separate modules. The generator produces a plausible initial poem, the detector learns where revision is useful, and the prompter learns how to alter a local position coherently under author and rhyme conditioning. The paper’s framing is explicitly human-inspired: poems are not produced in “just one breath” but are repeatedly revisited and corrected.

3. Reward shaping, optimization, and empirical behavior in poem revision

Rhyme detection uses the Pronouncing library to map each word $N$0 to a rhyming class $N$1 based on the final phoneme sequence. The instantaneous reward at timestep $N$2 is

$N$3

where matching means that for every pair of lines $N$4 with the same rhyme label in $N$5, $N$6. The cumulative undiscounted return is

$N$7

with $N$8, terminating either when the rhyme scheme is satisfied or when the maximum revision length is reached (Zugarini et al., 2021).

The policy is optimized with PPO using the clipped surrogate objective

$N$9

where $a$0 and $a$1. Advantage estimation uses GAE,

$a$2

with $a$3 and $a$4 tuned in $a$5. The implementation may attach a separate value head $a$6 to the detector and can add an entropy bonus; PPO epochs are early-stopped when the KL divergence exceeds a trust-region threshold (Zugarini et al., 2021).

Training proceeds through on-policy sample collection in “volleys.” Start states are sampled from the generator as $a$7. In experiments with a fixed number of poems, the procedure cycles through a fixed set of $a$8 distinct generated drafts; in the “dynamic” setting, fresh drafts are sampled each episode. Episodes run for at most $a$9 revision steps. After collecting $r$0 episodes—$r$1 for $r$2 poems and $r$3 for $r$4—the method computes normalized per-batch advantages, then performs $r$5 PPO epochs with mini-batches of size approximately $r$6, clipping with $r$7 and early-stopping on KL. Training spans $r$8 volleys, with the average episode return per volley as the monitored quantity (Zugarini et al., 2021).

The dataset comprises $r$9 quatrains from Project Gutenberg, restricted to English and automatic rhyme annotation; each quatrain has $i\in\{1,\ldots,N\}$0 lines and a maximum of $i\in\{1,\ldots,N\}$1 tokens per line. The vocabulary size is $i\in\{1,\ldots,N\}$2, the embedding size is $i\in\{1,\ldots,N\}$3, the number of author IDs is $i\in\{1,\ldots,N\}$4, and the number of rhyme-scheme types is $i\in\{1,\ldots,N\}$5. The conditional generator improves over the vanilla generator from validation/test perplexity $i\in\{1,\ldots,N\}$6 to $i\in\{1,\ldots,N\}$7, and the conditional prompter improves from $i\in\{1,\ldots,N\}$8 to $i\in\{1,\ldots,N\}$9. In the “poem reconstruction” ablation with a perfect prompter oracle, PPO average total return improves from approximately $i$0 for $i$1–$i$2 poems and from $i$3 for $i$4 poems. In the full generate-and-revise setting, PPO steadily improves average return per volley, while Vanilla Policy Gradient is unstable; for $i$5, PPO moves from $i$6, whereas VPG remains near random performance at $i$7 (Zugarini et al., 2021).

Qualitative examples with AABB and ABBB schemes show that the model learns to concentrate edits on line-end words despite not being told explicitly to do so, and it often alters interior words to preserve fluency around newly introduced rhymes. The paper presents this as evidence that shortest-path revision under sparse rhyme feedback is feasible without direct supervision over which tokens “should rhyme.”

4. Systems architecture of RhymeRL for LLM reinforcement learning

The 2025 RhymeRL is a systems framework rather than a poem-editing policy. It inherits the disaggregated rollout–reward–train pipeline of modern LLM RL systems such as veRL. Rollout workers perform LLM inference to generate reasoning chains and use HistoSpec during decoding; reward workers score completed rollouts with rule-based or learned reward models and forward $i$8 tuples to the replay buffer; train workers sample from the replay buffer, perform policy optimization such as GRPO or DAPO, and publish updated model weights to rollout workers’ weight buffers (He et al., 26 Aug 2025).

A central controller assigns prompt sub-batches to rollout workers in a streaming fashion, using historical length-rank data to invoke HistoPipe. CPU-side history workers maintain suffix-tree indexes of past rollouts together with the length rankings used by rollout-time scheduling; these workers run opportunistically on idle CPUs and ingest completed rollouts asynchronously. Weight updates, prompt dispatch, rollouts, reward scoring, and training therefore proceed in a pipelined fashion with high overlap (He et al., 26 Aug 2025).

The motivating systems problem is GPU underutilization in LLM RL. The paper identifies two primary causes: the dominance of rollout time because of test-time scaling, and “GPU bubbles” induced by imbalances in rollout lengths within a batch. Earlier mitigations such as asynchronous execution and truncation are described as offering partial relief while potentially compromising training accuracy for efficiency. RhymeRL addresses these two bottlenecks through two modules: HistoSpec for accelerating rollout generation, and HistoPipe for balancing rollout workloads (He et al., 26 Aug 2025).

This formulation relocates the use of RL from the object level to the infrastructure level. The underlying policy optimization algorithm may still be GRPO or DAPO, but the contribution lies in how rollouts are generated, scheduled, and synchronized across workers.

5. Historical reuse mechanisms: HistoSpec and HistoPipe

HistoSpec is motivated by an empirical observation: in RLHF, when the same prompt is rolled out repeatedly across adjacent epochs with small clipped model updates, $i$9–$w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$0 of tokens in a rollout at epoch $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$1 appear as contiguous subsequences in the previous epoch’s rollout for that prompt. The system therefore reuses historical token sequences as speculative drafts. At each decoding step for prompt $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$2, it takes the last $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$3 decoded tokens as a prefix, queries a suffix-tree index for matching nodes, extracts up to $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$4 speculative continuation tokens, verifies the draft in a single LLM forward pass through the KV-cache, accepts the longest verified prefix, appends accepted tokens, and repeats (He et al., 26 Aug 2025).

Each prompt maintains a suffix tree $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$5 built over all rollouts from the last epoch. Nodes correspond to substrings, edges represent one or more tokens, and each node stores a priority equal to the sum of reward scores of all rollouts containing that substring as a suffix. This priority guides branch selection when several suffix continuations are available. History workers build the index asynchronously in $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$6 time and memory per epoch, where $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$7 is the total number of tokens (He et al., 26 Aug 2025).

The speculation window uses an AIMD-like policy. The system initializes $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$8; after full acceptance of a $w' \sim p(w' \mid o_t \setminus \{w_i\}, a, r)$9-token draft, it updates $w_i$0 with default $w_i$1; after any rejection, it resets $w_i$2. The prefix length is similarly reduced from $w_i$3 toward $w_i$4 if no suffix-tree match exists. The corresponding recurrence is

$w_i$5

with $w_i$6, $w_i$7, and $w_i$8 (He et al., 26 Aug 2025).

HistoPipe is motivated by a second empirical regularity: when prompts are ranked by rollout length in epoch $w_i$9, $(o_t, a, r)$00–$(o_t, a, r)$01 of prompts remain in or near their previous rank in epoch $(o_t, a, r)$02. The system partitions prompts into $(o_t, a, r)$03 equally sized ranking groups sorted by increasing historical length. Tier 1 alternates ascending and descending group assignment across steps: on odd steps, worker $(o_t, a, r)$04 is assigned to group $(o_t, a, r)$05, worker $(o_t, a, r)$06 to group $(o_t, a, r)$07, and so forth; on even steps, worker $(o_t, a, r)$08 is assigned to group $(o_t, a, r)$09, worker $(o_t, a, r)$10 to group $(o_t, a, r)$11, and so forth. This creates inter-step complementarity between short and long rollouts (He et al., 26 Aug 2025).

Tier 2 reshapes GPU allocation because equal GPU counts per group still leave residual bubbles under long-tailed length distributions. The system solves

$(o_t, a, r)$12

subject to

$(o_t, a, r)$13

where $(o_t, a, r)$14 is obtained by pre-profiling and represents expected rollout time for length $(o_t, a, r)$15 with data-parallel width $(o_t, a, r)$16. A binary search over $(o_t, a, r)$17 finds the minimal allocation plan $(o_t, a, r)$18 satisfying these constraints. Additional migration-based rebalancing handles anomalously long outliers through intra-step reassignment of the last $(o_t, a, r)$19 of tasks in a group when their length exceeds $(o_t, a, r)$20 the historic maximum, and through inter-step deferral of very long rollouts (He et al., 26 Aug 2025).

Two misconceptions are directly addressed in this design. One is that speculative decoding necessarily changes the model’s sampling distribution; the paper states that speculative decoding preserves the exact output distribution because the historical drafts are only verified, not imposed. The other is that efficiency must be purchased by increasing off-policyness; RhymeRL states that it strictly limits off-policyness to one RL step, matching veRL’s one-step stale-weights paradigm (He et al., 26 Aug 2025).

6. Experimental results, correctness claims, and interpretive context

The LLM-systems evaluation uses a $(o_t, a, r)$21-node cluster, each node containing $(o_t, a, r)$22 NVIDIA H100 GPUs, dual-socket Sapphire Rapids with $(o_t, a, r)$23 cores, $(o_t, a, r)$24 TB DRAM, and InfiniBand. The evaluated models are Qwen3-8B, Qwen3-14B, and Qwen2.5-32B; the RL algorithms are GRPO by default and DAPO; rollout group size is $(o_t, a, r)$25 samples per prompt. Baselines are veRL v0.4.1 and AReaL with off-policyness $(o_t, a, r)$26 and $(o_t, a, r)$27 (He et al., 26 Aug 2025).

Against these baselines, RhymeRL reports up to $(o_t, a, r)$28 throughput improvement over veRL, with average $(o_t, a, r)$29 at $(o_t, a, r)$30K token maximum and $(o_t, a, r)$31 at $(o_t, a, r)$32K; up to $(o_t, a, r)$33 over AReaL with off-policyness $(o_t, a, r)$34, with average $(o_t, a, r)$35 at $(o_t, a, r)$36K and $(o_t, a, r)$37 at $(o_t, a, r)$38K; and up to $(o_t, a, r)$39 over AReaL with off-policyness $(o_t, a, r)$40. In ablations, the hybrid rollout pipeline contributes a $(o_t, a, r)$41 performance boost, two-tier scheduling contributes a further $(o_t, a, r)$42, and speculative decoding contributes a further $(o_t, a, r)$43. HistoSpec alone yields up to $(o_t, a, r)$44 per-step rollout speedups, the speculation rate grows to approximately $(o_t, a, r)$45, and the acceptance rate remains between $(o_t, a, r)$46 and $(o_t, a, r)$47. HistoPipe shortens $(o_t, a, r)$48-step training time by up to $(o_t, a, r)$49, with two-tier scheduling yielding up to $(o_t, a, r)$50 further improvement; only approximately $(o_t, a, r)$51–$(o_t, a, r)$52 of samples are migrated as outliers (He et al., 26 Aug 2025).

On accuracy, reward-curve trajectories for Math and Code tasks on $(o_t, a, r)$53B models are reported to nearly overlap with veRL, and off-policyness $(o_t, a, r)$54 is stated to be strictly preserved, with no degradation in training accuracy. The correctness argument for HistoSpec is that verifying speculative drafts in bundle preserves the exact autoregressive distribution, since historical drafts only reorder inference calls and do not alter logits. The scheduling argument is that the observed rank stability across epochs is sufficiently high to support bounded bubble time under mild assumptions on rollout-length drift (He et al., 26 Aug 2025).

Taken together, the two RhymeRL lines of work illustrate two different uses of reinforcement learning over structured textual processes. The 2021 framework treats revision itself as the decision problem, with actions over edit positions and rewards defined by rhyme satisfaction. The 2025 framework treats rollout generation history and rollout-length history as reusable signals for systems optimization. This suggests a broader interpretation of “history rhymes” across both usages: prior textual structure can serve either as the object of control, as in poem revision, or as a systems prior for accelerating repeated RL computation.