Paragraph-level Relative Policy Optimization (PRPO)

• The paper introduces PRPO as a framework that segments responses into meaningful paragraphs for fine-grained reward assignment.
• It has been applied in both multimodal deepfake detection and long-form reasoning, showing improvements in F1 scores and accuracy metrics.
• The method combines visual consistency, prediction consistency, and entropy-based segmentation to compute relative advantages for stable PPO-style updates.

Paragraph-level Relative Policy Optimization (PRPO) denotes policy-optimization schemes that assign learning signal to semantically meaningful subunits of a generated response, especially paragraphs or segment-like reasoning units, rather than treating the entire response as a single unit or broadcasting one scalar reward to all tokens. In current arXiv usage, however, the acronym is overloaded. The explicit phrase “Paragraph-level Relative Policy Optimization” appears in a multimodal deepfake-detection setting, where PRPO is a test-time reinforcement-learning algorithm that optimizes paragraph likelihoods under visual-grounding and consistency rewards (Nguyen et al., 30 Sep 2025). A separate reasoning paper uses PRPO to mean “Process Relative Policy Optimization,” a critic-free method that segments multi-step reasoning and aligns dense process rewards with sparse outcome rewards (Ding et al., 12 Jan 2026). In RLHF, “PRPO” instead means “Pessimistic Reward-based Policy Optimization,” and that paper explicitly states that a paragraph-level PRPO formulation is not defined there (Gupta et al., 10 Mar 2025). The term therefore refers less to a single canonical algorithm than to a recurring design principle: shifting credit assignment from whole responses to structured intermediate units.

1. Terminological scope and acronym ambiguity

The literature uses closely related names for materially different objectives. This ambiguity is not incidental; it reflects several contemporaneous attempts to refine credit assignment beneath the response level while preserving PPO- or GRPO-style stability.

Usage	Expansion	Scope
PRPO	Paragraph-level Relative Policy Optimization	Test-time RL for multimodal deepfake detection with paragraph-level rewards and paragraph-level PPO-style updates
PRPO	Process Relative Policy Optimization	Critic-free reasoning optimization with entropy-based segmentation and fused process/outcome advantages
PRPO	Pessimistic Reward-based Policy Optimization	RLHF objective derived from pessimistic preference optimization; paragraph-level form not defined
RPO	Relative Policy Optimization	Policy-transfer method for MDPs; unrelated to paragraph-level language-model alignment

The deepfake-detection formulation is the clearest literal instance of paragraph-level PRPO. It defines outputs as multi-paragraph explanations plus a final classification, computes rewards per paragraph, normalizes those rewards within the sampled group, and optimizes paragraph likelihoods directly (Nguyen et al., 30 Sep 2025). The reasoning formulation is different in name and mechanism but shares the same granularity shift: “paragraph-level” there means step- or segment-level groups identified by semantic clues rather than fixed-length slices (Ding et al., 12 Jan 2026). By contrast, the RLHF paper “Mitigating Preference Hacking in Policy Optimization with Pessimism” states that PRPO stands for Pessimistic Reward-based Policy Optimization and that a “paragraph-level PRPO” formulation is not defined in that work (Gupta et al., 10 Mar 2025). An older reinforcement-learning paper defines RPO as Relative Policy Optimization for fast policy transfer between MDPs and contains no PRPO construct (Xu et al., 2022).

2. Paragraph-level PRPO in multimodal deepfake detection

The explicit paragraph-level algorithm is introduced for vision-language deepfake detection, motivated by the observation that off-the-shelf multimodal LLMs often jump to conclusions prematurely, produce shallow or generic explanations not anchored in visual evidence, and exhibit poor consistency between explanation and final classification (Nguyen et al., 30 Sep 2025). The task input is an image $x$ together with a textual prompt $v$. The model encodes the image into tokens $z$ using a frozen CLIP ConvNeXT encoder and a learned projector, then decodes text with a Vicuna-based LLM. Each output $o$ is segmented into paragraphs $\{p_1,\dots,p_K\}$, with the last paragraph $p_K$ designated as the final decision sentence.

The architecture is implemented in a LLaVA-style system adapted for fine-grained visual artifacts, denoted DX-LLaVA. Its vision encoder is CLIP ConvNeXT-Large-D 320, frozen during training; Stage 3 features produce a $10\times10$ grid of $100$ pixel embeddings with $1536$-dimensional vectors, which are projected by a linear map into the $4096$-dimensional Vicuna token space. The LLM is Vicuna in a LLaVA-7B configuration, and the system also includes a classifier head on pooled visual features:

$v$0

Before PRPO is applied, the backbone is fine-tuned with a supervised multimodal objective

$v$1

with $v$2. The language-model term is

$v$3

and the binary term is

$v$4

The distinctive step is test-time reinforcement learning at paragraph granularity. For a given image and prompt, the policy samples multiple outputs, splits each output into paragraphs, scores each paragraph individually, computes paragraph-level relative advantages across the sampled group, and performs a small number of PPO-style updates on paragraph likelihoods. This design makes the paragraph, rather than the token or full response, the principal credit-assignment unit (Nguyen et al., 30 Sep 2025).

3. Reward design, paragraph-level objective, and optimization procedure

The paragraph-level reward is the average of two components. The first is the Visual Consistency Reward (VCR), intended to ground each paragraph in image evidence. Representative keywords $v$5 are extracted from paragraph $v$6 using YAKE, encoded by CLIP, and compared to image features:

$v$7

The second is the Prediction Consistency Reward (PCR), which enforces consistency between the final decision paragraph and the trend expressed in the earlier reasoning. For intermediate paragraphs, $v$8. For the final paragraph $v$9, the reward is determined by majority voting over labels inferred from preceding paragraphs using dictionary-based scoring with real terms $z$0, fake terms $z$1, and negation terms $z$2:

$z$3

The overall paragraph reward is then

$z$4

Rewards are normalized across all paragraphs in the current sampled group. If $z$5 and $z$6 are the mean and standard deviation over all paragraph rewards in the group, the paragraph-level relative advantage is

$z$7

This is the sense in which the method is “relative”: reward is contextualized within the set of candidate paragraphs produced for the current input.

The optimization objective is PPO-style clipping applied at paragraph granularity. With paragraph-level importance ratio

$z$8

the loss is

$z$9

A KL penalty toward a reference policy stabilizes updates:

$o$0

and the total objective is

$o$1

In the reported implementation, $o$2 for PRPO, the optimization is performed in the verl/HybridFlow framework, and the test-time procedure samples $o$3 outputs per input, typically $o$4–$o$5, runs $o$6–$o$7 RL steps per test input, and uses a small learning rate of $o$8 to avoid drift. No explicit length penalty is added; instead, irrelevant or hallucinated paragraphs are implicitly penalized by low VCR, while contradictions are penalized by PCR (Nguyen et al., 30 Sep 2025).

4. Segment-wise PRPO for multi-step reasoning

A separate line of work introduces PRPO as “Process Relative Policy Optimization” for long-form mathematical reasoning, where the central problem is sparse and delayed reward under critic-free optimization (Ding et al., 12 Jan 2026). In GRPO-style training, each sampled completion receives a single normalized outcome reward, and that scalar is applied to all tokens. The paper argues that this produces weak temporal credit assignment in long chains of thought. It also argues that PRM-only optimization can collapse prematurely, because early segments with negative advantages can dominate the gradient before later high-reward steps can compensate.

In that formulation, “paragraph-level” means semantically delimited reasoning segments rather than fixed-size windows. The segmentation mechanism uses token-entropy spikes:

$o$9

Segments $\{p_1,\dots,p_K\}$0 are formed by selecting the top-$\{p_1,\dots,p_K\}$1 entropy spikes, with $\{p_1,\dots,p_K\}$2 in experiments, subject to a minimum gap of $\{p_1,\dots,p_K\}$3 tokens. The resulting segments often coincide with logical connectors, line breaks, “Step”, “Therefore”, or equation delimiters. Each segment $\{p_1,\dots,p_K\}$4 receives a PRM score $\{p_1,\dots,p_K\}$5 from Qwen2.5-Math-PRM-7B.

The process score is normalized against a fixed prior rather than per-rollout statistics:

$\{p_1,\dots,p_K\}$6

so that

$\{p_1,\dots,p_K\}$7

Outcome reward is separately mean-centered over the rollout group:

$\{p_1,\dots,p_K\}$8

The fused token-level advantage is then a location-only shift,

$\{p_1,\dots,p_K\}$9

which preserves the segment-wise relative structure of the PRM signal while anchoring it to the trajectory-level outcome.

The training objective is critic-free and KL-regularized:

$p_K$0

Implementations may replace the log-probability term with PPO-style clipping; the reported setting uses KL coefficient $p_K$1, PPO clip ratio $p_K$2, $p_K$3 rollouts per prompt, batch size $p_K$4, learning rate $p_K$5, maximum new tokens $p_K$6, and the VeRL framework on $p_K$7H200 GPUs.

Empirically, this PRPO improves Qwen2.5-Math-1.5B on MATH500 from $p_K$8 to $p_K$9 over GRPO with eight rollouts and no value network. The segmentation ablation is unusually sharp: random splits yield $10\times10$0 accuracy, uniform splits yield $10\times10$1, GRPO yields $10\times10$2, and entropy-based PRPO yields $10\times10$3. Fixed-prior normalization reaches $10\times10$4 and remains stable across epochs, whereas per-rollout relative normalization reaches a $10\times10$5 peak but later exhibits catastrophic drops (Ding et al., 12 Jan 2026).

5. Relation to GRPO, pairwise relative methods, and pessimistic PRPO

The primary contrast class for paragraph-level PRPO is GRPO. In the multimodal setting, GRPO assigns one normalized score per sampled response and applies that advantage uniformly to all tokens of the response. Paragraph-level PRPO replaces that single response-level signal with paragraph-specific rewards and paragraph-specific normalized advantages, thereby amplifying visually grounded reasoning paragraphs and suppressing misaligned ones (Nguyen et al., 30 Sep 2025). In the reasoning setting, GRPO likewise broadcasts a single scalar outcome advantage to all tokens, whereas PRPO injects segment-wise process structure into token-level credit assignment (Ding et al., 12 Jan 2026).

LambdaPO is conceptually adjacent but not identical. It replaces GRPO’s cohort-mean baseline with a Pairwise Decomposed Advantage built from reward differentials against all peers in the cohort and weighted by the policy’s own confidence:

$10\times10$6

It also augments sparse outcome supervision with a semantic density reward based on ROUGE-L. That paper does not define PRPO explicitly, but it states that the pairwise mechanism and dense semantic alignment are directly relevant to paragraph-level relative optimization, because they can supply finer-grained relational signals for segment-level updates (Yuan et al., 19 May 2026).

PAPO provides another neighboring perspective. It composes the advantage from an outcome term normalized over all responses and a process term normalized only among correct responses:

$10\times10$7

Its detailed exposition then connects this design to a paragraph-level extension in which paragraph-conditioned process advantages are normalized over correct responses only, with weights summing to one across paragraphs to avoid verbosity inflation. That presentation is a proposed extension rather than a separately evaluated paragraph-level PRPO algorithm, but it formalizes a decoupled-normalization template for paragraph-conditioned credit assignment (Tan et al., 27 Mar 2026).

Acronym collision is most consequential in RLHF. In “Mitigating Preference Hacking in Policy Optimization with Pessimism,” PRPO means Pessimistic Reward-based Policy Optimization, the Bradley–Terry–Luce reward-model specialization of a general pessimistic preference objective. The paper explicitly states that a paragraph-level PRPO formulation is not defined there, although it outlines how the objective could be decomposed over paragraphs if the pipeline provides paragraph-level pairs, paragraph-conditional sampling, and paragraph-level uncertainty estimates (Gupta et al., 10 Mar 2025). This is distinct again from the earlier RPO of fast policy transfer, which optimizes relative returns across source and target MDP dynamics and is unrelated to paragraph-level language-model alignment (Xu et al., 2022).

6. Empirical profile, limitations, and open directions

The multimodal paragraph-level formulation reports strong empirical gains in cross-domain deepfake detection. On held-out domain transfer, average F1 rises from $10\times10$8 for LLaVA and $10\times10$9 for DX-LLaVA to $100$0 for PRPO; Gemini-2.5 averages $100$1. Explanation quality, evaluated by GPT-4o on a $100$2–$100$3 scale across CAC, EGIA, RQ, CC, and CU, improves from $100$4 for DX-LLaVA and $100$5 for Gemini-2.5 to $100$6 for PRPO. On the $100$7DDIM RL-method comparison, PRPO achieves F1 $100$8 versus PPO $100$9 and GRPO $1536$0. Reward ablations on $1536$1SD show that VCR alone yields F1 $1536$2, PCR alone yields $1536$3, and the full reward yields $1536$4, indicating that grounding and intra-response consistency act complementarily rather than interchangeably (Nguyen et al., 30 Sep 2025).

The reasoning-domain formulation shows a different empirical profile. Its main gain is not multimodal grounding but stable fine-grained credit assignment under critic-free optimization. PRPO improves MATH500 accuracy from $1536$5 to $1536$6, prevents the premature truncation associated with PRM-only optimization, and relies on semantically informed segmentation rather than tokenwise or rolloutwise heuristics. The method does, however, introduce substantial overhead: average step time increases from $1536$7s without PRM to $1536$8s for single training and $1536$9s for three concurrent trainings (Ding et al., 12 Jan 2026).

Limitations differ by instantiation. In multimodal deepfake detection, VCR depends on CLIP similarity over YAKE keywords, so missed keywords or weak forensic sensitivity in CLIP can inject reward noise; PCR is based on dictionary heuristics and may miss nuanced language; dataset construction may reflect biases from annotation distillation; and test-time RL adds sampling and optimization overhead (Nguyen et al., 30 Sep 2025). In reasoning, performance depends on PRM quality, entropy spikes may fail to align with genuine semantic boundaries, adversarial formatting can disrupt segmentation or PRM scoring, and process guidance becomes weak when correctness is so low that there are too few useful positive trajectories (Ding et al., 12 Jan 2026).

The broader research direction suggested by these papers is hierarchical relative optimization: response-level correctness anchoring, paragraph- or segment-level process rewards, and normalization schemes that preserve local structure without inviting reward hacking. The literature already sketches several routes toward that goal: pairwise decomposed cohort comparisons with semantic density rewards (Yuan et al., 19 May 2026), correct-subset paragraph normalization to suppress verbosity exploitation (Tan et al., 27 Mar 2026), and pessimistic paragraph-level decomposition when uncertainty over reward or preference models can be estimated reliably (Gupta et al., 10 Mar 2025). This suggests that “paragraph-level PRPO” is best understood not as a single settled algorithmic object, but as an emerging design space for structured credit assignment in aligned generation.