Progress Advantage in Multi-Domain Systems

• Progress advantage is a notion of progress-sensitive supervisory signals that discern actions moving systems toward success versus those that stall or regress.
• It is applied in robot learning, LLM reasoning, and quantum technology by subtracting state baselines from future success estimates to guide action selection.
• Its implementations range from reward shaping and policy reweighting in robotics to step-level advantage estimation and analytical policy divergence in agentic systems.

Progress advantage denotes a family of progress-sensitive supervisory signals used to distinguish actions, transitions, chunks, or reasoning steps that move a system closer to successful completion from those that stall, regress, or merely preserve a promising state. Across current literature, the term is not used uniformly. In robot learning, it commonly appears as a phase-aware reweighting signal, a shaped reward, or a self-supervised advantage proxy; in LLM reasoning, it denotes a step-level change in future correctness or an implicit advantage recovered from RL post-training; in quantum-technology roadmapping, it denotes programmatic acceleration toward practical quantum advantage rather than an algorithmic value function (Kim et al., 27 May 2026, Setlur et al., 2024, Oh et al., 24 Jun 2026, Alsing et al., 2022).

1. Conceptual scope and formal motifs

A recurrent formal pattern is the subtraction of a state baseline from an action- or step-conditioned estimate of future success. In ProgVLA, the core quantity is

$A_p(c_t, a_t) = Q_p(c_t, a_t) - V_p(c_t),$

where both heads regress a normalized remaining-horizon target from the policy’s own compact multimodal context. In process-verifier work for LLM reasoning, the analogous definition is

$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$

with the crucial difference that the value functions are defined under a prover policy $\mu$ distinct from the base policy (Kim et al., 27 May 2026, Setlur et al., 2024).

The literature also repeatedly contrasts progress advantage with absolute progress. ARM explicitly argues that absolute progress attempts to assign a global value $P_t$ to each state and is sensitive to stage boundaries and non-monotonic behaviors such as backtracking and recovery. It therefore reframes supervision as local relative progress, using tri-state labels—Progressive, Regressive, and Stagnant—on short transitions rather than dense scalar progress scores (Mao et al., 3 Apr 2026). A related distinction appears in ProgVLA, where the target

$r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$

is described as a purely temporal signal, and “progress-aware” refers to the training objective rather than verified semantic progress estimation (Kim et al., 27 May 2026).

A common misconception is that all uses of progress advantage are standard RL advantages in the strict off-policy sense. The surveyed work does not support that interpretation. Some methods use detached sample weights for supervised imitation, some use self-supervised advantage proxies without learned $Q/V$ functions, some use potential-based reward shaping, and some recover an optimal advantage implicitly from policy ratios after RL post-training (Kim et al., 27 May 2026, Yu et al., 26 Jun 2026, Tan et al., 8 Mar 2026, Oh et al., 24 Jun 2026).

2. Offline robot manipulation, imitation learning, and data curation

ProgVLA gives one of the most explicit robot-manipulation formulations. At time $t$, the multimodal encoder produces control-ready context tokens

$c_t = \pi_\phi(I_t, q_t, l_n),$

and the progress heads learn an action-conditioned critic $Q_p(c_t,a_t)$, a value baseline $V_p(c_t)$ trained via expectile regression, and a success head $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$0. The detached progress and success weights are

$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$1

and the total objective combines weighted flow matching with auxiliary value, critic, and success losses. Architecturally, this is coupled to a multi-modal encoder with two-stage Perceiver resampling that compresses long visual, language, and proprioceptive sequences into a small set of control-ready context tokens. On LIBERO, the 0.1B model reports success rates of $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$2, $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$3, $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$4, $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$5, and $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$6 across Spatial, Object, Goal, Long, and Avg; on Meta-World it reports $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$7, $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$8, $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$9, $\mu$0, and $\mu$1 across Easy, Medium, Hard, Very hard, and Avg. Ablations show that removing progress objectives lowers LIBERO average success from $\mu$2 to $\mu$3, with the largest drops on Object and Long (Kim et al., 27 May 2026).

ARM replaces dense progress rewards with local relative labels. Its interval head predicts $\mu$4, a completion head predicts terminal success, and dense progress curves $\mu$5 are reconstructed by accumulating predicted local gains and anchoring them with a completion signal. Policy reweighting is based on the length-adaptive gain

$\mu$6

followed by either clamp-based weighting or the thresholding rule $\mu$7, else $\mu$8. On the towel-folding task, ARM reports reward-model MSE $\mu$9 versus SARM’s $P_t$0, and AW-BC with ARM reaches a $P_t$1 success rate (Mao et al., 3 Apr 2026).

WARP and STEAM generalize the same intuition to self-supervised data curation. WARP-RM learns dense signed relative progress magnitudes from time-warped successful demonstrations and defines a chunk-level proxy

$P_t$2

with WARP-BC using

$P_t$3

On the D2 T-shirt-folding dataset, WARP-BC reports $P_t$4 success versus vanilla BC’s $P_t$5; on bottle-in-bin it reports $P_t$6 successful placements versus $P_t$7 for vanilla BC (Yu et al., 26 Jun 2026). STEAM instead trains an ensemble of temporal-offset predictors and converts each predictor’s distribution over offset bins into a scalar frame-level advantage, then takes a conservative minimum across the ensemble:

$P_t$8

When combined with CFGRL, STEAM improves policy success rate by $P_t$9, $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$0, $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$1, and $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$2 on bimanual towel folding, chip checkout, cola restocking, and single-arm pick-and-place, respectively (Liu et al., 29 Jun 2026).

3. Progress-shaped rewards and differentiable progress guidance in robotics

ProgAgent uses progress advantage in a more classical RL sense. It trains a perceptual model $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$3 on expert-video triplets $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$4 to predict a Gaussian distribution over the normalized progress ratio $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$5, interprets the resulting progress estimate as a state potential $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$6, and shapes rewards by

$r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$7

The policy then computes the usual PPO/GAE advantage under the shaped reward,

$r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$8

while stability across tasks is maintained by coreset replay and Synaptic Intelligence. To reduce overconfident predictions on out-of-distribution states, the reward model is regularized by an adversarial push-back term that pulls predictions on policy rollouts toward a conservative prior. On ContinualBench, ProgAgent reports, for example, a button-press success rate of $r_t = \max\!\left(0,\, 1 - \frac{T-t}{T_M}\right)$9 and is described as outperforming Rank2Reward, TCN, and even a Perfect Memory baseline (Tan et al., 8 Mar 2026).

ProgressVLA uses progress as differentiable guidance rather than as a detached imitation weight. It pre-trains a progress estimator

$Q/V$0

with normalized-time targets $Q/V$1 and reports a residual MAE of $Q/V$2 on CALVIN. It then learns an inverse-dynamics world model in DINO feature space and defines an evaluator

$Q/V$3

whose gradient guides diffusion sampling through a maximal progress regularization. On CALVIN, introducing progress guidance increases the Pearson correlation between predicted progress and a linear ramp from $Q/V$4 to $Q/V$5, reduces average steps from $Q/V$6 to $Q/V$7, and raises success rate from $Q/V$8 to $Q/V$9. On LIBERO, the full model reports $t$0 average success, and in real-world deployment it reports $t$1 success with classifier guidance, versus $t$2 without guidance and $t$3 for the Octo baseline (Yan et al., 29 Mar 2026).

These robotic formulations establish a broad taxonomy. Progress advantage may be a critic-derived sample weight, a local relative-gain label, a self-supervised advantage proxy calibrated to expert pace, a conservative ensemble score, a potential-based shaping term, or a differentiable evaluator over latent future states. The shared role is not identical parameterization but selective emphasis on transitions associated with forward task completion.

4. Process advantages in LLM reasoning

In LLM reasoning, the central formalization is the process advantage verifier. “Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning” defines the correct step-level process reward as the change in a prover’s probability of eventual correctness caused by taking a step:

$t$4

The paper argues that using $t$5 under the base policy is unhelpful for RL, because policy gradients already subtract a baseline, while overly strong provers collapse $t$6 toward zero. It therefore advocates a complementary prover $t$7, often Best-of-$t$8 with moderate $t$9, and scores expansions with

$c_t = \pi_\phi(I_t, q_t, l_n),$0

Empirically, PAV beam search is reported to be more than $c_t = \pi_\phi(I_t, q_t, l_n),$1 more accurate and $c_t = \pi_\phi(I_t, q_t, l_n),$2–$c_t = \pi_\phi(I_t, q_t, l_n),$3 more compute-efficient than ORM-based search, while online RL with dense PAV rewards yields $c_t = \pi_\phi(I_t, q_t, l_n),$4–$c_t = \pi_\phi(I_t, q_t, l_n),$5 gains in sample efficiency and more than $c_t = \pi_\phi(I_t, q_t, l_n),$6 accuracy gains over ORM-RL (Setlur et al., 2024).

SPAE develops a distinct step-level formulation for RLVR in mathematics reasoning. It probes intermediate confidence and correctness at every step boundary, then combines them into a Step Potential

$c_t = \pi_\phi(I_t, q_t, l_n),$7

Once $c_t = \pi_\phi(I_t, q_t, l_n),$8 exceeds the saturation threshold $c_t = \pi_\phi(I_t, q_t, l_n),$9, subsequent tokens are treated as checking rather than solving. The token-level SPAE advantage multiplies a group-relative outcome advantage by a saturation penalty and adds a shaped term based on $Q_p(c_t,a_t)$0:

$Q_p(c_t,a_t)$1

On DeepSeek-R1-Distill-Qwen-7B, SPAE reports average accuracy $Q_p(c_t,a_t)$2 and average length $Q_p(c_t,a_t)$3, versus base accuracy $Q_p(c_t,a_t)$4 and length $Q_p(c_t,a_t)$5; on AIME2024 it reduces Right-to-Wrong failures from $Q_p(c_t,a_t)$6 to $Q_p(c_t,a_t)$7 and checking tokens from $Q_p(c_t,a_t)$8 to $Q_p(c_t,a_t)$9 (Wu et al., 7 Jan 2026).

A related, though not identically named, line is EDGE-GRPO. It addresses advantage collapse in GRPO by entropy-driven reweighting

$V_p(c_t)$0

and Guided Error Correction to ensure mixed-quality groups. On DeepScaleR-Random-1K, Qwen2.5-Math-7B improves from $V_p(c_t)$1 with vanilla GRPO to $V_p(c_t)$2 with EDGE-GRPO; on DeepScaleR-Hard-1K, it improves from $V_p(c_t)$3 to $V_p(c_t)$4 (Zhang et al., 29 Jul 2025). This suggests that progress-sensitive advantages in reasoning are closely tied to the broader problem of recovering non-degenerate, step-discriminative training signals under sparse outcome rewards.

5. RL post-training and implicit progress advantage in agentic LLM systems

“Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents” gives the most explicit theorem-level identification of progress advantage with an optimal RL quantity. Under a KL-regularized RL objective with reference policy $V_p(c_t)$5, the paper defines

$V_p(c_t)$6

and proves that

$V_p(c_t)$7

In this formulation, progress advantage is not learned by a separate process reward model; it is a byproduct of ordinary RL post-training. The practical recipe is correspondingly simple: compute per-token or per-action log-probability differences between the RL-trained policy and its reference, then aggregate them across a step or trajectory (Oh et al., 24 Jun 2026).

The reported use cases are test-time scaling, uncertainty quantification, and failure attribution. In Best-of-8 trajectory selection, the paper reports average success $V_p(c_t)$8 for Gemma4-4B and $V_p(c_t)$9 for Qwen3.5-9B, compared with ThinkPRM-14B at $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$00 and $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$01, respectively. For trajectory-level uncertainty on $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$02-bench Airline, PA AUROC reaches $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$03 on Gemma4 and $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$04 on Olmo3; for off-policy uncertainty, using Gemma4-4B PA to score Qwen trajectories on Airline yields AUROC $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$05 for Qwen3.5-9B and $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$06 for Qwen3-14B. The paper also reports that PA outperforms confidence-only baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models across the evaluated agentic benchmarks (Oh et al., 24 Jun 2026).

This line marks an important conceptual shift. In verifier-based reasoning work, progress advantage is usually predicted from rollouts or Monte Carlo estimates. In agentic post-training work, it is recovered analytically from the divergence between the learned policy and the reference policy. The underlying role is the same—step-level discrimination of helpful and harmful actions—but the estimator is fundamentally different.

6. Quantum-technology usage of the term

In the QTDP roadmap, “progress advantage” does not refer to an RL-style baseline-subtracted signal. It refers to programmatic acceleration achieved by focusing investment on intermediate Technology Readiness Levels. The roadmap defines practical quantum advantage as user-motivated advantage translated from laboratory research to higher TRLs, and defines progress advantage as the advantage gained in progress itself by standing up Quantum Technology Demonstration Projects at TRLs $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$07–$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$08. The central development loop is the scaling cycle $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$09, with advantage demonstration at stage $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$10, user access at stage $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$11, scientific applications at stage $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$12, and commercial development at stage $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$13 (Alsing et al., 2022).

The roadmap’s metrics are milestone- and KPI-based rather than advantage-estimation formulas. Illustrative targets include distributed computational capability scaling as $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$14 for two entangled $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$15-qubit processors, square-root-of-$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$16 sensing enhancement for distributed entangled clocks, ps-level timing and synchronization for metropolitan-scale entanglement distribution, and repeater demonstrations with entanglement distribution rates surpassing simple direct transmission distribution rates (Alsing et al., 2022). The term therefore carries a governance and systems-engineering meaning: progress advantage is faster, coordinated movement from curiosity-driven proofs of concept to open, user-accessible systems that can demonstrate practical quantum advantage.

Related quantum work measures progress toward advantage through hardware, algorithmic, and verification milestones rather than through a progress advantage signal. “Mind the gaps: The fraught road to quantum advantage” organizes this progression around four transitions—error mitigation to active error correction, rudimentary error correction to scalable fault tolerance, early heuristics to mature algorithms, and exploratory simulators to credible advantage—and uses waypoints such as megaquop, gigaquop, and teraquop operation, surface-code logical error scaling, and cross-platform verification (Eisert et al., 22 Oct 2025). “Runtime Quantum Advantage with Digital Quantum Optimization” operationalizes progress through time-to-approximate-solution and time-to-solution, reporting, for example, a $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$17-qubit case with TT$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$18 s versus TT$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$19 s and a hardest-instance demonstration with BF-DCQO TT$A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$20 s (Chandarana et al., 13 May 2025). This suggests that, in quantum technology, progress advantage functions as a translational and benchmarking concept rather than as a local training signal.

7. Recurrent limitations, ambiguities, and open questions

A recurring limitation is semantic calibration. ProgVLA explicitly notes that its target $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$21 is purely temporal and that the advantage signal is weakly identified strictly as an RL advantage; the paper recommends viewing it as a trajectory-phase reweighting term rather than a strict off-policy advantage (Kim et al., 27 May 2026). ARM similarly assumes a discernible terminal success signal and requires a small seed of tri-state labels; tasks with ambiguous goals, repeated cyclical subgoals, or poor visual observability may reduce anchor reliability (Mao et al., 3 Apr 2026). WARP’s negative-progress supervision is generated by reversed playback and is therefore synthetic, while STEAM’s min-ensemble is designed to suppress overestimation on mixed-quality rollouts but does not eliminate errors under severe visual aliasing or distribution shift (Yu et al., 26 Jun 2026, Liu et al., 29 Jun 2026).

Another recurring issue is estimator dependence. ProgAgent’s learned potential depends on expert-video quality and careful tuning of $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$22, $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$23, and $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$24, and the paper identifies scalability to very long horizons and complex multi-task interference as open problems (Tan et al., 8 Mar 2026). In LLM reasoning, PAV gains depend on choosing a complementary prover; provers that are too weak or too strong collapse $A^\mu(s_h, a_h) = Q^\mu(s_h, a_h) - V^\mu(s_h),$25 toward zero, while Monte Carlo estimation and verifier miscalibration can misguide search or RL (Setlur et al., 2024). For post-training-derived PA in agentic systems, the signal depends on reference-policy choice, KL calibration, and the extent of distribution shift between the trajectories being scored and the trajectories that shaped the RL policy (Oh et al., 24 Jun 2026). SPAE adds a different constraint: its correctness probe assumes structured, canonicalizable answers and incurs additional probing overhead (Wu et al., 7 Jan 2026).

The term itself also remains domain-dependent. In some papers, progress advantage is a learned critic difference; in others it is a step-level shift in eventual correctness, a policy-ratio identity, a fixed-baseline proxy calibrated to expert pace, or a programmatic acceleration concept. This suggests that “progress advantage” is best understood as an umbrella notion for progress-sensitive discrimination rather than as a single settled formal object. Its unifying theme is local preference for moves that increase the probability, pace, or credibility of completion; its mathematical instantiation depends strongly on whether the underlying problem is offline imitation, continual RL, diffusion control, chain-of-thought search, RL post-training, or quantum-technology translation (Kim et al., 27 May 2026, Setlur et al., 2024, Oh et al., 24 Jun 2026, Alsing et al., 2022).