Rank-Then-Act: Reward-Free Control from Frame-Order Progress

Abstract: We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains a Vision-LLM (VLM) offline as a progress-based ordinal scorer, using a Group Relative Policy Optimization (GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function for reinforcement learning: at each interaction window, we compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms prior video-based reward learning methods and rank-based baselines, while demonstrating strong cross-task reuse of a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design.

• The paper introduces a novel RTA framework that learns control policies using ordinal progress signals derived from expert video demonstrations.
• The method employs a two-stage process where a VLM progress scorer is fine-tuned with GRPO and then used to generate a correlation-based reward for RL.
• Empirical evaluations in both discrete and continuous environments show strong transferability, enhanced training efficiency, and robustness over traditional reward models.

Rank-Then-Act: Reward-Free Control from Frame-Order Progress

Problem Formulation and Motivations

This paper introduces Rank-Then-Act (RTA), a reward-free control framework that enables learning policies solely from expert video demonstrations using progress-based ordinal signals instead of handcrafted or extrinsic environment rewards. The core challenge addressed is the elimination of explicit reward engineering in RL, particularly for visually-rich domains like games and robotics, where reward signals are either absent, brittle, or subject to reward hacking. Instead, the approach leverages vision–LLMs (VLMs) to derive a dense “progress” notion from expert-annotated video, moving supervision from scalar prediction to alignment between temporal order and inferred progress.

The authors identify three main obstacles with prior methods: (1) trivial temporal shortcuts in progress estimation due to chronological inputs, (2) ambiguity and poor transfer in absolute progress scales, and (3) the need for informative, robust, low-cost reward signals suitable for online RL. RTA addresses all three by decoupling progress scoring from absolute calibration and using rank-based, scale-invariant correlation as the reward.

Figure 1: RTA’s two-stage protocol—GRPO-based offline training for an ordinal VLM progress scorer on shuffled clips (stage 1), followed by online RL using only progress–time Spearman correlation as reward (stage 2).

Methodology

Stage 1: Listwise VLM Progress Scorer via GRPO

Stage 1 formulates the progress estimation task as ordinal ranking rather than scalar regression. Given expert video trajectories, a VLM is fine-tuned using Group Relative Policy Optimization (GRPO) on sets of shuffled frames (with the first frame as an anchor). The shuffling is crucial to prevent the “later-is-better” shortcut and ensures the model captures genuine temporal structure from visual semantics rather than absolute frame indices.

A listwise reward is defined as the Spearman rank correlation between the VLM’s predicted per-frame progress ranks and the ground-truth frame order. The GRPO objective directly optimizes this correlation reward, treating VLM text generation as a policy. The result is a frozen VLM capable of assigning ordinal progress scores to arbitrary frame sets, invariant to the precise visual appearance or scale of the underlying environment.

Figure 2: Training curves for per-level and cross-game progress scoring; rapid convergence to high progress–time rank correlation, with generalization demonstrated across different levels and domains.

Stage 2: Correlation-Only Reward for RL

Stage 2 operationalizes the reward function for RL. The frozen VLM scorer is evaluated online over sliding windows of agent-generated observations: for each window (with random shuffles of non-anchor frames), Spearman’s $\rho$ between predicted progress and the underlying timestamps is used as the sole, bounded reward signal. This signal, being strictly ordinal and scale-free, is robust to cross-task shifts and does not require any reference to ground-truth rewards, actions, or environment APIs. Policy optimization is performed using standard policy-gradient methods (e.g., PPO variants), leveraging the correlation-only feedback.

The reward frequency and window size introduce a critical bias–variance and computational tradeoff, thoroughly analyzed in ablations and cyclic trajectory experiments.

Figure 3: Accumulated VOC (Spearman correlation) reward aligns closely with agent task success rates, validating the informativeness and stability of the reward signal.

Empirical Evaluation

Discrete Control: GameBoy Environments

Comprehensive evaluation on PyBoy (Catrap) and VideoGameBench (Kirby) benchmarks demonstrates that RTA surpasses competitive video-based reward models (e.g., Rank2Reward, VLM-RM) and direct regression approaches in training sample efficiency and final success rates, even in challenging settings with long-horizon reasoning or non-monotonic progress. Notably, RTA is the only method achieving non-zero success on the harder, long-horizon Kirby environment without specialized environment rewards.

A key empirical finding is the strong transferability of a single, pretrained progress scorer across domains, games, and levels. Scorers trained on pooled YouTube playthroughs or unrelated game videos still enable successful policy learning downstream.

Figure 4: Performance curves on PointMaze-UMaze, showing RTA outperforming Rank2Reward and its mixture with GAIL across seeds and training regimes.

Figure 5: Training efficiency comparison: RTA remains competitive or superior to Rank2Reward under aligned compute budgets, demonstrating that its gains are architectural rather than resource-driven.

Continuous Control: MetaWorld and PointMaze

Extensions to continuous domains (MetaWorld, PointMaze) show that RTA outperforms Rank2Reward in pure rank-reward settings and achieves competitive returns with GAIL-augmented reward combinations, despite using a frozen, task-generic scorer. The correlation-based reward remains sufficiently shaped for policy improvement and is robust to reward sparsity and the absence of ground-truth rewards.

Figure 6: PointMaze UMaze—RTA demonstrates higher training efficiency and absolute performance than ranking-based and classifier-augmented baselines.

Robustness and Ablations

Window size and reward computation frequency are systematically ablated. Intermediate window sizes provide the best trade-off between local sensitivity and global progress, while extremely small windows are prone to local artifacts and large windows dilute the reward. Providing the correlation reward only at rollout termination benefits less-expressive policies (e.g., MLPs), while richer shaping is helpful for VLM-policy agents.

Cyclic trajectory analysis further reveals that windowed rank-correlation avoids spurious reward assignment in the absence of true progress, provided appropriate window sizing.

Figure 7: The impact of start-point refreshing in long-horizon tasks, enabling successful trajectory completion when reward frequency misaligns with episode horizons.

Theoretical and Practical Implications

RTA fills a notable gap between “reward-free” RL and traditional imitation-from-observation, leveraging only passive expert video and without requiring action, reward, or language annotation during either reward model training or policy learning. The key theoretical contribution is the ordinal supervision structure: by aligning visual progress rankings with temporal evolution via correlation, RTA circumvents the calibration and scale-limitations of prior reward regressors. This design ensures reward-model transfer and robustness to distribution shift.

Practically, the approach suggests a scalable protocol for LLM/VLM-driven control, sidestepping reward hacking and enabling rapid generalization to new environments and tasks via universal progress scouts. This is especially relevant for robotics, embodied AI, and any domain where dense, shaped reward signals are unavailable.

The windowed, correlation-only reward signal also shows promise as a diagnostic for guiding intrinsic motivation or exploration bonuses, complementing broader trends in reward modeling for RL from video.

Limitations and Future Directions

Some limitations persist. The framework assumes access to sufficiently rich and diverse expert video data—models underperform with biased or sparse demonstration sets and exhibit asymmetric generalization in highly compositional or visually-distinct domains. Windowed correlation can be sensitive to the temporal structure of the task, requiring careful hyperparameterization. Further, the approach is reliant on the computational cost of VLM backends, though the authors note that distillation or caching can partially address this.

Future developments likely include integrating subgoal-aware or multi-scale aggregation into the reward, more efficient reward model architectures, and compositional adaptation for highly non-monotonic or cyclic tasks. Combining the approach with additional intrinsic or adversarial reward signals (e.g., GAIL) could yield further improvements in exploration-heavy domains.

Conclusion

Rank-Then-Act (RTA) provides a robust, dual-stage protocol for purely reward-free RL from expert video—training a listwise VLM progress scorer and leveraging correlation-based ordinal rewards for online control. The method achieves strong empirical results in both discrete and continuous settings, demonstrating generality, scalability, and stability not matched by direct regression or adversarial imitation approaches. RTA consequently marks an important advance towards generalist, instruction-following control agents that learn exclusively from observational demonstrations, significantly reducing the need for manual reward engineering or calibration.

For full details, see "Rank-Then-Act: Reward-Free Control from Frame-Order Progress" (2607.01897).