Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling

Abstract: Robot policy evaluation often relies on a binary success signal for task completion. In real-world deployment, however, desired execution behaviors are also defined by quality attributes such as execution speed and motion smoothness, which need to be dynamically controlled according to task requirements. To address this, we introduce Quality-Controllable Policy (QCP), a robot policy that enables inference-time control over both individual quality attributes and their combinations. To extend the controllable quality range beyond initial demonstrations, we further propose an iterative hindsight relabeling loop: QCP collects online rollouts under target quality labels, adds successful trajectories to the dataset, relabels the accumulated trajectories in hindsight according to their measured quality, and fine-tunes the policy. Experiments on Robomimic and RoboEval demonstrate that QCP consistently improves both task execution quality and success rates across iterations, producing successful rollouts that exceed the quality of the initial demonstrations. Finally, we validate QCP on real-world manipulation tasks, where controlling speed increases throughput in conveyor operations and controlling transport stability improves success when carrying wine glasses, demonstrating that quality-controllable policies yield practical gains in physical deployment.

• The paper demonstrates the Quality-Controllable Policy (QCP) with iterative hindsight relabeling, effectively extending the range and improving control over robotic task execution.
• QCP context-trains a policy on learned metrics using a flow object and enables an agent to changed its quality of execution at run-time far beyond the target metrics.
• The method consistently achieves better quality and success rates over baselines in most targeted tasks, emphasizing the value of quantized quality conditioning.

Problem formulation and central contribution

The paper addresses a limitation of behavior cloning (BC) for robot manipulation: successful demonstrations are treated as equivalent even when their execution differs substantially in speed, smoothness, efficiency, or stability. A policy trained by ordinary BC therefore models the empirical mixture of behaviors but provides no inference-time mechanism for selecting among them. This is problematic in deployment settings where task completion is necessary but insufficient—for example, conveyor manipulation requires high throughput, whereas transporting wine glasses requires low tray tilt.

The proposed Quality-Controllable Policy (QCP) augments each successful trajectory with automatically computed quality measurements and conditions the policy on discrete quality labels. The quality labels are obtained by partitioning the empirical distribution of trajectory-level scores into equal-frequency bins. During inference, the requested bin acts as a behavioral control variable: the same policy can be asked to produce slower or faster, smoother or less smooth, or otherwise differently executed trajectories.

The second contribution is iterative hindsight relabeling, which is intended to overcome the bounded quality support of static demonstrations. At each iteration, QCP collects rollouts under the highest quality label, retains successful trajectories, measures their realized quality, relabels the entire accumulated dataset according to the updated empirical quality distribution, and retrains the policy. This procedure differs from simple data accumulation because the quality semantics of the labels are recomputed after every collection round. Consequently, the highest label can progressively represent behavior beyond the quality range of the original demonstrations.

Figure 1: QCP conditions policy execution on quantized quality labels and uses iterative hindsight relabeling to move the controllable quality range toward higher-quality successful behavior.

The method relies on a strong but explicit assumption: the relevant quality attributes can be measured by rule-based score functions. This eliminates human preference annotation, but it transfers the problem to metric design. The resulting behavior is controllable only with respect to the selected score functions, and a poorly specified metric can be optimized without improving the operational property that matters.

Quality-conditioned policy and iterative relabeling

For each trajectory $\tau$, QCP computes a scalar score $m(\tau)$ for an attribute such as speed, action jerk, joint path length, or tray tilt. Scores are transformed so that larger values always denote the preferred direction and are then discretized into $B$ equal-frequency bins. In the main experiments, $B=4$. Every action chunk extracted from a trajectory receives the trajectory’s quality label, and the policy is trained with a flow-matching objective conditioned on observation history and the label.

This design intentionally separates task learning from quality selection. Lower-quality successful trajectories remain useful because they provide supervision for task completion, while the label specifies which execution regime should be sampled at inference time. This is the paper’s main explanation for why QCP can avoid the task-quality trade-off encountered by methods that discard data or weight only trajectories aligned with the desired style.

At inference, QCP normally requests the highest bin. The policy is stochastic, so rollouts conditioned on this bin produce a distribution of realized qualities rather than a deterministic score. Successful rollouts are added to the replay dataset, and their labels are assigned retrospectively from their actual scores. The empirical quantile boundaries are then recomputed over all accumulated successful trajectories. Retraining from scratch is used because relabeling changes the meaning of each bin; continuing optimization from the preceding policy could preserve an obsolete label-to-behavior mapping.

This procedure has two interacting effects. First, all successful trajectories preserve coverage of the task while remaining distinguishable by quality. Second, successful samples generated near the current upper-quality boundary become training examples for a new, higher-quality upper bin. The method therefore performs a form of supervised policy improvement in which the improvement signal is generated by the policy’s own successful rollouts and represented through hindsight labels.

Experimental design and baselines

The evaluation covers Robomimic Square and Transport, four RoboEval tasks, and two physical manipulation tasks. Robomimic experiments target execution speed and smoothness; RoboEval experiments target completion time and motion efficiency; physical experiments target end-effector speed in conveyor work and tray stability during wine-glass transport.

The principal comparisons distinguish QCP from several simpler or more optimization-oriented alternatives:

• Accumulated BC retains all successful online rollouts but has no quality condition.
• Filtered BC trains only on the best fraction of successful trajectories for the target attribute.
• SORL and SCIQL use offline RL-style task and style advantages.
• DSRL and EXPO optimize quality-augmented rewards through online RL applied to flow-matching policies.

Simulation uses 300 rollouts per task and seed at each iteration, with six iterations in the main experiments. All reported results average three random seeds. The RL baselines receive an interaction budget matched to QCP’s 1,800 rollouts, although the comparison necessarily depends on implementation and reward-design choices. In the real-robot experiments, QCP is implemented by inserting the quality label into the language prompt of the $\pi_{0.5}$ VLA model rather than modifying its architecture.

Simultaneous improvement of quality and success

The central empirical claim is that QCP improves execution quality and task success jointly, whereas the baselines generally improve one at the expense of the other. Accumulated BC improves task completion but does not directly select a quality regime. Filtered BC and SORL remove or downweight much of the successful data, reducing task coverage. SCIQL retains broader data but introduces a weighting trade-off between style and task advantages.

The final numerical comparison against the RL baselines is particularly informative. Across eight task-attribute combinations, QCP attains the strongest or near-strongest combination of success and quality. For example, on Square speed, QCP reaches an 82.2% success rate with 120.3 steps to success, compared with 53.2% and 156.0 steps for SORL, 37.0% and 208.8 steps for DSRL, and 96.0% and 162.5 steps for EXPO. On Transport speed, QCP achieves 22.2% success at 432.8 steps, while SORL reaches only 4.6% at 466.8 steps, SCIQL 10.6% at 540.8 steps, DSRL 4.0% at 549.5 steps, and EXPO 11.0% at 567.7 steps.

The results are not uniformly dominant on every individual metric. EXPO achieves higher Square speed success than QCP, and SCIQL slightly exceeds QCP’s Square smoothness success rate, although its smoothness is substantially worse. The stronger claim supported by the data is therefore not universal Pareto dominance, but a more favorable quality-success balance across heterogeneous tasks and attributes.

Figure 2: QCP improves quality and success across iterations more consistently than accumulated BC, filtered BC, SORL, and SCIQL.

The paper attributes QCP’s advantage over DSRL and EXPO to the representation of quality supervision. Those methods receive quality through a scalar reward added to the task reward, whereas QCP uses the quality measurement as a supervised conditional variable attached to successful trajectories. This interpretation is plausible within the reported experiments, but it does not establish that reward-based methods are intrinsically unable to exploit quality signals; performance depends on reward normalization, coefficient selection, critic estimation, and policy parameterization.

Inference-time control and range expansion

QCP retains controllability across iterative updates rather than merely improving one fixed operating point. On Square, the initial policy already separates the requested bins: the mean steps to success are 193.7 for the highest-speed bin and 308.1 for the lowest-speed bin, with intermediate bins ordered between them. The ordering remains intact across iterations even though new data are collected only under the highest bin.

This is an important property of the relabeling scheme. It raises the quality represented by the upper bin without collapsing the lower bins. The result is not simply a policy that becomes faster or smoother; it is a policy whose controllable range moves toward improved behavior while preserving multiple selectable execution regimes.

Figure 3: Requested quality bins remain ordered across iterations while the entire controllable range shifts toward higher quality.

The distributional analysis supports the claim that QCP reaches beyond the initial demonstration support. For Square smoothness and RoboEval joint path length, the normalized Wasserstein-1 distance between the initial demonstration distribution and final successful-rollout distribution exceeds 1.5 when normalized by the initial interquartile range. Speed distributions also extend toward shorter completion times than those present in the demonstrations. The authors note that the fastest rollouts may approach the physical speed limit of the particular setup, but no independent estimate of that limit is provided.

Figure 4: Successful rollout distributions shift beyond the initial demonstration range as iterative hindsight relabeling proceeds.

The fixed-policy control experiment isolates the role of iterative retraining. With the same total budget of 1,800 rollouts, a policy frozen after iteration 0 does not produce the quality reached by the iteratively retrained policy. On Square smoothness, the frozen policy has a 54.5% success rate and median jerk of 0.0335, whereas iteration 5 reaches 73.9% success and median jerk of 0.0098. The smoothest frozen rollout has jerk 0.0171, which is worse than the 75th percentile of the final iterative distribution. Thus the reported improvement is not explained by rare-event sampling from a fixed conditional policy; it requires policy updates on relabeled data.

Multi-attribute control

QCP supports multiple independently quantized attributes by assigning each attribute its own label embedding. On Square, the authors jointly control execution speed and lift amplitude, requesting all four combinations of faster or slower execution and smaller or larger lift. The joint policies approach the quality achieved by policies trained with only one attribute, although slower commands exhibit lower success.

The six-attribute RoboEval experiment provides a stronger test of simultaneous conditioning. QCP improves most of the selected attributes over both initial BC and accumulated BC across Lift Pot, Stack, Handover, and Valve. The numerical improvements are substantial. On Lift Pot, QCP raises success from 92% for initial BC and 99% for accumulated BC to 100%, while reducing joint path length from 4.37 to 2.82, completion time from 2.04 to 1.72, and mean joint jerk from 228.8 to 130.5. On Handover, QCP reaches 100% success and reduces mean joint jerk from 169.6 for initial BC to 36.2, while also reducing bimanual arm velocity difference from 0.293 to 0.127.

Figure 5: QCP jointly improves six selected quality attributes across four RoboEval tasks while maintaining or increasing success rates.

These results should be interpreted with attention to attribute selection. The authors remove metrics that are redundant, nearly invariant, geometrically constrained, or in conflict with task requirements. Thus the experiment demonstrates multi-attribute control for carefully selected compatible metrics, not arbitrary simultaneous optimization of all available attributes. The reported result is nevertheless significant because conditioning on six labels does not cause the severe degradation that would be expected if the policy could not disentangle the attributes.

Physical deployment

The physical experiments test whether quality control produces operational benefits rather than only improved simulator metrics. A Franka Research 3 arm with the DROID hardware and camera setup is used for conveyor manipulation and wine-glass transport. QCP is applied to a pretrained VLA by inserting a textual quality instruction into the prompt.

The wine-serving task is especially difficult: only 50 of 382 teleoperated attempts succeeded during data collection, a 13.1% success rate. QCP conditioned on the highest stability bin reduces mean tray tilt from 0.0660 rad for BC to 0.0482 rad, a 27.0% reduction, while success increases from 30.7% to 42.0%. The final QCP policy therefore improves the safety-relevant quality measure and task completion simultaneously. The filtered BC baseline initially improves but subsequently degrades, supporting the paper’s argument that retaining lower-quality successful examples preserves task coverage.

For conveyor work, QCP increases mean end-effector speed from 188.6 to 257.6 mm/s, a 36.6% increase, and raises throughput from 8.2 to 13.0 cans per minute, a 59.3% improvement. This is operationally more meaningful than speed alone because throughput incorporates grasp success under the fixed arrival schedule. QCP also exceeds the reported human teleoperation speed of 173.5 mm/s, although this comparison is not a controlled comparison of human and robot policies under identical trial conditions.

Figure 6: On real robots, QCP reduces tray tilt during wine serving and increases end-effector speed and throughput in conveyor manipulation.

The physical results establish that a quality metric can be connected to a deployment-level objective, but they remain limited in scale. Each task uses a single hardware configuration, three seeds, and a relatively narrow set of operating conditions. The results therefore validate feasibility and practical relevance rather than robustness across hardware, operators, objects, or environmental disturbances.

Representation and labeling analyses

The paper compares quantized labels with continuous score conditioning. Four bins provide the strongest overall balance in the reported Square speed experiment. At iteration 6, the four-, eight-, sixteen-, and thirty-two-bin variants reach success rates of 70.3%, 63.2%, 57.7%, and 28.1%, with 139, 120, 116, and 112 steps, respectively. More bins produce modestly faster execution but sharply reduce success, particularly at 32 bins. Continuous targets reach 55.0% and 67.0% success with 246 and 209 steps, depending on the requested target.

The result supports quantized labels as a practical default because each label is supported by many trajectories. It does not demonstrate that continuous conditioning is fundamentally inferior: the continuous variants use fixed normalization and target-selection procedures that differ from the adaptively recomputed quantile labels. The experiment establishes an empirical advantage for the tested discretization protocol, not a general theorem about discrete versus continuous quality representations.

The segment-label analysis exposes a more substantive limitation. Assigning quality labels to shorter temporal windows does not improve performance. On Square at iteration 9, complete-trajectory labels achieve 84.3% success and 6.08 mm per step, compared with 48.0% and 4.80 mm per step for segment labels and 71.0% and 5.07 mm per step for hybrid labels. On Transport, complete-trajectory labels achieve 13.3% success, versus 3.7% for segment labels and 4.0% for hybrid labels.

The explanation is that local quality measurements are entangled with task phase. A segment can appear fast or smooth because it corresponds to a particular stage of the manipulation rather than because it represents globally desirable execution. This finding limits the current formulation to trajectory-level quality and leaves open how to construct temporally local metrics that separate execution quality from task progress.

Limitations and open questions

The method depends on successful-trajectory filtering and rule-based quality metrics. The authors report that including failed rollouts degraded performance, so QCP does not presently exploit negative examples to learn failure boundaries or quality-success trade-offs. This makes the method particularly dependent on obtaining enough successful online trajectories, which may be difficult for tasks with low initial success or expensive physical interaction.

The adaptive quantile representation also makes quality labels relative to the current dataset. A label such as “highest quality” has no fixed physical meaning across iterations, tasks, or deployments. This is useful for expanding the empirical range but complicates cross-iteration calibration and makes it difficult to specify an absolute target, such as a maximum allowable tilt angle or a required throughput.

The iterative process can also amplify measurement and selection biases. Only successful rollouts are retained, and new data are collected under the current highest label. If a quality metric is correlated with an unmodeled failure mode, repeated relabeling could concentrate data in a narrow behavioral region. The experiments do not characterize performance under distribution shifts, metric misspecification, adversarial object configurations, or quality attributes that conflict strongly with task completion.

Finally, the study uses relatively small numbers of seeds and task-specific experimental choices, including selected RoboEval attributes and manually designed real-robot score functions. The physical results do not establish broad robustness, and the paper does not provide a principled criterion for choosing the number of bins, determining whether attributes are compatible, or deciding when iterative improvement has reached a physical or safety constraint. These are specific unresolved questions for the proposed framework.

Conclusion

QCP presents a coherent supervised approach to quality-controllable robot policy learning. Quantized trajectory-level quality labels preserve successful task supervision while enabling inference-time behavioral selection, and iterative hindsight relabeling extends the controllable range through successful online rollouts whose realized qualities are recomputed into the accumulated dataset. Across simulation and physical manipulation, the method improves quality and success more consistently than the evaluated BC, offline RL, and quality-reward RL baselines.

The strongest evidence concerns the separation of iterative improvement from mere sampling, the preservation of ordered quality control across iterations, and the practical gains in wine-serving stability and conveyor throughput. The principal constraints are equally clear: QCP requires reliable rule-based metrics, sufficient successful rollouts, and quality attributes that can be represented at the trajectory level without undermining task completion. Within those conditions, the paper establishes iterative hindsight relabeling as an effective mechanism for expanding quality-controlled robot behavior beyond the support of the initial demonstrations (2610.37393).