- The paper introduces MileGPO, a three-stage method that discovers milestones, calibrates their reliability, and contrasts sibling actions using grouped on-policy rollouts without extra models or environment interactions.
- MileGPO improves success to 94.60% on ALFWorld and 78.58% on WebShop, outperforming reproduced GraphGPO by 3.13 and 3.78 percentage points while reducing the ALFWorld ID–OOD gap to 1.69 points.
- The results show that local rollout evidence can resolve distance-based credit ties, although MileGPO depends on recurring states, environment-specific thresholds, and comparisons that do not establish causal effects.
MileGPO addresses a specific weakness in graph-based credit assignment for long-horizon LLM agent training: the final-goal-distance returns used by GraphGPO capture reachability but cannot distinguish locally competing actions or verify that success-visited states represent genuine progress. The paper proposes MileGPO, which derives process-level credit entirely from grouped on-policy rollouts through three nested mechanisms—Milestone Discovery (MD), Reliability-Calibrated Shaping (RCS), and Progress-Contrastive Calibration (PCC)—without auxiliary models, critics, reward models, or additional environment interaction (2608.19803).
Motivation and problem structure
The starting observation is empirical. In ALFWorld and WebShop, roughly 74% of transitions originate from states shared across rollouts within a task group, providing recurring states and sibling branches from which intermediate credit can be inferred. However, GraphGPO's distance-based return assigns identical credit to 54.4% of same-state action pairs in ALFWorld and 72.7% in WebShop. Identical transitions also appear in both successful and failed trajectories, so success-visited states are not automatically reliable progress anchors. This motivates three questions: how to discover credit anchors, how to weight them by reliability, and how to validate them against local evidence.
Method
MileGPO builds on the grouped rollout graph of GraphGPO: for each task, K trajectories are merged into a directed graph over canonicalized observations, and each transition receives a dense return cγGd(v,g) based on shortest-path distance to the successful final state. MileGPO preserves this term and adds an intermediate correction.
Milestone Discovery identifies milestone candidates as nonfinal states visited by successful trajectories, scored by the excess conditional success rate ℓ+(v)=max(p+(v)−pq+,0), coverage m(v) among successful trajectories, and normalized connectivity C(v). Trap candidates—states visited only by failures, recurring across failures or revisited—are scored analogously via excess failure rate and revisit frequency.
Reliability-Calibrated Shaping replaces MD's unit weights with max-normalized candidate scores in two potential maps over graph distance, then converts discounted potential increases along each transition into one-sided positive and negative shaping terms δt+ and δt−. The paper is explicit that this is a one-sided credit rule rather than policy-invariant potential shaping.
Progress-Contrastive Calibration refines positive candidates with two transition-level signals: Branch-Counterfactual Credit compares each edge's success rate against sibling edges from the same source state, and a local-progress score combines goal-distance reduction, success-rate gain, and excess edge-level failure rate. Candidates with local support are amplified; those without are shrunk by a factor ρ. The final step return adds cλ(δt+−δt−) to the GraphGPO return, and the policy update combines separately normalized graph and shaped advantages plus their residual, weighted by η, together with an episode-level group-normalized term. The appendix establishes three properties on a fixed rollout graph: the update is a convex interpolation between GraphGPO and fully shaped components controlled by cγGd(v,g)0; any strict ordering produced by the shaped advantage resolves graph-distance ties; and the raw shaping correction is bounded—for reported hyperparameters, within cγGd(v,g)1 relative to the graph return.
Empirical results
All local comparisons use Qwen2.5-1.5B-Instruct under shared model, rollout, optimizer, and environment configurations, with baselines trained for 300 steps because validation curves had not converged under shorter schedules—a concession worth noting when comparing to originally reported numbers.
| Benchmark |
Metric |
GiGPO |
GraphGPO |
MileGPO |
| ALFWorld (OOD) |
Success (%) |
90.17 ± 0.1 |
91.47 ± 0.5 |
94.60 ± 0.3 |
| ALFWorld |
ID–OOD gap (pts) |
1.89 |
3.78 |
1.69 |
| WebShop |
Success (%) |
76.17 ± 1.6 |
74.80 ± 2.4 |
78.58 ± 1.2 |
| WebShop |
Task score (%) |
89.81 ± 0.9 |
88.24 ± 1.4 |
90.29 ± 0.8 |
On ALFWorld, MileGPO raises overall success by 3.13 points over reproduced GraphGPO and 4.43 over GiGPO; on WebShop it improves success by 3.78 points over GraphGPO. The reduced ID–OOD gap suggests the learned preferences transfer rather than fit the evaluation distribution. MileGPO also exhibits lower variance than reproduced GraphGPO on both benchmarks.
A mechanism analysis ties gains to the environment property the method targets: ALFWorld and WebShop have nearly identical shared-state transition coverage (73.7% vs. 73.9%), but WebShop has far more final-distance ties (72.7% vs. 54.4% of same-state pairs). MileGPO's clearest gain occurs precisely on WebShop, supporting the claim that the improvement comes from resolving local credit ambiguity rather than graph reuse alone.
Cumulative ablations show each component matters: removing PCC costs 3.71 points of ALFWorld OOD success and widens the ID–OOD gap by 3.32 points; further removing RCS causes the largest additional drop in WebShop success (3.32 points). Notably, milestone discovery alone can introduce noisy credit—RCS's reliability weighting recovers outcome-consistent preferences. An offline replay on fixed GraphGPO traces shows RCS corrects 26.5% of eligible opposite-outcome distance-tied sibling pairs versus 17.0% for uniform MD; the authors correctly flag this replay as descriptive mechanism evidence, not a causal intervention. A qualitative paired trace illustrates the mechanism concretely: two rollouts sharing the same initial observation diverge only in whether the search query preserves the price constraint, and RCS favors the successful branch by +1.533 where GraphGPO assigns equal advantage.
Computationally, MileGPO reuses the rollout graph and adds no environment interactions or model passes; candidate statistics cost cγGd(v,g)2 and sibling comparisons cγGd(v,g)3, negligible relative to rollout and policy-update costs.
Limitations and open questions
Several caveats bear directly on the results. First, the method depends on sufficient state recurrence within rollout groups; environments without recurring states would offer little branch evidence, and the paper does not test such regimes. Second, BCC margins are explicitly not causal-effect estimates—they compare observed branches whose outcomes may be confounded by downstream behavior—and the retention rule relies on hand-set thresholds (cγGd(v,g)4, cγGd(v,g)5, cγGd(v,g)6) that differ between environments, so the recipe requires per-environment tuning even though the core mechanism is unchanged. Third, the shaping rule is one-sided rather than policy-invariant, leaving open whether the correction could bias learning in settings where potential-based guarantees matter. Fourth, all experiments use a single 1.5B-scale policy on two benchmarks; scaling behavior and applicability beyond embodied navigation and web shopping remain untested. Finally, the tie-correction rate of 26.5%, while improved over uniform weighting, indicates most ambiguous pairs remain unresolved—an explicit question about what additional evidence could close the remaining gap.
Conclusion
MileGPO demonstrates that grouped on-policy rollouts already contain sufficient local evidence—outcome-conditioned visitation statistics, sibling-branch contrasts, and goal-distance changes—to supply calibrated process-level credit without external supervision. Its contributions are a diagnosis of distance-based credit ambiguity, a rollout-native algorithm addressing it in three nested stages, and state-of-the-art results on ALFWorld and WebShop with a correspondingly small ID–OOD gap. The main open questions concern generalization beyond recurrent-state environments, the non-causal nature of branch comparisons, and sensitivity to the environment-specific retention thresholds.