Papers
Topics
Authors
Recent
Search
2000 character limit reached

Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL

Published 10 May 2026 in cs.LG | (2605.09335v1)

Abstract: Sparse goal-conditioned reinforcement learning can produce policies whose failures are hidden by aggregate success rates. We analyze trained goal-conditioned value policies through the deterministic functional graphs induced by greedy evaluation: for each goal, every state maps to a single successor, decomposing behavior into attractors and basins. This reveals a local-to-global structure in learned policies. We define local goal support (LGS), a one-step statistic measuring the fraction of valid neighboring states whose greedy successor is the goal. In deterministic sparse GridWorlds, zero LGS exactly precludes goal entry from non-goal starts. Empirically, weak LGS is a strong diagnostic of goal-level failure across update rules, curricula, larger grids, and bottleneck geometries: the fixed rule LGS <= 0.5 identifies low-success goals with precision 0.921, recall 0.929, and F1 0.925 in the main 8x8 TD setting, with similar performance across variants. However, local support is not sufficient for global success: some supported goals still fail because distant states are captured by competing attractors or fragmented basin structure. We therefore introduce a compact post-hoc taxonomy of policy-induced graphs -- goal-dominant, competitor-dominated, partial/contested, and fragmented -- to characterize residual failure modes beyond local support. These results show that sparse GCRL failures can be understood as structured policy-induced dynamics, and that local one-step policy structure provides a cheap post-training diagnostic for goal-level failure.

Authors (1)

Summary

  • The paper models each deterministic greedy goal-conditioned policy as a functional graph, using attractors and basins to explain per-goal success and failure beyond aggregate metrics.
  • A fixed local goal support threshold of 0.5 predicts low-success goals with F1 above 0.88 across five settings, while zero support provides an exact certificate that the goal cannot be reached.
  • The paper’s taxonomy identifies competitor-dominated and fragmented basins as major residual failure modes, offering actionable post-hoc targets for improving local entry, suppressing attractors, and reshaping curricula.

Motivation: aggregate metrics conceal structural failure

Sparse goal-conditioned reinforcement learning (GCRL) is conventionally evaluated through aggregate quantities such as mean success rate, yet these statistics obscure why a policy succeeds for some goals and fails for others. This paper, "Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL" (2605.09335), addresses that gap by analyzing the deterministic greedy policy of a trained goal-conditioned value function as a finite discrete dynamical system. The motivating observation is concrete: in an open 8×88\times 8 GridWorld trained with TD learning under uniform goal sampling, mean evaluation success across 20 seeds is $0.44$, but seed-level success ranges from $0.26$ to $0.67$ despite identical environment, architecture, horizon, update rule, and sampling distribution. An edge-biased goal curriculum, which triples the sampling rate of boundary goals, fails to reduce this variability and leaves mean success essentially unchanged ($0.46$). The paper's central question is therefore structural: for each goal, what closed-loop dynamics has the greedy policy induced, and how do those dynamics determine goal-level success or failure?

The functional-graph framework

For a fixed goal gg, the learned value function Vθ(s,g)V_\theta(s,g) and the deterministic environment jointly define a successor map fg(s)=T(s,πg(s))f_g(s) = T(s, \pi_g(s)) with exactly one outgoing edge per state. This map is a functional graph (mapping digraph), and the paper invokes the standard finite-map decomposition: every trajectory eventually enters a directed cycle, each weakly connected component contains exactly one cycle, and all remaining states form in-trees feeding into it. Cycles are attractors; predecessor sets are basins of attraction. Treating the goal as absorbing (fg(g)=gf_g(g)=g), success corresponds to membership in the goal basin, while failure manifests as spurious fixed points, non-goal cycles, dominant competing basins, or fragmentation across multiple attractors.

The framework distinguishes eventual goal basins Bg({g})\mathcal{B}_g(\{g\}) from the finite-horizon success set $0.44$0, noting that evaluation uses the training horizon $0.44$1, so states reaching the goal only after more than $0.44$2 steps are counted as failures even though they lie in the eventual goal basin. From the graph the paper extracts a compact set of metrics: finite-horizon success, local goal support, goal-basin fraction, largest competing-basin fraction, and a fragmentation index $0.44$3 over basin mass.

Local goal support: a provable necessary condition

The core local statistic is local goal support (LGS): the fraction of valid one-step neighbors of $0.44$4 whose greedy successor is $0.44$5. In a four-neighbor deterministic GridWorld, any trajectory reaching $0.44$6 from a non-goal start must pass through a valid neighbor, so zero LGS exactly precludes goal entry: $0.44$7 implies $0.44$8 for every horizon. This yields a hard, parameter-free failure certificate.

The paper is careful about the converse, proving an asymmetry (Proposition A.2): positive LGS guarantees only that at least one start reaches the goal in one step, and even full local support does not guarantee high success, since distant states may be captured by competing attractors. This is confirmed empirically: the appendix enumerates interior goals with $0.44$9 but $0.26$0 in every condition—for instance, 119 such cases in the $0.26$1 TD setting with mean success $0.26$2 and fragmentation $0.26$3.

Empirical strength of the LGS diagnostic

Despite its simplicity, LGS is a strong failure predictor. Using the fixed rule $0.26$4 to predict low-success goals ($0.26$5), with no fitted parameters, the main $0.26$6 TD setting achieves precision $0.26$7, recall $0.26$8, and F1 $0.26$9; F1 exceeds $0.67$0 in all five settings (TD, MC, edge-biased, $0.67$1, bottleneck). The zero-support rule ($0.67$2) attains perfect precision everywhere, matching the theory, but low recall because many failures retain weak nonzero support.

Per-seed robustness analysis shows this is not a pooled artifact: mean within-seed AUC exceeds $0.67$3 in every condition (minimum $0.67$4), with mean Spearman correlation between LGS and success above $0.67$5. LGS also stratifies global structure monotonically: low-support goals exhibit success between $0.67$6 and $0.67$7 with competing-basin fractions of $0.67$8–$0.67$9, whereas full-support goals reach success of $0.46$0–$0.46$1 with competitors reduced to $0.46$2–$0.46$3.

A post-hoc taxonomy of residual failure modes

Because local support is not sufficient for success, the paper introduces a rule-based taxonomy assigning each seed–goal functional graph to one of four regimes using goal-basin fraction, largest competing basin, and fragmentation. Across 7,780 seed–goal graphs, the regimes separate sharply:

Regime Share Mean success Mean LGS Comp. basin Frag.
Goal-dominant 17.4% 0.944 0.978 0.037 0.104
Partial/contested 4.0% 0.554 0.767 0.392 0.097
Competitor-dominated 54.1% 0.073 0.334 0.733 0.195
Fragmented 24.5% 0.244 0.537 0.326 0.565

The regime distribution varies systematically by condition: the $0.46$4 setting is 64.9% competitor-dominated, suggesting that larger state spaces make global basin organization harder; the bottleneck environment is 49.3% fragmented, consistent with constrained geometry splitting the state space into separated routing regions; and MC learning yields far fewer goal-dominant graphs (5.7%) than TD (34.1%), with much higher fragmentation. Notably, edge-biased sampling shifts where structure forms—edge-goal success rises from $0.46$5 to $0.46$6 while interior success falls from $0.46$7 to $0.46$8—without eliminating competitor-dominated failures. The taxonomy is explicitly descriptive rather than a fitted classifier.

Limitations and open questions

The paper's claims are bounded by its experimental scope. All results assume deterministic greedy evaluation, finite discrete state spaces, four-neighbor movement, and exhaustive enumeration of the successor map; the environments are small GridWorlds ($0.46$9 to gg0) with a small MLP UVFA trained for only 500 episodes per seed. The authors state plainly that extension to stochastic policies, continuous states, richer action spaces, and high-dimensional observations will require approximate graph construction or local dynamical diagnostics, and that the framework has not been tested beyond the bottleneck variant. Two specific questions remain open: whether the same structural failure modes (competitor dominance, fragmentation) persist in multi-room and MiniGrid-style geometries scaled in a controlled sequence, and whether the diagnostic taxonomy can be converted into targeted interventions—improving local entry, weakening competing attractors, or reshaping curricula to reduce fragmentation—that causally improve success rather than merely explaining it.

Conclusion

This paper contributes a structural evaluation framework, not a training algorithm. By decomposing the greedy goal-conditioned policy into goal-indexed functional graphs, it establishes that zero local goal support is an exact obstruction to goal entry, that a fixed threshold rule on LGS predicts goal-level failure with F1 above gg1 across five settings without any fitted parameters, and that residual failures among locally supported goals are explained by competitor-dominated and fragmented basin structure. The result reframes sparse GCRL failures as structured, exactly computable policy-induced dynamics, and provides a cheap post-training diagnostic that complements aggregate success rates with a mechanistic account of per-goal failure.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.