Goal Relational Graph (GRG) Framework

• GRG is a weighted, learnable graph that models soft, probabilistic affinities among goal states in both hierarchical RL and self-supervised learning.
• Its Bayesian and EM-style approaches aggregate multiple relation cues—like embedding similarity and adjacency—to improve sub-goal selection and representation invariance.
• Empirical evaluations in grid-world navigation and node classification demonstrate that GRG significantly boosts success rates and representation accuracy over traditional methods.

A Goal Relational Graph (GRG) is a weighted, learnable graph structure that encodes the latent relations among goals or nodes—either in the context of hierarchical reinforcement learning (HRL) or self-supervised graph representation learning. The GRG framework generalizes classical adjacency-based relations by representing soft, probabilistic or compositional affinities between goal states or embedding vectors, and is leveraged for both high-level planning (e.g., sub-goal selection in HRL) and representation invariance (e.g., graph-level SSL objectives). The GRG formalism has been independently introduced in hierarchical RL for goal-driven navigation (Ye et al., 2021) and for compositional self-supervised node representation learning (Naseri et al., 2024), with domain-specific variants tailored to the respective learning systems.

1. Formal Definitions and Properties

Hierarchical RL Setting

Let $G_d = \{g_1, g_2, \dots, g_N\}$ be the finite set of all possible goal descriptions. The GRG is a complete directed weighted graph $\mathcal{G} = (V, E, W)$ with node set $V = G_d$, directed edge set $E = \{(g_i, g_j): i \neq j\}$, and edge weights $W = [w_{ij}] \in \mathbb{R}^{N \times N}$, where $w_{ij} \in [0, 1]$ for $i \neq j$ and $w_{ii} = 1$. The key semantics of $w_{ij}$ is that it encodes “how quickly and how likely” goal $g_j$ becomes visible when pursuing $\mathcal{G} = (V, E, W)$0 under the current policy, formalized via a discounted observation process.

This weighted adjacency can equivalently be represented by a third-order tensor $\mathcal{G} = (V, E, W)$1, where $\mathcal{G} = (V, E, W)$2 gives the posterior probability that $\mathcal{G} = (V, E, W)$3 appears at step $\mathcal{G} = (V, E, W)$4 when targeting $\mathcal{G} = (V, E, W)$5.

Self-Supervised Representation Learning Setting

In ExGRG (Naseri et al., 2024), the GRG is a soft, compositional, and learnable symmetric matrix $\mathcal{G} = (V, E, W)$6 over the set of node embeddings $\mathcal{G} = (V, E, W)$7. Each $\mathcal{G} = (V, E, W)$8 encodes the degree to which representations $\mathcal{G} = (V, E, W)$9 and $V = G_d$0 should be aligned under the invariance objective. Unlike conventional SSL graphs that rely on augmentation-based identity (i.e., $V = G_d$1 only for positive pairs), here $V = G_d$2 is explicitly constructed from multi-source graph cues (representation-space similarity, original adjacency, structural/positional encodings, and online clustering).

2. Relation Learning Methodologies

Dirichlet–Categorical Process in HRL

For each edge $V = G_d$3, relation discovery is cast as learning the distribution of first appearance times for $V = G_d$4 when following the optimal policy toward $V = G_d$5. Event sampling is modeled as $V = G_d$6, where $V = G_d$7, and the (discounted) value for step $V = G_d$8 is $V = G_d$9, with $E = \{(g_i, g_j): i \neq j\}$0 indicating non-observation. The vector $E = \{(g_i, g_j): i \neq j\}$1 is given a Dirichlet prior and updated with observed counts, yielding a conjugate Dirichlet posterior. The edge weight $E = \{(g_i, g_j): i \neq j\}$2 is set to the posterior expectation:

$E = \{(g_i, g_j): i \neq j\}$3

EM-Style Compositional Construction in SSL

ExGRG applies an expectation-maximization-inspired approach. At each iteration:

• E-step: Multiple candidate relation graphs $E = \{(g_i, g_j): i \neq j\}$4 are built based on: $E = \{(g_i, g_j): i \neq j\}$5-NN similarity in embedding space, adjacency-filtered edges, positional/structural encodings (e.g., Laplacian eigenmaps, RWSE, SignNet), and online deep clustering. A tiny learned hypernetwork, based on summary statistics of each $E = \{(g_i, g_j): i \neq j\}$6, generates mixing coefficients $E = \{(g_i, g_j): i \neq j\}$7 for aggregation:

$E = \{(g_i, g_j): i \neq j\}$8

• M-step: The encoder, projector, and hypernetwork parameters $E = \{(g_i, g_j): i \neq j\}$9 are updated to minimize the total loss

$W = [w_{ij}] \in \mathbb{R}^{N \times N}$0

with explicit invariance loss $W = [w_{ij}] \in \mathbb{R}^{N \times N}$1 and a graph regularizer $W = [w_{ij}] \in \mathbb{R}^{N \times N}$2 to prevent degenerate solutions.

3. Integration within Hierarchical and Graph Learning Systems

Hierarchical RL: Sub-Goal Selection and Action Control

The high-level controller observes the current state $W = [w_{ij}] \in \mathbb{R}^{N \times N}$3, computes the set of visible candidate sub-goals $W = [w_{ij}] \in \mathbb{R}^{N \times N}$4, and for each, plans the maximal-product path to the final goal using the GRG:

$W = [w_{ij}] \in \mathbb{R}^{N \times N}$5

with sub-goal input embedding modulated by the plan cost $W = [w_{ij}] \in \mathbb{R}^{N \times N}$6. Sub-goal selection is via $W = [w_{ij}] \in \mathbb{R}^{N \times N}$7.

The low-level policy receives the current sub-goal $W = [w_{ij}] \in \mathbb{R}^{N \times N}$8 and operates in a fully observable goal-conditioned MDP. Early termination is triggered if any future node along $W = [w_{ij}] \in \mathbb{R}^{N \times N}$9 appears, transferring control back to the high-level. GRG statistics and edge-weights are updated online at each sub-goal execution.

SSL: Relation Graph in Representation Losses

In ExGRG, the explicit GRG replaces the binary augmentation-identity graph in the standard invariance term. The constructed $w_{ij} \in [0, 1]$0 is held fixed by stop-gradient during parameter updates, ensuring stable E-M style training. Additional losses enforce variance and covariance constraints (VICReg), cluster alignment (via optimal transport), and regularization of the relation graph. The system is highly modular, allowing sources and hypernetwork configuration to vary with the application.

4. Empirical Evaluation and Ablation Findings

Hierarchical RL

Evaluations on partially observable tasks, including grid-worlds (16×16 grid, 16 goals) and robotic object search in simulation (AI2-THOR, House3D), demonstrate substantial gains:

Domain	Method	Success Rate (SR)	SPL
Grid-world (unseen goals)	GRG-based	0.70	0.45
	DQN	≈0.20	≈0.15
	h-DQN	≈0.19	≈0.08
House3D	GRG-based	0.62	0.10
	A3C	0.47	0.03
	hRL	0.28	0.02

Ablation studies found that removing either relation-based sub-goal weighting or early low-level termination dropped unseen goal SR from 0.70 to 0.35/0.58 and SPL from 0.45 to 0.14/0.32, respectively. This demonstrates that both relation learning and dynamic control are essential for generalization to new environments and goals (Ye et al., 2021).

Self-Supervised Graph Learning

ExGRG was benchmarked on nine node classification datasets (e.g., WikiCS, Amazon Computers, Coauthor CS), showing outperformance of prior methods:

Dataset	ExGRG Accuracy	Best Baseline
WikiCS	82.09 ± 0.67	79.98 ± 0.10 (BGRL)
AmzComp	93.37 ± 0.48	90.34 ± 0.19 (BGRL)

Ablations indicate each relation source and the $w_{ij} \in [0, 1]$1 regularizer are critical; removing relation sources or regularization causes collapse or subpar accuracy. This confirms the importance of soft, compositional relations for robust invariance in graph SSL (Naseri et al., 2024).

5. Algorithmic Details and Practical Implications

In hierarchical RL (Ye et al., 2021), the full training loop initializes the Dirichlet pseudo-counts for the GRG, high- and low-level policy parameters, and iteratively observes, plans, updates policies, and refines GRG statistics online at every episode and time-step. Policy improvement is driven by Q-learning (high-level) and either DQN or A3C (low-level) losses.

In ExGRG-style SSL (Naseri et al., 2024), the E-step aggregates multiple candidate graphs, each constructed with distinct semantics (embedding similarity, adjacency, spectral or walk-based encodings, clustering structure), with sparse, learnable convex weights. The M-step uses the frozen composite relation to shape the invariance objective, supported by variance, covariance, and clustering regularizers. Hypernetwork and regularization parameters are tuned following empirical guidelines (e.g., $w_{ij} \in [0, 1]$2, mini-batch size $w_{ij} \in [0, 1]$3, $w_{ij} \in [0, 1]$4 for $w_{ij} \in [0, 1]$5NN = 10–20).

All operations scale linearly in the number of nodes and sources, and ExGRG is robust to small mini-batches.

6. Theoretical and Practical Significance

The GRG provides a unifying paradigm for representing, learning, and exploiting soft task or embedding dependencies beyond hard adjacency. In hierarchical RL, GRG-driven policies generalize across previously unseen configurations by explicitly modeling the structure of goal relationships, integrating relational evidence via a Bayesian process. In graph SSL, GRG enables more semantically precise invariance objectives by combining geometric, relational, and clustering evidence—bridging the gap between traditional Laplacian-based objectives and modern deep invariance learning.

A plausible implication is that GRG-style compositional relation modeling may become standard for tasks requiring generalization across dynamic, unseen environments, or when learning representations in highly heterogeneous graphs, as it supports both inductive bias from domain knowledge and online adaptation.

7. Limitations and Future Directions

The GRG framework, while empirically effective, raises questions about optimality of relation graph sources and aggregation strategies in the SSL context. Stability may depend on hyperparameter selection (e.g., regularization scales, number and type of PSEs, clustering prototypes). For hierarchical RL, the practicality of Dirichlet–categorical estimation may be impacted by the number and diversity of goals, as well as partial observability constraints. Extensions to continuous goal spaces or multi-agent settings remain to be fully explored. Both frameworks suggest further theoretical investigation into the spectral and convergence properties of learned relation graphs in complex temporal and structural domains.