Hyper-GoalNet: Adaptive Robotic Policy

• Hyper-GoalNet is a goal-conditioned manipulation framework that uses hypernetworks to generate task-specific policy parameters from visual goal specifications.
• It employs behavior cloning on offline demonstrations, augmenting its latent space with forward dynamics and monotonic progress constraints.
• Evaluations show substantial gains in diverse robotic tasks, outperforming fixed-parameter baselines in both simulation and real-robot experiments.

Hyper-GoalNet is a goal-conditioned manipulation policy learning framework in which a hypernetwork generates task-specific policy parameters from goal information, rather than appending the goal to a fixed-parameter policy as an additional input. In the formulation introduced in “Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworks,” the goal determines how current observations are processed, while the generated target policy maps recent observations to continuous robot actions. The method is trained by behavior cloning on offline demonstrations, uses a single future image from the same trajectory as the goal specification, and augments its learned visual latent space with a forward dynamics constraint and a distance-based monotonic progress constraint. Reported evaluations cover Robosuite / MimicGen manipulation tasks under environmental randomization and single-arm real-robot experiments, with substantial gains over fixed-parameter goal-conditioned baselines, especially in higher-variability settings (Zhou et al., 26 Nov 2025).

1. Problem setting and formalization

Hyper-GoalNet addresses goal-conditioned policy learning for robotic manipulation from demonstrations. The training dataset is

$\mathcal{D} = \{ \tau_i \}_{i=1}^{M},$

with each trajectory

$\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$

where each observation contains an RGB image and proprioception,

$o_j = (I_j, s_j),$

and each action $a_j$ is continuous-valued. Although the appendix recalls a standard finite-horizon MDP and a goal-conditioned RL objective, Hyper-GoalNet is explicitly not trained with rewards: it uses reward-free supervised policy learning from offline demonstrations, not online RL, offline RL in the reward-learning sense, or an imitation-plus-RL hybrid (Zhou et al., 26 Nov 2025).

The goal-conditioned training signal is constructed by future-image sampling within the same demonstration trajectory. For a current timestep $t$, a future timestep $t' > t$ from the same trajectory is sampled, and the future observation serves as the goal observation. The method therefore uses feasible goals by construction. Goals are represented as single future RGB images rather than goal sequences, and proprioceptive goal states are not assumed available.

The central conceptual claim is that fixed-weight goal-conditioned policies create a bottleneck in diverse manipulation settings. If the same weights must simultaneously interpret the goal, process the current observation, and map all current-goal combinations to actions, generalization can degrade under large visual variability, contact-rich dynamics, partial observability, and qualitatively different goals. Hyper-GoalNet replaces that regime with goal-conditioned parameter generation:

$$\theta = \mathcal{H}(z_t, z_g), \qquad \hat a_t = \pi(o_{t-L:t}; \theta),$$

where $z_t = \mathcal{E}(I_t)$ and $z_g = \mathcal{E}(I_g)$ are latent encodings of current and goal images, $\mathcal{H}$ is the hypernetwork, and $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$0 is the generated target policy.

2. Architectural decomposition

The architecture has three main components: an image encoder $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$1, a hypernetwork $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$2, and a lightweight target policy $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$3. The encoder maps images into latent features,

$\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$4

The implementation uses a pretrained R3M visual encoder, frozen for the first 20 epochs and then unfrozen for fine-tuning. The hypernetwork takes latent current/goal information and generates the parameters of the target policy; at inference time the conditioning form is described as concatenated latent features $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$5. A variant, Hyper-GoalNet(G), conditions the hypernetwork only on the goal image (Zhou et al., 26 Nov 2025).

The target policy is described in the appendix as a 3-layer MLP. The hypernetwork follows the HyPoGen architecture with 8 optimization blocks. Rather than directly regressing weights in one shot, it iteratively refines them:

$\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$6

with updates

$\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$7

Here $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$8 behaves like a learned step size and $\tau_i = \{(o^i_j, a^i_j)\}_{j=1}^{N_i},$9 like a learned update direction. This gives the model an optimization-inspired inductive bias.

The generated policy consumes a short observation history $o_j = (I_j, s_j),$0, not merely a single current observation. In practice, the context length is 2: the current frame and one historical frame. Each timestep combines temporal image features with proprioception through a compact proprioceptive encoder, and the integrated features are processed by the target MLP to produce actions. This separates goal interpretation from state processing. In conventional conditioning, one writes

$o_j = (I_j, s_j),$1

with one shared $o_j = (I_j, s_j),$2 across all goals. In Hyper-GoalNet, the goal changes the function itself:

$o_j = (I_j, s_j),$3

Several implementation details are explicit: Adam, a cosine learning rate schedule, no weight decay or dropout in the hypernetwork, image resolution $o_j = (I_j, s_j),$4, simulation proprioception dimension 9, and context length 2. The latent dimensionality, activation functions, normalization layers, and exact parameter-count breakdown are not specified in the paper text.

3. Latent-space shaping and training objective

A distinctive feature of Hyper-GoalNet is that policy generation is coupled to explicit shaping of the visual latent space. Two complementary constraints are introduced. The first is a forward dynamics model, intended to make latent states predictive under actions:

$o_j = (I_j, s_j),$5

implemented deterministically as

$o_j = (I_j, s_j),$6

Its predictive loss is

$o_j = (I_j, s_j),$7

This encourages the encoder to produce action-relevant latents rather than latents governed only by visual similarity (Zhou et al., 26 Nov 2025).

The second constraint imposes monotonic progress toward a future goal along successful trajectories. The desired property is

$o_j = (I_j, s_j),$8

and the implemented loss is

$o_j = (I_j, s_j),$9

with

$a_j$0

The paper states that $a_j$1 is a margin and that $a_j$2 is sufficient empirically. This latent geometry is used not only to improve policy generation but also to support autonomous goal completion detection by thresholding latent distance to the goal.

The policy itself is trained by behavior cloning. For sampled current-goal pairs from the same trajectory,

$a_j$3

with $a_j$4 equal to MSE. The full objective is

$a_j$5

and the experiments use $a_j$6 for all components.

At test time, the current image and goal image are encoded, policy weights are regenerated online as the current observation changes, and success may be declared when

$a_j$7

The exact threshold $a_j$8 is not provided in the paper text.

4. Evaluation regime and empirical performance

The simulation study uses Robosuite / MimicGen tasks spanning contact-rich tabletop manipulation and longer-horizon problems: Coffee, Threading, Mug cleanup, Nut assembly, Three-piece assembly, Coffee preparation, and Kitchen manipulation. Difficulty levels $a_j$9, $t$0, and $t$1 introduce increasing variability in initial object poses, including position and orientation randomization. Training uses 950 demonstrations per task and 50 validation demonstrations, with front-view RGB images of size $t$2, 9-D proprioception, batch size 256, 500 epochs, Adam, and initial learning rate $t$3. Evaluation uses 50 independent rollouts per task/setting on unseen randomized initializations. Maximum trajectory lengths are $t$4 or $t$5 for contact-rich tasks and $t$6 for long-horizon tasks (Zhou et al., 26 Nov 2025).

The main quantitative results show large gains over fixed-parameter baselines. On Coffee, average success is 0.77 for Hyper-GoalNet, versus 0.55 for C-BeT, 0.24 for MimicPlay, and 0.00 for GCBC and Play-LMP. On Mug cleanup, the average is 0.62 for Hyper-GoalNet versus 0.40 for C-BeT. On Three-piece assembly, 0.25 versus 0.01. On Threading, 0.46 versus 0.32. On Nut assembly, 0.55 versus 0.34. On long-horizon tasks, Coffee Preparation improves from 0.43 for C-BeT to 0.65, and Kitchen from 0.74 to 0.90; the overall average rises from 0.59 to 0.78. The paper emphasizes that gains are especially pronounced under $t$7 and $t$8, where environmental randomization is stronger.

Real-robot experiments use a RealMan RMC-DA dual-arm platform in single-arm mode, with each arm having 7 DoF and a parallel-jaw gripper, and an overhead Intel RealSense D435i RGB-D sensor. Inputs follow the same temporal protocol as simulation: two most recent observations and a single goal image. RGB-D is concatenated into a 4-channel $t$9 input normalized to $t' > t$0, and because explicit end-effector pose is unavailable, proprioception is represented by the previous action, an 8-D vector containing 7 joint angles and 1 gripper command. Across approximately 70–100 teleoperation trajectories per task and 15 evaluation trials per task, success reaches 14/15 on Pickplace, 15/15 on Pull, 14/15 on Stack, and 15/15 on Sweep, compared with 2/15, 6/15, 5/15, and 8/15 for C-BeT, and near-zero rates for GCBC and Play-LMP.

The latent-space completion detector is also evaluated. On Coffee, latent-threshold success detection achieves mean accuracy 86.6% and mean recall 94.3%, indicating that the shaped latent geometry functions as a usable completion signal in addition to supporting control.

5. Ablations, efficiency, and architectural interpretation

The ablations isolate both the hypernetwork formulation and the latent-shaping components. On Coffee, removing latent shaping reduces performance from $t' > t$1 to $t' > t$2, dropping the average from 0.77 to 0.31. Unfreezing the visual encoder from epoch 0 reduces the average from 0.77 to 0.51, which is why the frozen-then-unfrozen schedule is retained. Replacing goal-relative Euclidean distance with distance to the start image yields average 0.45, and using cosine distance yields 0.59, both below the full method’s 0.77. Applying the same shaping to C-BeT improves that baseline to 0.69, but it still remains below Hyper-GoalNet at 0.77. Against HyperZero variants on Coffee, HyperZero + ScalarInit and HyperZero + Bias-Init each report average 16%, whereas Hyper-GoalNet reports 77%, supporting the particular optimization-inspired hypernetwork choice (Zhou et al., 26 Nov 2025).

Despite online parameter generation, deployment is fast. Reported latency is 6.33 ms/step for Hyper-GoalNet and 1.46 ms/step for Hyper-GoalNet(G), versus 15.47 ms/step for GCBC, 22.78 ms/step for Play-LMP, and 13.61 ms/step for C-BeT. Training cost is somewhat higher than HyperZero, at approximately 104 s/epoch versus approximately 90 s/epoch. Memory use is 3038 MB versus 4916 MB with frozen encoders, and 13452 MB versus 14844 MB with unfrozen encoders, in the comparison reported in the appendix.

These findings support a specific interpretation of the method. Latent shaping is not merely auxiliary supervision; in the reported ablations it is functionally necessary for robustness in harder settings. At the same time, the generated-policy architecture contributes beyond representation learning alone, because a shaped fixed-parameter baseline still trails the hypernetwork model.

6. Relation to adjacent work, limitations, and scope

Hyper-GoalNet differs from standard UVFA-style or goal-conditioned behavior cloning systems because the goal does not merely enter as an additional feature; it determines the policy parameters. The work also differs from prior hypernetwork applications in robotics and control by focusing on reward-free behavior cloning for goal-conditioned manipulation, coupled to latent shaping for predictability and monotonic progress (Zhou et al., 26 Nov 2025).

Similarity in naming should not obscure domain differences. “GoalNet: Goal Areas Oriented Pedestrian Trajectory Prediction” studies first-person-view pedestrian forecasting in autonomous driving, predicts goal areas and trajectories, and does not mention Hyper-GoalNet (Lee et al., 2024). “GoalNet: Inferring Conjunctive Goal Predicates from Human Plan Demonstrations for Robot Instruction Following” is a neuro-symbolic instruction-following system that infers symbolic goal predicates for a planner, and likewise does not mention Hyper-GoalNet (Sharma et al., 2022). “GANet: Goal Area Network for Motion Forecasting” is a goal-area method for autonomous-driving motion forecasting with map-conditioned feature cropping rather than goal-conditioned policy-parameter generation (Wang et al., 2022). Hyper-GoalNet is therefore specifically a robotic manipulation architecture centered on hypernetwork-based parameter generation.

Several limitations are stated explicitly. Performance depends strongly on learning a well-structured latent space, demonstrations must exhibit clear goal progression, out-of-distribution goals can produce erratic generated policies, and the offline training regime offers no explicit safety guarantees on novel states. Absolute success can also remain modest on difficult tasks; Three-piece assembly shows clear relative gains, but the hardest settings are still challenging. A plausible implication is that the method is best suited to domains where single-image goal specification is natural, demonstration coverage is reasonably good, and latent geometry can be shaped by coherent goal-reaching trajectories.