ReLAM for Visual Robotic Manipulation

• ReLAM is a framework that learns dense, geometric rewards by predicting keypoint-based subgoals from action-free video demonstrations.
• It employs a two-stage pipeline that extracts visual keypoints and uses a hierarchical control method with reinforcement learning to refine the reward signal.
• Empirical evaluations on Meta-World and ManiSkill demonstrate near-optimal performance by aligning subgoal progression with task geometry.

Reward Learning with Anticipation Model (ReLAM) is a framework for visual robotic manipulation that learns dense, structured rewards from action-free video demonstrations by predicting intermediate keypoint-based subgoals and converting distances to those anticipated subgoals into stage-wise reward signals for a goal-conditioned policy. It is designed for settings in which dense simulator-state rewards are unavailable because the agent observes only RGB images, and it combines a high-level anticipation model with low-level reinforcement learning under a hierarchical formulation equipped with a sub-optimality bound (Tang et al., 26 Sep 2025).

1. Scope and problem formulation

ReLAM addresses a specific bottleneck in visual reinforcement learning for robotic manipulation: reward design when exact geometric state is not directly observable. In simulation, dense rewards are often defined from distances between privileged simulator variables such as object poses, end-effector positions, and target configurations. In real-world visual settings, however, such precise positional information is often unavailable because of sensory noise, occlusion, perceptual ambiguity, viewpoint dependence, and the lack of calibrated state estimation. ReLAM is proposed for precisely this regime: it seeks to automatically generate dense rewards for long-horizon manipulation directly from images and action-free videos rather than from privileged state or manually engineered geometric formulas (Tang et al., 26 Sep 2025).

The method is organized as a two-stage pipeline. First, it converts demonstrations into a sequence of keypoint-based subgoals and trains an anticipation model that predicts those subgoals from the initial observation. Second, it uses the predicted subgoals to define a dense reward for a low-level, goal-conditioned policy. At the start of an episode, the anticipation model outputs a sequence $P_0, P_1, \dots, P_k$ of keypoint configurations; during policy learning, the agent is rewarded according to its progress toward the current subgoal, receives additional milestone rewards when subgoals are reached, and then advances to the next stage. The resulting curriculum is explicitly aligned with the task’s geometric progression rather than with image-level similarity alone (Tang et al., 26 Sep 2025).

A central design choice is that ReLAM does not learn reward by adversarial discrimination over full videos, nor by directly generating future images and comparing them in representation space. Instead, it infers task structure through sparse keypoints and uses an anticipation model as a planner over those keypoints. This makes the framework both a reward-learning method and a hierarchical control method: the learned reward is induced by predicted intermediate task states, and the policy is trained to realize those states sequentially (Tang et al., 26 Sep 2025).

2. Visual state abstraction and subgoal extraction

ReLAM represents manipulation state through tracked keypoints rather than raw pixels. Given action-free demonstrations

$$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$

the method first applies grounded SAM to the initial frame $I_0$ to segment task-relevant objects, then tracks pixels over time with CoTracker. It retains only sufficiently moving points and selects a sparse representative subset by Farthest Point Sampling: $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$ Appendix details add a further filtering rule: a point is kept only if an $L\times L$ square centered on it lies inside the segmentation mask, which reduces contamination from SAM errors (Tang et al., 26 Sep 2025).

The paper treats the resulting keypoints as a high-level geometric abstraction of task state. A keypoint coordinate is denoted $p_t^k$, and the dataset of keypoint configurations at selected keyframes is written as

$$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$

This abstraction is crucial because it allows ReLAM to infer an image-based proxy for spatial progress using Euclidean distances between current and target keypoint configurations, instead of requiring explicit 3D pose or simulator coordinates (Tang et al., 26 Sep 2025).

Subgoals are extracted by selecting keyframes that correspond to meaningful transitions between motion segments. The paper assumes that manipulation trajectories can often be decomposed into approximately linear-motion segments, and it detects boundaries between such segments by choosing the timestep with the largest direction change within a bounded interval: $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$ Because cosine similarity is close to $1$ under straight motion and smaller at turning points, minimizing the sum identifies likely transitions between motion primitives. The keypoint configurations at these keyframes become the subgoal sequence used to supervise the anticipation model (Tang et al., 26 Sep 2025).

3. Anticipation model

The anticipation model predicts a sequence of future keypoint subgoals rather than future images. Its output is a planned sequence $P_0, P_1, \dots, P_k$, where $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$0 is the inferred current keypoint state and $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$1 are intermediate targets on the path to the final goal. In the paper’s terminology, the anticipation model “serves as a planner” because it proposes intermediate waypoints that decompose a long-horizon manipulation problem into shorter geometric stages (Tang et al., 26 Sep 2025).

Its inputs are the initial observation $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$2, optionally a task-indication frame $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$3 in multi-task settings, and previously generated keypoint tokens under autoregressive decoding. The specified architecture uses a frozen DINOv2 visual encoder with RGB inputs of size $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$4, divides each image into $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$5 patch embeddings, embeds normalized point coordinates through an MLP, and passes image and point tokens through a 12-layer causal transformer. Predicted point tokens are decoded by an MLP in residual form. Appendix hyperparameters give embedding dimension $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$6, $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$7 layers, $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$8 heads, dropout $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$9, batch size $I_0$0, learning rate $I_0$1, and keypoint number $I_0$2 (Tang et al., 26 Sep 2025).

Training is autoregressive and uses teacher forcing with mean squared error on future keypoint coordinates. The paper does not present a single closed-form loss equation for the anticipation model in the main text, but it states that predicted coordinates are compared with ground-truth coordinates and optimized by MSE. This distinguishes ReLAM from image-generation approaches: the learned planner operates in sparse geometric coordinates rather than in full visual space, which the reported experiments later identify as a major practical advantage (Tang et al., 26 Sep 2025).

4. Reward construction and hierarchical control

Once a subgoal sequence $I_0$3 has been predicted, ReLAM converts it into a stage-wise reward. If the current stage is $I_0$4, the average keypoint distance to the stage target is

$I_0$5

where $I_0$6 is the current position of keypoint $I_0$7 and $I_0$8 is its target position in subgoal $I_0$9. This distance is transformed into a dense reward by a piecewise linear function: $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$0 The paper states that the slopes satisfy $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$1, so the reward becomes more sensitive as the agent approaches the target. This produces stronger incentive near the subgoal while preserving learning stability farther away (Tang et al., 26 Sep 2025).

A subgoal is considered reached when the distance satisfies $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$2. At that point, the current stage is marked successful, the target is updated to the next subgoal, and the agent receives an additional milestone reward. The overall reward is written as

$$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$3

with the text clarifying that $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$4 includes stage-success and final task-success rewards. The reward is therefore both dense and structured: dense because it tracks geometric proximity continuously, and structured because it is organized around an ordered sequence of anticipated subgoals (Tang et al., 26 Sep 2025).

This reward is embedded in a hierarchical reinforcement-learning interpretation. The high-level component is the anticipation model $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$5, which proposes the keypoint subgoal sequence, and the low-level component is a goal-conditioned policy conditioned on the current RGB observation, current keypoint coordinates, and target keypoint coordinates. ReLAM does not introduce a new Bellman objective; instead, it optimizes the low-level policy with standard RL algorithms using the ReLAM reward. For online RL, it uses PPO and modifies it so that terminal=True is set whenever either a subgoal is reached or the episode ends, after which GAE is computed separately for each low-level episode segment. For offline RL, it generates subgoals for each trajectory, relabels transitions with the ReLAM reward, and trains with IQL (Tang et al., 26 Sep 2025).

The paper’s training recipe is correspondingly divided into three steps: generate the keypoint-subgoal dataset from action-free videos, train the anticipation model with teacher forcing, and then train the policy using rewards computed from the predicted subgoals. For the anticipation model, the reported data requirement is $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$6 video trajectories per task. For offline RL, the reported control dataset contains $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$7 trajectories per task, split into $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$8 expert and $$\mathcal{P} = \mathbf{FPS}\left( \{p=(x,y) \in SAM(I_0): \max_{0\leq t,t' \leq T} (x_t - x_{t'})^2 + (y_t - y_{t'})^2 \geq \Theta \} \right).$$9 noisy expert trajectories (Tang et al., 26 Sep 2025).

5. Theoretical analysis

The theoretical analysis is formulated in keypoint state space and yields a sub-optimality bound rather than a full deep-RL convergence theorem. The assumptions are: $L\times L$0 is continuous, $L\times L$1 has a Euclidean distance metric $L\times L$2, and the transition function $L\times L$3 is deterministic. The paper further assumes bounded reachability: for each transition from $L\times L$4 to $L\times L$5 there exists a constant $L\times L$6 such that $L\times L$7, and for every reachable $L\times L$8 there exists a unique action $L\times L$9 satisfying $p_t^k$0, denoted $p_t^k$1 (Tang et al., 26 Sep 2025).

Within this framework, the paper defines linear reachability and introduces two rewards for a final goal $p_t^k$2. The time reward is

$p_t^k$3

and the distance reward is

$p_t^k$4

where $p_t^k$5 is monotonically decreasing and always negative. The paper proves that if the goal is linearly reachable from $p_t^k$6, then the optimal policy under distance reward is also optimal for time reward. The corresponding optimal policy under time reward is

$p_t^k$7

The interpretation given in the paper is that optimal behavior moves to the farthest reachable point along the straight line to the goal (Tang et al., 26 Sep 2025).

ReLAM’s anticipation model is then analyzed through a shortest-path sequence $p_t^k$8 and a predicted path $p_t^k$9 satisfying

$$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$0

If each low-level policy $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$1 is also close to its subtask optimum in the sense that

$$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$2

then the resulting hierarchical policy satisfies the bound

$$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$3

The theorem implies that ReLAM is near-optimal when the anticipation model predicts waypoints close to the true shortest-path waypoints and when each low-level stage policy is well solved; the approximation error scales linearly with the number of subgoals $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$4 (Tang et al., 26 Sep 2025).

6. Empirical performance, ablations, and limitations

The empirical evaluation spans two benchmark suites. For online RL, ReLAM is evaluated on Meta-World with a Sawyer arm on Drawer Open, Door Open, and Button Press Wall. For offline RL, it is evaluated on ManiSkill with a Franka Panda on Push Cube and Pick Cube. All tasks use RGB observations at resolution $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$5 from a fixed third-person camera. The main metric is average success rate over $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$6 episodes, reported across $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$7 random seeds with half-standard-deviation error bars. Baselines are DACfO, Diffusion Reward (DR), Image Subgoal (IS), and Oracle, where Oracle uses ground-truth subgoal images as an upper bound for image-subgoal methods (Tang et al., 26 Sep 2025).

On Meta-World, the appendix reports final mean success rates of $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$8 for ReLAM on Drawer Open, $$\mathcal{K} = \bigcup_{i=1}^N \mathcal{K}_i = \bigcup_{i=1}^N \{ p_{i,j}^k = (x_{i,j}^k, y_{i,j}^k)\}.$$9 on Door Open, and $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$0 on Button Press Wall. The corresponding Oracle values are $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$1, $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$2, and $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$3; DR achieves $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$4, $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$5, and $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$6; DACfO achieves $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$7, $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$8, and $$t_j = \arg \min_{\,t \in [t_{j-1}+m,\; t_{j-1}+M]} \; \sum_{k=1}^{K} \frac{\langle p_t^k - p_{t-1}^k, p_{t+1}^k - p_t^k \rangle}{\|p_t^k - p_{t-1}^k \| \|p_{t+1}^k - p_t^k \|}.$$9; and IS achieves $1$0, $1$1, and $1$2. The paper attributes the weakness of IS to noisy generated images and unstable similarity thresholds, and describes DACfO and DR as unstable on some runs because exploration and discriminator-like reward estimation can fluctuate substantially (Tang et al., 26 Sep 2025).

On ManiSkill, ReLAM is second only to Oracle and remains close to it: on Push Cube, ReLAM reaches $1$3 versus Oracle $1$4, DACfO $1$5, DR $1$6, and IS $1$7; on Pick Cube, ReLAM reaches $1$8 versus Oracle $1$9, DACfO $P_0, P_1, \dots, P_k$0, DR $P_0, P_1, \dots, P_k$1, and IS $P_0, P_1, \dots, P_k$2. The paper interprets this as evidence that keypoint-based subgoals are substantially easier to anticipate and exploit than image-based subgoals. A t-SNE analysis of rewards is reported to show that ReLAM’s rewards cluster closer to environment rewards, whereas DR and DACfO cluster together because they both emphasize closeness to the expert distribution rather than geometric task progress (Tang et al., 26 Sep 2025).

The ablations reinforce that the method’s gains depend on its geometric representation. On Drawer Open, $P_0, P_1, \dots, P_k$3 keypoints perform better than $P_0, P_1, \dots, P_k$4 or $P_0, P_1, \dots, P_k$5, with the paper attributing the drop at larger keypoint counts to more difficult anticipation and overconstrained policy behavior. Among reward transforms, piecewise linear performs best, pure linear is second, and exponential and logistic are worse; the explanation given is that exponential and logistic induce continuously changing slopes that make adaptation harder, while linear provides insufficient incentive near the target. The gap between policies trained on autoregressively generated keypoint subgoals and on ground-truth keypoint subgoals is reported to be small, which the paper takes as evidence that the anticipation model is sufficiently accurate. By contrast, the gap between IS and Oracle is much larger, again emphasizing the relative difficulty of image-subgoal generation (Tang et al., 26 Sep 2025).

The paper also states several limitations explicitly. Performance is viewpoint-dependent because the anticipation model is trained from a single camera viewpoint. Heavy occlusions can break reliable keypoint tracking. The experiments are conducted at relatively modest data and model scale. Proposed future directions include multi-view observations, point-cloud representations, larger pretrained VLMs as anticipation models, and larger datasets such as Open X-Embodiment (Tang et al., 26 Sep 2025).

7. Relation to adjacent research and common misconceptions

A recurrent source of confusion is the similarity between ReLAM and the distinct method "Reinforcement Learning with Anticipation" (RLA). RLA is a hierarchical RL framework in which a high-level anticipation model proposes intermediate subgoals by enforcing value geometric consistency,

$P_0, P_1, \dots, P_k$6

but it is not presented as a reward-learning method and does not use the name ReLAM. Its high-level module learns a deterministic subgoal proposer $P_0, P_1, \dots, P_k$7, whereas ReLAM learns rewards from anticipated keypoint subgoals extracted from action-free videos (Yu, 6 Sep 2025).

ReLAM is also distinct from RLAF, which addresses non-Markovian RL by deriving preferences over trajectories from a deterministic finite automaton and then learning a reward model $P_0, P_1, \dots, P_k$8 from those preferences. RLAF’s central signal is symbolic trajectory ranking, including scores such as

$P_0, P_1, \dots, P_k$9

and the paper explicitly states that this automaton-derived score is only an ordinal signal and is never added to returns or used as shaping. ReLAM, by contrast, uses an anticipation model to generate subgoals and then directly defines a dense geometric reward from distances to those subgoals (Alinejad et al., 17 Oct 2025).

Another nearby line is anticipatory reward shaping for collision avoidance. In "Deep Reinforcement Learning with anticipatory reward in LSTM for Collision Avoidance of Mobile Robots," an LSTM predicts near-future robot positions and the predicted inter-robot distance is converted into an auxiliary term $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$00 that is added to the DQN target,

$$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$01

That method is ReLAM-like in the sense of using a learned anticipation model to modulate reward online, but the risk-to-reward mapping remains manually engineered and the task is decentralized collision avoidance rather than visual manipulation from demonstrations (Poulet et al., 11 Aug 2025).

At a broader level, ReLAM belongs to a family of methods that anticipate reward before full task completion. "Expected Reward Prediction, with Applications to Model Routing" studies a different anticipation problem: predicting

$$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$02

from the prompt alone, prior to generation, in order to route prompts to LLMs. That work anticipates expected reward at the model level rather than constructing a control reward for embodied RL, but it shares the upstream principle of forecasting reward-relevant outcomes before execution (Hasanaliyev et al., 3 Mar 2026).

Finally, ReLAM differs from learned reward-machine approaches. "Learning Reward Machines: A Study in Partially Observable Reinforcement Learning" learns a discrete automaton $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$03 from traces so that the augmented representation $$\mathcal{D}=\{V_i=(I_1^i,I_2^i,\cdots,I_{T_i}^i)\}_{i=1,\cdots,N},$$04 becomes more predictive under partial observability. That line is about symbolic memory and reward decomposition in POMDPs rather than keypoint-based anticipation from images. The conceptual overlap lies in learning reward-relevant temporal structure from data, but the learned representation in reward machines is an automaton over abstract events, not an autoregressive planner over visual keypoint subgoals (Icarte et al., 2021).

In its most specific sense, ReLAM denotes the visual manipulation framework of (Tang et al., 26 Sep 2025): action-free video demonstrations are converted into keypoint trajectories, a transformer-based anticipation model predicts intermediate keypoint subgoals, and those anticipated subgoals induce a dense, stage-structured reward for goal-conditioned policy learning.