CoMOK: Chain of Moving Oriented Keypoints

• The paper introduces CoMOK as a novel action representation that uses task-relevant SE(3) keypoints to encapsulate semantic affordances for tasks like grasping and pouring.
• CoMOK unifies diverse robotic manipulation tasks by predicting precise, object-centric frames that guide both initial grasping and subsequent motion sequences.
• The approach leverages a diffusion-based, transformer model achieving sub-centimeter accuracy and supporting multi-modal actions for both rigid and deformable objects.

Searching arXiv for the specified papers to ground the article and verify bibliographic details. arxiv_search(query="^^^^^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^^^^^", max_results=5, sort_by="^^^^^^^^3submittedDate3 arxiv_search(query="^^^^^^^^3ti:\3 Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation\"", max_results=3, sort_by="relevance") Chain of Moving Oriented Keypoints (CoMOK) is an action representation for robotic manipulation in which a policy predicts a task-specific oriented keypoint attached to a manipulated object or local patch, together with a target motion for that keypoint, rather than predicting only a direct end-effector or joint-space action. In the formulation introduced in "ImaginationPolicy: Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation" (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&), CoMOK is intended to occupy a middle level of abstraction: it is more expressive and general than a single end-effector pose, while remaining precise and physically grounded. The representation is designed to unify grasping, placement, pouring, hanging, insertion, and deformable-object manipulation within an end-to-end policy, while supporting multi-stage tasks, multi-modal behaviors, and sub-centimeter accuracy (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

^^^^3submittedDate3^^^^. Concept and motivation

CoMOK is motivated by a limitation of standard end-effector pose actions: they directly encode the robot tool frame, but not necessarily the object part that is semantically relevant to the task. The motivating examples are object-centric. In grasping, the relevant target is a grasp pose; in pouring, the mug rim and pouring orientation matter; in hanging a mug, the handle center relative to the rack matters; in insertion, the grasp point determines what part of a deformable rope becomes controllable later; and in stable placement, the object’s bottom-center and orientation relative to the table matter (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). In this view, the task is not adequately described by “move the gripper here,” but rather by “move this semantically meaningful local part of the object in this way.”

The central response is to represent the affordance explicitly as a local object frame and the action as where that frame should move. The paper characterizes this as a more interpretable, more general, and more accurate parameterization because the network predicts a geometrically meaningful frame instead of a coarse category-level action (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Existing keypoint-based manipulation methods are described as often task-specific; CoMOK is presented as a general action abstraction that extends the standard end-effector pose representation and supports diverse manipulation tasks in a unified manner (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

A common misunderstanding is to treat CoMOK as a simple list of keypoints. In the paper’s formulation, the keypoint is not merely positional, and the “chain” is not merely temporal indexing. The representation is explicitly object-centric, pose-valued, and stage-aware. This suggests that CoMOK should be understood less as sparse geometric annotation and more as a structured action language over controllable local frames.

^^^^3ti:\3^^^^. Oriented keypoints as PRESERVED_PLACEHOLDER_^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^ affordance frames

An oriented keypoint in CoMOK is a full PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^-DoF local frame in PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^, not just a point in $3$D space. The paper denotes the task-relevant oriented keypoint by $T_\text{affordance} \in SE(3)$ (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). This frame includes both position and orientation, and is semantically attached to a rigid object, a local patch of a deformable object, or the gripper TCP when the manipulated “object” is the robot’s own tool (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

This affordance-centric semantics is task dependent. For a mug, the bottom-center, rim, or handle center can each serve as the oriented keypoint, depending on whether the task is placement, pouring, or hanging (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The paper’s generalization argument is geometric: “bottom-center,” “rim,” “handle center,” or “insertable end” can be recognized across varied shapes and sizes because the representation is tied to local affordance geometry rather than object identity (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The oriented keypoint therefore captures task-relevant local geometry while ignoring irrelevant details.

The basic affordance-action mapping is written as

$$\text{network}(o_\text{scene}, f_\text{task}) \xrightarrow{} (o_\text{manipulated}, T_\text{affordance}, T_\text{action}),$$

where $o_\text{scene}$ is the scene observation, $f_\text{task}$ is the task description, $o_\text{manipulated}$ is the object or region the robot can control, $T_\text{affordance}$ is the task-relevant oriented keypoint, and PRESERVED_PLACEHOLDER_^^^^3submittedDate3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^ is the target pose of that keypoint (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The semantics are explicit: PRESERVED_PLACEHOLDER_^^^^3submittedDate3submittedDate3^^^^ is “the thing the robot can control” in a task-relevant way, and PRESERVED_PLACEHOLDER_^^^^3submittedDate3ti:\3^^^^ is where that affordance frame should be moved to accomplish the task (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

Task context	Task-relevant oriented keypoint	Role in CoMOK
Stable placement	Bottom-center	Placement affordance
Pouring	Rim	Pouring orientation
Mug hanging	Handle center	Alignment with rack
Deformable insertion	Local grasped patch / insertable end	Controllable local region

The implementation note that the score-matching network predicts denoising vectors for each PRESERVED_PLACEHOLDER_^^^^3submittedDate33^^^^ element in PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^4 is consistent with a PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^5D pose parameterization, although the text does not separately decompose PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^6 into translation and rotation (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

3. From single affordances to chains

CoMOK becomes a chain when the target is extended from a single pose to a sequence of poses. The paper writes

PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^7

with

PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^8

where PRESERVED_PLACEHOLDER_^^^^3submittedDate3^^^^9 is the horizon length (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). In this form, the robot is not given only a final target pose; it is given a trajectory in pose space for the task-relevant keypoint. The paper gives sparse trajectories such as “pre-grasp PRESERVED_PLACEHOLDER_^^^^3ti:\3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^ grasp” and “pre-place PRESERVED_PLACEHOLDER_^^^^3ti:\3submittedDate3^^^^ place,” as well as dense trajectories for cutting, welding, or drawing (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Its intuition is direct: if the keypoint is moved along this chain of target frames, the task is completed (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

The same representational logic extends to long-horizon manipulation. For multi-stage tasks, the system predicts a list of staged actions,

PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^

where

PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^

This allows a global task such as “pour water into the container” to be decomposed into sub-tasks such as grasp, pour, and place (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The paper explicitly argues that CoMOK naturally decomposes a task into a chain of stage-wise affordance-action pairs, with the output of one stage becoming a prerequisite for the next (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

A further property is multi-modality. The paper notes that many valid affordance-action pairs may exist for the same task, because of multiple manipulated object instances, multiple feasible keypoint locations, or multiple valid grasp and placement poses (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). For that reason, the policy uses a diffusion-like score-matching model to generate a distribution over possible actions rather than a single deterministic output (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). This design is central to the claim that CoMOK can represent grasp diversity, multiple placement poses, and multiple feasible insertion configurations.

The representation also extends to deformable objects. In that setting, the affordance is attached to a local grasped patch rather than a rigid object frame, so the system reasons about a PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4-DoF controllable local part even when the entire object is not rigidly controllable (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). This is especially important in the paper’s cable insertion example, where the rope is deformable and the grasp point determines the controllable region for the later insertion stage (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

4. Policy architecture and supervision

The end-to-end policy consists of two major components: a task-planning network and an action-prediction network (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The task-planning network consumes RGBD scene input and a global language task description, and is based on a fine-tuned Groma vision-LLM (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Its role is to decompose the global task into stage-level subtasks, identify the manipulated object, and optionally identify an environment region relevant to the action. The corresponding mapping is

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5

where

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6

The optional PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7 indicates the region the action should attend to (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

The action-prediction network takes the scene point cloud and stage information and predicts CoMOK actions using a score-matching / diffusion-style model (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The architecture is transformer-based: point cloud features are encoded into a sequence; noisy tokens representing PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8 and PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9 are flattened and processed; self-attention predicts denoising vectors for each $3$^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^ element; and cross-attention injects visual and language features, possibly masked by stage-specific regions (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The model predicts all stages jointly and does not backpropagate through the task-stage hidden feature $3$^^^^3submittedDate3^^^^ (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

Supervision is end-to-end over task decomposition, object or region localization, affordance frame prediction, and action frame or action-sequence prediction (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Although the excerpt does not provide a full loss formula, it states that the action network is trained with score-matching / denoising, similar to diffusion-based pose prediction, so the network learns to denoise from noisy pose samples toward the correct action distribution (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

At inference time, the task-planning network predicts staged subtasks and manipulated regions, the action network samples one or more CoMOK candidates, and a downstream motion generator converts CoMOK into executable robot motion (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Two motion-generation choices are described: a learned joint-space trajectory generator in simulation and a conventional task-and-motion planner on hardware, which selects among candidates based on collision, reachability, and path length (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). This separation is conceptually important. CoMOK specifies semantically meaningful task geometry; the motion generator handles robot-specific feasibility. A common misconception is therefore to treat CoMOK itself as a complete low-level controller. The paper instead presents it as the action representation supplied to downstream execution modules.

5. Generalization, precision, and empirical scope

The paper’s argument for generalization has several components. First, shape and size generalization follow from defining the keypoint in terms of local affordance geometry rather than object identity (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Second, the representation is claimed to achieve sub-centimeter-level accuracy because the network predicts a precise $3$^^^^3ti:\3^^^^-DoF frame rather than a coarse class label or a low-resolution heatmap (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Third, multi-stage tasks are naturally represented as chains of stage-wise affordance-action pairs (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Fourth, multi-modal behavior is represented by generating a distribution over CoMOK candidates through diffusion / score-matching (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Fifth, deformable-object handling follows from attaching the affordance to a controllable local patch (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

The experimental program is intended to show that the formulation works across task types, object variations, and physical settings (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). On a standard parallel-jaw grasp dataset, the model predicts grasp poses plus a pre-grasp frame, making grasping a special case of the formulation; in this case only the action sequence is meaningful because the manipulated object is the gripper itself (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). In stable placement in simulation, the robot must pick and place bottles, boxes, and tripods of many shapes and sizes while avoiding obstacles; the method predicts a grasp stage, a placement stage with a stable bottom-oriented affordance, and a pre-place $3$3 placement action chain (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

Cable insertion demonstrates deformable-object handling. The system must jointly predict grasp and insertion actions, showing that the affordance representation can reason about local controllable patches on nonrigid objects (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Mug hanging on a rack emphasizes precise alignment: the model must identify the mug handle center and align it with the rack stick, with centimeter-level accuracy requirements and an action sequence that uses several frames along the stick axis (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The paper presents this as evidence that the formulation supports fine geometric control rather than only coarse manipulation.

The ablation logic in the excerpt is representational rather than purely metric. Using task-specific oriented keypoints is described as better than direct end-effector action because it provides object-centric semantics; using a sequence of frames improves long-horizon and multi-stage behavior; using a diffusion/score-matching generator improves multimodal prediction over deterministic regression; and jointly predicting the grasp and downstream manipulation stage is necessary for tasks such as insertion and hanging, where the grasp determines future controllability (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Reported failures are described as mostly due to upstream task-planning errors or downstream physical feasibility limits, not because CoMOK itself is too weak (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). This suggests that the representation is intended as the semantic and geometric substrate of manipulation, rather than the sole determinant of execution success.

6. Interpretation and relation to adjacent trajectory-grounded reasoning

CoMOK can be interpreted as a structured trajectory representation over object-centric $3$4 affordances. In manipulation terms, a robot does not simply predict where the gripper goes; it predicts what object part matters, how that part is oriented, and how it should move through space over time (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). The “chain” therefore denotes an ordered sequence of desired motions for a controllable local frame, rather than an unstructured temporal trace.

A useful point of comparison appears in "Motion-o: Trajectory-Grounded Video Reasoning" (&&&^^^^3submittedDate3^^^^&&&). That work formalizes Spatial-Temporal-Trajectory reasoning and introduces Motion Chain of Thought, in which successive grounded observations are explicitly connected into trajectories through structured motion descriptors (&&&^^^^3submittedDate3^^^^&&&). The domains differ—robotic manipulation in CoMOK, video reasoning in Motion-o—but the shared conceptual move is to make the linkage between observations or states first-class rather than implicit. This suggests a broader methodological pattern: trajectory structure can be represented explicitly, supervised directly, and used as a verifiable intermediate object.

That comparison also helps clarify what CoMOK is not. It is not merely a language description of intent, because its primitives are $3$5 frames and pose sequences. It is not merely a perception module, because the representation includes target motions. It is not identical to full motion planning, because execution feasibility is delegated to a downstream motion generator (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Nor is it restricted to rigid objects, since the paper explicitly attaches the affordance to a local patch for deformable objects (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&).

Within the paper’s framing, CoMOK is best understood as a general manipulation action representation built from semantically meaningful $3$6-DoF local object frames and their target motions (&&&^^^^3id:(Lu et al., 25 Sep 2025) OR id:(Galoaa et al., 19 Mar 2026)3^^^^&&&). Its significance lies in recasting manipulation policy output around controllable affordances rather than only robot-centric poses, with the aim of unifying diverse tasks while retaining geometric precision.