---
title: 'CoMOK: Chain of Moving Oriented Keypoints'
url: https://www.emergentmind.com/topics/chain-of-moving-oriented-keypoints-comok
type: topic
---

# CoMOK: Chain of Moving Oriented Keypoints

Searching arXiv for the specified papers to ground the article and verify bibliographic details.
arxiv_search(query="id:2509.20841 OR id:2603.18856", max_results=5, sort_by="submittedDate")
arxiv_search(query="ti:\"ImaginationPolicy: Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation\"", max_results=3, sort_by="relevance")
Chain of Moving Oriented Keypoints (CoMOK) is an action representation for robotic manipulation in which a policy predicts a task-specific oriented keypoint attached to a manipulated object or local patch, together with a target motion for that keypoint, rather than predicting only a direct end-effector or joint-space action. In the formulation introduced in "ImaginationPolicy: Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation" [2509.20841], CoMOK is intended to occupy a middle level of abstraction: it is more expressive and general than a single end-effector pose, while remaining precise and physically grounded. The representation is designed to unify grasping, placement, pouring, hanging, insertion, and deformable-object manipulation within an end-to-end policy, while supporting multi-stage tasks, multi-modal behaviors, and sub-centimeter accuracy [2509.20841].

## 1. Concept and motivation

CoMOK is motivated by a limitation of standard end-effector pose actions: they directly encode the robot tool frame, but not necessarily the object part that is semantically relevant to the task. The motivating examples are object-centric. In grasping, the relevant target is a grasp pose; in pouring, the mug rim and pouring orientation matter; in hanging a mug, the handle center relative to the rack matters; in insertion, the grasp point determines what part of a deformable rope becomes controllable later; and in stable placement, the object’s bottom-center and orientation relative to the table matter [2509.20841]. In this view, the task is not adequately described by “move the gripper here,” but rather by “move this semantically meaningful local part of the object in this way.”

The central response is to represent the affordance explicitly as a local object frame and the action as where that frame should move. The paper characterizes this as a more interpretable, more general, and more accurate parameterization because the network predicts a geometrically meaningful frame instead of a coarse category-level action [2509.20841]. Existing keypoint-based manipulation methods are described as often task-specific; CoMOK is presented as a general action abstraction that extends the standard end-effector pose representation and supports diverse manipulation tasks in a unified manner [2509.20841].

A common misunderstanding is to treat CoMOK as a simple list of keypoints. In the paper’s formulation, the keypoint is not merely positional, and the “chain” is not merely temporal indexing. The representation is explicitly object-centric, pose-valued, and stage-aware. This suggests that CoMOK should be understood less as sparse geometric annotation and more as a structured action language over controllable local frames.

## 2. Oriented keypoints as \(SE(3)\) affordance frames

An oriented keypoint in CoMOK is a full \(6\)-DoF local frame in \(SE(3)\), not just a point in \(3\)D space. The paper denotes the task-relevant oriented keypoint by \(T_\text{affordance} \in SE(3)\) [2509.20841]. This frame includes both position and orientation, and is semantically attached to a rigid object, a local patch of a deformable object, or the gripper TCP when the manipulated “object” is the robot’s own tool [2509.20841].

This affordance-centric semantics is task dependent. For a mug, the bottom-center, rim, or handle center can each serve as the oriented keypoint, depending on whether the task is placement, pouring, or hanging [2509.20841]. The paper’s generalization argument is geometric: “bottom-center,” “rim,” “handle center,” or “insertable end” can be recognized across varied shapes and sizes because the representation is tied to local affordance geometry rather than object identity [2509.20841]. The oriented keypoint therefore captures task-relevant local geometry while ignoring irrelevant details.

The basic affordance-action mapping is written as
$$
\text{network}(o_\text{scene}, f_\text{task}) \xrightarrow{} (o_\text{manipulated}, T_\text{affordance}, T_\text{action}),
$$
where \(o_\text{scene}\) is the scene observation, \(f_\text{task}\) is the task description, \(o_\text{manipulated}\) is the object or region the robot can control, \(T_\text{affordance}\) is the task-relevant oriented keypoint, and \(T_\text{action}\) is the target pose of that keypoint [2509.20841]. The semantics are explicit: \(T_\text{affordance}\) is “the thing the robot can control” in a task-relevant way, and \(T_\text{action}\) is where that affordance frame should be moved to accomplish the task [2509.20841].

| Task context | Task-relevant oriented keypoint | Role in CoMOK |
|---|---|---|
| Stable placement | Bottom-center | Placement affordance |
| Pouring | Rim | Pouring orientation |
| Mug hanging | Handle center | Alignment with rack |
| Deformable insertion | Local grasped patch / insertable end | Controllable local region |

The implementation note that the score-matching network predicts denoising vectors for each \(SE(3)\) element in \(\mathbb{R}^6\) is consistent with a \(6\)D pose parameterization, although the text does not separately decompose \(SE(3)\) into translation and rotation [2509.20841].

## 3. From single affordances to chains

CoMOK becomes a chain when the target is extended from a single pose to a sequence of poses. The paper writes
$$
\text{network}(o_\text{scene}, f_\text{task}) \xrightarrow{} (o_\text{manipulated}, T_\text{affordance}, T_\text{action\_seq}),
$$
with
$$
T_\text{action\_seq} \in SE(3) \times K,
$$
where \(K\) is the horizon length [2509.20841]. In this form, the robot is not given only a final target pose; it is given a trajectory in pose space for the task-relevant keypoint. The paper gives sparse trajectories such as “pre-grasp \(\rightarrow\) grasp” and “pre-place \(\rightarrow\) place,” as well as dense trajectories for cutting, welding, or drawing [2509.20841]. Its intuition is direct: if the keypoint is moved along this chain of target frames, the task is completed [2509.20841].

The same representational logic extends to long-horizon manipulation. For multi-stage tasks, the system predicts a list of staged actions,
$$
\text{network}(o_\text{scene}, f_\text{task\_global}) \xrightarrow{} \text{List}[a_\text{stage}],
$$
where
$$
a_\text{stage} = (f_\text{task\_stage}, o_\text{manipulated}, T_\text{affordance}, T_\text{action}).
$$
This allows a global task such as “pour water into the container” to be decomposed into sub-tasks such as grasp, pour, and place [2509.20841]. The paper explicitly argues that CoMOK naturally decomposes a task into a chain of stage-wise affordance-action pairs, with the output of one stage becoming a prerequisite for the next [2509.20841].

A further property is multi-modality. The paper notes that many valid affordance-action pairs may exist for the same task, because of multiple manipulated object instances, multiple feasible keypoint locations, or multiple valid grasp and placement poses [2509.20841]. For that reason, the policy uses a diffusion-like score-matching model to generate a distribution over possible actions rather than a single deterministic output [2509.20841]. This design is central to the claim that CoMOK can represent grasp diversity, multiple placement poses, and multiple feasible insertion configurations.

The representation also extends to deformable objects. In that setting, the affordance is attached to a local grasped patch rather than a rigid object frame, so the system reasons about a \(6\)-DoF controllable local part even when the entire object is not rigidly controllable [2509.20841]. This is especially important in the paper’s cable insertion example, where the rope is deformable and the grasp point determines the controllable region for the later insertion stage [2509.20841].

## 4. Policy architecture and supervision

The end-to-end policy consists of two major components: a task-planning network and an action-prediction network [2509.20841]. The task-planning network consumes RGBD scene input and a global language task description, and is based on a fine-tuned Groma vision-language model [2509.20841]. Its role is to decompose the global task into stage-level subtasks, identify the manipulated object, and optionally identify an environment region relevant to the action. The corresponding mapping is
$$
\text{task\_planning}(o_\text{scene\_RGB}, f_\text{task\_global}) \xrightarrow{} \text{List}[t_\text{stage}],
$$
where
$$
t_\text{stage} = (f_\text{task\_stage}, o_\text{manipulated}, o_\text{env}).
$$
The optional \(o_\text{env}\) indicates the region the action should attend to [2509.20841].

The action-prediction network takes the scene point cloud and stage information and predicts CoMOK actions using a score-matching / diffusion-style model [2509.20841]. The architecture is transformer-based: point cloud features are encoded into a sequence; noisy tokens representing \(T_\text{affordance}\) and \(T_\text{action\_seq}\) are flattened and processed; self-attention predicts denoising vectors for each \(SE(3)\) element; and cross-attention injects visual and language features, possibly masked by stage-specific regions [2509.20841]. The model predicts all stages jointly and does not backpropagate through the task-stage hidden feature \(h_\text{task\_stage}\) [2509.20841].

Supervision is end-to-end over task decomposition, object or region localization, affordance frame prediction, and action frame or action-sequence prediction [2509.20841]. Although the excerpt does not provide a full loss formula, it states that the action network is trained with score-matching / denoising, similar to diffusion-based pose prediction, so the network learns to denoise from noisy pose samples toward the correct action distribution [2509.20841].

At inference time, the task-planning network predicts staged subtasks and manipulated regions, the action network samples one or more CoMOK candidates, and a downstream motion generator converts CoMOK into executable robot motion [2509.20841]. Two motion-generation choices are described: a learned joint-space trajectory generator in simulation and a conventional task-and-motion planner on hardware, which selects among candidates based on collision, reachability, and path length [2509.20841]. This separation is conceptually important. CoMOK specifies semantically meaningful task geometry; the motion generator handles robot-specific feasibility. A common misconception is therefore to treat CoMOK itself as a complete low-level controller. The paper instead presents it as the action representation supplied to downstream execution modules.

## 5. Generalization, precision, and empirical scope

The paper’s argument for generalization has several components. First, shape and size generalization follow from defining the keypoint in terms of local affordance geometry rather than object identity [2509.20841]. Second, the representation is claimed to achieve sub-centimeter-level accuracy because the network predicts a precise \(6\)-DoF frame rather than a coarse class label or a low-resolution heatmap [2509.20841]. Third, multi-stage tasks are naturally represented as chains of stage-wise affordance-action pairs [2509.20841]. Fourth, multi-modal behavior is represented by generating a distribution over CoMOK candidates through diffusion / score-matching [2509.20841]. Fifth, deformable-object handling follows from attaching the affordance to a controllable local patch [2509.20841].

The experimental program is intended to show that the formulation works across task types, object variations, and physical settings [2509.20841]. On a standard parallel-jaw grasp dataset, the model predicts grasp poses plus a pre-grasp frame, making grasping a special case of the formulation; in this case only the action sequence is meaningful because the manipulated object is the gripper itself [2509.20841]. In stable placement in simulation, the robot must pick and place bottles, boxes, and tripods of many shapes and sizes while avoiding obstacles; the method predicts a grasp stage, a placement stage with a stable bottom-oriented affordance, and a pre-place \(\rightarrow\) placement action chain [2509.20841].

Cable insertion demonstrates deformable-object handling. The system must jointly predict grasp and insertion actions, showing that the affordance representation can reason about local controllable patches on nonrigid objects [2509.20841]. Mug hanging on a rack emphasizes precise alignment: the model must identify the mug handle center and align it with the rack stick, with centimeter-level accuracy requirements and an action sequence that uses several frames along the stick axis [2509.20841]. The paper presents this as evidence that the formulation supports fine geometric control rather than only coarse manipulation.

The ablation logic in the excerpt is representational rather than purely metric. Using task-specific oriented keypoints is described as better than direct end-effector action because it provides object-centric semantics; using a sequence of frames improves long-horizon and multi-stage behavior; using a diffusion/score-matching generator improves multimodal prediction over deterministic regression; and jointly predicting the grasp and downstream manipulation stage is necessary for tasks such as insertion and hanging, where the grasp determines future controllability [2509.20841]. Reported failures are described as mostly due to upstream task-planning errors or downstream physical feasibility limits, not because CoMOK itself is too weak [2509.20841]. This suggests that the representation is intended as the semantic and geometric substrate of manipulation, rather than the sole determinant of execution success.

## 6. Interpretation and relation to adjacent trajectory-grounded reasoning

CoMOK can be interpreted as a structured trajectory representation over object-centric \(SE(3)\) affordances. In manipulation terms, a robot does not simply predict where the gripper goes; it predicts what object part matters, how that part is oriented, and how it should move through space over time [2509.20841]. The “chain” therefore denotes an ordered sequence of desired motions for a controllable local frame, rather than an unstructured temporal trace.

A useful point of comparison appears in "Motion-o: Trajectory-Grounded Video Reasoning" [2603.18856]. That work formalizes Spatial-Temporal-Trajectory reasoning and introduces Motion Chain of Thought, in which successive grounded observations are explicitly connected into trajectories through structured motion descriptors [2603.18856]. The domains differ—robotic manipulation in CoMOK, video reasoning in Motion-o—but the shared conceptual move is to make the linkage between observations or states first-class rather than implicit. This suggests a broader methodological pattern: trajectory structure can be represented explicitly, supervised directly, and used as a verifiable intermediate object.

That comparison also helps clarify what CoMOK is not. It is not merely a language description of intent, because its primitives are \(SE(3)\) frames and pose sequences. It is not merely a perception module, because the representation includes target motions. It is not identical to full motion planning, because execution feasibility is delegated to a downstream motion generator [2509.20841]. Nor is it restricted to rigid objects, since the paper explicitly attaches the affordance to a local patch for deformable objects [2509.20841].

Within the paper’s framing, CoMOK is best understood as a general manipulation action representation built from semantically meaningful \(6\)-DoF local object frames and their target motions [2509.20841]. Its significance lies in recasting manipulation policy output around controllable affordances rather than only robot-centric poses, with the aim of unifying diverse tasks while retaining geometric precision.

Source: https://www.emergentmind.com/topics/chain-of-moving-oriented-keypoints-comok