---
title: Marking Open-vocabulary Keypoint Affordances (MOKA)
url: https://www.emergentmind.com/topics/marking-open-vocabulary-keypoint-affordances-moka
type: topic
---

# Marking Open-vocabulary Keypoint Affordances (MOKA)

Marking Open-vocabulary Keypoint Affordances (MOKA) is a methodology for robotic manipulation in open-world environments that leverages the semantic and commonsense reasoning abilities of large-scale vision-language models (VLMs) by reducing physical affordance prediction to a problem of selecting discrete keypoint candidates in a mark-based visual prompting framework. The approach enables robots to interpret diverse, free-form language instructions and physically interact with previously unseen objects and tasks by constructing a compact, semantic, point-based representation of affordances in images and grounding them as end-effector poses for manipulation [2403.03174]. MOKA operates without end-to-end policy training, instead delegating high-level grounding to pre-trained VLMs and employing downstream distillation for efficient real-time deployment. Related frameworks, such as AFFORD2ACT, advance this line by jointly optimizing vision-language and action pipelines for scalable and generic keypoint-based robotic policies [2510.01433].

## 1. Problem Formulation and Affordance Representation

MOKA assumes an open-world robotic manipulation scenario where, at each timestep $t$, the robot receives a perceptual state $s_t$ consisting of an RGB–D image $I_t$ and proprioceptive measurements, along with a free-form natural language instruction $L$ (e.g., “Swipe the snack package off the table, but first move the eyeglasses to their case.”). The target is to generate a low-level control sequence $u = (u_0, u_1, \dots, u_T)$ that implements the described behavior.

The workflow decomposes into two principal mappings:
1. **Affordance prediction**: An affordance function $f_a$ predicts a set of keypoint candidates $\mathbf{P} = \{p_i\}$ (2D pixel locations) and associated semantic affordance labels $\mathbf{A} = \{a_i\}$:
   $$
   (\mathbf{P}, \mathbf{A}) = f_a(I, L).
   $$
   Types of keypoints include $x_\text{grasp}$ (grasp location), $x_\text{function}$ (for tool-object interactions), $x_\text{target}$ (where to act upon or place), and free-space waypoints $x_\text{pre}, x_\text{post}$ for trajectory preconditions.
2. **Control generation**: A controller $g$ lifts these 2D points to 3D poses $x_i\in SE(3)$ and synthesizes continuous SE(3) trajectories:
   $$
   u = g(\mathbf{P}, \mathbf{A}),
   $$
   with the end-to-end mapping:
   $$
   u = (g \circ f_a)(I, L).
   $$

This two-level structure explicitly decouples visual-linguistic affordance inference from physical actuation, enabling modular integration of large-scale pre-trained models [2403.03174].

## 2. Mark-Based Visual Prompting and Keypoint Selection

Generating reliable continuous spatial coordinates with zero-shot VLMs is unstable. MOKA reformulates coordinate selection as a discrete visual multiple-choice problem via mark-based prompting:
- **Candidate-mark generation**: Using segmentation (e.g. GroundedSAM), $K$ contour points and the centroid are densely sampled on the object mask, overlaid as colored dots, each assigned an index (e.g., $P_1, \dots, P_{K+1}$ for the grasped object).
- **Spatial grid for free-space points**: The workspace is discretized into an $m \times n$ grid (typically $5 \times 5$), labeled with chessboard indices (e.g., $a1$–$e5$), representing candidates for pre/post waypoints.
- **Hierarchical prompting**: A high-level prompt decomposes complex instructions into subtasks (specifying objects, actions, directions). For each subtask, the VLM receives: the annotated image, dictionary of roles, brief definitions of keypoint/waypoint concepts, and strictly enforced JSON output format. The VLM outputs the indices of selected marks for each affordance.
- **VLM output distribution**:
   $$
   P(m|I, L) = \text{softmax}_m\, s(m;I,L)
   $$
   where $s(m;I,L)$ is the unnormalized log-probability (cross-attention) score from the VLM on candidate mark $m$ [2403.03174].

This explicit visual prompting bypasses the need for direct coordinate regression while harnessing open-world concept coverage from VLMs.

## 3. Integration with Vision-Language Models, In-Context Learning, and Policy Distillation

MOKA exploits both zero-shot and few-shot capabilities of VLMs (e.g., GPT-4V) through structured prompting:
- **Zero-shot**: Prompts comprise only task/subtask and annotated candidates.
- **In-context learning**: Enhances VLM accuracy by prepending 2–3 prior annotated (image, marks, JSON) pairs to the prompt before inference.
- **Policy distillation**: While MOKA does not require reinforcement learning, successful VLM-guided rollouts ($D = \{(s_t, u_t)\}$) supervise a "student" policy $\pi_\theta$ (as in the Octo transformer-diffusion architecture), using an imitation loss:
  $$
  \mathcal{L}(\theta) = \mathbb{E}_{(s,u) \in D} \sum_t \|u_t - \pi_\theta(s_t, L)\|_2^2
  $$
This yields a trainable actor for real-time, prompt-free control, amortizing VLM inference and reducing system latency [2403.03174].

## 4. Implementation and System Components

Key implementation details include:
- **Hardware**: 7-DoF Franka Emika manipulator, 2F-85 gripper, dual ZED 2.0 RGBD cameras, and a wrist camera for distillation data.
- **Perception**: Object segmentation uses GroundedSAM within GroundingDINO bounds; keypoint sampling selects nine contour points plus centroid.
- **Prompt management**: Maximum 20 keypoint marks and 25 spatial grid cells per prompt; JSON output is strictly parsed, with malformed outputs retried.
- **Student policy**: Multimodal transformer encoder with tokenized RGB and text embeddings; 3-layer MLP diffusion decoder.
- **Optimization**: Learning rate $3 \times 10^{-4}$, batch size 256, weight decay 0.01, $2 \times 10^5$ training steps [2403.03174].

Related frameworks, such as AFFORD2ACT, modularize affordance-guided filtering, category-level keypoint construction, and lightweight transformer-gating policies. The AFFORD2ACT recipe details the joint training of vision-text affordance masks, semantic keypoint clustering, and compact transformer-gated control policies with less than 20 keypoints (38-D attention-bottleneck state), achieving 25 ms per-frame inference efficiency on a single NVIDIA RTX 3090 [2510.01433].

## 5. Empirical Evaluation and Benchmarks

MOKA is benchmarked on four two-stage tabletop manipulation domains:
1. Table Wiping: eyeglass relocation, sweeping debris with broom
2. Watch Cleaning: watch placement, ultrasonic cleaner activation
3. Gift Preparation: fill box, place perfume
4. Laptop Packing: unplug cable, close lid

Comparison with Code-as-Policies and VoxPoser yields the following per-subtask success rates (extract):

| Method             | TW-1 | TW-2 | WC-1 | WC-2 | GP-1 | GP-2 | LP-1 | LP-2 |
|--------------------|------|------|------|------|------|------|------|------|
| Code-as-Policies   | 0.7  | 0.6  | 0.6  | 1.0  | 1.0  | 0.7  | 0.4  | 0.8  |
| VoxPoser           | 0.6  | 0.0  | 0.6  | 0.8  | 1.0  | 0.6  | 0.5  | 0.8  |
| MOKA Zero-Shot     | 0.6  | 0.6  | 0.7  | 1.0  | 1.0  | 0.7  | 0.5  | 0.8  |
| MOKA Distilled     | 1.0  | 0.7  | 0.8  | 0.8  | 1.0  | 0.7  | 1.0  | 1.0  |
| MOKA In-Context    | 0.9  | 0.9  | 0.9  | 1.0  | 1.0  | 0.9  | 1.0  | 0.9  |

A plausible implication is that distillation and in-context learning respectively yield robust improvements in subtask-level success, reducing the share of VLM-driven reasoning errors by over 50% compared to unprompted zero-shot operation. Robustness to object geometry, pose variation, and linguistic paraphrasing is demonstrated qualitatively [2403.03174].

AFFORD2ACT achieves an 82% success rate on unseen objects, categories, backgrounds, and distractors on comparable manipulation settings [2510.01433].

## 6. Limitations and Outlook

Limitations of the MOKA approach include:
- **2D spatial reasoning**: Current VLMs operate on 2D image marks, constraining the manipulation to 4-DoF end-effector trajectories, which is insufficient for tasks requiring precise 6-DoF or bimanual coordination.
- **API and real-time constraints**: Use of proprietary VLM APIs (e.g., GPT-4V) introduces inference latency and operational cost.
- **Generalization**: While mark-based prompting and keypoint clustering mitigate dataset and instruction bias, fine-grained 3D affordance reasoning and dynamic contact manipulation remain challenging.

Future research directions include learning 6-DoF semantic keypoints, developing trainable visual prompt templates, and integrating physics-informed simulation or scene flow for contact-rich and dynamic interactions. Combining open-vocabulary mark-based keypoint systems with end-to-end trainable policies may further enhance generalization and autonomy [2403.03174][2510.01433].

Source: https://www.emergentmind.com/topics/marking-open-vocabulary-keypoint-affordances-moka