---
title: Coarse-to-Fine Language-Aligned Manipulation Policy
url: https://www.emergentmind.com/topics/coarse-to-fine-language-aligned-manipulation-policy-clap
type: topic
---

# Coarse-to-Fine Language-Aligned Manipulation Policy

Searching arXiv for the specified papers and closely related work to ground the article.
I’ll look up the named arXiv records and a few directly related hierarchical language-conditioned manipulation papers.
Coarse-to-fine Language-Aligned Manipulation Policy (CLAP) is a hierarchical robotic manipulation framework in which a coarse branch predicts a language-aligned region of interest or intermediate intent, and a fine branch conditions on that output to generate precise manipulation actions. In the formulation introduced for robotic 3D manipulation, CLAP integrates **task decomposition**, **VLM fine-tuning for 3D keypoint prediction**, and **3D-aware representation** to improve precision, sample efficiency, and generalization to novel instructions and environment variations [2509.23575]. Closely related work connects the same coarse-to-fine principle to **language-guided grasp detection** via a mask-then-grasp decomposition [2512.21065] and to **Vision-Language-Action** systems via a hybrid action space that separates macro-directional intent from micro-pose alignment [2604.24921].

## 1. Definition and conceptual scope

CLAP denotes a **coarse-to-fine language-aligned manipulation policy** in which manipulation is decomposed into two stages: a coarse stage that grounds language into an actionable spatial or semantic intermediate, and a fine stage that uses that intermediate to predict precise control outputs. In the 3D manipulation formulation, the coarse module is a **VLM-based coarse task planner** that predicts **step-wise language instructions** and **language-aligned 3D keypoints**, while the fine module is a **3D-aware fine action predictor** that uses the coarse output and the selected step instruction to produce **6-DoF actions and gripper commands** [2509.23575].

The motivation is explicitly tied to the limitations of prior hierarchical and pretrained manipulation systems. The cited formulation states that hierarchical coarse-to-fine policies are **sample-efficient** because a coarse module first isolates a **region-of-interest (ROI)**, thereby reducing the visual search space, and are **precise** because the fine module operates on **zoomed, canonical projections or 3D crops with explicit geometric context**. It also identifies three recurring failure modes in prior approaches: **domain shift** between internet-scale pretraining and robot observations, **insufficient 3D grounding from 2D-only inputs**, and **conditioning on a single high-level instruction across an entire trajectory**, which hinders compositional generalization for long-horizon tasks [2509.23575].

Within the broader literature represented here, CLAP is not restricted to one action representation. In 3D manipulation it is centered on **language-aligned 3D keypoints** and ROI-gated action prediction [2509.23575]. In language-guided grasping, the same idea is realized as a **mask-then-grasp decomposition** over dense planar grasp maps [2512.21065]. In Libra-VLA, the coarse-to-fine principle is realized as a **dual-system** over a **Hybrid Action Space**, where the coarse component predicts discrete macro-directional action tokens and the fine component predicts continuous micro-pose actions [2604.24921]. This suggests that CLAP is best understood as a policy design principle rather than a single architecture.

## 2. Formal structure of the coarse-to-fine policy

In the original 3D manipulation setting, the policy objective at key-frame $t_k$ is to predict the action $a_{t_k}$ from the current observation and language:
$$
\pi(o_t, L_i) \to a_{t_k}.
$$
The dataset is written as $D = \{(\tau_i, L_i)\}$, where demonstrations are segmented by **key-frame imitation learning** into times $t_1 < \dots < t_K$ at which discrete actions such as **grasp** or **place** occur [2509.23575].

The coarse stage predicts a 3D keypoint $p_{t_k} \in \mathbb{R}^3$, and the ROI is defined as an **axis-aligned box** centered at that keypoint with side lengths $s = (s_x, s_y, s_z)$:
$$
B(p_{t_k}, s) = \{x \in \mathbb{R}^3 \mid |x_x - p_{t_k,x}| \le s_x/2,\ |x_y - p_{t_k,y}| \le s_y/2,\ |x_z - p_{t_k,z}| \le s_z/2\}.
$$
A binary gate $M_B(x)$ masks the fused point cloud or canonical-view pixels, after which cropping and zooming are applied before the fine module [2509.23575]. This explicitly formalizes the coarse stage as a language-aligned spatial restriction on subsequent action prediction.

A closely related probabilistic factorization appears in language-guided grasp detection:
$$
p(g \mid s, l) = p(M_{\text{coarse}} \mid s, l)\cdot p(g \mid M_{\text{coarse}}, s, l),
$$
where $M_{\text{coarse}}$ is a **language-conditioned segmentation mask** of the referred object or region [2512.21065]. For planar grasping, the grasp is parameterized as
$$
g = \{(x, y), \theta, w, q\},
$$
with dense grasp maps
$$
G = \{q,\cos(2\theta),\sin(2\theta),w\},
$$
and the double-angle encoding is used to avoid angular discontinuities at $\theta = \pm \pi/2$ [2512.21065].

Libra-VLA gives a third formalization, now at the action level rather than the spatial ROI level. It decomposes action into coarse and fine components:
$$
\mathbf{a}_t = \Phi(\mathbf{a}_t^c,\mathbf{a}_t^f),
$$
with a hybrid action space
$$
\mathcal{A} = \mathcal{Z} \times \mathcal{U},
$$
and policy factorization
$$
P(\mathbf{a}_t \mid \mathbf{o}_t, L) \approx P(\mathbf{a}_t^c \mid \mathbf{o}_t, L)\cdot P(\mathbf{a}_t^f \mid \mathbf{a}_t^c, \mathbf{o}_t).
$$
Here the coarse policy models discrete macro-intent and the fine policy models continuous micro-pose alignment [2604.24921]. Taken together, these formulations show that CLAP can be instantiated through **ROI gating**, **mask-then-action decomposition**, or **hybrid action factorization**, provided that language alignment is introduced in the coarse stage and preserved in the fine stage.

## 3. CLAP as language-aligned 3D keypoint prediction

The formulation in "Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints" places **language-aligned 3D keypoint prediction** at the center of CLAP [2509.23575]. The coarse task planner is implemented by **LoRA-fine-tuning Qwen2.5-VL-3B** using three supervision sources: **robot trajectories**, an **auxiliary language plan dataset**, and an **object position dataset**. Its inference is explicitly **two-round**. First, the model performs text-only planning:
$$
f_\theta(L_i) \to \mathcal{L},
$$
where $\mathcal{L} = (\ell_1,\dots,\ell_K)$ is a plan of step-wise language instructions. Second, it performs visual grounding:
$$
f_\theta(o_{t_{k-1}}, \mathcal{L}, \hat{\ell}) \to (p_{\text{obj}}, \ell_{t_k}, p_{t_k}),
$$
where $p_{\text{obj}}$ are object 3D positions or per-view pixels, $\ell_{t_k}$ is the current step instruction, and $p_{t_k}$ is the keypoint for the next action [2509.23575].

A notable aspect of this formulation is **task decomposition** into sub-plans and the use of **short-term memory** $\hat{\ell}$ to disambiguate repeated subgoals. The planner scores steps in $\mathcal{L}$ by attention between step embeddings and per-view visual tokens; tokens describing a phrase such as “blue block” select object pixels via cross-attention, and the associated pixel’s 3D coordinate becomes $p_{t_k}$ [2509.23575]. This suggests that CLAP’s coarse stage is not only spatially selective but also temporally selective, because it resolves which subgoal is currently active.

The fine-grained action predictor consumes **cropped canonical views gated by $B(p_{t_k}, s)$** and the selected step instruction $\ell_{t_k}$. Its representation is explicitly **3D-aware**: multi-view RGB-D is fused into a point cloud and re-rendered into **three orthogonal canonical views (front, left, top)** such that each pixel carries RGB, depth, and its world 3D coordinate. Back-projection is given by
$$
x_{\text{cam}} = d(u,v)K^{-1}[u,v,1]^\top,
$$
followed by
$$
x_{\text{world}} = R x_{\text{cam}} + t,
$$
and 3D position embeddings $\phi(x_{\text{world}})$ are added to tokens in the fine module [2509.23575].

The training losses make the hierarchy explicit. The coarse loss is
$$
L_{\text{coarse}} = \lambda_{\text{obj}}L_{\text{obj}} + \lambda_{\text{step}}L_{\text{step}} + \lambda_{\text{kp}}L_{\text{kp}} + \lambda_{\text{align}}L_{\text{align}},
$$
where $L_{\text{align}}$ is an InfoNCE-style language-vision alignment loss between step-instruction embeddings and visual keypoint features. The fine loss is
$$
L_{\text{fine}} = L_{\text{pose}} + \lambda_g L_{\text{grip}},
$$
with translation $L_2$, rotation geodesic loss on $SO(3)$, and binary cross-entropy for the gripper state [2509.23575]. This architecture gives CLAP a concrete meaning: the coarse stage predicts a language-aligned geometric anchor, and the fine stage predicts manipulation relative to that anchor.

## 4. LGGD as a CLAP-style language-guided grasping system

"Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulation" presents **LGGD**, and its structured description explicitly states that it connects naturally to a **Coarse-to-fine Language-Aligned Manipulation Policy (CLAP)** [2512.21065]. The policy view is written as
$$
\pi(g \mid s, l) \equiv p(g \mid s, l),
$$
and its coarse-to-fine factorization is the mask-then-grasp decomposition already noted:
$$
p(g \mid s, l) = p(M_{\text{coarse}} \mid s, l)\cdot p(g \mid M_{\text{coarse}}, s, l).
$$
Here the coarse stage predicts a **language-conditioned segmentation mask** of the referred object or region, while the fine stage predicts grasp maps and then refines both mask and grasp outputs by residual correction [2512.21065].

LGGD’s coarse language alignment is implemented through three main mechanisms. First, it uses **CLIP-derived visual and textual features**, with a **CLIP ResNet-50** image encoder and CLIP text encoder producing **word-level** and **sentence-level** embeddings. Second, it introduces a **Dual Cross Vision-Language Fusion (DCVLF)** bottleneck performing **bidirectional cross-attention** between visual tokens and word tokens, followed by fusion and refinement. Third, it reconstructs higher-resolution visual features through **hierarchical language-guided upsampling** using **FiLM** and **LMAFN**, where FiLM is defined as
$$
\text{FiLM}(x,s) = (1+\gamma)\odot x + \beta,
$$
with $(\gamma,\beta)$ generated from the sentence embedding $s = y_{\text{sentence}}$ [2512.21065].

Instruction adaptivity is pushed further in the output heads through the **Language-Conditioned Dynamic Convolution Head (LDCH)**. For sample $i$ with sentence embedding $s_i$, gating is defined as
$$
\alpha_i = \text{softmax}(\text{MLP}(s_i)) \in \mathbb{R}^K,
$$
and expert mixing as
$$
W_{i,t} = \sum_{k=1}^K \alpha_{i,k} W_t^{(k)}, \qquad
b_{i,t} = \sum_{k=1}^K \alpha_{i,k} b_t^{(k)}.
$$
The paper defines the expert-mixture formulation but does not fix $K$ [2512.21065]. The resulting coarse outputs are
$$
Y_{\text{mask}}^{\text{coarse}} \in \mathbb{R}^{B\times 1\times H\times W}, \qquad
Y_{\text{grasp}}^{\text{coarse}} \in \mathbb{R}^{B\times 4\times H\times W},
$$
which are then refined by residual modules:
$$
Y_{\text{mask}} = Y_{\text{mask}}^{\text{coarse}} + \text{Mask}_{\text{refine}}(Y_{\text{mask}}^{\text{coarse}}),
$$
$$
Y_{\text{grasp}} = Y_{\text{grasp}}^{\text{coarse}} + \text{Grasp}_{\text{refine}}(Y_{\text{grasp}}^{\text{coarse}}).
$$

The losses are likewise coarse-to-fine. LGGD uses multi-stage deep supervision:
$$
L_{\text{total}} = L_{\text{refine}} + \lambda L_{\text{coarse}}, \qquad \lambda = 0.5,
$$
with weighted BCE-with-logits for segmentation and a Smooth L1 loss for the grasp maps, masked to the referred object pixels. The structured description explicitly notes that **no additional CLIP-style contrastive alignment loss** is used; language-vision alignment is achieved through **frozen CLIP encoders** and the explicit cross-modal modules **DCVLF**, **FiLM**, and **LDCH** [2512.21065].

Within a CLAP interpretation, LGGD realizes coarse grounding through **DCVLF and LMAFN**, instruction-adaptive prediction through **LDCH**, and robustness through **residual refinement** [2512.21065]. A plausible implication is that CLAP’s coarse stage need not be a symbolic planner or an explicit 3D keypoint predictor; it can also be a dense, language-conditioned perceptual mask that constrains downstream action generation.

## 5. Hybrid-action and dual-system realizations

Libra-VLA extends the CLAP idea into a **Vision-Language-Action** setting by replacing spatial ROI prediction with a **hybrid action-space decomposition** [2604.24921]. Its motivation is the **semantic–actuation gap** in monolithic VLA policies that directly map visual-language features to high-frequency continuous motor commands. Libra-VLA states that such flat architectures force a single model to handle both semantic intent and precise motion control, producing a representational bottleneck [2604.24921].

The proposed solution is a **Coarse-to-Fine Dual-System VLA architecture** with two components. The **Semantic Planner (System 2)** is a low-frequency planner that predicts **discrete macro-directional intent tokens aligned with high-level language semantics**. The **Action Refiner (System 1)** is a high-frequency **conditional diffusion policy** that generates continuous micro-pose actions conditioned on those tokens and fine-grained geometry [2604.24921]. The planner objective is a cross-entropy loss over quantized action bins,
$$
\mathcal{L}_{plan} = \mathcal{L}_{CE}(P(\mathbf{a}_t^c), \mathbf{y}_t^{gt}),
$$
while the refiner objective is a diffusion noise-prediction loss,
$$
\mathcal{L}_{diff} = \mathbb{E}_{k,\mathbf{x}_0,\epsilon}\big[\|\epsilon - \epsilon_\theta(\mathbf{x}_k,\mathbf{F}_t^{geo},\mathbf{e}_{intent})\|^2\big].
$$
The joint loss is
$$
\mathcal{L}_{total} = \lambda_{diff}\mathcal{L}_{diff} + \lambda_{plan}\mathcal{L}_{plan}.
$$
The description explicitly notes that **no additional KL/contrastive terms are introduced** [2604.24921].

A distinctive contribution is the **granularity analysis** of the coarse stage. The paper defines a bin size $N$ for quantization of the normalized continuous action and reports that performance follows an **inverted-U curve** with respect to $N$, peaking at the **“Libra point”** $N=10$, where the learning difficulty is balanced between the two subsystems; at that setting the average success on LIBERO is **97.2%** [2604.24921]. This provides a concrete empirical account of what “coarse” and “fine” should mean in a CLAP-like decomposition: neither too little structure nor too much discretization.

Libra-VLA also uses the hierarchy to implement **asynchronous execution**. The planner predicts an expanded macro-horizon
$$
L_{macro} = M \times H_{chunk},
$$
stores predicted coarse tokens in a FIFO **Intent Buffer** $\mathcal{Q}$, and the refiner consumes slices of length $H_{chunk}$ at control frequency. On an RTX 4090, the baseline monolithic latency is **220 ms per step**, while Libra-VLA achieves **122 ms** at $M=2$, **112 ms** at $M=3$, **107 ms** at $M=4$, and **104 ms** at $M=5$, with average success remaining **95.3–97.2%** [2604.24921]. This suggests a second interpretation of CLAP: beyond improving alignment and generalization, hierarchy can also be used to amortize expensive semantic inference.

## 6. Empirical performance, generalization, and limitations

The original CLAP paper reports results on **GemBench**, a benchmark for generalization. Training uses **16 tasks with 31 variations**, where baselines typically use **100 trajectories per variation**, while CLAP uses **20 trajectories per variation (1/5)**. Evaluation spans four generalization levels: **L1 placements**, **L2 novel rigid objects**, **L3 novel articulated objects**, and **L4 long-horizon tasks**, with **20 episodes per variation per seed and 5 seeds** [2509.23575]. CLAP achieves **62.0% average success** versus **BridgeVLA 50.0%**, a **12% absolute improvement**, with per-level averages of **83.9%**, **83.2%**, **49.6%**, and **31.4%** for L1–L4, respectively [2509.23575]. In real robot experiments using only **10 demonstrations per task** across four training tasks, CLAP averages **77.3%** across conditions versus **RVT2 22.5%**, a **54.8% higher** result [2509.23575].

LGGD provides an alternative empirical profile in language-guided grasping. On **OCID-VLG (Multiple-Split)**, LGGD reports **J@1 = 85.4**, **J@5 = 90.2**, **IoU = 83.1**, **Pr@50 = 98.0**, **Pr@60 = 93.7**, **Pr@70 = 91.3**, **Pr@80 = 66.7**, and **Pr@90 = 26.1**; on **Novel-Instances**, **J@1 = 57.6**, **J@5 = 64.1**, **IoU = 67.6**; and on **Novel-Classes**, **J@1 = 46.0**, **J@5 = 54.4**, **IoU = 63.1** [2512.21065]. On **Grasp-Anything++**, LGGD reports **seen = 0.59** and **unseen = 0.31**, outperforming the baselines listed in the structured description [2512.21065]. Its ablations show a progression from **IoU 31.30, J@1 9.73** for a simple concatenation baseline to **IoU 83.14, J@1 85.36** for the full model, with **DCVLF** contributing the largest jump [2512.21065].

Libra-VLA reports **97.2% average success** on LIBERO, including **Spatial 98.6%**, **Object 99.4%**, **Goal 98.0%**, and **Long 92.8%**, and **79.5% zero-shot average** on LIBERO-Plus, with robustness figures across **Camera, Robot, Language, Light, Background, Noise, and Layout** perturbations [2604.24921]. Its ablations show **97.2%** for the full model, **95.1%** for Libra-Refinement, **88.3%** for Libra-Base, and **87.0%** for Libra-VE, while dynamic curriculum outperforms both teacher forcing and no teacher forcing [2604.24921].

The limitations stated across these works are consistent. The original CLAP formulation notes dependence on **depth quality**, difficulty with **continuous tasks** such as wiping, possible **error cascades**, and **language domain shift** under unusual phrasings or ambiguous instructions [2509.23575]. LGGD identifies reliance on **CLIP embeddings**, challenges from **ambiguity and compositional language**, difficulty under **dense clutter and occlusion**, the absence of an **explicit alignment loss**, and scaling issues due to model size [2512.21065]. Libra-VLA notes attenuation as the horizon expansion factor grows, the use of **uniform per-dimension binning** for the discrete coarse vocabulary, and the absence of **explicit language–action contrastive objectives** [2604.24921]. Taken together, these limitations indicate that CLAP-style hierarchy improves grounding and control, but does not remove the need for stronger ambiguity handling, better uncertainty estimation, and more robust geometric sensing.

## 7. Significance and extensions

Across these papers, CLAP consistently denotes a design in which language first constrains **where**, **what**, or **which macro-intent** the system should pursue, and only then conditions **how** to execute the manipulation. In the 3D keypoint formulation, the coarse output is a **language-aligned 3D keypoint** and ROI [2509.23575]. In LGGD, it is a **language-conditioned segmentation mask** [2512.21065]. In Libra-VLA, it is a sequence of **discrete macro-directional action tokens** [2604.24921]. This suggests that the defining feature of CLAP is not the exact intermediate representation, but the preservation of language alignment through a hierarchical bottleneck that reduces the burden on fine control.

The structured description accompanying LGGD explicitly outlines how the same philosophy could be extended beyond planar grasping. It proposes **pick-and-place** by adding a language-conditioned placement head, **tool use/part-aware manipulation** through language-conditioned affordance maps, **sequencing primitives** such as approach, pregrasp, grasp, lift, transport, place, and release, and **6-DoF** extensions using depth or multi-view inputs with full $SE(3)$ poses [2512.21065]. The original CLAP paper already embodies several of these elements through key-frame decomposition, multi-view RGB-D, and 6-DoF action prediction [2509.23575]. Libra-VLA, in turn, demonstrates that the hierarchy can be moved into the action domain itself, where discrete coarse intent and continuous refinement support both long-horizon execution and reduced inference latency [2604.24921].

A plausible implication is that CLAP functions as a unifying abstraction across language-conditioned manipulation systems that would otherwise appear heterogeneous: 2D grasp detection, 3D keypoint-guided manipulation, and hybrid-action VLA all instantiate the same principle of **coarse language grounding followed by fine action generation**. Under that interpretation, CLAP is a general policy pattern for addressing the semantic–geometric and semantic–actuation gaps in robotic manipulation.

Source: https://www.emergentmind.com/topics/coarse-to-fine-language-aligned-manipulation-policy-clap