Coarse-to-Fine Language-Aligned Manipulation Policy

• The paper introduces CLAP, a framework that decomposes robotic manipulation into a coarse stage for language-guided region selection and a fine stage for precise 3D action prediction.
• It integrates task decomposition, VLM fine-tuning for 3D keypoint prediction, and 3D-aware representations to enhance precision, sample efficiency, and generalization.
• The approach unifies various methods from grasp detection to hybrid vision-language-action systems, addressing challenges like domain shift and spatial search complexity.

Searching arXiv for the specified papers and closely related work to ground the article. I’ll look up the named arXiv records and a few directly related hierarchical language-conditioned manipulation papers. Coarse-to-fine Language-Aligned Manipulation Policy (CLAP) is a hierarchical robotic manipulation framework in which a coarse branch predicts a language-aligned region of interest or intermediate intent, and a fine branch conditions on that output to generate precise manipulation actions. In the formulation introduced for robotic 3D manipulation, CLAP integrates task decomposition, VLM fine-tuning for 3D keypoint prediction, and 3D-aware representation to improve precision, sample efficiency, and generalization to novel instructions and environment variations (Hu et al., 28 Sep 2025). Closely related work connects the same coarse-to-fine principle to language-guided grasp detection via a mask-then-grasp decomposition (Jiang et al., 24 Dec 2025) and to Vision-Language-Action systems via a hybrid action space that separates macro-directional intent from micro-pose alignment (Wei et al., 27 Apr 2026).

1. Definition and conceptual scope

CLAP denotes a coarse-to-fine language-aligned manipulation policy in which manipulation is decomposed into two stages: a coarse stage that grounds language into an actionable spatial or semantic intermediate, and a fine stage that uses that intermediate to predict precise control outputs. In the 3D manipulation formulation, the coarse module is a VLM-based coarse task planner that predicts step-wise language instructions and language-aligned 3D keypoints, while the fine module is a 3D-aware fine action predictor that uses the coarse output and the selected step instruction to produce 6-DoF actions and gripper commands (Hu et al., 28 Sep 2025).

The motivation is explicitly tied to the limitations of prior hierarchical and pretrained manipulation systems. The cited formulation states that hierarchical coarse-to-fine policies are sample-efficient because a coarse module first isolates a region-of-interest (ROI), thereby reducing the visual search space, and are precise because the fine module operates on zoomed, canonical projections or 3D crops with explicit geometric context. It also identifies three recurring failure modes in prior approaches: domain shift between internet-scale pretraining and robot observations, insufficient 3D grounding from 2D-only inputs, and conditioning on a single high-level instruction across an entire trajectory, which hinders compositional generalization for long-horizon tasks (Hu et al., 28 Sep 2025).

Within the broader literature represented here, CLAP is not restricted to one action representation. In 3D manipulation it is centered on language-aligned 3D keypoints and ROI-gated action prediction (Hu et al., 28 Sep 2025). In language-guided grasping, the same idea is realized as a mask-then-grasp decomposition over dense planar grasp maps (Jiang et al., 24 Dec 2025). In Libra-VLA, the coarse-to-fine principle is realized as a dual-system over a Hybrid Action Space, where the coarse component predicts discrete macro-directional action tokens and the fine component predicts continuous micro-pose actions (Wei et al., 27 Apr 2026). This suggests that CLAP is best understood as a policy design principle rather than a single architecture.

2. Formal structure of the coarse-to-fine policy

In the original 3D manipulation setting, the policy objective at key-frame $t_k$ is to predict the action $a_{t_k}$ from the current observation and language:

$\pi(o_t, L_i) \to a_{t_k}.$

The dataset is written as $D = \{(\tau_i, L_i)\}$, where demonstrations are segmented by key-frame imitation learning into times $t_1 < \dots < t_K$ at which discrete actions such as grasp or place occur (Hu et al., 28 Sep 2025).

The coarse stage predicts a 3D keypoint $p_{t_k} \in \mathbb{R}^3$, and the ROI is defined as an axis-aligned box centered at that keypoint with side lengths $s = (s_x, s_y, s_z)$:

$$B(p_{t_k}, s) = \{x \in \mathbb{R}^3 \mid |x_x - p_{t_k,x}| \le s_x/2,\ |x_y - p_{t_k,y}| \le s_y/2,\ |x_z - p_{t_k,z}| \le s_z/2\}.$$

A binary gate $M_B(x)$ masks the fused point cloud or canonical-view pixels, after which cropping and zooming are applied before the fine module (Hu et al., 28 Sep 2025). This explicitly formalizes the coarse stage as a language-aligned spatial restriction on subsequent action prediction.

A closely related probabilistic factorization appears in language-guided grasp detection:

$$p(g \mid s, l) = p(M_{\text{coarse}} \mid s, l)\cdot p(g \mid M_{\text{coarse}}, s, l),$$

where $a_{t_k}$0 is a language-conditioned segmentation mask of the referred object or region (Jiang et al., 24 Dec 2025). For planar grasping, the grasp is parameterized as

$a_{t_k}$1

with dense grasp maps

$a_{t_k}$2

and the double-angle encoding is used to avoid angular discontinuities at $a_{t_k}$3 (Jiang et al., 24 Dec 2025).

Libra-VLA gives a third formalization, now at the action level rather than the spatial ROI level. It decomposes action into coarse and fine components:

$a_{t_k}$4

with a hybrid action space

$a_{t_k}$5

and policy factorization

$a_{t_k}$6

Here the coarse policy models discrete macro-intent and the fine policy models continuous micro-pose alignment (Wei et al., 27 Apr 2026). Taken together, these formulations show that CLAP can be instantiated through ROI gating, mask-then-action decomposition, or hybrid action factorization, provided that language alignment is introduced in the coarse stage and preserved in the fine stage.

3. CLAP as language-aligned 3D keypoint prediction

The formulation in "Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints" places language-aligned 3D keypoint prediction at the center of CLAP (Hu et al., 28 Sep 2025). The coarse task planner is implemented by LoRA-fine-tuning Qwen2.5-VL-3B using three supervision sources: robot trajectories, an auxiliary language plan dataset, and an object position dataset. Its inference is explicitly two-round. First, the model performs text-only planning:

$a_{t_k}$7

where $a_{t_k}$8 is a plan of step-wise language instructions. Second, it performs visual grounding:

$a_{t_k}$9

where $\pi(o_t, L_i) \to a_{t_k}.$0 are object 3D positions or per-view pixels, $\pi(o_t, L_i) \to a_{t_k}.$1 is the current step instruction, and $\pi(o_t, L_i) \to a_{t_k}.$2 is the keypoint for the next action (Hu et al., 28 Sep 2025).

A notable aspect of this formulation is task decomposition into sub-plans and the use of short-term memory $\pi(o_t, L_i) \to a_{t_k}.$3 to disambiguate repeated subgoals. The planner scores steps in $\pi(o_t, L_i) \to a_{t_k}.$4 by attention between step embeddings and per-view visual tokens; tokens describing a phrase such as “blue block” select object pixels via cross-attention, and the associated pixel’s 3D coordinate becomes $\pi(o_t, L_i) \to a_{t_k}.$5 (Hu et al., 28 Sep 2025). This suggests that CLAP’s coarse stage is not only spatially selective but also temporally selective, because it resolves which subgoal is currently active.

The fine-grained action predictor consumes cropped canonical views gated by $\pi(o_t, L_i) \to a_{t_k}.$6 and the selected step instruction $\pi(o_t, L_i) \to a_{t_k}.$7. Its representation is explicitly 3D-aware: multi-view RGB-D is fused into a point cloud and re-rendered into three orthogonal canonical views (front, left, top) such that each pixel carries RGB, depth, and its world 3D coordinate. Back-projection is given by

$\pi(o_t, L_i) \to a_{t_k}.$8

followed by

$\pi(o_t, L_i) \to a_{t_k}.$9

and 3D position embeddings $D = \{(\tau_i, L_i)\}$0 are added to tokens in the fine module (Hu et al., 28 Sep 2025).

The training losses make the hierarchy explicit. The coarse loss is

$D = \{(\tau_i, L_i)\}$1

where $D = \{(\tau_i, L_i)\}$2 is an InfoNCE-style language-vision alignment loss between step-instruction embeddings and visual keypoint features. The fine loss is

$D = \{(\tau_i, L_i)\}$3

with translation $D = \{(\tau_i, L_i)\}$4, rotation geodesic loss on $D = \{(\tau_i, L_i)\}$5, and binary cross-entropy for the gripper state (Hu et al., 28 Sep 2025). This architecture gives CLAP a concrete meaning: the coarse stage predicts a language-aligned geometric anchor, and the fine stage predicts manipulation relative to that anchor.

4. LGGD as a CLAP-style language-guided grasping system

"Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulation" presents LGGD, and its structured description explicitly states that it connects naturally to a Coarse-to-fine Language-Aligned Manipulation Policy (CLAP) (Jiang et al., 24 Dec 2025). The policy view is written as

$D = \{(\tau_i, L_i)\}$6

and its coarse-to-fine factorization is the mask-then-grasp decomposition already noted:

$D = \{(\tau_i, L_i)\}$7

Here the coarse stage predicts a language-conditioned segmentation mask of the referred object or region, while the fine stage predicts grasp maps and then refines both mask and grasp outputs by residual correction (Jiang et al., 24 Dec 2025).

LGGD’s coarse language alignment is implemented through three main mechanisms. First, it uses CLIP-derived visual and textual features, with a CLIP ResNet-50 image encoder and CLIP text encoder producing word-level and sentence-level embeddings. Second, it introduces a Dual Cross Vision-Language Fusion (DCVLF) bottleneck performing bidirectional cross-attention between visual tokens and word tokens, followed by fusion and refinement. Third, it reconstructs higher-resolution visual features through hierarchical language-guided upsampling using FiLM and LMAFN, where FiLM is defined as

$D = \{(\tau_i, L_i)\}$8

with $D = \{(\tau_i, L_i)\}$9 generated from the sentence embedding $t_1 < \dots < t_K$0 (Jiang et al., 24 Dec 2025).

Instruction adaptivity is pushed further in the output heads through the Language-Conditioned Dynamic Convolution Head (LDCH). For sample $t_1 < \dots < t_K$1 with sentence embedding $t_1 < \dots < t_K$2, gating is defined as

$t_1 < \dots < t_K$3

and expert mixing as

$t_1 < \dots < t_K$4

The paper defines the expert-mixture formulation but does not fix $t_1 < \dots < t_K$5 (Jiang et al., 24 Dec 2025). The resulting coarse outputs are

$t_1 < \dots < t_K$6

which are then refined by residual modules:

$t_1 < \dots < t_K$7

$t_1 < \dots < t_K$8

The losses are likewise coarse-to-fine. LGGD uses multi-stage deep supervision:

$t_1 < \dots < t_K$9

with weighted BCE-with-logits for segmentation and a Smooth L1 loss for the grasp maps, masked to the referred object pixels. The structured description explicitly notes that no additional CLIP-style contrastive alignment loss is used; language-vision alignment is achieved through frozen CLIP encoders and the explicit cross-modal modules DCVLF, FiLM, and LDCH (Jiang et al., 24 Dec 2025).

Within a CLAP interpretation, LGGD realizes coarse grounding through DCVLF and LMAFN, instruction-adaptive prediction through LDCH, and robustness through residual refinement (Jiang et al., 24 Dec 2025). A plausible implication is that CLAP’s coarse stage need not be a symbolic planner or an explicit 3D keypoint predictor; it can also be a dense, language-conditioned perceptual mask that constrains downstream action generation.

5. Hybrid-action and dual-system realizations

Libra-VLA extends the CLAP idea into a Vision-Language-Action setting by replacing spatial ROI prediction with a hybrid action-space decomposition (Wei et al., 27 Apr 2026). Its motivation is the semantic–actuation gap in monolithic VLA policies that directly map visual-language features to high-frequency continuous motor commands. Libra-VLA states that such flat architectures force a single model to handle both semantic intent and precise motion control, producing a representational bottleneck (Wei et al., 27 Apr 2026).

The proposed solution is a Coarse-to-Fine Dual-System VLA architecture with two components. The Semantic Planner (System 2) is a low-frequency planner that predicts discrete macro-directional intent tokens aligned with high-level language semantics. The Action Refiner (System 1) is a high-frequency conditional diffusion policy that generates continuous micro-pose actions conditioned on those tokens and fine-grained geometry (Wei et al., 27 Apr 2026). The planner objective is a cross-entropy loss over quantized action bins,

$p_{t_k} \in \mathbb{R}^3$0

while the refiner objective is a diffusion noise-prediction loss,

$p_{t_k} \in \mathbb{R}^3$1

The joint loss is

$p_{t_k} \in \mathbb{R}^3$2

The description explicitly notes that no additional KL/contrastive terms are introduced (Wei et al., 27 Apr 2026).

A distinctive contribution is the granularity analysis of the coarse stage. The paper defines a bin size $p_{t_k} \in \mathbb{R}^3$3 for quantization of the normalized continuous action and reports that performance follows an inverted-U curve with respect to $p_{t_k} \in \mathbb{R}^3$4, peaking at the “Libra point” $p_{t_k} \in \mathbb{R}^3$5, where the learning difficulty is balanced between the two subsystems; at that setting the average success on LIBERO is 97.2% (Wei et al., 27 Apr 2026). This provides a concrete empirical account of what “coarse” and “fine” should mean in a CLAP-like decomposition: neither too little structure nor too much discretization.

Libra-VLA also uses the hierarchy to implement asynchronous execution. The planner predicts an expanded macro-horizon

$p_{t_k} \in \mathbb{R}^3$6

stores predicted coarse tokens in a FIFO Intent Buffer $p_{t_k} \in \mathbb{R}^3$7, and the refiner consumes slices of length $p_{t_k} \in \mathbb{R}^3$8 at control frequency. On an RTX 4090, the baseline monolithic latency is 220 ms per step, while Libra-VLA achieves 122 ms at $p_{t_k} \in \mathbb{R}^3$9, 112 ms at $s = (s_x, s_y, s_z)$0, 107 ms at $s = (s_x, s_y, s_z)$1, and 104 ms at $s = (s_x, s_y, s_z)$2, with average success remaining 95.3–97.2% (Wei et al., 27 Apr 2026). This suggests a second interpretation of CLAP: beyond improving alignment and generalization, hierarchy can also be used to amortize expensive semantic inference.

6. Empirical performance, generalization, and limitations

The original CLAP paper reports results on GemBench, a benchmark for generalization. Training uses 16 tasks with 31 variations, where baselines typically use 100 trajectories per variation, while CLAP uses 20 trajectories per variation (1/5). Evaluation spans four generalization levels: L1 placements, L2 novel rigid objects, L3 novel articulated objects, and L4 long-horizon tasks, with 20 episodes per variation per seed and 5 seeds (Hu et al., 28 Sep 2025). CLAP achieves 62.0% average success versus BridgeVLA 50.0%, a 12% absolute improvement, with per-level averages of 83.9%, 83.2%, 49.6%, and 31.4% for L1–L4, respectively (Hu et al., 28 Sep 2025). In real robot experiments using only 10 demonstrations per task across four training tasks, CLAP averages 77.3% across conditions versus RVT2 22.5%, a 54.8% higher result (Hu et al., 28 Sep 2025).

LGGD provides an alternative empirical profile in language-guided grasping. On OCID-VLG (Multiple-Split), LGGD reports J@1 = 85.4, J@5 = 90.2, IoU = 83.1, Pr@50 = 98.0, Pr@60 = 93.7, Pr@70 = 91.3, Pr@80 = 66.7, and Pr@90 = 26.1; on Novel-Instances, J@1 = 57.6, J@5 = 64.1, IoU = 67.6; and on Novel-Classes, J@1 = 46.0, J@5 = 54.4, IoU = 63.1 (Jiang et al., 24 Dec 2025). On Grasp-Anything++, LGGD reports seen = 0.59 and unseen = 0.31, outperforming the baselines listed in the structured description (Jiang et al., 24 Dec 2025). Its ablations show a progression from IoU 31.30, J@1 9.73 for a simple concatenation baseline to IoU 83.14, J@1 85.36 for the full model, with DCVLF contributing the largest jump (Jiang et al., 24 Dec 2025).

Libra-VLA reports 97.2% average success on LIBERO, including Spatial 98.6%, Object 99.4%, Goal 98.0%, and Long 92.8%, and 79.5% zero-shot average on LIBERO-Plus, with robustness figures across Camera, Robot, Language, Light, Background, Noise, and Layout perturbations (Wei et al., 27 Apr 2026). Its ablations show 97.2% for the full model, 95.1% for Libra-Refinement, 88.3% for Libra-Base, and 87.0% for Libra-VE, while dynamic curriculum outperforms both teacher forcing and no teacher forcing (Wei et al., 27 Apr 2026).

The limitations stated across these works are consistent. The original CLAP formulation notes dependence on depth quality, difficulty with continuous tasks such as wiping, possible error cascades, and language domain shift under unusual phrasings or ambiguous instructions (Hu et al., 28 Sep 2025). LGGD identifies reliance on CLIP embeddings, challenges from ambiguity and compositional language, difficulty under dense clutter and occlusion, the absence of an explicit alignment loss, and scaling issues due to model size (Jiang et al., 24 Dec 2025). Libra-VLA notes attenuation as the horizon expansion factor grows, the use of uniform per-dimension binning for the discrete coarse vocabulary, and the absence of explicit language–action contrastive objectives (Wei et al., 27 Apr 2026). Taken together, these limitations indicate that CLAP-style hierarchy improves grounding and control, but does not remove the need for stronger ambiguity handling, better uncertainty estimation, and more robust geometric sensing.

7. Significance and extensions

Across these papers, CLAP consistently denotes a design in which language first constrains where, what, or which macro-intent the system should pursue, and only then conditions how to execute the manipulation. In the 3D keypoint formulation, the coarse output is a language-aligned 3D keypoint and ROI (Hu et al., 28 Sep 2025). In LGGD, it is a language-conditioned segmentation mask (Jiang et al., 24 Dec 2025). In Libra-VLA, it is a sequence of discrete macro-directional action tokens (Wei et al., 27 Apr 2026). This suggests that the defining feature of CLAP is not the exact intermediate representation, but the preservation of language alignment through a hierarchical bottleneck that reduces the burden on fine control.

The structured description accompanying LGGD explicitly outlines how the same philosophy could be extended beyond planar grasping. It proposes pick-and-place by adding a language-conditioned placement head, tool use/part-aware manipulation through language-conditioned affordance maps, sequencing primitives such as approach, pregrasp, grasp, lift, transport, place, and release, and 6-DoF extensions using depth or multi-view inputs with full $s = (s_x, s_y, s_z)$3 poses (Jiang et al., 24 Dec 2025). The original CLAP paper already embodies several of these elements through key-frame decomposition, multi-view RGB-D, and 6-DoF action prediction (Hu et al., 28 Sep 2025). Libra-VLA, in turn, demonstrates that the hierarchy can be moved into the action domain itself, where discrete coarse intent and continuous refinement support both long-horizon execution and reduced inference latency (Wei et al., 27 Apr 2026).

A plausible implication is that CLAP functions as a unifying abstraction across language-conditioned manipulation systems that would otherwise appear heterogeneous: 2D grasp detection, 3D keypoint-guided manipulation, and hybrid-action VLA all instantiate the same principle of coarse language grounding followed by fine action generation. Under that interpretation, CLAP is a general policy pattern for addressing the semantic–geometric and semantic–actuation gaps in robotic manipulation.