---
title: 'ContactPrompt: Zero-Shot Dense Hand Contact Estimation'
url: https://www.emergentmind.com/topics/contactprompt
type: topic
---

# ContactPrompt: Zero-Shot Dense Hand Contact Estimation

ContactPrompt refers to a training-free, zero-shot framework for dense hand contact estimation built on multi-modal large language models (MLLMs), as formally presented in "Training-Free Dense Hand Contact Estimation with Multi-Modal Large Language Models" [2605.05886]. By encoding 3D hand geometry as structured part-wise segmentation and vertex-grid representations and decomposing the estimation task into a multi-stage reasoning process, ContactPrompt enables precise, vertex-level hand contact labeling in hand-object interaction scenes. Remarkably, this approach surpasses supervised methods on standard benchmarks without any model-specific fine-tuning or dataset-specific training.

## 1. Problem Formulation and Technical Foundations

Dense hand contact estimation involves assigning a binary label to each vertex $v_i$ of a 3D hand mesh (usually MANO, $V=778$ vertices) describing whether it is in contact with an object. This task requires high-level semantic reasoning (e.g., understanding grasp type, object class, and occlusion) and fine-grained geometric discrimination (contact at the scale of individual finger phalanges and webspaces).

MLLMs, such as GPT-5.5, operate primarily on vision and language, lacking built-in mechanisms for explicit 3D geometric reasoning. ContactPrompt addresses these limitations by introducing:

- **Hand-part segmentation** at fine granularity ($K=103$ regions).
- **Part-wise vertex-grid representations** for encoding local structure in language-friendly grids.
- **Multi-stage, structured MLLM prompting** to decompose reasoning from global semantics to part-level and vertex-level contact.

## 2. Hand-Part Segmentation and Geometric Abstractions

ContactPrompt begins with a semantic decomposition of the hand mesh:

- **Part Set:** $\mathcal{P} = \{ p_1,\dots,p_K \}$, with each part representing a specific anatomic structure (e.g., thumb distal, palm center-distal, thenar, webspaces); $K=103$ for high granularity.
- **Segmentation Map:** $S: \{1,\dots,V\} \to \mathcal{P}$, where $S(i) = p$ assigns vertex $i$ to part $p$.
- **Prompt Construction:** The MLLM receives a base64-encoded segmentation map image, with each pixel color-coded and overlaid with a numeric part index, as well as a JSON mapping of part names and vertex membership.

This segmentation is substantially finer than previous work, and ablation studies show a $+35.2\%$ absolute improvement in F$_1$ score compared to coarser (e.g., DIGIT) partitions, as well as a $-32.4\%$ reduction in MLLM output length [2605.05886].

## 3. Part-Wise Vertex-Grid Representation

Each hand part $p$ is organized into a local 2D grid, rendering the set of vertices $\mathcal{V}_p = \{ i : S(i) = p \}$ in row-major order, following mesh topology from the distal end toward the base:

- **Grid Structure:** $\mathcal{G}_p = \{ \mathbf{g}_p^{(1)}, \dots, \mathbf{g}_p^{(R_p)} \}$, with $R_p$ rows and each row an ordered sequence.
- **Local Embedding:** Vertices projected to $\mathbb{Z}^2$ via $g_p(i) = \lfloor (v_i - \mathbf{c}_p)/\Delta \rfloor$, where $\mathbf{c}_p$ is the centroid and $\Delta$ the quantization.
- **Feature Description:** Each vertex $v_i$ embeds as $\phi(v_i) = W [v_i \| n_i] + b$ (concatenated position and normal), but zero-shot inference uses only visual (dot-and-line) grid illustrations.

Only grid shape (rows, row lengths) is transmitted; explicit coordinates are not required. Inclusion of this grid representation lifts recall by $+55.7\%$ ($+21.8\%$ in F$_1$) at minimal token cost, indicating strong efficacy for conveying geometry to MLLMs.

## 4. Multi-Stage Structured Contact Reasoning with MLLMs

ContactPrompt operationalizes dense contact inference as three chained MLLM calls:

1. **Stage 0 (Global Free-Form Reasoning):**  
   $z = f^{(0)}(I, T^{(0)})$  
   The model receives an RGB image $I$ and reasoning prompt $T^{(0)}$ and produces a concise paragraph on hand pose, viewpoint, object, occlusion, and grasp.
2. **Stage 1 (Part-Level Contact Prediction):**  
   $\hat{\mathcal{P}} = f^{(1)}(I, T^{(1)}, S_\text{part}, z)$  
   Given the image, segmentation prompt, and global context, the model outputs a set $\hat{\mathcal{P}}\subseteq\mathcal{P}$ of parts in contact (as a JSON array).
3. **Stage 2 (Dense Vertex-Level Estimation):**  
   $\widehat{\mathbf{G}} = f^{(2)}(I, T^{(2)}, S_\text{full}, z, \hat{\mathcal{P}}, \mathbf{Q}_{\hat{\mathcal{P}}})$  
   For each selected part $p$, the model fills out a $0/1$ grid (rows $\times$ columns) reflecting per-vertex contact, given a visual grid prompt and explicit specification of grid dimensions.

The final dense contact vector $\hat{c}_i$ is assembled by mapping per-part grids back to vertex-space:
$$
\hat{c}_i = \begin{cases}
(\widehat{\mathbf{G}_p})_{r,c} & \text{if } S(i)=p \text{ and } i \text{ is the } (r, c)\text{ grid cell} \\
0 & \text{if } S(i) \notin \hat{\mathcal{P}}
\end{cases}
$$

Part conditioning—where only predicted-contact parts are forwarded to Stage 2—reduces inference tokens by $-20.7\%$ and raises precision by $+10.5\%$.

## 5. Zero-Shot Inference Pipeline and Pseudocode

The decision pipeline operates entirely in zero-shot mode, with no sample-specific or per-dataset prompt adaptation:

- **Preprocessing:** Input RGB image as base64-JPEG. Generate segmentation and grid skeleton visual prompts. Assemble a JSON grid spec for each part.
- **Prompting:** Fixed system and user prompts for each stage, following specified protocols (e.g., bullet reasoning, explicit grid-filling instructions, and JSON format).
- **MLLM Calls:** Three-stage pipeline as above, with deterministic JSON output parsing.
- **Postprocessing:** Map output grids to per-vertex contact vector.

Pseudocode for inference execution is as follows:
```python
function ContactPrompt(I: RGB image, M: MANO mesh):
    S_part_img ← render_segmentation_map(M)
    S_grid_img ← render_grid_skeleton(M)
    JSON_Q ← { for each p: (R_p, row_lengths[p]) }
    # Stage 0
    T0 ← fixed free-form prompt
    z  ← MLLM.call(inputs=[I, S_part_img], prompt=T0)
    # Stage 1
    T1 ← fixed part-prediction prompt
    out1 ← MLLM.call(inputs=[I, S_part_img], prompt=T1 + z)
    P_hat ← parse_JSON(out1)["contact_parts"]
    # Stage 2
    T2 ← fixed dense-prediction prompt
    Q_sel ← JSON_Q filtered to parts in P_hat
    out2 ← MLLM.call(inputs=[I, S_part_img, S_grid_img], prompt=T2 + z + Q_sel)
    G_hat ← parse_JSON(out2)
    for i in 1…V:
        p ← S(i)
        if p ∈ P_hat:
            (r,c) ← grid_position_of_vertex(i,p)
            c[i] ← G_hat[p][r][c]
        else:
            c[i] ← 0
    return c
```
[2605.05886]

## 6. Quantitative Evaluation and Comparative Results

ContactPrompt is evaluated on the MOW hand–object benchmark (92 test samples), measuring vertex-wise precision, recall, and F$_1$:

| Method              | Precision | Recall | F$_1$  |
|---------------------|-----------|--------|--------|
| ContactPrompt       | 0.473     | 0.710  | 0.526  |
| POSA                | –         | –      | 0.101  |
| BSTRO               | –         | –      | 0.112  |
| DECO                | –         | –      | 0.197  |
| HACO                | –         | –      | 0.522  |

ContactPrompt (GPT-5.5) not only outperforms these supervised baselines in F$_1$, but also demonstrates competitive inference cost (approximately 3.6K output tokens or $\$0.11$ per sample), making it an efficient solution for dense contact estimation without any supervised training [2605.05886].

Ablation studies highlight that detailed segmentation and vertex-grid encodings are critical for performance, each yielding substantial F$_1$/recall improvements. The three-stage, part-conditioned pipeline provides superior balance compared to 1- or 2-stage ablations.

## 7. Significance and Implications

ContactPrompt demonstrates that with careful structuring of 3D geometry and a multi-step reasoning protocol, state-of-the-art dense hand contact estimation is achievable using generalist MLLMs in a training-free, zero-shot setting. This suggests the potential for MLLMs to enable high-precision, reasoning-intensive perception tasks without the need for dataset-specific model training, provided the task structure is sufficiently formalized via prompts and geometric abstraction. A plausible implication is that similar structured abstraction and prompt chaining could generalize to other geometry-intensive tasks beyond hand contact analysis [2605.05886].

Source: https://www.emergentmind.com/topics/contactprompt