ContactPrompt: Zero-Shot Dense Hand Contact Estimation

• ContactPrompt is a training-free framework for dense hand contact estimation that uses multi-modal LLMs for vertex-level labeling.
• It employs detailed hand-part segmentation and vertex-grid representations to achieve significant improvements in F1 score and recall without fine-tuning.
• The multi-stage prompting pipeline decomposes reasoning from global context to part-wise predictions, ensuring efficient and precise hand contact inference.

ContactPrompt refers to a training-free, zero-shot framework for dense hand contact estimation built on multi-modal LLMs (MLLMs), as formally presented in "Training-Free Dense Hand Contact Estimation with Multi-Modal LLMs" (Jung et al., 7 May 2026). By encoding 3D hand geometry as structured part-wise segmentation and vertex-grid representations and decomposing the estimation task into a multi-stage reasoning process, ContactPrompt enables precise, vertex-level hand contact labeling in hand-object interaction scenes. Remarkably, this approach surpasses supervised methods on standard benchmarks without any model-specific fine-tuning or dataset-specific training.

1. Problem Formulation and Technical Foundations

Dense hand contact estimation involves assigning a binary label to each vertex $v_i$ of a 3D hand mesh (usually MANO, $V=778$ vertices) describing whether it is in contact with an object. This task requires high-level semantic reasoning (e.g., understanding grasp type, object class, and occlusion) and fine-grained geometric discrimination (contact at the scale of individual finger phalanges and webspaces).

MLLMs, such as GPT-5.5, operate primarily on vision and language, lacking built-in mechanisms for explicit 3D geometric reasoning. ContactPrompt addresses these limitations by introducing:

• Hand-part segmentation at fine granularity ($K=103$ regions).
• Part-wise vertex-grid representations for encoding local structure in language-friendly grids.
• Multi-stage, structured MLLM prompting to decompose reasoning from global semantics to part-level and vertex-level contact.

2. Hand-Part Segmentation and Geometric Abstractions

ContactPrompt begins with a semantic decomposition of the hand mesh:

• Part Set: $\mathcal{P} = \{ p_1,\dots,p_K \}$, with each part representing a specific anatomic structure (e.g., thumb distal, palm center-distal, thenar, webspaces); $K=103$ for high granularity.
• Segmentation Map: $S: \{1,\dots,V\} \to \mathcal{P}$, where $S(i) = p$ assigns vertex $i$ to part $p$.
• Prompt Construction: The MLLM receives a base64-encoded segmentation map image, with each pixel color-coded and overlaid with a numeric part index, as well as a JSON mapping of part names and vertex membership.

This segmentation is substantially finer than previous work, and ablation studies show a $+35.2\%$ absolute improvement in F$V=778$0 score compared to coarser (e.g., DIGIT) partitions, as well as a $V=778$1 reduction in MLLM output length (Jung et al., 7 May 2026).

3. Part-Wise Vertex-Grid Representation

Each hand part $V=778$2 is organized into a local 2D grid, rendering the set of vertices $V=778$3 in row-major order, following mesh topology from the distal end toward the base:

• Grid Structure: $V=778$4, with $V=778$5 rows and each row an ordered sequence.
• Local Embedding: Vertices projected to $V=778$6 via $V=778$7, where $V=778$8 is the centroid and $V=778$9 the quantization.
• Feature Description: Each vertex $K=103$0 embeds as $K=103$1 (concatenated position and normal), but zero-shot inference uses only visual (dot-and-line) grid illustrations.

Only grid shape (rows, row lengths) is transmitted; explicit coordinates are not required. Inclusion of this grid representation lifts recall by $K=103$2 ($K=103$3 in F$K=103$4) at minimal token cost, indicating strong efficacy for conveying geometry to MLLMs.

4. Multi-Stage Structured Contact Reasoning with MLLMs

ContactPrompt operationalizes dense contact inference as three chained MLLM calls:

1. Stage 0 (Global Free-Form Reasoning):

$K=103$5 The model receives an RGB image $K=103$6 and reasoning prompt $K=103$7 and produces a concise paragraph on hand pose, viewpoint, object, occlusion, and grasp.

2. Stage 1 (Part-Level Contact Prediction):

$K=103$8 Given the image, segmentation prompt, and global context, the model outputs a set $K=103$9 of parts in contact (as a JSON array).

3. Stage 2 (Dense Vertex-Level Estimation):

$\mathcal{P} = \{ p_1,\dots,p_K \}$0 For each selected part $\mathcal{P} = \{ p_1,\dots,p_K \}$1, the model fills out a $\mathcal{P} = \{ p_1,\dots,p_K \}$2 grid (rows $\mathcal{P} = \{ p_1,\dots,p_K \}$3 columns) reflecting per-vertex contact, given a visual grid prompt and explicit specification of grid dimensions.

The final dense contact vector $\mathcal{P} = \{ p_1,\dots,p_K \}$4 is assembled by mapping per-part grids back to vertex-space:

$\mathcal{P} = \{ p_1,\dots,p_K \}$5

Part conditioning—where only predicted-contact parts are forwarded to Stage 2—reduces inference tokens by $\mathcal{P} = \{ p_1,\dots,p_K \}$6 and raises precision by $\mathcal{P} = \{ p_1,\dots,p_K \}$7.

5. Zero-Shot Inference Pipeline and Pseudocode

The decision pipeline operates entirely in zero-shot mode, with no sample-specific or per-dataset prompt adaptation:

• Preprocessing: Input RGB image as base64-JPEG. Generate segmentation and grid skeleton visual prompts. Assemble a JSON grid spec for each part.
• Prompting: Fixed system and user prompts for each stage, following specified protocols (e.g., bullet reasoning, explicit grid-filling instructions, and JSON format).
• MLLM Calls: Three-stage pipeline as above, with deterministic JSON output parsing.
• Postprocessing: Map output grids to per-vertex contact vector.

Pseudocode for inference execution is as follows: $K=103$3 (Jung et al., 7 May 2026)

6. Quantitative Evaluation and Comparative Results

ContactPrompt is evaluated on the MOW hand–object benchmark (92 test samples), measuring vertex-wise precision, recall, and F$\mathcal{P} = \{ p_1,\dots,p_K \}$8:

Method	Precision	Recall	F$\mathcal{P} = \{ p_1,\dots,p_K \}$9
ContactPrompt	0.473	0.710	0.526
POSA	–	–	0.101
BSTRO	–	–	0.112
DECO	–	–	0.197
HACO	–	–	0.522

ContactPrompt (GPT-5.5) not only outperforms these supervised baselines in F$K=103$0, but also demonstrates competitive inference cost (approximately 3.6K output tokens or $K=103$10.11$$per sample), making it an efficient solution for dense contact estimation without any supervised training (<a href="/papers/2605.05886" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Jung et al., 7 May 2026</a>).</p> <p>Ablation studies highlight that detailed segmentation and vertex-grid encodings are critical for performance, each yielding substantial F$$K=103$2/recall improvements. The three-stage, part-conditioned pipeline provides superior balance compared to 1- or 2-stage ablations.

7. Significance and Implications

ContactPrompt demonstrates that with careful structuring of 3D geometry and a multi-step reasoning protocol, state-of-the-art dense hand contact estimation is achievable using generalist MLLMs in a training-free, zero-shot setting. This suggests the potential for MLLMs to enable high-precision, reasoning-intensive perception tasks without the need for dataset-specific model training, provided the task structure is sufficiently formalized via prompts and geometric abstraction. A plausible implication is that similar structured abstraction and prompt chaining could generalize to other geometry-intensive tasks beyond hand contact analysis (Jung et al., 7 May 2026).