FunHSI: Functional 3D Human-Scene Interaction

• FunHSI is a training-free, functionality-driven framework that synthesizes a 3D SMPL-X human in a scene by reasoning over object functionality and contact relations.
• It decomposes the generation process into functionality-aware contact reasoning, body initialization, and geometry-aware optimization to ensure physically plausible interactions.
• Empirical evaluations show that FunHSI achieves superior functional contact accuracy and collision avoidance compared to existing methods.

Searching arXiv for the specified FunHSI paper and closely related HSI generation work. FunHSI is a training-free, functionality-driven framework for generating a 3D human body interacting with a 3D scene from an open-vocabulary task prompt and a small set of posed RGB-D views. Its defining objective is not merely to synthesize a plausible human placement, but to produce an interaction that is simultaneously functionally correct for the requested task and physically plausible with respect to scene geometry. In the formulation introduced in "Open-Vocabulary Functional 3D Human-Scene Interaction Generation" (Liu et al., 28 Jan 2026), FunHSI targets both coarse interactions such as "sitting on a sofa" and fine-grained functional interactions such as "increasing the room temperature," where success depends on reasoning about object functionality, contact location, body-part assignment, and scene support structure.

1. Problem formulation and scope

FunHSI addresses functional 3D human-scene interaction generation under open-vocabulary prompting (Liu et al., 28 Jan 2026). The input consists of a set of RGB-D images of a scene, corresponding camera parameters or poses, and a task prompt in natural language. The output is a 3D SMPL-X human body placed in the reconstructed scene so that it appears to perform the requested task.

The problem is substantially harder than generic human placement. The task prompt may be a high-level goal such as "increase the room temperature," "open the window," or "dial a number." Such prompts do not explicitly specify the interacted object, the exact body part to use, or the interaction style. The method must infer which object or object part is relevant, what body parts should contact it, and what supporting contacts are needed, such as feet on the floor or buttocks on a chair (Liu et al., 28 Jan 2026).

A central distinction in the paper is between general and functional interaction prompts. General prompts explicitly specify an action-object pair, such as "sit on the chair" or "stand in front of the window." Functional prompts specify only the goal, such as "open the door" or "adjust the temperature," and therefore require explicit reasoning about object functionality rather than object category alone (Liu et al., 28 Jan 2026). The paper argues that existing training-free methods are mainly effective for general interactions and often lack explicit reasoning over object functionality and corresponding human-scene contact, which leads to contact with the wrong object, missing fine-grained contact on functional parts, or physically implausible body placement.

FunHSI is explicitly single-step. It generates one static pose that accomplishes a task, rather than a multi-step sequence such as opening a door and then walking through it (Liu et al., 28 Jan 2026). This defines both its scope and one of its main limitations.

2. Architectural organization

The FunHSI pipeline is organized into three major modules: functionality-aware contact reasoning, functionality-aware body initialization, and optimization-based body refinement (Liu et al., 28 Jan 2026). The framework is described as training-free because it does not train a new interaction generation model on paired human-scene interaction data; instead, it composes pretrained modules and foundation models in zero-shot form.

The first module infers task-relevant functional scene elements from the prompt, localizes and segments them in the RGB views, reconstructs them in 3D, and produces a contact graph encoding which human body parts should contact which scene elements. The second module synthesizes a human in the image performing the task, estimates an initial 3D SMPL-X body pose and hand pose, and refines the contact graph if the synthesized image implies a different left/right-hand assignment. The third module places the initialized human into the 3D scene and refines translation, orientation, pose, and contacts through a two-stage optimization procedure that enforces collision avoidance and contact consistency (Liu et al., 28 Jan 2026).

This decomposition is important because the paper does not treat scene understanding, body initialization, and geometric refinement as a single undifferentiated generation problem. Instead, it separates semantic reasoning from pose estimation and from geometry-aware optimization. A plausible implication is that this modularization is what enables FunHSI to support open-vocabulary prompts without any paired task-specific training data.

3. Functionality-aware scene reasoning and contact graphs

The core novelty of FunHSI lies in functionality-aware contact reasoning (Liu et al., 28 Jan 2026). Given a task such as "adjust the temperature," the method first identifies candidate functional elements in the RGB views. The implementation uses Gemini-2.5-Flash for inferring candidate functional elements conditioned on the task description; the supplementary implementation details additionally state that Gemini Robotics-ER-1.5 is used for bounding box localization and SAM-ViT-H for segmentation (Liu et al., 28 Jan 2026). Once masks are available, the method back-projects each RGB-D frame using known camera parameters, fuses the views into a scene point cloud, and back-projects the 2D masks to obtain corresponding 3D masks for functional elements and supporting elements such as floors and chairs.

FunHSI represents high-level interaction structure with a contact graph

$\mathcal{G} = (\mathcal{V}, \mathcal{E}), \quad \mathcal{V} = \mathcal{V}_{\text{body} \cup \mathcal{V}_{\text{scene},$

where $\mathcal{V}_{\text{body}}$ is a predefined set of human body-part nodes and $\mathcal{V}_{\text{scene}}$ is a set of scene-element nodes, including both functional elements and supporting elements (Liu et al., 28 Jan 2026). Each edge $(b,o)\in\mathcal{E}$ denotes a contact relation between body part $b$ and scene element $o$.

The body representation is semantically structured. According to the supplementary material, the SMPL-X body is decomposed into 15 coarse parts: head, left upper arm, right upper arm, left forearm, right forearm, left hand, right hand, back, buttocks, left thigh, right thigh, left calf, right calf, left foot, and right foot. For finer hand interactions, each hand is subdivided into palm and five fingers (Liu et al., 28 Jan 2026). This fine-grained decomposition is necessary for tasks involving knobs, dials, switches, or buttons.

The graph itself is generated by an LLM, such as GPT-4o or Gemini, conditioned on the task description, detected functional elements, the predefined body-part set, and structured instructions encouraging complete and human-like interaction (Liu et al., 28 Jan 2026). This graph serves as an explicit intermediate representation linking natural-language task intent to geometric optimization targets. The paper presents it as the mechanism by which semantic task grounding is transformed into concrete contact constraints.

4. Body initialization, inpainting, and pose estimation

FunHSI does not optimize body configuration from scratch. Instead, it first synthesizes a human into an RGB image using a contact-aware prompting strategy with Gemini (Liu et al., 28 Jan 2026). The inpainting prompt includes the original human-free image, the task description, the inferred contact graph, and detected object bounding boxes. This is intended to bias the image generator to place body parts near relevant object regions.

To reduce hallucinations, the method uses an iterative generator-evaluator loop inspired by "LLMs as optimizers" (Liu et al., 28 Jan 2026). A separate Gemini model acts as a critic and checks whether the synthesized human performs the specified task, whether contact with functional elements is plausible, and whether irrelevant or nonexistent objects are introduced. If any condition fails, the generator is prompted again. In practice, 3–4 iterations are usually sufficient, and the supplementary details state that at most four iterations are used.

From the inpainted image, FunHSI estimates SMPL-X parameters

$\mathcal{M}(\beta, r, \varphi, \theta),$

where $\beta \in \mathbb{R}^{10}$ are body shape parameters, $r \in \mathbb{R}^3$ is root translation, $\varphi \in \mathbb{R}^3$ is root orientation, and $\mathcal{V}_{\text{body}}$0 denotes pose parameters with $\mathcal{V}_{\text{body}}$1 for body pose and $\mathcal{V}_{\text{body}}$2 for hand pose (Liu et al., 28 Jan 2026). SMPL-X outputs a mesh with 10,475 vertices. CameraHMR is used for global translation, root orientation, and body pose; WiLoR is used for hand pose; if hands are occluded, the hand pose is initialized to the default relaxed hand pose of SMPL-X (Liu et al., 28 Jan 2026).

A practical complication is left-right inconsistency between LLM reasoning and image synthesis. To handle this, FunHSI projects the left and right wrist joints into 2D and compares their distances to the target object box center $\mathcal{V}_{\text{body}}$3: $\mathcal{V}_{\text{body}}$4 where $\mathcal{V}_{\text{body}}$5 is the 3D-to-2D projection operator (Liu et al., 28 Jan 2026). If the thresholded comparison indicates the opposite hand is closer, the hand-related nodes in the contact graph are swapped, producing a refined graph $\mathcal{V}_{\text{body}}$6. The ablation study shows that removing contact graph refinement degrades supporting-contact quality, even though functional distance changes only slightly (Liu et al., 28 Jan 2026).

5. Optimization objective and two-stage refinement

The final stage of FunHSI is a geometry-aware two-stage optimization over the SMPL-X body in the reconstructed scene (Liu et al., 28 Jan 2026). The method uses VolumetricSMPL to compute a body signed distance field

$\mathcal{V}_{\text{body}}$7

Because both the SMPL-X mesh function $\mathcal{V}_{\text{body}}$8 and the body SDF $\mathcal{V}_{\text{body}}$9 are differentiable, inverse-kinematics-style optimization can be carried out by backpropagation.

The collision loss over a scene point cloud $\mathcal{V}_{\text{scene}}$0 is

$\mathcal{V}_{\text{scene}}$1

which penalizes scene points lying inside the body (Liu et al., 28 Jan 2026). The contact loss defined by the refined contact graph $\mathcal{V}_{\text{scene}}$2 is

$\mathcal{V}_{\text{scene}}$3

Here $\mathcal{V}_{\text{scene}}$4 denotes the body vertices of part $\mathcal{V}_{\text{scene}}$5, and $\mathcal{V}_{\text{scene}}$6 denotes the scene points of element $\mathcal{V}_{\text{scene}}$7 (Liu et al., 28 Jan 2026). This is a single-sided Chamfer-style term that pulls each relevant body part toward its target scene element. For foot contacts, the loss is computed only on toe/heel-near vertices, allowing finer foot-ground configurations.

To regularize pose realism, the method adds a VPoser prior

$\mathcal{V}_{\text{scene}}$8

which penalizes large deviations from the learned body-pose manifold (Liu et al., 28 Jan 2026). The total objective is

$\mathcal{V}_{\text{scene}}$9

although the paper does not provide explicit numerical values for the $(b,o)\in\mathcal{E}$0 weights.

The optimization proceeds in two stages rather than jointly. Stage 1 optimizes translation $(b,o)\in\mathcal{E}$1, global body orientation around the gravity axis $(b,o)\in\mathcal{E}$2, and arm pose parameters $(b,o)\in\mathcal{E}$3, while freezing remaining pose parameters and non-gravity components of root orientation (Liu et al., 28 Jan 2026). This stage uses only $(b,o)\in\mathcal{E}$4 and $(b,o)\in\mathcal{E}$5. Stage 2 then optimizes translation $(b,o)\in\mathcal{E}$6 and full body pose $(b,o)\in\mathcal{E}$7, with emphasis on ankle joints, while freezing shape $(b,o)\in\mathcal{E}$8 and non-gravity components of $(b,o)\in\mathcal{E}$9. This stage uses $b$0, $b$1, and $b$2, and a smaller learning rate

$b$3

The supplementary implementation details specify AdamW, $b$4 iterations and $b$5 for Stage 1, and $b$6 iterations with $b$7 for Stage 2 (Liu et al., 28 Jan 2026).

The paper is explicit about what is not present: there is no separate reprojection loss in the final refinement objective, no explicit hand-object alignment loss beyond the graph-guided contact loss, and no separately named functionality-consistency loss. Functionality is enforced indirectly through functional-element grounding, LLM contact-graph reasoning, and graph-guided contact optimization (Liu et al., 28 Jan 2026).

6. Evaluation, empirical behavior, and limitations

FunHSI is evaluated on a curated benchmark derived from SceneFun3D, using 30 indoor scenes spanning living rooms, bedrooms, kitchens, and bathrooms, each with three RGB-D views and mask annotations for key affordance elements (Liu et al., 28 Jan 2026). The benchmark contains 60 curated tasks: 30 functional and 30 general. The paper also evaluates qualitative generalization on real-world city scenes captured with an iPhone 14 Pro Max, using GeoCalib for camera intrinsics and gravity direction and MapAnything for camera poses, depth maps, and 3D scene point clouds (Liu et al., 28 Jan 2026).

The evaluation metrics are Semantic Consistency Score (SCS), Non-Collision Score (NCS), Non-Functional Contact Distance (N-FCD), and Functional Contact Distance (FCD) (Liu et al., 28 Jan 2026). SCS measures image-text cosine similarity with CLIP ViT-B/32 over rendered views; NCS measures absence of penetration between the SMPL-X mesh and the scene point cloud; N-FCD measures Chamfer distance between the body and supporting scene elements such as floor or chair; FCD measures Chamfer distance between the task-relevant functional element and interacting human hands.

On general HSI, FunHSI obtains the best NCS and N-FCD among the compared methods, with SCS remaining competitive (Liu et al., 28 Jan 2026). On functional HSI, its main gains are functional: compared with adapted GenZI* and GenHSI*, FunHSI achieves the best FCD and the best N-FCD while remaining essentially comparable in NCS. The reported functional HSI results are:

Method	SCS ↑	NCS ↑	N-FCD ↓	FCD ↓
GenZI*	0.2501	0.9823	0.2027	0.6262
GenHSI*	0.2607	0.9925	0.5415	0.4199
FunHSI	0.2540	0.9917	0.1837	0.2968

These numbers support the paper’s claim that FunHSI is particularly advantageous when the task requires precise hand contact with small functional parts (Liu et al., 28 Jan 2026). A user study on Amazon Mechanical Turk reports 71.1% overall preference over the GenHSI baseline, including 76.8% preference on functional HSI and 66.0% on general HSI (Liu et al., 28 Jan 2026).

The ablations indicate that iterative body refinement is the most important component. Removing iterative refinement drops NCS from $b$8 to $b$9, worsens N-FCD from $o$0 to $o$1, and worsens FCD from $o$2 to $o$3 (Liu et al., 28 Jan 2026). Removing body/hand pose estimation also substantially degrades FCD, from $o$4 to $o$5, showing that initialization quality is critical. Oracle functional detection improves FCD to $o$6, indicating that upstream functional-element detection remains a bottleneck (Liu et al., 28 Jan 2026).

The paper’s limitations are clear. FunHSI handles only single-step functional interactions. It depends heavily on external foundation models, including Gemini-2.5-Flash, Gemini Robotics-ER-1.5, GPT-4o, SAM-ViT-H, CameraHMR, WiLoR, and VolumetricSMPL (Liu et al., 28 Jan 2026). Upstream detection errors propagate to final interaction quality, prompt interpretation may be ambiguous, and inpainting can hallucinate scene changes, which is why the critic loop is necessary. Runtime is not reported, but the method uses multiple external model calls, optimization over 400 + 200 iterations, and inpainting that may repeat up to four times on a single NVIDIA A6000 GPU, so it is likely nontrivial in computational cost (Liu et al., 28 Jan 2026).

A related direction is represented by "FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework" (Mu et al., 1 Sep 2025), which addresses long-horizon 4D human-scene interaction through a graph-based multi-agent system rather than single-step functionality-aware pose generation. This suggests a broader emerging split in the literature between static functional interaction generation, exemplified by FunHSI (Liu et al., 28 Jan 2026), and long-horizon task-oriented 4D synthesis, exemplified by FantasyHSI (Mu et al., 1 Sep 2025).

7. Significance and position within human-scene interaction research

FunHSI advances open-vocabulary human-scene interaction generation by shifting emphasis from generic semantic plausibility to explicit functional correctness (Liu et al., 28 Jan 2026). Its key contribution is not a new learned generative prior, but a structured zero-shot pipeline that combines foundation-model-based functional reasoning with explicit contact-graph constraints and geometry-aware body refinement.

This position is important because many previous training-free methods can generate coarse interactions such as sitting or standing near objects, yet lack explicit mechanisms for selecting the correct functional subpart, assigning the correct body part, and ensuring scene-grounded contact. FunHSI addresses exactly that gap through functional-element grounding, contact-graph representation, and task-constrained optimization (Liu et al., 28 Jan 2026).

The framework also clarifies a methodological principle: for open-vocabulary interaction synthesis, language reasoning alone is insufficient unless it is converted into explicit geometric constraints. In FunHSI, the contact graph is that conversion layer. This suggests a broader design pattern for future HSI systems: semantic task inference should be represented in a form that can directly supervise collision-aware and support-aware optimization.

At the same time, the paper does not claim to solve long-horizon planning, dynamic interaction, or multi-step manipulation. A plausible implication is that FunHSI is best understood as a strong zero-shot baseline for static, functionally grounded interaction generation rather than a complete general-purpose embodied interaction system.