TeHOR: 3D Human-Object Reconstruction

• TeHOR is a single-image reconstruction framework that jointly estimates textured 3D humans and objects using text-guided semantic alignment.
• It integrates text descriptions with appearance supervision, contact supervision, and collision regularization for robust human-object interaction reconstruction.
• The method leverages 3D Gaussian representations and diffusion-based optimization to achieve state-of-the-art performance on benchmarks like Open3DHOI and BEHAVE.

Searching arXiv for the TeHOR paper and closely related HOI reconstruction work to ground the article in current literature. arxiv_search: {"query":"(Nam et al., 23 Feb 2026) TeHOR Text-Guided 3D Human and Object Reconstruction with Textures", "max_results": 5} arxiv_search: {"query":"HOI-Gaussian arXiv human object interaction 3D Gaussian", "max_results": 5} TeHOR, short for Text-guided 3D Human and Object Reconstruction with Textures, is a single-image reconstruction framework that jointly estimates a textured 3D human and a textured 3D object. Its defining premise is that 3D human-object interaction reconstruction should not depend only on physical contact or on local geometric proximity, because many interactions are non-contact and many semantically important cues are carried by appearance, pose, gaze, and object orientation rather than by touching surfaces alone. TeHOR therefore combines text descriptions of human-object interactions, appearance-aware rendering supervision, contact supervision, and collision regularization in a unified HOI optimization procedure, and is reported to achieve state-of-the-art performance on the evaluated benchmarks (Nam et al., 23 Feb 2026).

1. Definition, naming, and scope

TeHOR is introduced specifically for joint reconstruction of 3D human and object from a single image. The method outputs a reconstructed textured 3D human and textured 3D object, with the optimization designed to preserve both geometric plausibility and semantic coherence of the interaction (Nam et al., 23 Feb 2026).

The name is easy to confuse with several unrelated arXiv systems. In the CTA Observatory software stack, the relevant system is the Transients Handler (TH) rather than “TeHOR” (Collins et al., 28 Aug 2025). In planetary science, THOR denotes a global circulation model (Mendonça et al., 2016). In collider phenomenology, HATHOR is a framework for inclusive top-quark cross sections (Kant et al., 2014). In a simulation-only cancer-therapy paper, the relevant term is TOR rather than TeHOR (Mello et al., 11 Oct 2025). Within 3D vision and HOI reconstruction, however, TeHOR refers specifically to the text-guided reconstruction framework introduced in “TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures” (Nam et al., 23 Feb 2026).

A central aspect of the method is its deliberate extension beyond contact-driven reasoning. The paper frames earlier HOI reconstruction systems as limited by two tendencies: dependence on physical contact information and over-reliance on local geometric proximity instead of holistic interaction context. TeHOR is proposed as a direct response to those limitations (Nam et al., 23 Feb 2026).

2. Problem formulation and motivation

The underlying problem is the reconstruction of human-object interactions from a single RGB image. Earlier methods are described as working well when contact provides a strong geometric cue, as in interactions such as holding, sitting on, or pushing. The difficulty arises for non-contact cases such as gazing at a monitor, pointing at something, reaching to catch a frisbee, or orienting the body toward an object with no visible contact (Nam et al., 23 Feb 2026).

The paper’s argument is that contact cannot serve as the dominant supervisory signal in such cases. When there is little or no contact, the optimization has little strong geometric evidence to enforce. Even when contact exists, concentrating mainly on local touching regions can produce globally implausible reconstructions, for example an object with an incorrect orientation or a human gaze direction that contradicts the interaction semantics (Nam et al., 23 Feb 2026).

TeHOR’s core response is text-guided semantic alignment. The system extracts text descriptions from the input image and uses them as high-level cues for the interaction. At the same time, it uses the rendered appearance of the current 3D human-object configuration, rather than relying only on sparse geometric relations, so that optimization can exploit cues such as human gaze direction, body orientation, object orientation, and broader scene context. This suggests a shift from “contact-only” reasoning toward a supervision regime in which semantics and appearance jointly constrain the reconstructed 3D scene (Nam et al., 23 Feb 2026).

3. Pipeline and scene representation

TeHOR is a two-stage pipeline. In the reconstruction stage, the method constructs four initial elements from a single image: a text description of the interaction, an initial 3D human, an initial 3D object, and a 2D background (Nam et al., 23 Feb 2026).

This stage uses several off-the-shelf modules. GPT-4 produces two captions: a holistic prompt $P_{\text{holistic}}$, which describes the overall HOI semantics, and a contact prompt $P_{\text{contact}}$, which identifies which body parts are physically touching the object. The human is isolated by removing the object with SmartEraser, then segmented and reconstructed with LHM. Human pose and shape are estimated with Multi-HMR. The object is isolated using SmartEraser and SAM, reconstructed as a textured mesh with InstantMesh, and then converted to 3D Gaussians. Its pose $(R,t,s)$ is estimated by aligning the reconstructed object with a depth map from ZoeDepth. The background is extracted by removing both human and object from the input image (Nam et al., 23 Feb 2026).

In the HOI optimization stage, the initial human and object are jointly refined using reconstruction supervision on the original view, text-guided appearance supervision on novel views, contact supervision on the predicted contacting body parts, and collision regularization. The optimization runs for 200 steps, updating both human and object Gaussian parameters (Nam et al., 23 Feb 2026).

The method represents both entities as sets of 3D Gaussians:

• $\Phi_{\text{h}}$ for the human
• $\Phi_{\text{o}}$ for the object

Each Gaussian carries 3D position, scale, rotation, opacity, and appearance features. Human Gaussians are anchored in canonical SMPL-X body space and articulated by linear blend skinning driven by the body pose $\theta$. The human model is parameterized by Gaussian attributes $\phi_{\text{h}}$, pose $\theta$, and shape $\beta$. The paper follows ExAvatar-style handling of hands and face versus other body parts for skinning weights. Object Gaussians are defined in canonical object space and transformed by rotation $R$, translation $P_{\text{contact}}$0, and scale $P_{\text{contact}}$1. Final rendering is performed with Mip-Splatting-style rendering (Nam et al., 23 Feb 2026).

The input is a single RGB image of a person interacting with an object, and the output is a reconstructed textured 3D human and textured 3D object. For evaluation and downstream use, the final Gaussian representation is converted into meshes. Because Gaussians and base meshes may disagree in contact regions, the method applies a local shift in the contact area so that mesh vertices remain geometrically consistent with the optimized Gaussian contacts (Nam et al., 23 Feb 2026).

4. Objective function and text-guided optimization

The most distinctive component of TeHOR is the use of a pretrained text-conditioned diffusion model, specifically Stable Diffusion v2.1, to guide optimization of the 3D reconstruction. The system renders the current 3D human-object configuration from random viewpoints, composites the result over the background, and applies score-based guidance conditioned on the holistic caption $P_{\text{contact}}$2 (Nam et al., 23 Feb 2026).

The paper writes the appearance/text alignment gradient as

$P_{\text{contact}}$3

Here, $P_{\text{contact}}$4 denotes the optimized 3D Gaussian parameters, $P_{\text{contact}}$5 is the diffusion timestep or noise level, $P_{\text{contact}}$6 is the noisy rendered image at timestep $P_{\text{contact}}$7, $P_{\text{contact}}$8 is the added Gaussian noise, $P_{\text{contact}}$9 is the diffusion model’s predicted noise conditioned on the prompt, and $(R,t,s)$0 is a timestep-dependent weighting term. The paper identifies this as score distillation sampling. Because supervision is applied over multiple viewpoints, the final 3D solution is required to remain text-consistent beyond the input camera view (Nam et al., 23 Feb 2026).

The full objective is

$(R,t,s)$1

The reconstruction loss $(R,t,s)$2 is a front-view fitting loss composed of two MSE terms: an RGB reconstruction loss and a silhouette/mask reconstruction loss. The appearance loss $(R,t,s)$3 is the diffusion-based text-guided term. The contact loss is defined as

$(R,t,s)$4

where $(R,t,s)$5 is the set of human Gaussian center points belonging to body parts indicated by $(R,t,s)$6, $(R,t,s)$7 is the set of object points, and the threshold is $(R,t,s)$8. The collision loss discourages human-object interpenetration (Nam et al., 23 Feb 2026).

Implementation is in PyTorch and uses Adam. The reported learning rates are $(R,t,s)$9 for object pose $\Phi_{\text{h}}$0, $\Phi_{\text{h}}$1 for human pose $\Phi_{\text{h}}$2, and $\Phi_{\text{h}}$3 for human and object Gaussian attributes. The classifier-free guidance scale is 15.0, diffusion timesteps are sampled in $\Phi_{\text{h}}$4, gradient clipping is applied at norm 1.0, and the method runs on a single RTX 8000 with an average optimization time of 134 seconds per sample. Rendering uses both global viewpoints and zoomed upper-body views, reflecting the importance of head and hand regions for HOI semantics (Nam et al., 23 Feb 2026).

5. Datasets, evaluation protocol, and quantitative performance

TeHOR is evaluated on Open3DHOI and BEHAVE. Open3DHOI is described as an open-vocabulary, in-the-wild HOI evaluation set with over 2.5K images and 133 object categories, and it is used for evaluation only. BEHAVE is described as an indoor HOI dataset with around 4.5K test images (Nam et al., 23 Feb 2026).

On Open3DHOI, TeHOR achieves:

• CD$\Phi_{\text{h}}$5 = 4.941
• CD$\Phi_{\text{h}}$6 = 16.701
• Contact = 0.412
• Collision = 0.047

The paper compares these results against HOI-Gaussian, InteractVLM, PHOSA, and LEMON+PICO. The reported best prior baseline numbers listed in the paper include HOI-Gaussian with CD$\Phi_{\text{h}}$7=5.111, CD$\Phi_{\text{h}}$8=19.363, Contact=0.348, and Collision=0.070. The front-view Gaussian evaluation variant TeHOR$\Phi_{\text{h}}$9 is also reported, with CD$\Phi_{\text{o}}$0=4.403, CD$\Phi_{\text{o}}$1=16.697, and Collision=0.045 (Nam et al., 23 Feb 2026).

On BEHAVE, TeHOR achieves:

• CD$\Phi_{\text{o}}$2 = 5.615
• CD$\Phi_{\text{o}}$3 = 17.339
• Contact = 0.412
• Collision = 0.016

The paper states that it is best or tied best across metrics on this dataset (Nam et al., 23 Feb 2026).

A particularly important evaluation is the non-contact subset of Open3DHOI, where samples with physical contact are removed. This probes whether a method can reason from gaze, body posture, and object orientation rather than from touching surfaces. In that setting, the paper reports the following object-CD values:

• PHOSA: 65.537
• LEMON+PICO: 33.073
• InteractVLM: 46.819
• HOI-Gaussian: 25.374
• TeHOR: 17.546

In the same non-contact evaluation, Collision drops to 0.005, the best among the compared methods. The paper presents this as evidence that TeHOR is not simply a stronger contact estimator, but a method with improved non-contact semantic reasoning (Nam et al., 23 Feb 2026).

6. Ablations, limitations, and significance

The ablation studies are used to isolate the contribution of the framework’s main design choices. When text prompts are removed from the appearance loss, performance degrades: after optimization, the paper reports CD$\Phi_{\text{o}}$4=20.348, Contact=0.374, and Collision=0.052. With text prompts, the corresponding values are CD$\Phi_{\text{o}}$5=16.701, Contact=0.412, and Collision=0.047. The paper interprets this as direct evidence that text-guided alignment contributes materially to recovery of global interaction semantics (Nam et al., 23 Feb 2026).

The appearance loss is also tested directly. Removing it leads to implausible reconstructions, and replacing the diffusion-based appearance loss with CLIP loss is reported as better than omitting appearance loss but still worse than the full method. The stated reason is that CLIP compresses text-image alignment into a single global embedding, which is too coarse for dense HOI reasoning, whereas diffusion-based supervision provides more spatially detailed gradients (Nam et al., 23 Feb 2026).

Representation matters as well. Replacing 3D Gaussians with meshes worsens performance substantially:

• human CD increases from 4.941 to 5.153
• object CD increases from 16.701 to 25.162
• contact decreases from 0.412 to 0.308

The paper attributes this to the flexibility and high-fidelity appearance modeling of Gaussians, and to their suitability for dense rendering supervision (Nam et al., 23 Feb 2026).

Scene context remains relevant beyond the human and object alone. Removing the 2D background degrades performance, increasing object CD to 18.196 and reducing contact to 0.389. Omitting the contact prompt also worsens results, although the paper emphasizes that contact functions mainly as a complement rather than as a sufficient basis for reconstruction (Nam et al., 23 Feb 2026).

The paper also states several limitations. Fine local details can be missed, particularly small accessories or subtle local surface deformations. Because the input is a single image, ambiguous occlusions or missing regions can still lead to incomplete geometry. The method is not designed for temporal consistency, so extension to video would require stable geometry and texture across frames. Finally, Gaussian-to-mesh conversion can introduce minor inconsistencies, with Gaussian renderings sometimes appearing slightly larger or blurrier than mesh renderings (Nam et al., 23 Feb 2026).

Taken together, these results place TeHOR within a broader shift in HOI reconstruction from purely geometric fitting toward methods that combine geometry with semantic alignment and appearance-aware optimization. A plausible implication is that the method is most significant not merely because it improves benchmark numbers, but because it treats non-contact interaction semantics as a first-class reconstruction signal rather than as an afterthought to contact estimation (Nam et al., 23 Feb 2026).