---
title: 'TeHOR: 3D Human-Object Reconstruction'
url: https://www.emergentmind.com/topics/tehor
type: topic
---

# TeHOR: 3D Human-Object Reconstruction

Searching arXiv for the TeHOR paper and closely related HOI reconstruction work to ground the article in current literature.
arxiv_search: {"query":"2602.19679 TeHOR Text-Guided 3D Human and Object Reconstruction with Textures", "max_results": 5}
arxiv_search: {"query":"HOI-Gaussian arXiv human object interaction 3D Gaussian", "max_results": 5}
TeHOR, short for **Text-guided 3D Human and Object Reconstruction with Textures**, is a **single-image reconstruction framework** that jointly estimates a **textured 3D human** and a **textured 3D object**. Its defining premise is that 3D human-object interaction reconstruction should not depend only on **physical contact** or on **local geometric proximity**, because many interactions are **non-contact** and many semantically important cues are carried by **appearance**, pose, gaze, and object orientation rather than by touching surfaces alone. TeHOR therefore combines **text descriptions of human-object interactions**, **appearance-aware rendering supervision**, **contact supervision**, and **collision regularization** in a unified HOI optimization procedure, and is reported to achieve **state-of-the-art performance** on the evaluated benchmarks [2602.19679].

## 1. Definition, naming, and scope

TeHOR is introduced specifically for **joint reconstruction of 3D human and object from a single image**. The method outputs a reconstructed **textured 3D human** and **textured 3D object**, with the optimization designed to preserve both geometric plausibility and semantic coherence of the interaction [2602.19679].

The name is easy to confuse with several unrelated arXiv systems. In the CTA Observatory software stack, the relevant system is the **Transients Handler (TH)** rather than “TeHOR” [2508.20726]. In planetary science, **THOR** denotes a global circulation model [1607.05535]. In collider phenomenology, **HATHOR** is a framework for inclusive top-quark cross sections [1406.4403]. In a simulation-only cancer-therapy paper, the relevant term is **TOR** rather than TeHOR [2510.09982]. Within 3D vision and HOI reconstruction, however, **TeHOR** refers specifically to the text-guided reconstruction framework introduced in “TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures” [2602.19679].

A central aspect of the method is its deliberate extension beyond contact-driven reasoning. The paper frames earlier HOI reconstruction systems as limited by two tendencies: dependence on **physical contact information** and over-reliance on **local geometric proximity** instead of **holistic interaction context**. TeHOR is proposed as a direct response to those limitations [2602.19679].

## 2. Problem formulation and motivation

The underlying problem is the reconstruction of human-object interactions from a **single RGB image**. Earlier methods are described as working well when contact provides a strong geometric cue, as in interactions such as **holding**, **sitting on**, or **pushing**. The difficulty arises for **non-contact** cases such as **gazing at a monitor**, **pointing at something**, **reaching to catch a frisbee**, or orienting the body toward an object with no visible contact [2602.19679].

The paper’s argument is that contact cannot serve as the dominant supervisory signal in such cases. When there is little or no contact, the optimization has little strong geometric evidence to enforce. Even when contact exists, concentrating mainly on local touching regions can produce globally implausible reconstructions, for example an object with an incorrect orientation or a human gaze direction that contradicts the interaction semantics [2602.19679].

TeHOR’s core response is **text-guided semantic alignment**. The system extracts **text descriptions** from the input image and uses them as high-level cues for the interaction. At the same time, it uses the rendered **appearance** of the current 3D human-object configuration, rather than relying only on sparse geometric relations, so that optimization can exploit cues such as **human gaze direction**, **body orientation**, **object orientation**, and broader scene context. This suggests a shift from “contact-only” reasoning toward a supervision regime in which semantics and appearance jointly constrain the reconstructed 3D scene [2602.19679].

## 3. Pipeline and scene representation

TeHOR is a **two-stage pipeline**. In the **reconstruction stage**, the method constructs four initial elements from a single image: a **text description of the interaction**, an initial **3D human**, an initial **3D object**, and a **2D background** [2602.19679].

This stage uses several off-the-shelf modules. **GPT-4** produces two captions: a **holistic prompt** $P_{\text{holistic}}$, which describes the overall HOI semantics, and a **contact prompt** $P_{\text{contact}}$, which identifies which body parts are physically touching the object. The human is isolated by removing the object with **SmartEraser**, then segmented and reconstructed with **LHM**. Human pose and shape are estimated with **Multi-HMR**. The object is isolated using **SmartEraser** and **SAM**, reconstructed as a textured mesh with **InstantMesh**, and then converted to **3D Gaussians**. Its pose $(R,t,s)$ is estimated by aligning the reconstructed object with a depth map from **ZoeDepth**. The background is extracted by removing both human and object from the input image [2602.19679].

In the **HOI optimization stage**, the initial human and object are jointly refined using **reconstruction supervision on the original view**, **text-guided appearance supervision on novel views**, **contact supervision on the predicted contacting body parts**, and **collision regularization**. The optimization runs for **200 steps**, updating both human and object Gaussian parameters [2602.19679].

The method represents both entities as **sets of 3D Gaussians**:
- $\Phi_{\text{h}}$ for the human
- $\Phi_{\text{o}}$ for the object

Each Gaussian carries **3D position**, **scale**, **rotation**, **opacity**, and **appearance features**. Human Gaussians are anchored in canonical **SMPL-X** body space and articulated by **linear blend skinning** driven by the body pose $\theta$. The human model is parameterized by Gaussian attributes $\phi_{\text{h}}$, pose $\theta$, and shape $\beta$. The paper follows **ExAvatar-style** handling of hands and face versus other body parts for skinning weights. Object Gaussians are defined in canonical object space and transformed by **rotation** $R$, **translation** $t$, and **scale** $s$. Final rendering is performed with **Mip-Splatting-style rendering** [2602.19679].

The input is a **single RGB image of a person interacting with an object**, and the output is a reconstructed **textured 3D human** and **textured 3D object**. For evaluation and downstream use, the final Gaussian representation is converted into **meshes**. Because Gaussians and base meshes may disagree in contact regions, the method applies a **local shift in the contact area** so that mesh vertices remain geometrically consistent with the optimized Gaussian contacts [2602.19679].

## 4. Objective function and text-guided optimization

The most distinctive component of TeHOR is the use of a **pretrained text-conditioned diffusion model**, specifically **Stable Diffusion v2.1**, to guide optimization of the 3D reconstruction. The system renders the current 3D human-object configuration from random viewpoints, composites the result over the background, and applies score-based guidance conditioned on the holistic caption $P_{\text{holistic}}$ [2602.19679].

The paper writes the appearance/text alignment gradient as

$$
\nabla_{\Phi} \mathcal{L}_{\text{appr}} =
\mathbb{E}\left[
w_t \left(
\hat{\epsilon}_t(\mathbf{x}_t; P_{\text{holistic}}) - \epsilon_t
\right)
\frac{\partial \mathbf{x}_t}{\partial \Phi}
\right].
$$

Here, $\Phi$ denotes the optimized 3D Gaussian parameters, $t$ is the diffusion timestep or noise level, $\mathbf{x}_t$ is the noisy rendered image at timestep $t$, $\epsilon_t$ is the added Gaussian noise, $\hat{\epsilon}_t(\cdot)$ is the diffusion model’s predicted noise conditioned on the prompt, and $w_t$ is a timestep-dependent weighting term. The paper identifies this as **score distillation sampling**. Because supervision is applied over **multiple viewpoints**, the final 3D solution is required to remain text-consistent beyond the input camera view [2602.19679].

The full objective is

$$
\mathcal{L} =
\mathcal{L}_{\text{recon}} +
\mathcal{L}_{\text{appr}} +
\mathcal{L}_{\text{contact}} +
\mathcal{L}_{\text{collision}}.
$$

The **reconstruction loss** $\mathcal{L}_{\text{recon}}$ is a front-view fitting loss composed of two MSE terms: an **RGB reconstruction loss** and a **silhouette/mask reconstruction loss**. The **appearance loss** $\mathcal{L}_{\text{appr}}$ is the diffusion-based text-guided term. The **contact loss** is defined as

$$
\mathcal{L}_{\text{contact}} =
\frac{1}{|V_{h,c}|}
\sum_{v_h \in V_{h,c}}
d(v_h, V_o)\cdot \mathbbm{1}[d(v_h, V_o) < \tau],
$$

where $V_{h,c}$ is the set of human Gaussian center points belonging to body parts indicated by $P_{\text{contact}}$, $V_o$ is the set of object points, and the threshold is $\tau = 10\text{ cm}$. The **collision loss** discourages human-object interpenetration [2602.19679].

Implementation is in **PyTorch** and uses **Adam**. The reported learning rates are $1\times10^{-2}$ for object pose $(R,t,s)$, $1\times10^{-4}$ for human pose $(\theta,\beta)$, and $1\times10^{-4}$ for human and object Gaussian attributes. The **classifier-free guidance scale** is **15.0**, diffusion timesteps are sampled in **$[0.02, 0.98]$**, gradient clipping is applied at norm **1.0**, and the method runs on a single **RTX 8000** with an **average optimization time of 134 seconds per sample**. Rendering uses both **global viewpoints** and **zoomed upper-body views**, reflecting the importance of head and hand regions for HOI semantics [2602.19679].

## 5. Datasets, evaluation protocol, and quantitative performance

TeHOR is evaluated on **Open3DHOI** and **BEHAVE**. **Open3DHOI** is described as an open-vocabulary, in-the-wild HOI evaluation set with **over 2.5K images and 133 object categories**, and it is used for **evaluation only**. **BEHAVE** is described as an indoor HOI dataset with **around 4.5K test images** [2602.19679].

On **Open3DHOI**, TeHOR achieves:
- **CD\(_{\text{human}}\) = 4.941**
- **CD\(_{\text{object}}\) = 16.701**
- **Contact = 0.412**
- **Collision = 0.047**

The paper compares these results against **HOI-Gaussian**, **InteractVLM**, **PHOSA**, and **LEMON+PICO**. The reported best prior baseline numbers listed in the paper include **HOI-Gaussian** with CD\(_{\text{human}}\)=**5.111**, CD\(_{\text{object}}\)=**19.363**, Contact=**0.348**, and Collision=**0.070**. The front-view Gaussian evaluation variant **TeHOR\(^\dagger\)** is also reported, with CD\(_{\text{human}}\)=**4.403**, CD\(_{\text{object}}\)=**16.697**, and Collision=**0.045** [2602.19679].

On **BEHAVE**, TeHOR achieves:
- **CD\(_{\text{human}}\) = 5.615**
- **CD\(_{\text{object}}\) = 17.339**
- **Contact = 0.412**
- **Collision = 0.016**

The paper states that it is **best or tied best across metrics** on this dataset [2602.19679].

A particularly important evaluation is the **non-contact** subset of **Open3DHOI**, where samples with physical contact are removed. This probes whether a method can reason from **gaze**, **body posture**, and **object orientation** rather than from touching surfaces. In that setting, the paper reports the following object-CD values:
- **PHOSA: 65.537**
- **LEMON+PICO: 33.073**
- **InteractVLM: 46.819**
- **HOI-Gaussian: 25.374**
- **TeHOR: 17.546**

In the same non-contact evaluation, **Collision** drops to **0.005**, the best among the compared methods. The paper presents this as evidence that TeHOR is not simply a stronger contact estimator, but a method with improved **non-contact semantic reasoning** [2602.19679].

## 6. Ablations, limitations, and significance

The ablation studies are used to isolate the contribution of the framework’s main design choices. When **text prompts are removed** from the appearance loss, performance degrades: after optimization, the paper reports **CD\(_{\text{object}}\)=20.348**, **Contact=0.374**, and **Collision=0.052**. With text prompts, the corresponding values are **CD\(_{\text{object}}\)=16.701**, **Contact=0.412**, and **Collision=0.047**. The paper interprets this as direct evidence that **text-guided alignment** contributes materially to recovery of global interaction semantics [2602.19679].

The **appearance loss** is also tested directly. Removing it leads to implausible reconstructions, and replacing the diffusion-based appearance loss with **CLIP loss** is reported as better than omitting appearance loss but still worse than the full method. The stated reason is that CLIP compresses text-image alignment into a single global embedding, which is too coarse for dense HOI reasoning, whereas diffusion-based supervision provides more spatially detailed gradients [2602.19679].

Representation matters as well. Replacing **3D Gaussians** with **meshes** worsens performance substantially:
- human CD increases from **4.941** to **5.153**
- object CD increases from **16.701** to **25.162**
- contact decreases from **0.412** to **0.308**

The paper attributes this to the flexibility and high-fidelity appearance modeling of Gaussians, and to their suitability for dense rendering supervision [2602.19679].

Scene context remains relevant beyond the human and object alone. Removing the **2D background** degrades performance, increasing object CD to **18.196** and reducing contact to **0.389**. Omitting the **contact prompt** also worsens results, although the paper emphasizes that contact functions mainly as a complement rather than as a sufficient basis for reconstruction [2602.19679].

The paper also states several limitations. **Fine local details can be missed**, particularly small accessories or subtle local surface deformations. Because the input is a **single image**, ambiguous occlusions or missing regions can still lead to incomplete geometry. The method is **not designed for temporal consistency**, so extension to video would require stable geometry and texture across frames. Finally, **Gaussian-to-mesh conversion** can introduce minor inconsistencies, with Gaussian renderings sometimes appearing slightly larger or blurrier than mesh renderings [2602.19679].

Taken together, these results place TeHOR within a broader shift in HOI reconstruction from purely geometric fitting toward methods that combine geometry with **semantic alignment** and **appearance-aware optimization**. A plausible implication is that the method is most significant not merely because it improves benchmark numbers, but because it treats **non-contact interaction semantics** as a first-class reconstruction signal rather than as an afterthought to contact estimation [2602.19679].

Source: https://www.emergentmind.com/topics/tehor