OneHOI: Unified HOI Generation & Editing

• OneHOI is a unified generative framework for human-object interaction image synthesis and editing that reformulates both tasks as a conditional denoising problem under structured constraints.
• Its architecture integrates a diffusion transformer with dedicated components such as Action Grounding, HOI Encoder, and Structured HOI Attention to enforce explicit role identities and spatial grounding.
• Empirical evaluations show significant gains in HOI editing and generation metrics, demonstrating reliable multi-interaction handling and improved physical contact consistency.

Searching arXiv for OneHOI and closely related HOI papers to ground the article with current citations. OneHOI is a unified generative framework for human-object interaction (HOI) image synthesis and editing. It treats HOI generation and HOI editing not as separate tasks but as instances of a single conditional denoising problem under structured interaction constraints, with HOI represented explicitly as triplets of the form $$\langle \text{person}, \text{action}, \text{object}\rangle$$. The framework is implemented as a diffusion transformer whose conditioning interface includes global text, structured HOI triplets, object-only entities, layouts, and arbitrary-shape masks, allowing layout-guided generation, layout-free generation and editing, mixed-condition control, and multi-HOI editing within one model (Hoe et al., 15 Apr 2026).

1. Problem definition and unification objective

OneHOI addresses a fragmentation in prior HOI synthesis research. Earlier HOI generation methods, exemplified in the paper by InteractDiffusion, synthesize scenes from triplets and layouts but depend strongly on layouts and do not naturally support mixed conditions such as HOI triplets together with object-only entities. Earlier HOI editing methods, such as HOIEdit and InteractEdit, modify interactions in existing images using text, but rely largely on implicit priors, struggle to decouple pose from physical contact, do not scale well to multiple simultaneous HOIs, and lack fine spatial control (Hoe et al., 15 Apr 2026).

The central claim of OneHOI is that both generation and editing can be expressed as the same structured denoising problem. Given a global text prompt $\mathcal{P}$ and either a set of structured interactions

$\{\langle s,o,a\rangle_n\}_{n=1}^N$

or independent objects

$\{\langle o\rangle_n\}_{n=1}^N,$

with optional layout

$$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$

the model produces an image realizing the specified targets. The structured interaction representation is denoted

$$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$

where $\mathcal{S}_n$, $\mathcal{A}_n$, and $\mathcal{O}_n$ are the subject, action, and object token sets for interaction instance $n$ (Hoe et al., 15 Apr 2026).

Generation and editing differ only in the latent input being denoised. For generation, OneHOI samples pure noise latents $\mathcal{P}$0. For editing, it encodes the source image into latents $\mathcal{P}$1, concatenates them with noise latents $\mathcal{P}$2, and applies the same conditional denoiser under new interaction targets. The paper argues that this joint formulation yields a “synergy effect”: generation contributes HOI priors such as pose patterns and contact geometry, while editing contributes identity preservation and scene consistency under partial observation (Hoe et al., 15 Apr 2026).

2. Structured interaction representation

OneHOI conditions on a more explicit interaction representation than plain text prompts. In its HOI encoder, role tokens are first embedded by T5. Let $\mathcal{P}$3 denote the T5 output dimension, with

$\mathcal{P}$4

For interaction instance $\mathcal{P}$5 and role $\mathcal{P}$6, the corresponding T5 embedding is

$\mathcal{P}$7

Three side signals are then attached:

$\mathcal{P}$8

where $\mathcal{P}$9 is a learnable role embedding, $\{\langle s,o,a\rangle_n\}_{n=1}^N$0 is a sinusoidal instance-index embedding, and $\{\langle s,o,a\rangle_n\}_{n=1}^N$1 is a Fourier embedding of the bounding box or shape-derived box. These cues are injected through a normalized, projected, gated residual update:

$\{\langle s,o,a\rangle_n\}_{n=1}^N$2

$\{\langle s,o,a\rangle_n\}_{n=1}^N$3

with $\{\langle s,o,a\rangle_n\}_{n=1}^N$4 learnable (Hoe et al., 15 Apr 2026).

This representation is designed to suppress two recurrent failure modes in multi-HOI synthesis. The first is role confusion, in which subject and object semantics are swapped. The second is instance mixing, in which multiple simultaneous HOIs contaminate one another. The role embedding specifies semantic function, the instance embedding specifies triplet identity, and the box embedding anchors a role spatially. The supplementary implementation details further specify a token-budgeting scheme. The model caps the total HOI tokens at $\{\langle s,o,a\rangle_n\}_{n=1}^N$5 and each sequence at $\{\langle s,o,a\rangle_n\}_{n=1}^N$6, assigning equal active sequence length

$\{\langle s,o,a\rangle_n\}_{n=1}^N$7

where $\{\langle s,o,a\rangle_n\}_{n=1}^N$8 is the number of active role sequences. The default values are $\{\langle s,o,a\rangle_n\}_{n=1}^N$9 and $\{\langle o\rangle_n\}_{n=1}^N,$0 (Hoe et al., 15 Apr 2026).

A notable consequence of this design is that OneHOI does not treat HOI as an unstructured textual attribute. Instead, HOI is encoded as a relational grammar with explicit role identities, explicit instance identities, and optional spatial grounding. This suggests that the model’s conditioning interface is intended not merely to describe scenes, but to impose topology on how the denoiser should realize interactions.

3. Relational Diffusion Transformer

The core architecture is the Relational Diffusion Transformer, or R-DiT. It augments a layout-conditioned DiT baseline, Eligen, with four interaction-aware components: Action Grounding, the HOI Encoder, Structured HOI Attention, and HOI RoPE. Together these components bias the denoiser toward relational reasoning rather than independent entity placement (Hoe et al., 15 Apr 2026).

Action Grounding introduces both semantic and spatial support for the action token. Prior work, especially InteractDiffusion, is criticized for defining the action region using a “between” operator. OneHOI instead defines the action region as the union of the subject and object regions. If $\{\langle o\rangle_n\}_{n=1}^N,$1 and $\{\langle o\rangle_n\}_{n=1}^N,$2 are rasterized subject and object regions from subject/object boxes or masks, then

$\{\langle o\rangle_n\}_{n=1}^N,$3

The paper motivates this by attention analysis: action-token attention tends to focus on the human and object themselves, including remote actions such as “throwing frisbee,” rather than on a thin band between them (Hoe et al., 15 Apr 2026).

Structured HOI Attention is the principal relational mechanism. At the HOI-token level, interaction topology is made verb-mediated. For interaction instance $\{\langle o\rangle_n\}_{n=1}^N,$4,

$\{\langle o\rangle_n\}_{n=1}^N,$5

All cross-instance HOI links are disabled for $\{\langle o\rangle_n\}_{n=1}^N,$6. This forces subject and object information to communicate through the action token rather than directly, encoding the structure of $\{\langle o\rangle_n\}_{n=1}^N,$7 inside the attention graph itself (Hoe et al., 15 Apr 2026).

When layout is available, HOI tokens are also grounded to designated image regions via a masked HOI-image attention map:

$\{\langle o\rangle_n\}_{n=1}^N,$8

The full masked attention is then written as

$\{\langle o\rangle_n\}_{n=1}^N,$9

where $$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$0 for allowed pairs and a large negative value otherwise. Prompt-to-HOI attention is blocked (Hoe et al., 15 Apr 2026).

HOI RoPE addresses cross-talk in multi-HOI scenes. The image stream uses 3D RoPE on an $$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$1 grid, while all HOI tokens belonging to interaction instance $$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$2 are assigned

$$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$3

Each interaction thus occupies a distinct positional slot, disjoint from both the image grid and other HOIs. The intended effect is to preserve instance-specific identities across layers and reduce interaction blending in multi-HOI scenes (Hoe et al., 15 Apr 2026).

4. Training regime, control modes, and data resources

OneHOI is trained jointly on generation and editing using the standard diffusion flow-matching objective. The paper does not print the explicit loss, but states that training follows standard flow matching. A key mechanism is modality dropout. During training, the layout is dropped with probability

$$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$4

HOI labels are dropped by replacing $$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$5 with object-only conditioning with probability

$$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$6

and the global text prompt is dropped with probability

$$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$7

while ensuring at least one modality remains (Hoe et al., 15 Apr 2026).

This training design enables the model’s broad control interface. Supported scenarios include layout-guided generation; layout-free generation and editing; arbitrary-mask control with irregular shapes; mixed-condition generation combining HOI-conditioned entities and object-only entities; single-HOI editing; multi-HOI editing; addition of new interactions or entities; and attribute editing. The paper illustrates a sequence in which OneHOI first generates from mixed conditions, then performs layout-free HOI editing, then layout-guided editing to add a new person and new interaction, and finally attribute editing, all under a single denoising workflow (Hoe et al., 15 Apr 2026).

A major data contribution is HOI-Edit-44K, introduced because large-scale paired HOI editing data had not previously existed. The supplement specifies the exact size as

$$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$8

paired examples. Each sample contains a source image, a target interaction triplet, an edited image, and the corresponding HOI layout for the edited image. Candidate pairs are synthesized from Flux.1-generated images with verified source HOIs and from real HICO-DET images, then filtered by two automatic tests: PViC must detect the target HOI in the edited image, and DINOv2 cosine similarity for both subject and object crops between source and edited images must exceed $$\mathcal{B}=\{b^s_n,b^o_n\}\quad \text{or} \quad \mathcal{B}=\{b^o_n\},$$9. Approximately $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$0 of initial candidates are rejected. The final dataset contains 79 unique object categories, 92 unique target actions, and 372 unique HOI triplets (Hoe et al., 15 Apr 2026).

The training corpus is supplemented by HICO-DET, from which 37K training images are used to learn HOI generation priors, and SA-1B, from which 35K sampled images with object masks converted to layouts provide object-only layout supervision. Evaluation additionally uses IEBench for layout-free HOI editing and a new MultiHOIEdit benchmark. MultiHOIEdit contains 103 unique source images and 200 distinct multi-interaction editing tasks, with scenes containing 2–3 HOIs, 54 object categories, 40 source actions, and 74 target actions. Only 200 valid source images were retained from 8,942 attempts (Hoe et al., 15 Apr 2026).

In implementation terms, OneHOI adapts an MM-DiT backbone from Flux.1 Kontext, fine-tuned with LoRA on the linear layers of each DiT block. The LoRA rank is 64. The system has about 0.3B trainable parameters, corresponding to 2.5% of a frozen 12B base model. The HOI Encoder has 17M parameters and is trained from scratch, while backbone adaptation contributes 344M trainable LoRA parameters. Training uses AdamW, $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$1 learning rate, 10K steps, and batch size 16 on two NVIDIA RTX 6000 ADA GPUs. Inference uses 28 sampling steps and classifier-free guidance scale 3.5 (Hoe et al., 15 Apr 2026).

5. Empirical performance and ablation evidence

OneHOI is evaluated on three tasks: layout-free HOI editing, layout-guided HOI editing, and HOI generation. For layout-free editing on IEBench, the model reports Editability-Identity $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$2, HOI Editability $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$3, PickScore $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$4, HPS $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$5, and ImageReward $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$6. The strongest open-source baseline, InteractEdit, reports Editability-Identity $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$7, HOI Editability $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$8, PickScore $$\mathcal{H}=\bigcup_{n=1}^N\{\mathcal{S}_n,\mathcal{A}_n,\mathcal{O}_n\},$$9, HPS $\mathcal{S}_n$0, and ImageReward $\mathcal{S}_n$1. The paper reports improvements of $\mathcal{S}_n$2 in Editability-Identity, $\mathcal{S}_n$3 in HOI Editability, $\mathcal{S}_n$4 in HPS, and $\mathcal{S}_n$5 in ImageReward over the strongest prior. It also states that OneHOI remains better than Nano Banana in Editability-Identity and HOI Editability (Hoe et al., 15 Apr 2026).

For layout-guided single-HOI editing, the baseline InteractEdit + InteractDiffusion yields EI $\mathcal{S}_n$6, HOI Editability $\mathcal{S}_n$7, and Spatial $\mathcal{S}_n$8, while OneHOI yields EI $\mathcal{S}_n$9, HOI Editability $\mathcal{A}_n$0, Spatial $\mathcal{A}_n$1, PickScore $\mathcal{A}_n$2, HPS $\mathcal{A}_n$3, and ImageReward $\mathcal{A}_n$4. For layout-guided multi-HOI editing on MultiHOIEdit, OneHOI reports EI $\mathcal{A}_n$5, HOI Editability $\mathcal{A}_n$6, Spatial $\mathcal{A}_n$7, PickScore $\mathcal{A}_n$8, HPS $\mathcal{A}_n$9, and ImageReward $\mathcal{O}_n$0. The paper notes that no meaningful baseline exists for layout-guided multi-HOI editing, so the benchmark and the baseline are established there (Hoe et al., 15 Apr 2026).

For HOI generation on 2000 HICO-DET test targets, OneHOI reports Spatial $\mathcal{O}_n$1, HOI Accuracy $\mathcal{O}_n$2, PickScore $\mathcal{O}_n$3, HPS $\mathcal{O}_n$4, and ImageReward $\mathcal{O}_n$5. InteractDiffusion, the best prior on HOI controllability, reports Spatial $\mathcal{O}_n$6 and HOI $\mathcal{O}_n$7, while Eligen, an object-level perceptual baseline, reports PickScore $\mathcal{O}_n$8, HPS $\mathcal{O}_n$9, and ImageReward $n$0. The paper uses these results to argue that unifying editing with generation does not harm generation; the supplementary unification ablation further reports that unified training improves generation from Spatial $n$1 / HOI Acc $n$2 to Spatial $n$3 / HOI Acc $n$4, layout-free editing from EI $n$5 / HE $n$6 to EI $n$7 / HE $n$8, and multi-HOI editing from EI $n$9 / HE $\mathcal{P}$00 to EI $\mathcal{P}$01 / HE $\mathcal{P}$02 (Hoe et al., 15 Apr 2026).

The ablation study isolates the effect of each architectural component, starting from Eligen:

Variant	HOI Gen Acc	Multi-HOI Edit EI
BL	0.3061	—
+ AG	0.4138	0.423
+ AG + Enc	0.4254	0.422
+ AG + Enc + Attn	0.4504	0.433
Full model	0.4528	0.435

The same ablation reports ImageReward for generation rising from $\mathcal{P}$03 in the baseline to $\mathcal{P}$04 in the full model, and multi-HOI edit ImageReward rising from $\mathcal{P}$05 after Action Grounding to $\mathcal{P}$06 in the full model. The paper interprets these gains as follows: Action Grounding provides the largest initial increase in HOI correctness; the HOI Encoder improves role and instance identity while substantially improving perceptual quality; Structured HOI Attention yields another major correctness gain by validating the verb-mediated topology; and HOI RoPE provides the final improvement, especially for disentangling simultaneous interactions such as “holding and petting bird” (Hoe et al., 15 Apr 2026).

Qualitatively, the paper emphasizes three recurring advantages. First, OneHOI more reliably recomposes pose and action in editing, such as transforming “walk” into “feed dog.” Second, it improves physical contact consistency, for example by rendering the hand actually grasping a ball or phone. Third, it supports multi-HOI compositionality, including mixed actions and mixed conditioning. A notable claim is that reliable multi-HOI editing emerges even without paired multi-HOI edit supervision, because joint training on multi-HOI generation plus the structured interaction architecture provides the relevant compositional prior (Hoe et al., 15 Apr 2026).

6. Position within the HOI research landscape

Within the broader HOI literature, OneHOI is positioned as a structured relational image-generation system rather than an HOI detector, a 4D reconstruction pipeline, or a 3D motion generator. Its core technical insight is that HOI should shape token identity, attention topology, and positional encoding, rather than be treated as an auxiliary text condition (Hoe et al., 15 Apr 2026).

This emphasis differs from the objective of physically grounded monocular HOI reconstruction in "Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos" (Zhao et al., 14 May 2026). That work is concerned with recovering 4D human-object trajectories from monocular video and projecting them into simulation via a human-anchored, object-follow pipeline and physics-based rollout. OneHOI, by contrast, operates in image generation and editing, not in recovering simulation-ready teacher trajectories. The contrast is instructive because both works elevate interaction structure over independent entity modeling, but they do so in different regimes: OneHOI in denoising-based image synthesis, HA-HOI in reconstruction-to-simulation (Zhao et al., 14 May 2026).

It also differs from open-ended HOI recognition frameworks such as "Towards Unconstrained Human-Object Interaction" (Tonini et al., 15 Apr 2026). That work defines U-HOI by removing predefined interaction vocabularies at training and inference time and uses multimodal LLMs plus language-to-graph conversion for unconstrained interaction prediction. OneHOI does not pursue unconstrained semantic decoding; instead, it assumes structured HOI conditioning and focuses on controllable generation and editing. A plausible implication is that the two lines of work attack complementary problems: OneHOI formalizes explicit relational control for synthesis, whereas U-HOI formalizes open-ended semantic recognition (Tonini et al., 15 Apr 2026).

The paper’s limitations are less explicit than its contributions, but several are stated or directly implied. Training and evaluation depend on external HOI detection and feature models for dataset curation and metrics, so detector biases propagate into both data quality and performance estimates. Multi-HOI editing remains substantially harder than single-HOI editing, as reflected by the drop from EI $\mathcal{P}$07 in layout-free editing to EI $\mathcal{P}$08 in layout-guided multi-HOI editing. The curation of HOI-Edit-44K is low-yield, with about $\mathcal{P}$09 of candidates rejected, which suggests that scaling paired HOI editing data remains expensive. The method also depends on a very large pretrained base model with LoRA adaptation rather than a lightweight backbone (Hoe et al., 15 Apr 2026).

In historical and conceptual terms, OneHOI may be read as part of a broader transition in HOI research away from fixed, object-prior-dominated recognition toward more explicit relational modeling. Earlier generalization work in HOI detection emphasized suppressing object priors for unseen triplets (Song et al., 2020), while more recent work has explored unconstrained language-based HOI recognition (Tonini et al., 15 Apr 2026), physically plausible HOI reconstruction (Zhao et al., 14 May 2026), and long-horizon 3D HOI generation from text (Geng et al., 21 Mar 2025). OneHOI’s specific contribution within this landscape is to show that image generation and editing can be consolidated into a single HOI-structured denoising framework whose gains arise not from treating HOI as generic prompt text, but from encoding role identity, verb-mediated topology, and instance-level disentanglement directly into the transformer (Hoe et al., 15 Apr 2026).