OneHOI: Unifying Human-Object Interaction Generation and Editing

Abstract: Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.

• The paper introduces OneHOI, a unified framework integrating HOI generation and editing through conditional denoising and explicit relational reasoning.
• It employs HOI tokenization, structured attention, and specialized embeddings to decouple pose, enhance spatial alignment, and support multi-interaction editing.
• Empirical evaluations show significant improvements in identity preservation, interaction correctness, and HOI accuracy over traditional methods.

Unified Human-Object Interaction Generation and Editing with OneHOI

Motivation and Problem Setting

Human-Object Interaction (HOI) modeling fundamentally concerns understanding and controlling the relationships between humans and objects, specified as $$\langle \text{person}, \text{action}, \text{object}\rangle$$ triplets. Traditional HOI research bifurcates into detection/recognition (identifying triplets from images) and generation/editing (synthesizing or modifying images conditioned on HOI structure). Existing generative approaches are fragmented into two distinct families: HOI scene generation—which synthesizes images from structured triplets and spatial layouts but lacks compositional flexibility—and HOI editing—which modifies interactions via text but struggles with decoupling pose from contact, reliable multi-interaction composition, and fine spatial control. No prior framework unifies these paradigms nor demonstrates robust multi-HOI editing.

OneHOI Framework Design

OneHOI introduces a single, unified framework for both HOI generation and editing, driven by a re-architected conditional diffusion transformer. The central insight is that both scene synthesis and editing can be realized as conditional denoising, parameterized over shared structured interaction representations. All interaction control—be it text, layout, arbitrary masks, or mixed conditions—maps to a unified set of HOI tokens. This consolidation is operationalized via the Relational Diffusion Transformer (R-DiT), a variant of the standard DiT backbone with explicit relational reasoning modules to model the semantics and geometry of role-specific (subject, action, object) relationships.

An overview of OneHOI's pipeline is depicted below.

Figure 1: An overview of OneHOI pipeline.

Architectural Innovations

1. HOI Encoder

The HOI Encoder embeds each structured HOI token (subject, action, object) with explicit role identity, instance identity, and spatial cues. Each role embedding is augmented with sinusoidal (instance) and Fourier (spatial box) embeddings, concatenated and passed through a learnable nonlinear projection. This enforces disambiguation of roles and instances, critical for multi-interaction disentanglement.

2. Structured HOI Attention

To explicitly model the topology of interactions, Structured HOI Attention strategically masks and routes attention flow: it enforces that subjects and objects communicate only via their linking action token, blocks direct subject-object links, and imposes spatial grounding according to region constraints from layouts or masks. This mechanism is central for verb-mediated relational reasoning and spatial compliance, especially under multiple interactions or partial controls.

3. HOI RoPE (Rotary Position Embedding) for Instance Separation

HOI RoPE assigns unique positional indices to each HOI instance, ensuring separation in the embedding space and mitigating 'cross-talk' between different $\langle s, a, o \rangle$ triplets. This is especially important for compositional generalization in multi-HOI and multi-object scenarios.

The combined effect of these modules is systematically validated via an ablation (see Figure 2).

Figure 2: Progressively adding components improves the interaction's plausibility; only the full model renders both “holding” and “petting“ correctly.

Unified Generation and Editing: Versatility and Capabilities

OneHOI enables comprehensive multi-step HOI workflows within a single model:

• Mixed-Condition Generation: Flexible generation with combinations of structured HOI triplets and object entities, using spatial layouts or arbitrary masks.
• Layout-Free and Layout-Guided HOI Editing: Modifying the interaction in an image either with or without explicit spatial layouts, supporting both single and multiple HOIs.
• Fine Attribute Editing: Targeted modifications at the instance or scene attribute level.

The diverse applications supported are visually summarized in the following figure, highlighting both generation and editing flexibility.

Figure 3: Unified HOI generation and editing in multi-step workflows: mixed-condition generation, layout-free and guided editing, and attribute manipulation.

Figure 4: OneHOI enables seamless, compositional workflows blending generation, multi-HOI editing, and attribute-level modifications within a single coherent interface.

Additional fine-grained control, such as arbitrary-shape region conditioning and mixed HOI/object-mode compositionality, are visualized in the example below.

Figure 5: Versatile control—arbitrary-shape masks and compositional HOI/object conditioning within a single scene.

Empirical Evaluation

Datasets and Benchmarks

The HOI-Edit-44K dataset—constructed as part of this work—provides 44K rigorously validated, identity-preserving paired examples for HOI editing, alongside established open datasets (HICO-DET, SA-1B) for HOI and object-centric supervision.

Quantitative Results

• Layout-Free HOI Editing: OneHOI achieves the best Editability-Identity (0.638) and HOI Editability (0.596), outperforming prior specialized editing and generalist baselines by +10.0% and +16.0%, respectively, while also improving perceptual alignment metrics.
• Layout-Guided Single- and Multi-HOI Editing: OneHOI demonstrates strong spatial (0.822 single, 0.675 multi-HOI), interaction correctness, and perceptual scores, directly establishing the first baseline for controllable multi-HOI edits.
• HOI Generation: The framework attains the highest HOI accuracy (0.4528) and spatial alignment (0.6104), with improvements in human-aligned quality metrics (PickScore, HPS, ImageReward) over leading predecessors (2604.14062).

Strikingly, no performance trade-off is observed when unifying tasks: joint training yields a “synergy effect”, where generative priors enhance editing robustness, and editing training regularizes generation, as confirmed by ablation.

Qualitative Analysis

In layout-free editing, OneHOI alone produces identity-consistent edits that accurately realize the requested interaction, whereas prior methods typically fail to alter pose or induce artifacts.

Figure 6: OneHOI renders the specified new interaction, preserving subject identity; baselines fail to update pose or introduce identity drift.

For HOI generation, object-level methods correctly localize participants but fail to capture intended interactions, while OneHOI yields superior semantic and geometric consistency.

Figure 7: Only OneHOI synthesizes spatially and semantically coherent multi-action scenes; baselines often miss specified relations.

Layout-guided editing demonstrates the exclusive capability for spatially localized, multi-instance edits while preserving context.

Figure 8: Single and multi-HOI editing confining changes to specified layout regions; scene coherence is maintained.

Human Evaluation

A rigorous side-by-side human preference study confirms the quantitative gains: OneHOI is preferred over QwenImageEdit and InteractEdit on HOI correctness/physical plausibility, identity preservation, and overall quality by 58–75% of raters.

Figure 9: Aggregate human preference scores substantiate the models' improvements on all critical axes.

Theoretical and Practical Implications

OneHOI demonstrates that a unified structured-relational formulation, realized via explicit HOI tokenization and relational attention, confers substantial benefits in compositionality, scalability, and task versatility. Architecturally, this work establishes that transformer-based diffusion models—when explicitly guided by symbolic interaction structure—can transcend entity-localized conditioning, addressing both relational plausibility and control. The new dataset curation pipeline provides a scalable and robust solution for high-fidelity, identity-preserving HOI editing supervision, crucial for community progress.

Practically, OneHOI enables precise, user-controllable image manipulation—critical for AR/VR, content creation, robotics simulation, and assistive technologies—by allowing direct, structured intent communication rather than prompt engineering or manual editing.

Future Directions

The unified approach invites extension to broader forms of structured scene understanding and control, incorporating more diverse semantic relations, multi-agent dynamics, and interaction graphs. Hybrid integration with video and 3D scene generation is a plausible next step, leveraging the explicit symbolic-relational interface. Moreover, further work on grounding interaction intent from richer linguistic or interactive instructions (beyond discrete labels) would broaden access and flexibility.

Conclusion

OneHOI sets a new state-of-the-art for unified, controllable HOI generation and editing, bridging a longstanding methodological divide and elevating scene compositionality in diffusion-transformer paradigms. The architectural and dataset innovations provide reproducible pathways for further advances in structured, relationally controllable visual generative models (2604.14062).