Controlled Entity and Interaction Diffusion Model

• CEIDM is a diffusion-based text-to-image generation framework that controls both entity appearance and interaction logic using explicit and implicit semantic cues.
• It employs dual control mechanisms, integrating action clustering with bidirectional offsets and the Interaction Enhance Attention module to refine spatial grounding and action accuracy.
• Operating as a training-free add-on to latent diffusion models, CEIDM leverages LLM-based reasoning and an Entity Control Network to mitigate common image synthesis errors.

Searching arXiv for CEIDM and closely related text-to-image control methods. CEIDM, short for Controlled Entity and Interaction Diffusion Model, is a diffusion-based text-to-image generation framework that simultaneously controls entities and their interactions. It is presented as a training-free augmentation built on top of existing text-to-image diffusion models, specifically InteractDiffusion, and introduces dual control: entity control, which keeps each object or person coherent, well-formed, and correctly localized, and interaction control, which aims to ensure that entities relate and act on each other in a logical, semantically accurate, and visually precise manner (Yang et al., 25 Aug 2025). Within the broader literature on grounded and controllable image synthesis, CEIDM is positioned alongside methods such as GLIGEN, ControlNet, Stable Diffusion, and InteractDiffusion, but is distinguished by combining explicit human-object interaction triplets, implicit relations mined by a LLM, action-semantic refinement via clustering and bidirectional offsets, and a dedicated entity control pathway (Yang et al., 25 Aug 2025, Li et al., 2023, Zhang et al., 2023, Peebles et al., 2022, Remy et al., 2023).

1. Definition and problem formulation

CEIDM was proposed to address a persistent difficulty in text-to-image generation: prompts with multiple entities and interaction-rich semantics often expose weaknesses in both spatial grounding and action fidelity. In the formulation described for CEIDM, the central problem is not only whether a model can place entities in an image, but whether it can render their relationships in a way that is consistent with commonsense, pose, contact, and scene logic (Yang et al., 25 Aug 2025).

The motivating examples are prompts such as “a person is feeding a dog” or “a person is riding a motorcycle while carrying a backpack.” The reported failure modes include interaction rationality errors, such as floating or overlapping objects and implausible facing directions; action accuracy errors, such as rendering “feeding” as “throwing food” or “carrying” as “pushing”; and entity quality and consistency errors, such as extra limbs, distorted faces, blurred hands, or malformed animals (Yang et al., 25 Aug 2025). This places CEIDM in a problem setting that overlaps with grounded generation and human-object interaction modeling, but with a stricter emphasis on simultaneous control of object appearance and inter-entity dynamics.

The method is described as an add-on to latent diffusion models, specifically InteractDiffusion built on Stable Diffusion, and introduces two principal modules: an Interaction Enhance Attention (IEA) module for interaction guidance and an Entity Control Network (ECN) for entity-level refinement (Yang et al., 25 Aug 2025). This suggests that CEIDM should be understood less as a replacement backbone than as an inference-time control stack layered over an existing latent diffusion architecture.

A potential source of confusion is acronym similarity with CIEM, the Contrastive Instruction Evaluation Method for hallucination analysis in large vision-LLMs (Hu et al., 2023). CEIDM and CIEM concern different tasks and different model families: CEIDM addresses text-to-image generation and interaction control, whereas CIEM addresses visual hallucination evaluation in vision-LLMs (Yang et al., 25 Aug 2025, Hu et al., 2023).

2. Architectural composition

The CEIDM framework is described as having five main components. First, it uses explicit interactive relationship embedding, which encodes given HOI triplets from HICO-DET in the form of subject phrase, action verb, and object phrase with bounding boxes (Yang et al., 25 Aug 2025). Second, it performs implicit interactive relationship mining and deep semantic embedding, in which a LLM expands the prompt into additional triplets that capture implicit relations, after which these triplets are embedded through linear projections and self-attention (Yang et al., 25 Aug 2025).

Third, CEIDM applies interactive action clustering and offset, where action embeddings are clustered into semantic classes and then perturbed with global and local bidirectional offsets to form enriched action representations (Yang et al., 25 Aug 2025). Fourth, these signals are injected through Interaction Enhance Attention, implemented as a Gated Self-Attention layer inside the latent diffusion model’s transformer, with a scaling coefficient $\delta$ to intensify interaction signals and a sampling-interval strategy controlling which timesteps use which information (Yang et al., 25 Aug 2025). Fifth, CEIDM adds the Entity Control Network, which produces semantically guided masks, enhances entity features with a multi-scale convolutional network, and dynamically fuses those features back into the diffusion model (Yang et al., 25 Aug 2025).

The high-level pipeline accepts a text prompt and HOI annotations from HICO-DET. Explicit HOI triplets and bounding boxes are embedded directly; an LLM infers extra triplets, which are then embedded as implicit interaction tokens; action verbs are embedded with CLIP, clustered, and offset into multiple action features; the denoising U-Net processes latent features through IEA and then through ECN; and the output image is intended to exhibit improved entity shape and interaction fidelity (Yang et al., 25 Aug 2025).

This modular design places CEIDM in direct methodological conversation with grounded generation systems such as GLIGEN, which introduces grounding tokens through gated self-attention (Li et al., 2023), and InteractDiffusion, which conditions generation on HOI triplets and bounding boxes (Remy et al., 2023). The CEIDM paper characterizes its own contribution as going beyond these by enriching interaction semantics and adding a dedicated entity-refinement branch (Yang et al., 25 Aug 2025).

3. LLM-based interaction reasoning

A defining feature of CEIDM is its use of a LLM for implicit interactive relationship mining. For each textual HOI description, CEIDM uses an LLM, reported as Qwen-Turbo in the experiments, to extract obvious explicit triplets and infer additional commonsense triplets through chain-of-thought reasoning (Yang et al., 25 Aug 2025). For the example “A person is blowing a cake,” the explicit triplet is given as $(\text{person}, \text{blowing}, \text{cake})$, while implicit triplets include relations such as $(\text{person}, \text{near}, \text{cake})$, $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$, and $(\text{cake}, \text{on}, \text{table})$ (Yang et al., 25 Aug 2025).

These triplets are then subjected to deep semantic embedding. For a relational triplet $(h, r, t)$, CEIDM applies three independent linear transformations:

$$(h_{emb}, r_{emb}, t_{emb}) = (W_h h,\, W_r r,\, W_t t)$$

It then stacks these transformed representations and applies self-attention:

$$resi = Self\_Attention(\text{Stack}(h_{emb}, r_{emb}, t_{emb})) \tag{4}$$

Finally, residual connections produce contextualized subject, relation, and object embeddings:

$$(e^{h}, e^{r}, e^{t}) = (h_{emb} + resi,\; r_{emb} + resi,\; t_{emb} + resi) \tag{5}$$

These embeddings are the implicit interactive information fed into the IEA module (Yang et al., 25 Aug 2025).

The significance of this mechanism is that the conditioning signal is no longer limited to explicit verb-object syntax. During denoising, the model is guided by supplementary relational structure, such as contact, orientation, and support relations, which the paper links to reductions in floating objects, incorrect body-part involvement, and nonsensical relative positions (Yang et al., 25 Aug 2025). A plausible implication is that CEIDM treats language-model reasoning as a way to densify sparse HOI annotations without retraining the diffusion backbone.

4. Action-semantic refinement and entity control

Action clustering and bidirectional offsets

CEIDM treats action verbs as a structured semantic space rather than as isolated labels. It embeds action phrases using CLIP and applies K-means over the action embedding set $A_i$ to obtain semantic action classes (Yang et al., 25 Aug 2025). The paper gives examples for motorcycle-related actions, distinguishing a Dynamic Motion Class containing “racing, riding, straddling, turning” from a Static Pose Class containing “sitting on” (Yang et al., 25 Aug 2025).

Given an action embedding $(\text{person}, \text{blowing}, \text{cake})$0 and a cluster center $(\text{person}, \text{blowing}, \text{cake})$1, CEIDM defines a global offset:

$(\text{person}, \text{blowing}, \text{cake})$2

and a local offset along the unit direction $(\text{person}, \text{blowing}, \text{cake})$3:

$(\text{person}, \text{blowing}, \text{cake})$4

These offsets are intended to globally emphasize category semantics and locally refine detail semantics (Yang et al., 25 Aug 2025).

The paper provides concrete lexical examples. For global offsets at $(\text{person}, \text{blowing}, \text{cake})$5, “carrying (+0.1)” maps toward “transporting, hauling, moving,” while “carrying (−0.1)” maps toward “releasing, dropping, placing”; “wearing (+0.1)” maps toward “dressing, accessorizing, adorning,” and “wearing (−0.1)” maps toward “removing, undressing, taking off” (Yang et al., 25 Aug 2025). For local offsets at $(\text{person}, \text{blowing}, \text{cake})$6, “carrying (+0.05)” maps toward “hand-carrying, shoulder-carrying, toting,” while “carrying (−0.05)” maps toward “light-carrying, brief-holding, partial-lifting”; similarly, “wearing (+0.05)” maps toward “tight-wearing, layered-wearing, strapping,” and “wearing (−0.05)” toward “loose-wearing, partial-covering, draping” (Yang et al., 25 Aug 2025).

These perturbed action features are then embedded using the explicit interaction equations:

$(\text{person}, \text{blowing}, \text{cake})$7

$(\text{person}, \text{blowing}, \text{cake})$8

$(\text{person}, \text{blowing}, \text{cake})$9

where $(\text{person}, \text{near}, \text{cake})$0 is the semantic embedding, $(\text{person}, \text{near}, \text{cake})$1 is the Fourier embedding of bounding boxes, $(\text{person}, \text{near}, \text{cake})$2 is the instance embedding, and $(\text{person}, \text{near}, \text{cake})$3 are role embeddings (Yang et al., 25 Aug 2025). The paper then constructs

$(\text{person}, \text{near}, \text{cake})$4

and concatenates $(\text{person}, \text{near}, \text{cake})$5 to produce the final action offset interactive information $(\text{person}, \text{near}, \text{cake})$6 (Yang et al., 25 Aug 2025).

Entity Control Network

The Entity Control Network addresses a complementary problem: even with improved interaction semantics, entities themselves may remain visually unstable. CEIDM therefore generates soft masks for subjects and objects using CLIP semantics and visual features from the IEA output (Yang et al., 25 Aug 2025). Subject and object names are embedded by CLIP and processed through an MLP; these semantic vectors are fused with latent visual features to produce spatial masks, conventionally denoted as $(\text{person}, \text{near}, \text{cake})$7 and $(\text{person}, \text{near}, \text{cake})$8 (Yang et al., 25 Aug 2025). Visual features are then masked:

• $(\text{person}, \text{near}, \text{cake})$9
• $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$0

Mask sharpness is governed by a temperature coefficient $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$1, conceptually represented as

$$(\text{person’s mouth}, \text{directed at}, \text{flame})$$2

with higher $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$3 yielding smoother masks and lower $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$4 yielding sharper focus (Yang et al., 25 Aug 2025).

The masked subject and object features are refined by a parallel multi-scale convolution network using different receptive fields, which the paper associates with capturing local details such as faces and hands, mid-range structures such as limbs and object shapes, and more global shape and context (Yang et al., 25 Aug 2025). A subsequent entity feature dynamic fusion network combines subject and object feature maps and reinserts them into the diffusion pipeline. The paper states that the fused features are combined with the visual features output by the IEA layer before cross-attention, and that the residual output of the cross-attention layer is directly connected to the IEA output to preserve coherence of the generated image content (Yang et al., 25 Aug 2025).

5. Integration with latent diffusion models

CEIDM is built on latent diffusion models such as Stable Diffusion and adopts the standard diffusion formalism (Yang et al., 25 Aug 2025, Rombach et al., 2021). The forward diffusion process is written as

$$(\text{person’s mouth}, \text{directed at}, \text{flame})$$5

where $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$6 (Yang et al., 25 Aug 2025). The reverse denoising objective is described in terms of predicting the added noise:

$$(\text{person’s mouth}, \text{directed at}, \text{flame})$$7

where the conditioning $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$8 now includes text prompt embeddings, explicit interaction tokens, implicit interaction tokens, and action offset interactive information $$(\text{person’s mouth}, \text{directed at}, \text{flame})$$9 (Yang et al., 25 Aug 2025).

The Interaction Enhance Attention module is inserted as a Gated Self-Attention layer inside the latent diffusion transformer. Let $(\text{cake}, \text{on}, \text{table})$0 denote visual tokens and $(\text{cake}, \text{on}, \text{table})$1 denote interaction tokens. CEIDM forms

$(\text{cake}, \text{on}, \text{table})$2

and defines

$(\text{cake}, \text{on}, \text{table})$3

where $(\text{cake}, \text{on}, \text{table})$4 is the interaction scaling coefficient (Yang et al., 25 Aug 2025). The interaction tokens supplied to this layer may be explicit interactive information, implicit interactive information, or action offset interactive information (Yang et al., 25 Aug 2025).

The method is explicitly described as training-free: it uses a pre-trained InteractDiffusion or Stable Diffusion backbone and does not define new explicit losses beyond the original diffusion training used by InteractDiffusion (Yang et al., 25 Aug 2025). Instead, the new components act as conditioning and architectural modifications during inference. This distinguishes CEIDM from methods that require additional supervised retraining, while aligning it with the broader trend of inference-time controllability mechanisms in text-to-image generation (Yang et al., 25 Aug 2025, Li et al., 2023, Zhang et al., 2023).

6. Experimental evaluation, comparative results, and limitations

The experiments are conducted on HICO-DET, described as containing 47,776 images, 600 HOI triplets, 80 object categories, and 117 verb classes, with 38,118 training and 9,658 test images (Yang et al., 25 Aug 2025). The paper states that it uses 33,405 HOI annotations from the test set as prompts and conditions (Yang et al., 25 Aug 2025). The base model is InteractDiffusion, and sampling uses the PLMS sampler with 50 steps on NVIDIA A100-SXM4-40GB, with inference over the full test set taking approximately 185 hours (Yang et al., 25 Aug 2025). Image quality is evaluated using FID and KID, while interaction controllability is measured by HOI detection mAP using the FGAHOI detector with Swin-Tiny and Swin-Large backbones, under both Default and Known Object settings (Yang et al., 25 Aug 2025).

The quantitative comparison reported in the paper is summarized below.

Model	Image quality	Selected HOI metrics
Stable Diffusion	FID 35.85, KID 0.01297	Swin-Tiny default full 0.63, rare 0.68
GLIGEN	FID 29.35, KID 0.01275	Swin-Tiny default full 21.73, rare 15.35
InteractDiffusion	FID 18.69, KID 0.00676	Swin-Tiny default full 29.53; Swin-Large default full 31.56
CEIDM	FID 16.12, KID 0.00485	Swin-Tiny default full 31.91; Swin-Large default full 33.69

More detailed results reported for CEIDM are Swin-Tiny default full 31.91, rare 25.27, known-object full 33.28, rare 27.24; and Swin-Large default full 33.69, rare 27.66, known-object full 34.61, rare 28.83 (Yang et al., 25 Aug 2025). Relative to InteractDiffusion, CEIDM improves FID from 18.69 to 16.12, Swin-Tiny default full mAP from 29.53 to 31.91, and Swin-Large default full mAP from 31.56 to 33.69 (Yang et al., 25 Aug 2025). The paper also gives HICO-DET ground truth values for reference, including Swin-Tiny default full 29.94, known-object full 32.48, Swin-Large default full 37.18, and known-object full 38.93 (Yang et al., 25 Aug 2025).

The qualitative analysis attributes these improvements to better action correctness, interaction rationality, and entity quality. The examples described in the paper report that Stable Diffusion and InteractDiffusion frequently misrender “sitting on,” “carrying,” and similar actions; that backpacks may float or hands may fail to contact objects correctly; and that CEIDM produces more plausible poses, ground contact, and body-object alignment (Yang et al., 25 Aug 2025). For complex scenes involving multiple interactions, the paper states that CEIDM is the only method among those compared that reasonably renders all relationships and accurate actions simultaneously (Yang et al., 25 Aug 2025).

The ablation study progressively adds components to the InteractDiffusion baseline. The baseline has FID 18.69, KID 0.00676, and Swin-Tiny mAP full 29.53. Adding Cot. yields FID 18.09 and mAP full 29.97; adding Cl. yields FID 17.85 and mAP full 31.18; adding Ec. yields FID 16.25 and mAP full 31.64; and adding $(\text{cake}, \text{on}, \text{table})$5 yields FID 16.12 and mAP full 31.91 (Yang et al., 25 Aug 2025). The paper also reports sensitivity analysis over $(\text{cake}, \text{on}, \text{table})$6 from 1.0 to 2.0, with the best results around $(\text{cake}, \text{on}, \text{table})$7 (Yang et al., 25 Aug 2025).

The method is also described as transferable to personalized Stable Diffusion models, where it preserves style while improving HOI correctness and entity quality (Yang et al., 25 Aug 2025). Hyperparameters reported in the paper include $(\text{cake}, \text{on}, \text{table})$8, IEA fusion weight $(\text{cake}, \text{on}, \text{table})$9, sampling strategy $(h, r, t)$0, $(h, r, t)$1, and mask temperature $(h, r, t)$2 (Yang et al., 25 Aug 2025).

The reported limitations are equally explicit. CEIDM relies on LLM quality, so poor prompts or unusual domains may yield incorrect triplets and misguide the generator (Yang et al., 25 Aug 2025). Very intricate multi-entity scenes with subtle actions remain difficult, and the paper states that there is “still room for further improvement in rendering finer action details” (Yang et al., 25 Aug 2025). The approach also introduces substantial computational overhead at inference time, with the cited 185 hours for full HICO-DET test-set inference (Yang et al., 25 Aug 2025). Finally, the method is built and tested on human-object interactions with HICO-DET and assumes clear subject-action-object structure, verbs amenable to CLIP embedding and clustering, and bounding-box annotations for explicit interactions (Yang et al., 25 Aug 2025). This suggests that adaptation may be required for abstract domains, non-human interaction settings, or cases lacking HOI-style supervision.