---
title: Controlled Entity and Interaction Diffusion Model
url: https://www.emergentmind.com/topics/ceidm
type: topic
---

# Controlled Entity and Interaction Diffusion Model

Searching arXiv for CEIDM and closely related text-to-image control methods.
CEIDM, short for **Controlled Entity and Interaction Diffusion Model**, is a diffusion-based text-to-image generation framework that simultaneously controls entities and their interactions. It is presented as a **training-free** augmentation built on top of existing text-to-image diffusion models, specifically InteractDiffusion, and introduces **dual control**: **entity control**, which keeps each object or person coherent, well-formed, and correctly localized, and **interaction control**, which aims to ensure that entities relate and act on each other in a logical, semantically accurate, and visually precise manner [2508.17760]. Within the broader literature on grounded and controllable image synthesis, CEIDM is positioned alongside methods such as GLIGEN, ControlNet, Stable Diffusion, and InteractDiffusion, but is distinguished by combining explicit human-object interaction triplets, implicit relations mined by a large language model, action-semantic refinement via clustering and bidirectional offsets, and a dedicated entity control pathway [2508.17760], [2301.07093], [2302.05543], [2212.09748], [2308.00157].

## 1. Definition and problem formulation

CEIDM was proposed to address a persistent difficulty in text-to-image generation: prompts with multiple entities and interaction-rich semantics often expose weaknesses in both spatial grounding and action fidelity. In the formulation described for CEIDM, the central problem is not only whether a model can place entities in an image, but whether it can render their relationships in a way that is consistent with commonsense, pose, contact, and scene logic [2508.17760].

The motivating examples are prompts such as “a person is feeding a dog” or “a person is riding a motorcycle while carrying a backpack.” The reported failure modes include **interaction rationality** errors, such as floating or overlapping objects and implausible facing directions; **action accuracy** errors, such as rendering “feeding” as “throwing food” or “carrying” as “pushing”; and **entity quality and consistency** errors, such as extra limbs, distorted faces, blurred hands, or malformed animals [2508.17760]. This places CEIDM in a problem setting that overlaps with grounded generation and human-object interaction modeling, but with a stricter emphasis on simultaneous control of object appearance and inter-entity dynamics.

The method is described as an add-on to latent diffusion models, specifically InteractDiffusion built on Stable Diffusion, and introduces two principal modules: an **Interaction Enhance Attention (IEA)** module for interaction guidance and an **Entity Control Network (ECN)** for entity-level refinement [2508.17760]. This suggests that CEIDM should be understood less as a replacement backbone than as an inference-time control stack layered over an existing latent diffusion architecture.

A potential source of confusion is acronym similarity with **CIEM**, the **Contrastive Instruction Evaluation Method** for hallucination analysis in large vision-language models [2309.02301]. CEIDM and CIEM concern different tasks and different model families: CEIDM addresses text-to-image generation and interaction control, whereas CIEM addresses visual hallucination evaluation in vision-language models [2508.17760], [2309.02301].

## 2. Architectural composition

The CEIDM framework is described as having five main components. First, it uses **explicit interactive relationship embedding**, which encodes given HOI triplets from HICO-DET in the form of subject phrase, action verb, and object phrase with bounding boxes [2508.17760]. Second, it performs **implicit interactive relationship mining and deep semantic embedding**, in which a large language model expands the prompt into additional triplets that capture implicit relations, after which these triplets are embedded through linear projections and self-attention [2508.17760].

Third, CEIDM applies **interactive action clustering and offset**, where action embeddings are clustered into semantic classes and then perturbed with **global and local bidirectional offsets** to form enriched action representations [2508.17760]. Fourth, these signals are injected through **Interaction Enhance Attention**, implemented as a **Gated Self-Attention** layer inside the latent diffusion model’s transformer, with a scaling coefficient $\delta$ to intensify interaction signals and a sampling-interval strategy controlling which timesteps use which information [2508.17760]. Fifth, CEIDM adds the **Entity Control Network**, which produces semantically guided masks, enhances entity features with a multi-scale convolutional network, and dynamically fuses those features back into the diffusion model [2508.17760].

The high-level pipeline accepts a text prompt and HOI annotations from HICO-DET. Explicit HOI triplets and bounding boxes are embedded directly; an LLM infers extra triplets, which are then embedded as implicit interaction tokens; action verbs are embedded with CLIP, clustered, and offset into multiple action features; the denoising U-Net processes latent features through IEA and then through ECN; and the output image is intended to exhibit improved entity shape and interaction fidelity [2508.17760].

This modular design places CEIDM in direct methodological conversation with grounded generation systems such as GLIGEN, which introduces grounding tokens through gated self-attention [2301.07093], and InteractDiffusion, which conditions generation on HOI triplets and bounding boxes [2308.00157]. The CEIDM paper characterizes its own contribution as going beyond these by enriching interaction semantics and adding a dedicated entity-refinement branch [2508.17760].

## 3. LLM-based interaction reasoning

A defining feature of CEIDM is its use of a large language model for **implicit interactive relationship mining**. For each textual HOI description, CEIDM uses an LLM, reported as **Qwen-Turbo** in the experiments, to extract obvious explicit triplets and infer additional commonsense triplets through **chain-of-thought** reasoning [2508.17760]. For the example “A person is blowing a cake,” the explicit triplet is given as $(\text{person}, \text{blowing}, \text{cake})$, while implicit triplets include relations such as $(\text{person}, \text{near}, \text{cake})$, $(\text{person’s mouth}, \text{directed at}, \text{flame})$, and $(\text{cake}, \text{on}, \text{table})$ [2508.17760].

These triplets are then subjected to **deep semantic embedding**. For a relational triplet $(h, r, t)$, CEIDM applies three independent linear transformations:
$$
(h_{emb}, r_{emb}, t_{emb}) = (W_h h,\, W_r r,\, W_t t)
$$
It then stacks these transformed representations and applies self-attention:
$$
resi = Self\_Attention(\text{Stack}(h_{emb}, r_{emb}, t_{emb})) \tag{4}
$$
Finally, residual connections produce contextualized subject, relation, and object embeddings:
$$
(e^{h}, e^{r}, e^{t}) = (h_{emb} + resi,\; r_{emb} + resi,\; t_{emb} + resi) \tag{5}
$$
These embeddings are the **implicit interactive information** fed into the IEA module [2508.17760].

The significance of this mechanism is that the conditioning signal is no longer limited to explicit verb-object syntax. During denoising, the model is guided by supplementary relational structure, such as contact, orientation, and support relations, which the paper links to reductions in floating objects, incorrect body-part involvement, and nonsensical relative positions [2508.17760]. A plausible implication is that CEIDM treats language-model reasoning as a way to densify sparse HOI annotations without retraining the diffusion backbone.

## 4. Action-semantic refinement and entity control

### Action clustering and bidirectional offsets

CEIDM treats action verbs as a structured semantic space rather than as isolated labels. It embeds action phrases using CLIP and applies **K-means** over the action embedding set $A_i$ to obtain semantic action classes [2508.17760]. The paper gives examples for motorcycle-related actions, distinguishing a **Dynamic Motion Class** containing “racing, riding, straddling, turning” from a **Static Pose Class** containing “sitting on” [2508.17760].

Given an action embedding $a$ and a cluster center $\mu_c$, CEIDM defines a **global offset**:
$$
a^{global}_{+} = a + \alpha (\mu_c - a), \quad
a^{global}_{-} = a - \alpha (\mu_c - a)
$$
and a **local offset** along the unit direction $u_c = (\mu_c - a)/\lVert \mu_c - a \rVert$:
$$
a^{local}_{+} = a + \beta u_c, \quad
a^{local}_{-} = a - \beta u_c
$$
These offsets are intended to globally emphasize category semantics and locally refine detail semantics [2508.17760].

The paper provides concrete lexical examples. For **global offsets** at $\alpha=0.1$, “carrying (+0.1)” maps toward “transporting, hauling, moving,” while “carrying (−0.1)” maps toward “releasing, dropping, placing”; “wearing (+0.1)” maps toward “dressing, accessorizing, adorning,” and “wearing (−0.1)” maps toward “removing, undressing, taking off” [2508.17760]. For **local offsets** at $\beta=0.05$, “carrying (+0.05)” maps toward “hand-carrying, shoulder-carrying, toting,” while “carrying (−0.05)” maps toward “light-carrying, brief-holding, partial-lifting”; similarly, “wearing (+0.05)” maps toward “tight-wearing, layered-wearing, strapping,” and “wearing (−0.05)” toward “loose-wearing, partial-covering, draping” [2508.17760].

These perturbed action features are then embedded using the explicit interaction equations:
$$
h_i^s, h_i^o = MLP_{Obj}([\text{CLIP(phrases)}, F(\text{boxes})]) \tag{1}
$$
$$
h_i^a = MLP_{Act}([\text{CLIP(phrases)}, F(\text{boxes})]) \tag{2}
$$
$$
(e_i^s, e_i^a, e_i^o) = (h_i^s + q_i + r^s,\; h_i^a + q_i + r^a,\; h_i^o + q_i + r^o) \tag{3}
$$
where $CLIP(\cdot)$ is the semantic embedding, $F(\cdot)$ is the Fourier embedding of bounding boxes, $q_i$ is the instance embedding, and $r^s, r^a, r^o$ are role embeddings [2508.17760]. The paper then constructs
$$
G_0 = \text{Cat}(e_i^s, e_i^a, e_i^o), \quad
G_j = \text{Cat}(e_i^s, e_i^{A^j}, e_i^o) \tag{6}
$$
and concatenates $G_0, G_1, \dots, G_m$ to produce the final **action offset interactive information** $G$ [2508.17760].

### Entity Control Network

The **Entity Control Network** addresses a complementary problem: even with improved interaction semantics, entities themselves may remain visually unstable. CEIDM therefore generates **soft masks** for subjects and objects using CLIP semantics and visual features from the IEA output [2508.17760]. Subject and object names are embedded by CLIP and processed through an MLP; these semantic vectors are fused with latent visual features to produce spatial masks, conventionally denoted as $subject\_mask$ and $object\_mask$ [2508.17760]. Visual features are then masked:
- $x\_subject = subject\_mask \odot v$
- $x\_object = object\_mask \odot v$

Mask sharpness is governed by a temperature coefficient $temp$, conceptually represented as
$$
mask = \text{softmax}( logits / temp )
$$
with higher $temp$ yielding smoother masks and lower $temp$ yielding sharper focus [2508.17760].

The masked subject and object features are refined by a **parallel multi-scale convolution network** using different receptive fields, which the paper associates with capturing local details such as faces and hands, mid-range structures such as limbs and object shapes, and more global shape and context [2508.17760]. A subsequent **entity feature dynamic fusion network** combines subject and object feature maps and reinserts them into the diffusion pipeline. The paper states that the fused features are combined with the visual features output by the IEA layer before cross-attention, and that the residual output of the cross-attention layer is directly connected to the IEA output to preserve coherence of the generated image content [2508.17760].

## 5. Integration with latent diffusion models

CEIDM is built on latent diffusion models such as Stable Diffusion and adopts the standard diffusion formalism [2508.17760], [2112.10752]. The forward diffusion process is written as
$$
q(x_t \mid x_0) = \mathcal{N}\left(x_t; \sqrt{\alpha_t} x_0,\, (1-\alpha_t) I \right)
$$
where $\alpha_t = \prod_{s=1}^t (1-\beta_s)$ [2508.17760]. The reverse denoising objective is described in terms of predicting the added noise:
$$
\mathcal{L}_{\text{DDPM} = \mathbb{E}_{x_0, t, \epsilon} \left[ \lVert \epsilon - \epsilon_\theta(x_t, t, c) \rVert^2 \right]
$$
where the conditioning $c$ now includes text prompt embeddings, explicit interaction tokens, implicit interaction tokens, and action offset interactive information $G$ [2508.17760].

The **Interaction Enhance Attention** module is inserted as a Gated Self-Attention layer inside the latent diffusion transformer. Let $v$ denote visual tokens and $info$ denote interaction tokens. CEIDM forms
$$
Q = W_Q \cdot [v, info], \quad K = W_K \cdot [v, info], \quad V = W_V \cdot [v, info]
$$
and defines
$$
StrenAtt(Q, K, V) = \text{Softmax}\left(\frac{Q K^\top}{\sqrt{d}\right) \cdot V \cdot \delta \tag{7}
$$
where $\delta$ is the **interaction scaling coefficient** [2508.17760]. The interaction tokens supplied to this layer may be explicit interactive information, implicit interactive information, or action offset interactive information [2508.17760].

The method is explicitly described as **training-free**: it uses a pre-trained InteractDiffusion or Stable Diffusion backbone and does not define new explicit losses beyond the original diffusion training used by InteractDiffusion [2508.17760]. Instead, the new components act as conditioning and architectural modifications during inference. This distinguishes CEIDM from methods that require additional supervised retraining, while aligning it with the broader trend of inference-time controllability mechanisms in text-to-image generation [2508.17760], [2301.07093], [2302.05543].

## 6. Experimental evaluation, comparative results, and limitations

The experiments are conducted on **HICO-DET**, described as containing **47,776 images**, **600 HOI triplets**, **80 object categories**, and **117 verb classes**, with **38,118 training** and **9,658 test images** [2508.17760]. The paper states that it uses **33,405 HOI annotations** from the test set as prompts and conditions [2508.17760]. The base model is **InteractDiffusion**, and sampling uses the **PLMS sampler** with **50 steps** on **NVIDIA A100-SXM4-40GB**, with inference over the full test set taking approximately **185 hours** [2508.17760]. Image quality is evaluated using **FID** and **KID**, while interaction controllability is measured by **HOI detection mAP** using the **FGAHOI** detector with **Swin-Tiny** and **Swin-Large** backbones, under both **Default** and **Known Object** settings [2508.17760].

The quantitative comparison reported in the paper is summarized below.

| Model | Image quality | Selected HOI metrics |
|---|---:|---:|
| Stable Diffusion | FID 35.85, KID 0.01297 | Swin-Tiny default full 0.63, rare 0.68 |
| GLIGEN | FID 29.35, KID 0.01275 | Swin-Tiny default full 21.73, rare 15.35 |
| InteractDiffusion | FID 18.69, KID 0.00676 | Swin-Tiny default full 29.53; Swin-Large default full 31.56 |
| CEIDM | **FID 16.12**, **KID 0.00485** | **Swin-Tiny default full 31.91**; **Swin-Large default full 33.69** |

More detailed results reported for **CEIDM** are **Swin-Tiny default full 31.91**, **rare 25.27**, **known-object full 33.28**, **rare 27.24**; and **Swin-Large default full 33.69**, **rare 27.66**, **known-object full 34.61**, **rare 28.83** [2508.17760]. Relative to **InteractDiffusion**, CEIDM improves **FID from 18.69 to 16.12**, **Swin-Tiny default full mAP from 29.53 to 31.91**, and **Swin-Large default full mAP from 31.56 to 33.69** [2508.17760]. The paper also gives HICO-DET ground truth values for reference, including **Swin-Tiny default full 29.94**, **known-object full 32.48**, **Swin-Large default full 37.18**, and **known-object full 38.93** [2508.17760].

The qualitative analysis attributes these improvements to better **action correctness**, **interaction rationality**, and **entity quality**. The examples described in the paper report that Stable Diffusion and InteractDiffusion frequently misrender “sitting on,” “carrying,” and similar actions; that backpacks may float or hands may fail to contact objects correctly; and that CEIDM produces more plausible poses, ground contact, and body-object alignment [2508.17760]. For complex scenes involving multiple interactions, the paper states that CEIDM is the only method among those compared that reasonably renders all relationships and accurate actions simultaneously [2508.17760].

The ablation study progressively adds components to the InteractDiffusion baseline. The baseline has **FID 18.69**, **KID 0.00676**, and **Swin-Tiny mAP full 29.53**. Adding **Cot.** yields **FID 18.09** and **mAP full 29.97**; adding **Cl.** yields **FID 17.85** and **mAP full 31.18**; adding **Ec.** yields **FID 16.25** and **mAP full 31.64**; and adding **$\delta$** yields **FID 16.12** and **mAP full 31.91** [2508.17760]. The paper also reports sensitivity analysis over $\delta$ from **1.0 to 2.0**, with the best results around **$\delta = 1.3$** [2508.17760].

The method is also described as **transferable** to personalized Stable Diffusion models, where it preserves style while improving HOI correctness and entity quality [2508.17760]. Hyperparameters reported in the paper include **$\delta = 1.3$**, **IEA fusion weight $w = 0.7$**, **sampling strategy $s_1 = 1$, $s_2 = 0.7$**, and **mask temperature $temp = 2.0$** [2508.17760].

The reported limitations are equally explicit. CEIDM relies on LLM quality, so poor prompts or unusual domains may yield incorrect triplets and misguide the generator [2508.17760]. Very intricate multi-entity scenes with subtle actions remain difficult, and the paper states that there is “still room for further improvement in rendering finer action details” [2508.17760]. The approach also introduces substantial computational overhead at inference time, with the cited **185 hours** for full HICO-DET test-set inference [2508.17760]. Finally, the method is built and tested on **human-object interactions** with HICO-DET and assumes clear subject-action-object structure, verbs amenable to CLIP embedding and clustering, and bounding-box annotations for explicit interactions [2508.17760]. This suggests that adaptation may be required for abstract domains, non-human interaction settings, or cases lacking HOI-style supervision.

Source: https://www.emergentmind.com/topics/ceidm