Adaptive Text-Image Harmony

• Adaptive Text-Image Harmony is a balance-optimization framework that fuses image and text inputs to create novel, semantically rich hybrid objects.
• It leverages diffusion models with adaptive mechanisms like cross-attention scaling and self-attention injection to balance image fidelity and text editability.
• Empirical results and human evaluations show ATIH outperforms traditional methods by achieving a harmonious blend of source image structure and textual directives.

Searching arXiv for the core ATIH paper and closely related follow-up uses of the term. Searching arXiv for "Adaptive Text-Image Harmony" and the C33D follow-up. Adaptive Text-Image Harmony (ATIH) denotes a diffusion-based method for novel object synthesis that takes an object image $O_I$ and an object text $O_T$ and generates a new object image $O$ that fuses both inputs into a single coherent object, rather than merely placing them together or editing one into the other. It was introduced in “Novel Object Synthesis via Adaptive Text-Image Harmony” (Xiong et al., 2024), where the central problem is the imbalance between text and image conditioning in diffusion models: standard pipelines often produce either text-dominated generation or image-dominated generation, instead of a genuinely fused object. In this formulation, “adaptive” refers to parameter selection that balances the two modalities during inversion and denoising, and “harmony” refers to maximizing similarity to both inputs while penalizing domination by either one (Xiong et al., 2024).

1. Definition and conceptual scope

ATIH is best understood as a balance-optimization framework for image-text object fusion. Its target output is a “novel and surprising” object: a result that is neither a copy of the source image nor a literal rendering of the prompt, but a coherent hybrid object with recognizable traits from both modalities. Representative examples reported for the method include colobus-glass jar, sea lion-glass jar, African chameleon-bird, and mud turtle-car (Xiong et al., 2024).

The method is built on the observation that standard diffusion editing and generation pipelines do not harmonize text and image contributions well. In the original formulation, this failure is attributed to two coupled mechanisms. First, cross-attention can allow text features to suppress image influence. Second, self-attention injection can preserve source-image structure, but excessive preservation harms editability while insufficient preservation harms fidelity. ATIH therefore treats multimodal fusion as a controlled trade-off between source-image fidelity and text-driven editability rather than as a one-shot prompt-conditioning problem (Xiong et al., 2024).

Within the literature, the term has both a strict and a broader use. In the strict sense, ATIH refers to the specific method introduced for novel object synthesis (Xiong et al., 2024). In a broader sense, later work uses the phrase to describe adaptive mechanisms for reconciling textual intent with visual generation, as in text-to-image RL post-training (Shen et al., 16 Jun 2026), adaptive prompt elicitation (Wen et al., 4 Feb 2026), and timestep-wise text-conditioning updates during diffusion sampling (Na et al., 28 Oct 2025). This broader usage suggests a family resemblance rather than a single unified formalism.

2. Core architecture and balancing mechanisms

The original ATIH pipeline builds on SDXL-Turbo and modifies a diffusion editing/inversion process at three points: cross-attention scaling, self-attention injection, and noise optimization (Xiong et al., 2024).

The method defines a Text-Image Diffusion Model (TIDM) with image latent $z_0=\mathcal{E}(O_I)$, text embedding $\tau=\mathcal{E}(O_T)$, and null-text embedding $\tau_N=\mathcal{E}(O_N)$. In the fusion denoising branch, the latent update is

$$z_{t-1} = \nu_t z_t + \beta_t \epsilon_\theta(z_t, t, \tau, \alpha, i) + \gamma_t \epsilon_t.$$

Here $\alpha$ is a scale factor controlling text strength in cross-attention, and $i$ is an injection step controlling how long inversion self-attention maps are reused (Xiong et al., 2024).

The first balancing mechanism is the scaled cross-attention term

$$\text{ScCross-Attn}\left(Q_t^c, K_t^c, V_t^c\right) = M_t^c \cdot \alpha \cdot V_t^c, \quad M_t^c = \text{Softmax}\left(Q_t^c(K_t^c)^T/\sqrt{d}\right),$$

where $O_T$0 comes from the latent stream and $O_T$1 come from the text embedding $O_T$2. Scaling the text-side value feature makes text influence an explicit control variable. Smaller $O_T$3 preserves more source-image traits; larger $O_T$4 injects more text semantics (Xiong et al., 2024).

The second mechanism is self-attention-map injection. Instead of using only the fusion-branch self-attention, ATIH reuses inversion-branch maps $O_T$5 up to a chosen denoising regime:

$O_T$6

with

$O_T$7

This preserves internal spatial relationships from the source image. Larger $O_T$8 implies more injection and thus stronger fidelity; smaller $O_T$9 implies more freedom for semantic transformation (Xiong et al., 2024).

The third mechanism is the optimization of per-step sampling noise $O$0. ATIH reformulates the inversion process in a ReNoise-style parameterization and defines a fidelity term

$O$1

together with a Gaussianity regularizer

$O$2

and combines them as

$O$3

In the paper’s interpretation, $O$4 preserves fidelity, while $O$5 improves editability by encouraging Gaussian-like noise statistics (Xiong et al., 2024).

3. Adaptive search and inference procedure

ATIH is not defined by fixed hyperparameters alone; its distinguishing feature is an adaptive search procedure over the image-preservation and text-injection controls. The paper uses Ancestral-Euler sampling, 4 denoising steps, image resolution $O$6, scale factor range $O$7, and injection-step range $O$8. The default initialization is $O$9 and $z_0=\mathcal{E}(O_I)$0 (Xiong et al., 2024).

The adaptive criterion uses two similarity terms for a fused image $z_0=\mathcal{E}(O_I)$1:

$z_0=\mathcal{E}(O_I)$2

$z_0=\mathcal{E}(O_I)$3

where $z_0=\mathcal{E}(O_I)$4 is cosine similarity computed using DINOv2 features for image-image similarity and CLIP features for text-image similarity (Xiong et al., 2024).

Injection-step adjustment is guided by an image-similarity band

$z_0=\mathcal{E}(O_I)$5

The paper contains a notation inconsistency in the update rule for $z_0=\mathcal{E}(O_I)$6: the main text and appendix give opposite increment/decrement directions. The intended interpretation, stated explicitly in the discussion, is that too little similarity requires more injection and too much similarity requires less injection (Xiong et al., 2024).

Once $z_0=\mathcal{E}(O_I)$7 is fixed, ATIH searches over $z_0=\mathcal{E}(O_I)$8 using the score

$z_0=\mathcal{E}(O_I)$9

with

$\tau=\mathcal{E}(O_T)$0

The first term rewards strong alignment to both inputs; the second penalizes imbalance. The paper maximizes this score with Golden Section Search, thereby turning text-image fusion into a one-dimensional balance search after the image-preservation regime has been fixed (Xiong et al., 2024).

The complete inference workflow is therefore: invert the source image, optimize per-step noise with the balanced loss, initialize $\tau=\mathcal{E}(O_T)$1, adjust $\tau=\mathcal{E}(O_T)$2 until image similarity falls within the desired band, search for the best $\tau=\mathcal{E}(O_T)$3 by maximizing $\tau=\mathcal{E}(O_T)$4, and finally generate the fused image with the selected $\tau=\mathcal{E}(O_T)$5 pair (Xiong et al., 2024).

4. Empirical profile, strengths, and failure modes

The original ATIH study evaluates on a Text-Image Fusion (TIF) dataset of 1,800 text-image pairs, built from 60 texts selected from ImageNet classes and 30 images selected from PIE-bench categories. Reported baselines include MasaCtrl, InfEdit, InstructPix2Pix, MagicMix, ConceptLab, and ControlNet (Xiong et al., 2024).

ATIH’s principal empirical claim is not that it maximizes any single raw similarity score. Instead, it maximizes balanced fusion quality. In the main quantitative table, ATIH reports DINO-I 0.756, CLIP-T 0.296, AES 6.124, HPS 0.383, $\tau=\mathcal{E}(O_T)$6score 1.362, and $\tau=\mathcal{E}(O_T)$7sim 0.075. By comparison, InfEdit has higher DINO-I (0.817), and InstructPix2Pix has higher CLIP-T (0.394), but both are worse on the paper’s balance-sensitive metrics, which is the intended point of the method (Xiong et al., 2024).

The user study includes 95 participants and 570 total votes. Against editing methods, ATIH receives 74.03% of votes; against mixing methods, it receives 79.47%. The paper interprets this as evidence that human judges prefer the harmonized hybrid objects over outputs that merely preserve the source more strongly or follow the text more aggressively (Xiong et al., 2024).

Ablations attribute distinct roles to the three main components. The balanced loss improves fidelity, texture quality, and editability relative to direct PnP-style inversion. The injection step produces a smoother transition between source image and target semantics. The scale factor $\tau=\mathcal{E}(O_T)$8 controls the monotonic trade-off in which image similarity decreases and text similarity increases as $\tau=\mathcal{E}(O_T)$9 grows (Xiong et al., 2024).

The method’s limitations are also explicit. Its performance depends on semantic correlation between the source image and target text in diffusion feature space. When that match is weak, ATIH may produce only texture changes rather than deeper semantic transformation. The appendix also reports notation inconsistencies, including one apparent typo around the parameter $\tau_N=\mathcal{E}(O_N)$0, though the experimental sections consistently support $\tau_N=\mathcal{E}(O_N)$1 as the operative value (Xiong et al., 2024).

5. ATIH as a reusable module in later systems

A major later use of ATIH appears in “Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusion,” where ATIH is the first stage of the C33D pipeline (Xiong et al., 2 Sep 2025). In that setting, the task is no longer 2D novel object synthesis alone, but category-aware 3D object composition: given a source 3D model $\tau_N=\mathcal{E}(O_N)$2 and a target category text $\tau_N=\mathcal{E}(O_N)$3, generate a new 3D object $\tau_N=\mathcal{E}(O_N)$4 that meaningfully blends both.

C33D renders the source 3D model into six RGB views and six normal maps, but ATIH is applied only to the front view $\tau_N=\mathcal{E}(O_N)$5. Its role is summarized as

$\tau_N=\mathcal{E}(O_N)$6

where $\tau_N=\mathcal{E}(O_N)$7 is a fused front-view image that becomes the anchor condition for later multi-view propagation and 3D reconstruction (Xiong et al., 2 Sep 2025).

In C33D, the authors state that they “follow the default configuration using SDXLturbo as the base model” and generate the fusion image by employing the golden-section search strategy to automatically find the optimal text-image harmony. They do not reproduce ATIH’s internal denoising equations or training formulation. The module is therefore treated as a black-box single-view harmonization primitive reused from prior work (Xiong et al., 2 Sep 2025).

Its downstream role is central. The ATIH output is encoded to a latent feature $\tau_N=\mathcal{E}(O_N)$8, which then conditions both texture multi-view diffusion (TMDiff) and shape multi-view diffusion (SMDiff). In TMDiff, $\tau_N=\mathcal{E}(O_N)$9 supplies keys and values for self-attention-based texture transfer to non-front views. In SMDiff, the same feature is injected as a global semantic and appearance control signal into RGB and normal-map denoising. C33D’s “Discussion 1” further reports that applying ATIH independently to all six views produces severe cross-view inconsistency and breaks 3D rendering, which motivates the design choice of restricting ATIH to a single anchor view (Xiong et al., 2 Sep 2025).

This later use clarifies an important conceptual boundary. ATIH is effective as a single-view compositional prior, but it is not itself a multiview-consistent or 3D-aware system. Its reusable strength is front-view image-text fusion; additional modules are needed when geometric coherence or multi-view consistency is required (Xiong et al., 2 Sep 2025).

6. Relation to adjacent adaptive text-image alignment research

Although ATIH in the strict sense refers to the object-fusion method above, later work explores closely related forms of adaptive text-image harmonization in prompting, sampling, RL post-training, and design-oriented generation.

One line of work treats harmony as adaptive credit assignment during training. “STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training” redistributes a scalar image-level reward over latent positions and denoising steps using internal text-image attention, thereby applying stronger updates to prompt-relevant regions while keeping the external reward unchanged (Shen et al., 16 Jun 2026). Another treats harmony as adaptive conditioning during sampling: “Diffusion Adaptive Text Embedding” updates the text embedding online at selected timesteps based on the current intermediate sample and a score on the predicted denoised image, rather than using a fixed prompt embedding throughout diffusion (Na et al., 28 Oct 2025).

A second line treats harmony as interactive intent inference or prompt adaptation. “Adaptive Prompt Elicitation for Text-to-Image Generation” frames misalignment as an information asymmetry problem and adaptively asks visual queries to infer latent user intent before compiling a model-specific final prompt (Wen et al., 4 Feb 2026). “Prompt Expansion for Adaptive Text-to-Image Generation” instead generates a set of expanded prompts that trade off semantics, aesthetics, and diversity, partially adapting prompt generation to downstream model behavior through image-conditioned filtering (Datta et al., 2023).

A third line treats harmony as layout- and design-aware generation. “TextCenGen” adapts generated backgrounds to predefined text regions by relocating conflicting objects and smoothing reserved blank areas for readability (Liang et al., 2024). “TextPainter” learns visually and semantically harmonious poster text generation from global-local background context and text semantics (Gao et al., 2023). In a different direction, “A$$z_{t-1} = \nu_t z_t + \beta_t \epsilon_\theta(z_t, t, \tau, \alpha, i) + \gamma_t \epsilon_t.$$0A” makes subject placement itself a text-conditioned variable for background inpainting, arguing that harmony sometimes requires moving the subject rather than only regenerating the background (Tang et al., 2 Apr 2025). “DreamFuse” similarly treats harmonious fusion as adaptive coordination among text, foreground identity, and scene context, with explicit mechanisms for spatial placement and localized preference refinement (Huang et al., 11 Apr 2025).

These neighboring formulations suggest that ATIH is best situated within a larger research movement toward adaptive multimodal coordination. The original ATIH paper makes this coordination explicit for single-image object fusion through the jointly optimized variables $$z_{t-1} = \nu_t z_t + \beta_t \epsilon_\theta(z_t, t, \tau, \alpha, i) + \gamma_t \epsilon_t.$$1, $$z_{t-1} = \nu_t z_t + \beta_t \epsilon_\theta(z_t, t, \tau, \alpha, i) + \gamma_t \epsilon_t.$$2, and $$z_{t-1} = \nu_t z_t + \beta_t \epsilon_\theta(z_t, t, \tau, \alpha, i) + \gamma_t \epsilon_t.$$3 (Xiong et al., 2024). Later work generalizes the same impulse to multiview conditioning, prompt interaction, reward routing, text-embedding updates, spatial composition, and design-aware generation (Xiong et al., 2 Sep 2025, Shen et al., 16 Jun 2026, Na et al., 28 Oct 2025, Wen et al., 4 Feb 2026, Liang et al., 2024, Gao et al., 2023).

In that broader sense, Adaptive Text-Image Harmony names both a specific method and a more general design principle: textual intent and visual evidence should not be treated as fixed, globally uniform conditions, but as signals whose relative strength, spatial scope, and temporal role must be adjusted to produce coherent multimodal outputs.