Textual Inversion in Diffusion Models

• Textual Inversion is a method that represents new concepts as learnable pseudo-words in the text embedding space of frozen diffusion or multimodal models.
• It efficiently personalizes image generation using only 3–5 images by optimizing a single embedding without fine-tuning the entire model.
• Variants extend TI by utilizing per-layer, multiresolution, and neural mapper strategies, enhancing reconstruction quality, editability, and cross-domain applications.

Textual Inversion (TI) is a family of methods that represent a novel concept as one or more learnable pseudo-words in the text embedding space of a frozen generative or multimodal model. In its canonical diffusion formulation, TI associates a new placeholder token $S_*$ with a learned embedding $v_*$ inside the text encoder of a frozen latent diffusion model, so that prompts containing $S_*$ can generate recognizable variations of a user-provided object, pet, style, or other concept from only $3\!-\!5$ images (Gal et al., 2022). Subsequent work reused the term for related operations such as per-layer tokens, multiresolution tokens, encoder-predicted tokens, and cross-modal pseudo-words, so TI now denotes both the original single-token personalization method and a broader design pattern for mapping external concepts into text space (Luo et al., 2024).

1. Canonical latent-diffusion formulation

In the original formulation, TI is defined for personalized text-to-image generation with a frozen Latent Diffusion Model (LDM). An image $x$ is encoded by an autoencoder $\mathcal{E}$ into a latent $z$, noise is added to obtain $z_t$, and a denoiser $\epsilon_\theta$ is trained to predict the noise conditioned on a text representation $c_\theta(y)$. The LDM loss is

$v_*$0

TI chooses the token embedding space of the text encoder as the inversion space: a new pseudo-word $v_*$1 is added to the tokenizer, its embedding lookup is overridden by a learnable vector $v_*$2, and all other embeddings and model parameters remain frozen. The optimization becomes

$v_*$3

Prompts such as “a photo of $v_*$4 on the beach” or “an oil painting of $v_*$5” then use $v_*$6 as if it were an ordinary vocabulary item (Gal et al., 2022).

A central empirical result of the original method is that a single word embedding is often sufficient to capture surprisingly rich and specific visual concepts, including identity, fine appearance, and personalized styles. Because the diffusion model and text encoder remain frozen, TI preserves the pretrained model’s generality and compositional language interface while avoiding the catastrophic forgetting, overfitting, and compute costs associated with full-model fine-tuning (Gal et al., 2022).

2. Training protocol, prompt design, and data regime

The canonical TI training recipe pairs each concept image with a neutral template prompt containing the placeholder token. The templates are derived from CLIP ImageNet templates and include forms such as “a photo of a $v_*$7,” “a rendering of a $v_*$8,” and “a close-up photo of the $v_*$9.” A template is sampled randomly at each iteration. This anchors $S_*$0 in syntactic contexts where ordinary object words appear, avoids long-caption failure modes, and improves later editability under prompts that add scenes, styles, or functions (Gal et al., 2022).

The learned embedding is initialized from a coarse descriptor word such as “cat” or “sculpture,” placing the optimization near an existing semantic region. The reported default setup uses batch size $S_*$1 on $S_*$2 NVIDIA V100 GPUs, base learning rate $S_*$3 scaled to an effective $S_*$4, and $S_*$5 optimization steps, with the same diffusion training scheme and noise schedule as the original LDM. Evaluation uses $S_*$6 DDIM steps, and training a single concept takes roughly $S_*$7 hours in that setup (Gal et al., 2022).

Data efficiency is a defining characteristic. The standard regime is $S_*$8 images per concept. The paper reports that increasing the dataset size beyond roughly $S_*$9 images tends to push the learned embedding farther from the distribution of real words, degrading editability while barely improving semantic reconstruction as measured by CLIP similarity. This places TI in a deliberately small-data, parameter-efficient regime: concept learning is performed by optimizing a single embedding row rather than modifying the large denoiser or text model (Gal et al., 2022).

3. Capabilities, evaluation, and failure modes

Once trained, TI supports several kinds of personalized generation. Reconstruction prompts such as “a photo of $3\!-\!5$0” yield variations that preserve identity and fine appearance while varying pose, background, and composition. The same token can be composed with scene modifiers (“$3\!-\!5$1 on the moon”), style modifiers (“watercolor painting of $3\!-\!5$2”), functional transformations (“a $3\!-\!5$3 backpack”), or other learned tokens (“photo of $3\!-\!5$4 in the style of $3\!-\!5$5”). Style-oriented training prompts can also produce style tokens that transfer to arbitrary subjects, and curated TI tokens can be used to reduce bias in prompts such as “doctor” by replacing the original word with a more diverse learned embedding (Gal et al., 2022).

Evaluation in the original paper separates semantic reconstruction from editability. Reconstruction quality is measured by generating $3\!-\!5$6 samples from “A photo of $3\!-\!5$7” and computing the average pair-wise cosine similarity in CLIP space between generated images and training images. Editability is measured by generating $3\!-\!5$8 samples for prompts such as “A photo of $3\!-\!5$9 on the moon” or “An oil painting of $x$0,” averaging the generated-image CLIP embeddings, and comparing them with the CLIP embedding of the prompt with the placeholder removed. On both automatic metrics and two human studies of $x$1 responses each, TI outperforms human-written descriptions and other baselines while lying on a distortion–editability trade-off curve whose sweet spot is the default single-token setting (Gal et al., 2022).

That trade-off remains central. Higher learning rates, more tokens, or embeddings that deviate farther from the natural word distribution improve reconstruction but reduce prompt adherence; lower learning rates or stronger regularization improve editability but weaken concept specificity. Canonical limitations include imperfect shape fidelity, optimization time of about two hours per concept in the original setup, weak handling of complex relational prompts such as “place $x$2 next to $x$3,” and inheritance of biases or misuse pathways from the base model (Gal et al., 2022). One response to misuse is concept watermarking, which embeds a serial-number-like message directly into the TI concept and decodes it from generated images; with $x$4-bit watermarks, the reported bit error rate is about $x$5 and the success rate about $x$6, while preserving high image and text alignment (Feng et al., 2023).

4. Structured variants and optimization strategies

A convenient way to organize later TI work is by asking which constraint of the canonical method is being relaxed: a single token, a single shared conditioning vector across the denoiser, gradient-based optimization, unconstrained embedding geometry, or a fixed manually chosen dataset. Representative variants are summarized below.

Variant	Core modification	Representative property
Extended TI / $x$7 (Voynov et al., 2023)	Per-layer textual conditions for the U-Net	Better reconstruction and editability; about $x$8 steps and $x$9 min per concept
Multiresolution TI (Daras et al., 2022)	Different pseudo-words for different diffusion-time resolutions	$\mathcal{E}$0 preserves exact object; $\mathcal{E}$1 preserves rough outlines and colors
Gradient-Free TI (Fei et al., 2023)	CLIP-based initialization plus subspace CMA-ES	Comparable quality without backprop; $\mathcal{E}$2 human pass rate
Controllable TI (Yang et al., 2023)	Active-learning data selection with weighted scoring	$\mathcal{E}$3 decrease in FID and $\mathcal{E}$4 boost in R-precision
BRAT (Baker, 2024)	Bonus orthogonal tokens; architecture-agnostic conditioning	Bonus token improves source adherence; vision transformer improves prompt adherence
Directional TI (Kim et al., 15 Dec 2025)	Fixed in-distribution norm, direction-only optimization on a sphere	Better text fidelity while maintaining subject similarity

Extended Textual Inversion places TI in an enlarged conditioning space $\mathcal{E}$5 with one learnable token per cross-attention layer of the denoising U-Net, rather than one token shared across all layers. This yields faster convergence, more regular inversions, and layerwise control that supports object–style mixing (Voynov et al., 2023). Multiresolution TI instead allocates different embeddings to different diffusion-time ranges, exposing coarse and fine concept structure separately and allowing prompts to request varying levels of agreement to the original concept (Daras et al., 2022).

Other work changes the optimization geometry rather than the conditioning structure. Gradient-Free TI replaces backpropagation with CLIP-guided initialization, low-dimensional subspace search, and CMA-ES, showing that forward-only access can still optimize a usable inversion (Fei et al., 2023). Controllable TI adds active data acquisition from large noisy web pools and a theoretically motivated weighted score combining aesthetics and concept matching, explicitly targeting the empirical brittleness of ordinary TI (Yang et al., 2023). Directional TI argues that many failures on complex prompts arise from norm inflation in pre-norm text encoders; it fixes embedding magnitude to an in-distribution scale and optimizes only direction via Riemannian SGD on the unit sphere, improving prompt fidelity and enabling meaningful spherical interpolation between learned concepts (Kim et al., 15 Dec 2025). This suggests that many TI pathologies are failures of conditioning geometry and data selection rather than failures of the underlying denoising objective itself.

5. From per-concept optimization to amortized embeddings and continuous control

A major later direction replaces per-concept embedding optimization with learned mappers that predict or refine TI embeddings from other signals. Viewpoint Neural Textual Inversion learns a small neural mapper $\mathcal{E}$6 from continuous camera viewpoint parameters to a view token $\mathcal{E}$7, showing that Stable Diffusion’s text space contains a continuous view-control manifold for particular scenes and evidence of a generalized view-control manifold across scenes; the same mechanism yields state-of-the-art LPIPS for single-image novel view synthesis on DTU (Burgess et al., 2023).

Encoder-based TI also appears outside image synthesis. In dance-to-music generation, dual-path rhythm–genre inversion maps dance motion into two pseudo-words, @ and *, inserted into prompts such as “a @ music with * as the rhythm.” The base text-to-music model stays frozen while learned rhythm and genre encoders generate dynamic token embeddings from motion, turning TI into a functional mapping rather than a static per-concept vector (Li et al., 2024).

Identity-preserving portrait personalization pushes this amortization further. ID-EA keeps the standard TI token $\mathcal{E}$8 but adds an ID-Enhancer that aligns face-recognition embeddings with a textual identity anchor and an ID-Adapter that injects the aligned identity signal into UNet cross-attention. On the reported person benchmark, ID-EA improves identity similarity to $\mathcal{E}$9, prompt similarity to $z$0, and IQA to $z$1, while requiring $z$2 seconds versus $z$3 seconds for vanilla TI (Jin et al., 16 Jul 2025). For arbitrary objects, a zero-shot personalization framework trains a $z$4-layer MLP to predict TI embeddings from concatenated CLIP image and text features, then fine-tunes only cross-attention blocks; on Custom101, the reported test-time cost is about $z$5 seconds per concept versus about $z$6 seconds for standard TI (Roy et al., 24 Mar 2026). A plausible implication is that TI increasingly functions as a target representation space for amortized concept extractors, not only as an optimization problem solved from scratch for each new concept.

6. Cross-domain generalizations and semantic broadening

TI has been adapted to domains far from open-domain text-to-image synthesis. In medical image generation, Stable Diffusion v2.0 can be adapted with TI embeddings trained on $z$7 images per concept for prostate MRI, chest X-ray, and histopathology. The paper reports that larger embeddings and more examples are necessary in this domain, and that adding TI-generated multi-modal prostate MRI data increases prostate-cancer classification AUC from $z$8 to $z$9; it also demonstrates disease interpolation, pathology composition, and inpainting-based lesion control (Wilde et al., 2023). In 3D reconstruction, MTFusion uses a multi-word TI prompt of the form “a <style> image of <object> <etc>” as a first stage for single-image 3D generation, and its ablations indicate that style, object, and residual tokens improve PSNR, LPIPS, and CLIP similarity over weaker tokenizations (Liu et al., 2024).

Other papers extend the term beyond user-visible concept tokens. In radiology report generation, TISR calls a $z_t$0-layer MLP that maps image patches into a sequence of pseudo words a form of textual inversion, then refines those pseudo words with contrastive self-supervision before decoding a report (Luo et al., 2024). In transferable adversarial attacks on person re-identification, AP-Attack trains five TI networks to map a pedestrian image into attribute-specific pseudo tokens inserted into an attribute-structured prompt, then perturbs images by pushing adversarial tokens away from benign ones and toward least similar attribute semantics (Bian et al., 27 Feb 2025). In classification, Multi-Class TI treats each class as a TI concept and adds a cosine-softmax discriminative regularizer so that the learned tokens serve as a semantic-agnostic classifier while preserving generation quality (Wang et al., 2024). This suggests that “Textual Inversion” has become a broader term for representing non-text inputs as token-like embeddings inside a text-conditioned backbone, even when the end task is reporting, retrieval, classification, or attack rather than personalized image synthesis.