---
title: 'EditInfinity: Text-Driven Image Editing'
url: https://www.emergentmind.com/topics/editinfinity
type: topic
---

# EditInfinity: Text-Driven Image Editing

EditInfinity is a text-driven image editing framework that adapts **Infinity**, a **binary-quantized autoregressive generative model**, for image editing rather than following the more common diffusion-model pipeline. Its defining premise is that **VQ-style discrete latent representations allow exact inversion supervision**, thereby avoiding the approximation errors that usually constrain diffusion-based editing. The method combines an inversion stage based on exact quantized token targets with **text prompting rectification**, **image style preservation** via **LoRA**, and a **holistic smoothing strategy** for token-space editing. On **PIE-Bench**, the framework is reported to achieve strong performance across **“add”, “change”, and “delete”** operations while preserving high fidelity to the source image and aligning edits precisely with target text prompts [2510.20217].

## 1. Problem formulation and motivation

EditInfinity is situated within the general paradigm of training-free image editing methods that first perform **image inversion** and then perform **editing** along the inferred generative trajectory. In this standard formulation, a source image is reconstructed into the model’s latent process and then modified under a target text prompt. The paper identifies a central limitation of this paradigm for diffusion models: **the exact forward trajectory that produced the source image is unknown**, so inversion must be approximated with deterministic samplers or optimization-based finetuning. These approximation errors in intermediate latent states degrade editing quality and create a persistent trade-off between source fidelity and editability [2510.20217].

The paper characterizes the downstream effects of inversion error as including **poor reconstruction of the source image**, **drift in background/unchanged regions**, **weak semantic alignment to the target prompt**, and a general **trade-off between fidelity and editability**. Within this framing, EditInfinity does not primarily claim a stronger base generator; rather, it attributes its improvement to replacing approximate diffusion inversion with an editing framework in which intermediate targets are exactly available in a discrete latent space [2510.20217].

This positions EditInfinity against a classical diffusion-editing bottleneck. A plausible implication is that the framework treats editing quality as fundamentally limited by the invertibility of the underlying generative representation, not only by the strength of the editing guidance.

## 2. Binary-quantized foundation and exact inversion supervision

The framework uses **Infinity**, described as a **binary-quantized autoregressive text-to-image model**, as its base generator. Its key property for editing is that images are tokenized into **discrete binary quantized residual tokens**, so the exact latent tokens of a source image can be obtained directly by quantization. The paper contrasts this with diffusion models, where inversion remains approximate [2510.20217].

Infinity is presented as a **bitwise multi-scale residual quantization model**. For an image $I \in \mathbb{R}^{H \times W \times 3}$ encoded into a feature $F$, residuals are formed across scales as

$$
z_k = \text{down}\left(F - F_{k-1}, (h_k, w_k)\right) \in \mathbb{R}^{h_k \times w_k \times d}.
$$

The model then applies **Binary Spherical Quantization (BSQ)**:

$$
R_k = \mathcal{Q}(z_k) = \frac{1}{\sqrt{d}} \operatorname{sign}\left(\frac{z_k}{\|z_k\|}\right),
$$

with cumulative feature reconstruction

$$
F_k = \sum_{i=1}^k \text{up}(R_i, (h_K,w_K)).
$$

Autoregressive modeling proceeds across scales:

$$
p(R_{1:K}|\Psi(t)) = \prod_{k=1}^K p\big(R_k \mid R_1,\dots,R_{k-1}, \Psi(t)\big),
$$

where $\Psi(t)$ is the text embedding from the text encoder, identified in the paper as **Flan-T5** [2510.20217].

These formulations matter because EditInfinity’s inversion stage operates against **exact token targets**. Given a source image $I_{sou}$ and source prompt $t_{sou}$, the method first quantizes the image into exact source tokens $R_{1:K}^{sou}$, then optimizes a learnable prompt to reconstruct those tokens using the inversion loss

$$
\mathcal{L}_{inv} = -\frac{1}{K}\sum_{k=1}^K
\left(
R_k^{sou} \cdot \log p(R_{k}^{inv}|R_{<k}^{sou},\Psi(t))
\right).
$$

The paper’s central claim is that, unlike diffusion inversion, this objective uses **cross-entropy over token predictions with exact supervision**. This suggests that inversion is recast from approximate latent trajectory recovery into supervised token reconstruction, which directly affects downstream edit fidelity [2510.20217].

## 3. Inversion mechanism: prompt rectification and style preservation

EditInfinity augments exact token supervision with two mechanisms: **text prompting rectification** and **image style preservation**. The first addresses the mismatch between a user-provided source prompt and the actual source image; the second addresses preservation of style and structure during inversion and editing [2510.20217].

For **text prompting rectification**, the method begins with the source prompt $t_{sou}$, adds an instruction prompt $t_{ins}$ such as **“the language style of this prompt is”**, and appends **20 learnable prompt tokens** $t_l$. The system encodes $t_{sou}$ and $t_{ins}$ through Infinity’s text encoder $\Psi(\cdot)$ and concatenates

$$
[\Psi(t_{sou}, t_{ins}), t_l].
$$

The backbone remains frozen, and only the learnable prompt tokens are optimized using $\mathcal{L}_{inv}$. The paper describes this as a way to rectify the semantic condition so that it better matches the actual source image, thereby reducing semantic mismatch during inversion [2510.20217].

For **image style preservation**, EditInfinity adds **LoRA** to the **FFN layers of Infinity**. The role of LoRA is described as preserving **global appearance and style** after prompt rectification. The paper motivates this with the low-rank bias of LoRA under the update

$$
W \rightarrow W + \Delta W.
$$

Only the low-rank matrices are updated, and the learned $\Delta W$ is retained during editing. The ablation summary in the paper states that removing both prompt rectification and LoRA causes serious inconsistency; prompt rectification alone improves prompt alignment but can shift style; adding LoRA restores fidelity [2510.20217].

Taken together, these two mechanisms split the inversion problem into semantic alignment and appearance preservation. A plausible implication is that EditInfinity treats inversion not as a single reconstruction objective, but as coordinated control over prompt semantics and source-style retention.

## 4. Editing stage and holistic smoothing

The editing stage uses a **token-level blending** strategy that mixes source and target tokens in a **spatially controlled, multi-scale** manner. For each scale $k$, the model generates target tokens $R_k^{tar}$ under the edited prompt, blends them with source tokens $R_k^{sou}$ using a smoothing kernel $\mathcal{G}$, and then feeds the blended tokens forward to the next scale. The paper emphasizes that smoothing is not applied only as a final compositing step; it is propagated autoregressively across scales [2510.20217].

The smoothing kernel is defined using Manhattan distance to the edit mask $M$:

$$
\mathcal{G}^{i,j} =
\begin{cases}
0, & d^{i,j} \leq \tau_1 \\
\frac{d^{i,j}-\tau_1}{\tau_2 - \tau_1}, & \tau_1 < d^{i,j} < \tau_2 \\
1,  & d^{i,j} \geq \tau_2
\end{cases}
$$

with

$$
d^{i,j} = \min_{(x,y) \in M} \left( |i - x| + |j - y| \right).
$$

The main paper uses $\tau_1=1$ and $\tau_2=4$ [2510.20217].

At each scale, token blending is given by

$$
E_k^{tar} = \mathrm{Upsample}(R_k^{tar}, (h_K,w_K)) \odot (1-\mathcal{G}) + \mathrm{Upsample}(R_k^{sou}, (h_K,w_K)) \odot \mathcal{G}.
$$

If $k < K$, the edited result is downsampled and used as the autoregressive state for the next scale [2510.20217].

The paper attributes two functions to this **holistic smoothing strategy**. First, it improves **fidelity** by preserving unedited background and structure. Second, it improves **semantic alignment** by keeping the edited region guided by the target prompt. The ablations reported in the paper state that removing smoothing leads to visible boundary discontinuities, that the **linear kernel** outperforms a **Gaussian** alternative in the chosen setting, and that **autoregressive blending at every scale** is better than blending only at the end [2510.20217].

This design distinguishes EditInfinity from editing systems that rely on approximate latent-noise inversion followed by guided denoising. Here, editing is closer to controlled autoregressive token replacement under exact source-token supervision.

## 5. Evaluation on PIE-Bench

The main empirical evaluation is conducted on **PIE-Bench**, described as containing **700 test cases** and **9 editing types**, with each case including a **source image**, **source prompt**, **target editing prompt**, and **editing mask**. The paper highlights qualitative results for **add**, **change**, and **delete**, and additionally reports edit types such as **change content**, **change pose**, **change color**, **change background**, **change material**, and **change style** [2510.20217].

Compared baselines include **P2P**, **MasaCtrl**, **P2P-Zero**, **NTI**, **PnP-Inv**, **NP**, **StableFlow**, **RF-Edit**, and **Gemini 2.0** as a closed-source reference. The reported evaluation criteria cover both preservation and alignment: **PSNR**, **LPIPS**, **MSE**, and **SSIM** for background preservation, and **CLIP score on whole image**, **CLIP score on edited region**, and **IR (ImageReward)** for text alignment [2510.20217].

The paper reports the following main scores for EditInfinity on PIE-Bench:

| Metric | Score | Role |
|---|---:|---|
| PSNR | 27.95 | Background preservation |
| LPIPS | 33.08 | Background preservation |
| MSE | 24.27 | Background preservation |
| SSIM | 92.12 | Background preservation |
| CLIP Whole | 26.41 | Text alignment |
| CLIP Edited | 23.47 | Edited-region alignment |
| IR | 5.88 | Overall reward-style evaluation |

These results are presented as the **best overall balance** on PIE-Bench, with the paper specifically stating that EditInfinity outperforms diffusion-based baselines across the trade-off between preserving background or unchanged regions and aligning edits to text prompts. It also states that the method outperforms **RF-Edit**, identified as the strongest diffusion-transformer baseline, on both preservation and alignment, and also exceeds the reported metrics of **Gemini 2.0**. The paper further stresses that this occurs **even though Infinity’s base generative model is not obviously stronger than FLUX on GenEval**, which it uses to argue that the improvement arises from the editing framework itself rather than only from the underlying generator [2510.20217].

The reported runtime characteristics show a marked asymmetry between inversion and editing: **inversion: 107.06 s** and **per-editing: 3.64 s**. The paper presents this as advantageous when multiple edits are performed on the same source image, since inversion is paid once and subsequent edits are relatively fast [2510.20217].

## 6. Ablations, relation to diffusion editing, and limitations

The ablation studies emphasize three components: **learnable prompt + LoRA**, **smoothing kernel**, and **multi-scale autoregressive blending**. The paper states that both learnable prompt and LoRA are important; prompt rectification improves semantic alignment, while LoRA restores source style and structure. It also notes that training LoRA too long overfits and hurts editability [2510.20217].

For the smoothing mechanism, the paper reports that performance is not overly sensitive to whether masks come from **user-provided masks** or from **cross-attention maps**. More importantly, it reports that **without $\mathcal{G}$, boundary artifacts appear**, that **linear smoothing** outperforms **Gaussian smoothing** in the chosen setting, and that **autoregressive blending at every scale** is superior to a non-autoregressive strategy [2510.20217].

In relation to diffusion-based pipelines, EditInfinity is explicitly framed as replacing **approximate latent-noise inversion + guided denoising** with **exact discrete reconstruction + controlled autoregressive token replacement**. That comparison aligns it with a broader set of editing systems that seek stronger control over localization and preservation, but its technical route is distinct: it relies on exact intermediate token supervision rather than time-step inversion in diffusion space [2510.20217].

The paper also states a limitation. EditInfinity can be **less ideal in extreme style-change cases**, especially when **no background preservation is needed** and the image contains **detailed structural patterns**. In such cases, token blending may constrain full transformation and slightly degrade fidelity. This suggests that the same mechanism that preserves local continuity and background structure may also limit radical global transformation when preservation is not the objective [2510.20217].

Within the paper’s own framing, EditInfinity’s main contributions are fivefold: a new editing framework for **binary-quantized generative models**; **exact-supervision inversion** using quantized source tokens; **text prompting rectification** and **LoRA-based style preservation**; **holistic smoothing** for spatially controlled editing; and strong empirical performance on **PIE-Bench** across localized editing operations. Its broader significance lies in showing that image editing quality can be improved by redesigning the inversion substrate itself rather than only improving prompt guidance or diffusion-time control [2510.20217].

Source: https://www.emergentmind.com/topics/editinfinity