---
title: Image-to-Ego Encoder (I2E Encoder)
url: https://www.emergentmind.com/topics/image-to-ego-encoder-i2e-encoder
type: topic
---

# Image-to-Ego Encoder (I2E Encoder)

The **Image-to-Ego Encoder (I2E Encoder)** denotes an encoder that maps input observations into an internal representation used to control generation, most directly in identity-conditioned image synthesis. In the literature considered here, the term is most naturally aligned with the **identity encoder** in personalized diffusion, which converts one or more reference images of a subject into an identity representation that conditions a diffusion generator [2304.07429]. In customized portrait generation (CPG), the same role is played by the image encoder \(\varepsilon\), which maps an input identity image \(x\) to an embedding \(z=\varepsilon(x)\) that anchors the generated portrait via \(y=D(z,t)\), with \(t\) the text prompt; this is precisely the locus at which identity is injected and, correspondingly, can be selectively suppressed [2606.29880]. The acronym is not uniform across the broader literature: it is also used for an image-to-environment decomposer in text-guided editing [2601.03741], and an analogous encoder-to-generator interface appears in EEG-to-image synthesis [2402.10115].

## 1. Definition and representational role

In personalized diffusion, the encoder-based formulation replaces per-identity fine-tuning with a shared pipeline consisting of an **identity encoder** \(E\) and a diffusion generator \(G\). Given a set of reference images for subject \(s_i\), \(Y_i=\{\mathbf{y}_i^j\}_{j=1}^N\), the objective is
\[
G(\epsilon, E(Y_i)) \sim P_i,
\]
where \(P_i\) is the distribution of images of that subject and \(\epsilon\) is random noise [2304.07429]. The encoder therefore implements an image-to-identity mapping: multiple images of the same identity are projected to a shared conditioning code, and the generator decodes that code into new samples of the same person.

The encoder is explicitly **set-based** rather than restricted to a single image. Per-image features are first extracted as
\[
\mathbf{z}=Enc(\mathbf{y}),
\]
and then aggregated across references:
\[
E(Y_i)=Aggregate\left(\{Enc(\mathbf{y}_i^j)\}_{j=1}^{N}\right).
\]
The aggregation used is **average pooling**, which permits an arbitrary number of reference images and is intended to produce a more robust identity representation [2304.07429]. During training, the model also uses a random weighted average embedding,
\[
\tilde{\mathbf{z}}_i=\sum_{j=1}^{N} w_j Enc(\mathbf{y}_i^j), \quad w_j\in[0,1], \quad \sum_j w_j=1,
\]
so that any point in the convex hull of embeddings from the same identity remains a valid identity representation [2304.07429].

The same representational logic appears in CPG. There, the image encoder converts an identity image into a **CLIP-style embedding** used by a conditional diffusion model, with the encoder functioning as the place “where identity information is injected into the generation process” [2606.29880]. This is why encoder-only interventions can alter what identity the generator sees without modifying the denoiser.

## 2. Architecture and training objectives

The personalized diffusion framework extends **Diffusion Autoencoder (DiffAE)** in two ways: a **set-based identity encoder** and a **personalized diffusion generator** [2304.07429]. The diffusion generator is a U-Net with stacked convolutional encoder/decoder, dense skip connections, base channels \(128\), channel multipliers \([1,1,2,2,4,4]\), attention at \(8\times 8\) and \(4\times 4\), and \(8\) heads. The identity encoder uses channel multipliers \([1,1,2,2,4,4,4]\), attention at \(8\times 8\), \(4\times 4\), and \(2\times 2\), and replaces some attention layers with **Vision Transformer-style** implementation [2304.07429].

The diffusion component is trained with the standard denoising loss
\[
\mathcal{L}_{diff}=\left\| \epsilon_\theta(\mathbf{x}_t,t,\mathbf{z})-\epsilon_t \right\|,
\]
where \(\mathbf{z}\) is the learned identity embedding rather than a semantic code from a single image [2304.07429]. The final objective combines three terms:
\[
L=L_{diff}(I_{id})+\alpha_1 L_{id}(I_{id})+\alpha_2 L_{diff}(I_g),
\]
with \(\alpha_1=\alpha_2=0.01\) in practice [2304.07429].

The term \(L_{id}\) is a **soft nearest-neighbor loss** that pulls together same-identity embeddings and pushes apart different identities within a batch [2304.07429]. The \(L_{diff}(I_g)\) term reflects multi-task learning with identity-agnostic images, where generic face images from **FFHQ** are treated as DiffAE-style reconstruction examples. This enlarges the training set beyond identity-labeled data and is reported to improve generalization [2304.07429]. The training procedure also uses **identity dropout**, with probability \(p\) decreasing from \(1.0\) to \(0.05\), so that the model first learns reconstruction and then progressively learns stronger identity representations [2304.07429].

A central design claim is that a standard image encoder’s one-to-one mapping is inadequate for personalized generation. The paper explicitly states that personalized generation requires a **many-to-many mapping**: one identity code must support many images of the same person, and different subsets of reference images should still correspond to the same identity [2304.07429]. Direct DiffAE-style training is said to degenerate into outputs with **little variation**, motivating the more structured identity space.

## 3. Inference regime, conditional extensions, and empirical behavior

At test time, a new identity is handled without subject-specific retraining. The procedure is: provide a few reference images \(Y_i\), compute \(E(Y_i)\), sample \(\epsilon\sim\mathcal{N}(0,I)\), and run the diffusion sampler conditioned on that embedding [2304.07429]. The method is designed to work with fewer than \(10\) references and sometimes even \(1\), and the reported inference configuration uses **DDPM sampling for 1000 steps** [2304.07429].

The framework also supports conditional generation tasks such as **super-resolution** and **inpainting**. For these tasks, a **condition encoder** extracts features from the input condition, and a **cross-attention layer** injects those features into the diffusion generator; the personalized generator is frozen, and only the new condition encoder and cross-attention layers are trained [2304.07429]. This conditioning mechanism is described as analogous to Stable Diffusion-style conditioning.

The reported training setup uses merged **CelebA**, **CelebRef-HQ**, and the MyStyle training split, with the MyStyle test split for evaluation, FFHQ for additional pretraining, image resolution **\(128\times 128\)**, **AdamW**, **1,000,000** training steps, batch size **128**, learning rate **\(5\times 10^{-5}\)**, warmup \(10^4\) linear steps, diffusion steps **1000**, and a cosine \(\beta\) schedule with \(\beta_0=0.0001\) and \(\beta_T=0.02\) [2304.07429].

Empirically, the encoder-based method is reported to outperform **DiffAE** and **MyStyle** in personalized generation. The main table gives **Ours: ID 0.119, FID 92.9, Diversity 0.115**; **MyStyle: ID 0.117, FID 214.9, Diversity 0.027**; and **DiffAE: ID 0.135, FID 118.0, Diversity 0.028** [2304.07429]. For inpainting and super-resolution, the reported numbers are **Inpainting: ID 0.110, FID 94.0, LPIPS 0.125** and **Super-resolution: ID 0.118, FID 98.6, LPIPS 0.146** [2304.07429]. The user study states preferences of **99.2%** over DiffAE for inpainting, **97.5%** over DiffAE for super-resolution, **95.8%** over MyStyle for inpainting, and **96.7%** over MyStyle for super-resolution [2304.07429]. The ablation further attributes the best results to the joint use of average embedding, multi-task learning, and identity loss [2304.07429].

## 4. Encoder-only identity unlearning in customized portrait generation

A second major use of the I2E concept is as the target of **identity unlearning**. In CPG, public generators can reproduce a person’s identity from an input image and a text prompt, which raises privacy risks such as impersonation, deepfakes, and unauthorized identity replication [2606.29880]. The paper argues that image-level defenses are insufficient because only protected images are affected, whereas unprotected photos of the same person may still be used. It therefore advocates **model-level unlearning**, specifically by modifying only the image encoder while freezing the diffusion backbone [2606.29880].

The CPG pipeline is formalized as
\[
z=\varepsilon(x), \qquad y=D(z,t),
\]
where \(\varepsilon\) is the image encoder and \(D\) is the conditional diffusion model [2606.29880]. The baseline unlearning objective separates identities into targets to forget, \(\mathcal{I}_{-}\), and retained identities, \(\mathcal{I}_{+}=\mathcal{I}\setminus\mathcal{I}_{-}\). For retained identities, it minimizes the discrepancy between the original and unlearned encoder outputs:
\[
\mathcal{L}_{r}=\bigl\| z_{+}-z'_{+}\bigr\|_2,
\]
where \(z_{+}=\varepsilon_u(x_{+})\) and \(z'_{+}=\varepsilon(x_{+})\) [2606.29880]. For target identities, the baseline pushes the embedding away from the original output across the entire feature space.

The paper identifies the weakness of this baseline as **global perturbation**. The encoder embedding is not purely identity-specific; it also carries pose, lighting, background, and other generation-relevant cues. Perturbing the whole vector therefore harms fidelity for retained identities, and the degradation becomes more severe as more target identities are unlearned [2606.29880].

To address this, **IREU** first locates identity-related features offline and perturbs only those coordinates. For a target image \(x_-\), it constructs a swapped image \(x_f=F(x_-,x_a)\), where \(x_a\) is randomly sampled from a retained identity and \(F(\cdot)\) is an off-the-shelf Face-Swap function. With \(z_-=\varepsilon(x_-)\) and \(z_f=\varepsilon(x_f)\), it computes
\[
\Delta z=z_- - z_f.
\]
Dimensions with large \(|\Delta z_i|\) are treated as identity-sensitive, and the mask is defined by
\[
\mathcal{M}=\mathbb{I}\big(\operatorname{rank}(|\Delta z|_{[i]})\le \lfloor k\cdot d\rfloor\big),
\]
where \(d\) is the embedding dimension and \(k\in(0,1]\) is the kept ratio [2606.29880]. The virtual identity target is then
\[
\tilde{z}_{-}=z_{-}+\alpha(\mathcal{M}\odot \Delta z),
\]
and the forgetting loss is
\[
\mathcal{L}_{f}=\bigl\| z_{-}-\tilde{z}_{-}\bigr\|_2.
\]
The full objective is
\[
\mathcal{L}_{u}=\lambda_{1}\mathcal{L}_{f}+\mathcal{L}_{r}.
\]
The diffusion model remains frozen, gradients do not flow through Face-Swap, and only the image encoder \(\varepsilon_u\) is optimized [2606.29880].

This encoder-only design is reported to generalize across generators that share the same encoder family. The paper explicitly transfers the unlearned encoder from one generator to **PhotoMaker** and from single-subject **FastComposer** to multi-subject FastComposer, with no additional fine-tuning [2606.29880]. On retained-identity fidelity, the oneID table reports **PSNR 28.88**, **SSIM 0.92**, **LPIPS 0.04**, and **\(\Delta\text{FID}=1.02\)** for IREU, while the baseline has lower fidelity [2606.29880]. The ablation states that increasing \(k\) worsens fidelity because more identity-agnostic dimensions are perturbed, motivating the choice \(k=0.4\), and that increasing \(\alpha\) strengthens forgetting until saturation, with \(\alpha=100\) selected as a practical default [2606.29880].

## 5. Related usages of the acronym and adjacent encoder paradigms

The acronym **I2E** is used differently in **“I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing”**, where it denotes a **Decomposer** that transforms an image \(I\) into a structured environment
\[
\mathcal{E}=(\{L_i\}_{i=1}^{N},B),
\qquad
L_i=\{\tilde{I}_i,m_i,D_i\},
\]
with object appearance \(\tilde{I}_i\), mask \(m_i\), depth score \(D_i\), and background \(B\) [2601.03741]. The system identifies and segments relevant instances, completes occluded content, reconstructs the background, and computes a physically consistent layer order using a DAG-based Spatial Constraint Propagation Algorithm. The resulting environment supports atomic actions such as `REMOVE`, `MOVE`, `KEEP`, `FALL`, `RESIZE`, `RETOUCH`, `EDIT`, and `INSERT` [2601.03741]. This is a structurally different use of “I2E”: it is image-to-environment rather than image-to-identity.

A related but modality-shifted example appears in EEG-to-image synthesis. There, a **Transformer-encoder based EEG encoder** called **C-former** is pretrained for EEG classification; after training, the softmax layer is removed and the **second-last fully connected layer output** is used as the EEG encoding fed to a GAN generator [2402.10115]. The generator takes this embedding as its **sole input**, without extra random noise, and produces a **\(64\times 64\times 3\)** RGB image [2402.10115]. The encoder has a convolution module with kernels of size \((1,5)\) and \((ch,1)\), a self-attention module with **\(h=8\)** heads, and a classifier head trained with cross-entropy loss [2402.10115]. The reported results give a mean class diversity score of **0.6501** versus **0.7897** for ThoughtViz, and inception scores of **5.1** and **4.62** under two evaluation conditions, described as comparable to prior methods while improving class specificity [2402.10115].

Taken together, these papers show that “I2E Encoder” does not denote a single canonical architecture across arXiv. In the identity-generation literature it refers to an identity-conditioning encoder; in text-guided editing it denotes image-to-environment decomposition; and in EEG-driven generation it describes a brain-signal-to-image-relevant latent mapper by analogy [2304.07429; 2601.03741; 2402.10115].

## 6. Conceptual significance, misconceptions, and limitations

A recurring misconception is that personalized generation can be reduced to a conventional image encoder plus a generative model. The personalized diffusion work explicitly rejects this: a standard encoder’s one-to-one mapping is insufficient because the problem requires a many-to-many relation between identity codes and possible images of the same subject [2304.07429]. Another misconception, addressed by the CPG unlearning paper, is that identity removal should occur in the diffusion U-Net. That paper argues the U-Net contains broad generative priors, so erasing identity information there can cause catastrophic forgetting and degrade image quality; encoder-only unlearning is proposed precisely to avoid that failure mode [2606.29880].

The privacy argument is also specific. The CPG paper does not present encoder-only unlearning merely as a modeling convenience, but as a deployment mechanism compatible with “right to be forgotten” style requirements, computational efficiency, and reuse across multiple generators with the same encoder interface [2606.29880]. This suggests that, in this line of work, the I2E Encoder is best understood as an **identity gateway**: it is the component through which subject identity enters the generator, and therefore the most direct place to implement both personalization and selective erasure.

At the same time, the current formulations remain bounded by their experimental settings. The unlearning work follows a **one-shot setting** for target identities, because prior work typically uses one image per identity for targets to be forgotten [2606.29880]. The original personalized diffusion system operates at **\(128\times 128\)** resolution and uses **1000** diffusion steps at inference [2304.07429]. The EEG-conditioned generator produces only **\(64\times 64\times 3\)** images and is evaluated with class-centric metrics rather than exact image reconstruction, since the EEG segments are not paired one-to-one with specific image instances [2402.10115]. These constraints do not negate the encoder’s role, but they delimit the regimes in which the reported results are established.

Across these usages, the shared principle is that generation quality and controllability depend critically on the structure of the intermediate representation. Whether the encoder maps images to identities, images to layered environments, or EEG to class-specific embeddings, the encoder is not treated as a generic feature extractor; it is the module that defines what latent variable the downstream generator or editor is allowed to act upon [2304.07429; 2606.29880; 2601.03741; 2402.10115].

Source: https://www.emergentmind.com/topics/image-to-ego-encoder-i2e-encoder