Image-to-Ego Encoder (I2E Encoder)

• Image-to-Ego Encoder (I2E Encoder) is an encoder that converts sets of input images into unified identity representations for personalized diffusion.
• It aggregates features using average pooling and employs soft nearest-neighbor loss to ensure a many-to-many mapping between identities and generated samples.
• The encoder also supports model-level identity unlearning, enabling selective perturbation of identity-specific features to enhance privacy in portrait generation.

The Image-to-Ego Encoder (I2E Encoder) denotes an encoder that maps input observations into an internal representation used to control generation, most directly in identity-conditioned image synthesis. In the literature considered here, the term is most naturally aligned with the identity encoder in personalized diffusion, which converts one or more reference images of a subject into an identity representation that conditions a diffusion generator (Su et al., 2023). In customized portrait generation (CPG), the same role is played by the image encoder $\varepsilon$, which maps an input identity image $x$ to an embedding $z=\varepsilon(x)$ that anchors the generated portrait via $y=D(z,t)$, with $t$ the text prompt; this is precisely the locus at which identity is injected and, correspondingly, can be selectively suppressed (Shi et al., 29 Jun 2026). The acronym is not uniform across the broader literature: it is also used for an image-to-environment decomposer in text-guided editing (Yu et al., 7 Jan 2026), and an analogous encoder-to-generator interface appears in EEG-to-image synthesis (Mishra et al., 2024).

1. Definition and representational role

In personalized diffusion, the encoder-based formulation replaces per-identity fine-tuning with a shared pipeline consisting of an identity encoder $E$ and a diffusion generator $G$. Given a set of reference images for subject $s_i$, $Y_i=\{\mathbf{y}_i^j\}_{j=1}^N$, the objective is

$G(\epsilon, E(Y_i)) \sim P_i,$

where $x$0 is the distribution of images of that subject and $x$1 is random noise (Su et al., 2023). The encoder therefore implements an image-to-identity mapping: multiple images of the same identity are projected to a shared conditioning code, and the generator decodes that code into new samples of the same person.

The encoder is explicitly set-based rather than restricted to a single image. Per-image features are first extracted as

$x$2

and then aggregated across references: $x$3 The aggregation used is average pooling, which permits an arbitrary number of reference images and is intended to produce a more robust identity representation (Su et al., 2023). During training, the model also uses a random weighted average embedding,

$x$4

so that any point in the convex hull of embeddings from the same identity remains a valid identity representation (Su et al., 2023).

The same representational logic appears in CPG. There, the image encoder converts an identity image into a CLIP-style embedding used by a conditional diffusion model, with the encoder functioning as the place “where identity information is injected into the generation process” (Shi et al., 29 Jun 2026). This is why encoder-only interventions can alter what identity the generator sees without modifying the denoiser.

2. Architecture and training objectives

The personalized diffusion framework extends Diffusion Autoencoder (DiffAE) in two ways: a set-based identity encoder and a personalized diffusion generator (Su et al., 2023). The diffusion generator is a U-Net with stacked convolutional encoder/decoder, dense skip connections, base channels $x$5, channel multipliers $x$6, attention at $x$7 and $x$8, and $x$9 heads. The identity encoder uses channel multipliers $z=\varepsilon(x)$0, attention at $z=\varepsilon(x)$1, $z=\varepsilon(x)$2, and $z=\varepsilon(x)$3, and replaces some attention layers with Vision Transformer-style implementation (Su et al., 2023).

The diffusion component is trained with the standard denoising loss

$z=\varepsilon(x)$4

where $z=\varepsilon(x)$5 is the learned identity embedding rather than a semantic code from a single image (Su et al., 2023). The final objective combines three terms: $z=\varepsilon(x)$6 with $z=\varepsilon(x)$7 in practice (Su et al., 2023).

The term $z=\varepsilon(x)$8 is a soft nearest-neighbor loss that pulls together same-identity embeddings and pushes apart different identities within a batch (Su et al., 2023). The $z=\varepsilon(x)$9 term reflects multi-task learning with identity-agnostic images, where generic face images from FFHQ are treated as DiffAE-style reconstruction examples. This enlarges the training set beyond identity-labeled data and is reported to improve generalization (Su et al., 2023). The training procedure also uses identity dropout, with probability $y=D(z,t)$0 decreasing from $y=D(z,t)$1 to $y=D(z,t)$2, so that the model first learns reconstruction and then progressively learns stronger identity representations (Su et al., 2023).

A central design claim is that a standard image encoder’s one-to-one mapping is inadequate for personalized generation. The paper explicitly states that personalized generation requires a many-to-many mapping: one identity code must support many images of the same person, and different subsets of reference images should still correspond to the same identity (Su et al., 2023). Direct DiffAE-style training is said to degenerate into outputs with little variation, motivating the more structured identity space.

3. Inference regime, conditional extensions, and empirical behavior

At test time, a new identity is handled without subject-specific retraining. The procedure is: provide a few reference images $y=D(z,t)$3, compute $y=D(z,t)$4, sample $y=D(z,t)$5, and run the diffusion sampler conditioned on that embedding (Su et al., 2023). The method is designed to work with fewer than $y=D(z,t)$6 references and sometimes even $y=D(z,t)$7, and the reported inference configuration uses DDPM sampling for 1000 steps (Su et al., 2023).

The framework also supports conditional generation tasks such as super-resolution and inpainting. For these tasks, a condition encoder extracts features from the input condition, and a cross-attention layer injects those features into the diffusion generator; the personalized generator is frozen, and only the new condition encoder and cross-attention layers are trained (Su et al., 2023). This conditioning mechanism is described as analogous to Stable Diffusion-style conditioning.

The reported training setup uses merged CelebA, CelebRef-HQ, and the MyStyle training split, with the MyStyle test split for evaluation, FFHQ for additional pretraining, image resolution $y=D(z,t)$8, AdamW, 1,000,000 training steps, batch size 128, learning rate $y=D(z,t)$9, warmup $t$0 linear steps, diffusion steps 1000, and a cosine $t$1 schedule with $t$2 and $t$3 (Su et al., 2023).

Empirically, the encoder-based method is reported to outperform DiffAE and MyStyle in personalized generation. The main table gives Ours: ID 0.119, FID 92.9, Diversity 0.115; MyStyle: ID 0.117, FID 214.9, Diversity 0.027; and DiffAE: ID 0.135, FID 118.0, Diversity 0.028 (Su et al., 2023). For inpainting and super-resolution, the reported numbers are Inpainting: ID 0.110, FID 94.0, LPIPS 0.125 and Super-resolution: ID 0.118, FID 98.6, LPIPS 0.146 (Su et al., 2023). The user study states preferences of 99.2% over DiffAE for inpainting, 97.5% over DiffAE for super-resolution, 95.8% over MyStyle for inpainting, and 96.7% over MyStyle for super-resolution (Su et al., 2023). The ablation further attributes the best results to the joint use of average embedding, multi-task learning, and identity loss (Su et al., 2023).

4. Encoder-only identity unlearning in customized portrait generation

A second major use of the I2E concept is as the target of identity unlearning. In CPG, public generators can reproduce a person’s identity from an input image and a text prompt, which raises privacy risks such as impersonation, deepfakes, and unauthorized identity replication (Shi et al., 29 Jun 2026). The paper argues that image-level defenses are insufficient because only protected images are affected, whereas unprotected photos of the same person may still be used. It therefore advocates model-level unlearning, specifically by modifying only the image encoder while freezing the diffusion backbone (Shi et al., 29 Jun 2026).

The CPG pipeline is formalized as

$t$4

where $t$5 is the image encoder and $t$6 is the conditional diffusion model (Shi et al., 29 Jun 2026). The baseline unlearning objective separates identities into targets to forget, $t$7, and retained identities, $t$8. For retained identities, it minimizes the discrepancy between the original and unlearned encoder outputs: $t$9 where $E$0 and $E$1 (Shi et al., 29 Jun 2026). For target identities, the baseline pushes the embedding away from the original output across the entire feature space.

The paper identifies the weakness of this baseline as global perturbation. The encoder embedding is not purely identity-specific; it also carries pose, lighting, background, and other generation-relevant cues. Perturbing the whole vector therefore harms fidelity for retained identities, and the degradation becomes more severe as more target identities are unlearned (Shi et al., 29 Jun 2026).

To address this, IREU first locates identity-related features offline and perturbs only those coordinates. For a target image $E$2, it constructs a swapped image $E$3, where $E$4 is randomly sampled from a retained identity and $E$5 is an off-the-shelf Face-Swap function. With $E$6 and $E$7, it computes

$E$8

Dimensions with large $E$9 are treated as identity-sensitive, and the mask is defined by

$G$0

where $G$1 is the embedding dimension and $G$2 is the kept ratio (Shi et al., 29 Jun 2026). The virtual identity target is then

$G$3

and the forgetting loss is

$G$4

The full objective is

$G$5

The diffusion model remains frozen, gradients do not flow through Face-Swap, and only the image encoder $G$6 is optimized (Shi et al., 29 Jun 2026).

This encoder-only design is reported to generalize across generators that share the same encoder family. The paper explicitly transfers the unlearned encoder from one generator to PhotoMaker and from single-subject FastComposer to multi-subject FastComposer, with no additional fine-tuning (Shi et al., 29 Jun 2026). On retained-identity fidelity, the oneID table reports PSNR 28.88, SSIM 0.92, LPIPS 0.04, and $G$7 for IREU, while the baseline has lower fidelity (Shi et al., 29 Jun 2026). The ablation states that increasing $G$8 worsens fidelity because more identity-agnostic dimensions are perturbed, motivating the choice $G$9, and that increasing $s_i$0 strengthens forgetting until saturation, with $s_i$1 selected as a practical default (Shi et al., 29 Jun 2026).

5. Related usages of the acronym and adjacent encoder paradigms

The acronym I2E is used differently in “I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing”, where it denotes a Decomposer that transforms an image $s_i$2 into a structured environment

$s_i$3

with object appearance $s_i$4, mask $s_i$5, depth score $s_i$6, and background $s_i$7 (Yu et al., 7 Jan 2026). The system identifies and segments relevant instances, completes occluded content, reconstructs the background, and computes a physically consistent layer order using a DAG-based Spatial Constraint Propagation Algorithm. The resulting environment supports atomic actions such as REMOVE, MOVE, KEEP, FALL, RESIZE, RETOUCH, EDIT, and INSERT (Yu et al., 7 Jan 2026). This is a structurally different use of “I2E”: it is image-to-environment rather than image-to-identity.

A related but modality-shifted example appears in EEG-to-image synthesis. There, a Transformer-encoder based EEG encoder called C-former is pretrained for EEG classification; after training, the softmax layer is removed and the second-last fully connected layer output is used as the EEG encoding fed to a GAN generator (Mishra et al., 2024). The generator takes this embedding as its sole input, without extra random noise, and produces a $s_i$8 RGB image (Mishra et al., 2024). The encoder has a convolution module with kernels of size $s_i$9 and $Y_i=\{\mathbf{y}_i^j\}_{j=1}^N$0, a self-attention module with $Y_i=\{\mathbf{y}_i^j\}_{j=1}^N$1 heads, and a classifier head trained with cross-entropy loss (Mishra et al., 2024). The reported results give a mean class diversity score of 0.6501 versus 0.7897 for ThoughtViz, and inception scores of 5.1 and 4.62 under two evaluation conditions, described as comparable to prior methods while improving class specificity (Mishra et al., 2024).

Taken together, these papers show that “I2E Encoder” does not denote a single canonical architecture across arXiv. In the identity-generation literature it refers to an identity-conditioning encoder; in text-guided editing it denotes image-to-environment decomposition; and in EEG-driven generation it describes a brain-signal-to-image-relevant latent mapper by analogy (Su et al., 2023, Yu et al., 7 Jan 2026, Mishra et al., 2024).

6. Conceptual significance, misconceptions, and limitations

A recurring misconception is that personalized generation can be reduced to a conventional image encoder plus a generative model. The personalized diffusion work explicitly rejects this: a standard encoder’s one-to-one mapping is insufficient because the problem requires a many-to-many relation between identity codes and possible images of the same subject (Su et al., 2023). Another misconception, addressed by the CPG unlearning paper, is that identity removal should occur in the diffusion U-Net. That paper argues the U-Net contains broad generative priors, so erasing identity information there can cause catastrophic forgetting and degrade image quality; encoder-only unlearning is proposed precisely to avoid that failure mode (Shi et al., 29 Jun 2026).

The privacy argument is also specific. The CPG paper does not present encoder-only unlearning merely as a modeling convenience, but as a deployment mechanism compatible with “right to be forgotten” style requirements, computational efficiency, and reuse across multiple generators with the same encoder interface (Shi et al., 29 Jun 2026). This suggests that, in this line of work, the I2E Encoder is best understood as an identity gateway: it is the component through which subject identity enters the generator, and therefore the most direct place to implement both personalization and selective erasure.

At the same time, the current formulations remain bounded by their experimental settings. The unlearning work follows a one-shot setting for target identities, because prior work typically uses one image per identity for targets to be forgotten (Shi et al., 29 Jun 2026). The original personalized diffusion system operates at $Y_i=\{\mathbf{y}_i^j\}_{j=1}^N$2 resolution and uses 1000 diffusion steps at inference (Su et al., 2023). The EEG-conditioned generator produces only $Y_i=\{\mathbf{y}_i^j\}_{j=1}^N$3 images and is evaluated with class-centric metrics rather than exact image reconstruction, since the EEG segments are not paired one-to-one with specific image instances (Mishra et al., 2024). These constraints do not negate the encoder’s role, but they delimit the regimes in which the reported results are established.

Across these usages, the shared principle is that generation quality and controllability depend critically on the structure of the intermediate representation. Whether the encoder maps images to identities, images to layered environments, or EEG to class-specific embeddings, the encoder is not treated as a generic feature extractor; it is the module that defines what latent variable the downstream generator or editor is allowed to act upon (Su et al., 2023, Shi et al., 29 Jun 2026, Yu et al., 7 Jan 2026, Mishra et al., 2024).