IA-RFT for Identity-Preserving T2I

• IA-RFT is a reward tuning mechanism that integrates human-preference appeal and structure rewards to enhance aesthetic quality while preserving identity in T2I.
• It employs a pairwise logistic loss framework to separately train appeal and structure reward models using both human-annotated and automatically constructed datasets.
• The method is implemented in both LoRA and Adapter pipelines, offering flexible integration with minimal changes to existing diffusion models.

Searching arXiv for the cited paper to ground the article in the source record. Identity-Aesthetic Reward Fine-Tuning (IA-RFT) is a reward-based fine-tuning mechanism introduced within ID-Aligner for identity-preserving text-to-image generation (ID-T2I). In that framework, IA-RFT addresses the observation that generated images can lose aesthetic appeal when identity retention is enforced. It does so by defining an aesthetic reward from two separately trained reward models—one for human-annotated “appeal” preferences and one for character-structure quality—and injecting that reward into diffusion-model fine-tuning alongside identity consistency terms. ID-Aligner presents this as part of a general feedback learning framework that can be applied to both LoRA-based and Adapter-based methods on SD1.5 and SDXL (Chen et al., 2024).

1. Position within identity-preserving text-to-image generation

ID-T2I aims to generate images that follow a text prompt while preserving the identity characteristics of a reference portrait. ID-Aligner frames the problem around three challenges: accurately maintaining identity characteristics, preserving aesthetic appeal while enforcing identity retention, and achieving compatibility with both LoRA-based and Adapter-based methods. IA-RFT is the component targeted specifically at aesthetic tuning, while identity consistency reward fine-tuning addresses identity preservation through face detection and recognition feedback (Chen et al., 2024).

Within this formulation, IA-RFT is not an isolated training criterion. It is one term in a broader reward-tuning scheme that combines identity similarity and aesthetics. This distinction is operationally important: the method does not replace the diffusion objective in all settings, and it does not define aesthetics as a single monolithic signal. Instead, it contributes a scalar reward assembled from two learned sub-rewards.

A common misunderstanding is to treat IA-RFT as equivalent to “identity reward tuning.” In ID-Aligner, those are separate but coordinated components. Identity consistency is computed from face embeddings, whereas IA-RFT derives its signal from preference data and automatically constructed structure data. The framework combines them only at the objective level.

2. Aesthetic reward formulation

IA-RFT trains two separate reward models and adds them at fine-tuning or inference time to obtain a scalar aesthetic reward. The first is an appeal reward, $R_{\rm appeal}(x,c)$, trained on a human-preference dataset $\mathcal{D}_{\rm pref}$ of triplets $(c, x^+, x^-)$, where annotators prefer $x^+$ over $x^-$ under prompt $c$. The reward model is parameterized by $f_\theta(x,c)$ and optimized with a Bradley–Terry / logistic pairwise loss (Chen et al., 2024).

The appeal loss is defined as

$$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$

with

$R_{\rm appeal}(x,c)=f_\theta(x,c).$

The second reward model is a structure reward, $R_{\rm struct}(x,c)$, trained on an automatically constructed dataset $\mathcal{D}_{\rm pref}$0 of $\mathcal{D}_{\rm pref}$1 pairs. In that dataset, $\mathcal{D}_{\rm pref}$2 are real human images and $\mathcal{D}_{\rm pref}$3 are identical-prompt but pose-twisted “ControlNet” outputs. The training objective is again pairwise logistic:

$\mathcal{D}_{\rm pref}$4

and

$\mathcal{D}_{\rm pref}$5

The combined aesthetic reward is then

$\mathcal{D}_{\rm pref}$6

This construction makes explicit that “aesthetic” in IA-RFT has two sources: human-annotated preference and an automatically induced structural prior. A plausible implication is that the method operationalizes aesthetics as both subjective appeal and anatomical or pose regularity, rather than as a single learned scalar with unspecified semantics.

3. Losses and optimization objective

ID-Aligner summarizes the losses used when fine-tuning a text-to-image diffusion model to preserve identity and improve aesthetics. The base denoising term is the diffusion reconstruction loss

$\mathcal{D}_{\rm pref}$7

where $\mathcal{D}_{\rm pref}$8 is the UNet noise predictor and $\mathcal{D}_{\rm pref}$9 (Chen et al., 2024).

Identity consistency is computed via cosine similarity between face embeddings of the generated and reference faces:

$(c, x^+, x^-)$0

This is converted into a loss by

$(c, x^+, x^-)$1

The IA-RFT term is the aesthetic reward loss

$(c, x^+, x^-)$2

These are combined into the reward-tuning objective

$(c, x^+, x^-)$3

with the summary stating that in the experiments for the Adapter, $(c, x^+, x^-)$4 and $(c, x^+, x^-)$5.

The total loss depends on the parameterization. For the Adapter-based model,

$(c, x^+, x^-)$6

For the LoRA-based model,

$(c, x^+, x^-)$7

This separation is central to understanding IA-RFT. It is not uniformly attached to all backbones in the same way: the Adapter formulation back-propagates only the reward tuning loss, while the LoRA formulation jointly optimizes denoising and reward terms.

4. Training pipeline and implementation variants

The IA-RFT pipeline has two stages. First, the reward models are trained: a small human-preference dataset $(c, x^+, x^-)$8 of prompt-guided pairwise annotations is collected; the structure dataset $(c, x^+, x^-)$9 is collected via pose twisting plus ControlNet; and $x^+$0 and $x^+$1 are optimized with the logistic losses. Second, the diffusion model—either Adapter or LoRA—is fine-tuned with the learned reward functions (Chen et al., 2024).

For the fine-tuning stage, the optimizer is AdamW. The reported settings are summarized below.

Variant	Optimization target	Settings
Adapter	$x^+$2	lr $x^+$3, batch $x^+$4, train steps $x^+$5 K
LoRA	$x^+$6	UNet-LR $x^+$7, TextEnc-LR $x^+$8, batch $x^+$9, steps $x^-$0 K
Inference	DDIM scheduler	20 sampling steps, guidance $x^-$1

The core per-iteration procedure is illustrated for the Adapter. Given pretrained Stable Diffusion plus Adapter weights $x^-$2, reward models $x^-$3 and $x^-$4, FaceDet and FaceEnc modules, and a dataset $x^-$5, the loop samples $x^-$6, chooses a random denoising step $x^-$7, runs the UNet without gradient from $x^-$8 to get $x^-$9, then runs one step with gradient to obtain $c$0 and predict $c$1. The predicted latent is decoded by the VAE to $c$2, the face is cropped as $c$3, and embeddings are formed for both $c$4 and the reference. The reward terms are then computed as

$c$5

followed by

$c$6

The Adapter weights are updated via AdamW. The LoRA version is identical except that the original denoising MSE $c$7 is added, and only the low-rank LoRA parameters are updated.

5. Empirical evaluation

ID-Aligner reports comparisons against FastComposer, IP-Adapter, InstantID, and PhotoMaker on SD1.5 and SDXL using Face Similarity, DINO score, CLIP-I, CLIP-T, LAION-Aesthetic, and a user study on text fidelity, image quality, and face resemblance (Chen et al., 2024).

On SD1.5, the abridged results are: FaceSim $c$8 for FastComposer, $c$9 for IP-Adapter, and $f_\theta(x,c)$0 for ID-Aligner; DINO $f_\theta(x,c)$1, $f_\theta(x,c)$2, and $f_\theta(x,c)$3; CLIP-I $f_\theta(x,c)$4, $f_\theta(x,c)$5, and $f_\theta(x,c)$6; Aes $f_\theta(x,c)$7, $f_\theta(x,c)$8, and $f_\theta(x,c)$9; and CLIP-T $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$0, $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$1, and $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$2. On SDXL, the abridged results are: FaceSim $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$3 for IP-Adapter, $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$4 for InstantID, $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$5 for PhotoMaker, and $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$6 for ID-Aligner; DINO $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$7, $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$8, $$\mathcal{L}_{\rm pref}(\theta) = -\,\mathbb{E}_{(c,x^+,x^-) \sim \mathcal{D}_{\rm pref}} \left[ \log \sigma\big(f_\theta(x^+,c)-f_\theta(x^-,c)\big) \right],$$9, and $R_{\rm appeal}(x,c)=f_\theta(x,c).$0; CLIP-I $R_{\rm appeal}(x,c)=f_\theta(x,c).$1, $R_{\rm appeal}(x,c)=f_\theta(x,c).$2, $R_{\rm appeal}(x,c)=f_\theta(x,c).$3, and $R_{\rm appeal}(x,c)=f_\theta(x,c).$4; Aes $R_{\rm appeal}(x,c)=f_\theta(x,c).$5, $R_{\rm appeal}(x,c)=f_\theta(x,c).$6, $R_{\rm appeal}(x,c)=f_\theta(x,c).$7, and $R_{\rm appeal}(x,c)=f_\theta(x,c).$8; and CLIP-T $R_{\rm appeal}(x,c)=f_\theta(x,c).$9, $R_{\rm struct}(x,c)$0, $R_{\rm struct}(x,c)$1, and $R_{\rm struct}(x,c)$2.

The summary also reports generalization on DreamShaper and RealVisXL, with $R_{\rm struct}(x,c)$3–$R_{\rm struct}(x,c)$4 FaceSim gains and $R_{\rm struct}(x,c)$5–$R_{\rm struct}(x,c)$6 CLIP-I gains. In a user study with 500 votes, the method ranked first in image quality and second in face resemblance and text fidelity. The stated interpretation is that the approach yields a well-balanced improvement.

These results should be read with the metric profile in mind. The highest FaceSim baseline on SDXL in the abridged table is InstantID at $R_{\rm struct}(x,c)$7, while ID-Aligner reports the highest Aes value at $R_{\rm struct}(x,c)$8. This suggests that the contribution of IA-RFT is best understood as part of a trade-off management strategy rather than as a single-metric maximization procedure.

6. Conceptual distinctions and scope

Several distinctions are necessary for precise use of the term. First, IA-RFT is specifically the aesthetic branch of reward fine-tuning; it is separate from identity consistency reward fine-tuning, which uses face detection and recognition models to preserve identity. Second, the “aesthetic reward” is explicitly decomposed into appeal and structure rewards. Third, the implementation differs across Adapter and LoRA variants, because only the latter jointly optimizes the denoising loss with the reward losses (Chen et al., 2024).

Another potential misconception is that the method requires architectural changes to the underlying diffusion model. The source characterizes ID-Aligner as a universal feedback fine-tuning framework that can be readily applied to both LoRA and Adapter models, and it closes by stating that IA-RFT injects identity-and-aesthetic rewards into diffusion fine-tuning “with minimal changes to existing Adapter or LoRA pipelines.” This suggests a modular design intent, although the exact engineering burden beyond the summarized pipeline is not specified in the provided description.

In the terminology of ID-Aligner, IA-RFT therefore denotes a narrowly defined reward-construction and optimization mechanism: pairwise-trained appeal and structure reward models, summed into $R_{\rm struct}(x,c)$9, then used through $\mathcal{D}_{\rm pref}$00 as part of a larger identity-preserving fine-tuning objective. Its significance lies in making aesthetic control an explicit optimization target in ID-T2I, rather than treating it as an indirect byproduct of identity-preserving generation.