WithAnyone: Towards Controllable and ID Consistent Image Generation (2510.14975v1)

Abstract: Identity-consistent generation has become an important focus in text-to-image research, with recent models achieving notable success in producing images aligned with a reference identity. Yet, the scarcity of large-scale paired datasets containing multiple images of the same individual forces most approaches to adopt reconstruction-based training. This reliance often leads to a failure mode we term copy-paste, where the model directly replicates the reference face rather than preserving identity across natural variations in pose, expression, or lighting. Such over-similarity undermines controllability and limits the expressive power of generation. To address these limitations, we (1) construct a large-scale paired dataset MultiID-2M, tailored for multi-person scenarios, providing diverse references for each identity; (2) introduce a benchmark that quantifies both copy-paste artifacts and the trade-off between identity fidelity and variation; and (3) propose a novel training paradigm with a contrastive identity loss that leverages paired data to balance fidelity with diversity. These contributions culminate in WithAnyone, a diffusion-based model that effectively mitigates copy-paste while preserving high identity similarity. Extensive qualitative and quantitative experiments demonstrate that WithAnyone significantly reduces copy-paste artifacts, improves controllability over pose and expression, and maintains strong perceptual quality. User studies further validate that our method achieves high identity fidelity while enabling expressive controllable generation.

• The paper introduces WithAnyone, a diffusion-based model that overcomes copy‑paste artifacts using contrastive identity supervision and a large-scale paired dataset.
• It presents MultiID-2M and MultiID-Bench, enabling robust training and evaluation that enhances identity fidelity while reducing unwanted replication of reference features.
• The comprehensive training pipeline—from reconstruction pre-training to paired tuning—demonstrates improved controllability and consistent, high-quality image synthesis.

WithAnyone: Controllable and Identity-Consistent Image Generation

Introduction and Motivation

Identity-consistent image generation, particularly in text-to-image and reference-driven synthesis, has advanced rapidly, yet remains fundamentally constrained by the lack of large-scale, diverse, and paired datasets. Most prior approaches rely on reconstruction-based training with single reference images, which leads to the "copy-paste" artifact: models overfit to the reference, directly replicating facial features and suppressing natural variations in pose, expression, and appearance. This over-similarity undermines controllability and limits the expressive power of generative models, as illustrated by the observed decrease in face similarity due to natural variations (Figure 1).

Figure 1: Natural variations in head pose, expression, and makeup can cause more face similarity decrease than expected; copying the reference image limits the model's ability to respond to prompts for expression and makeup adjustment.

MultiID-2M: Large-Scale Paired Multi-Person Dataset

To address the data bottleneck, the paper introduces MultiID-2M, a large-scale dataset comprising 500k group photos with 1–5 recognizable celebrities, each paired with hundreds of individual reference images capturing diverse expressions, poses, and viewpoints. The dataset construction pipeline involves: (1) collecting and clustering single-ID data, (2) targeted retrieval of multi-ID group photos, (3) identity assignment via ArcFace embedding matching, and (4) post-processing for quality control and stylization (Figure 2).

Figure 2: Overview of WithAnyone. MultiID-2M is constructed through a four-step pipeline, and training proceeds in four stages from pre-training to quality tuning.

This dataset enables the development and evaluation of models capable of multi-identity generation and provides the foundation for the MultiID-Bench benchmark, which standardizes evaluation protocols for identity customization.

MultiID-Bench: Comprehensive Benchmark for Multi-Identity Generation

MultiID-Bench is designed to evaluate both identity fidelity and generation quality in multi-identity scenarios. Unlike prior works that report only reference similarity, MultiID-Bench introduces metrics that penalize excessive copying and reward faithful realization of the prompted scene. The primary metric is similarity to the ground-truth (GT) image, not the reference, and a novel copy-paste metric quantifies the degree to which the generated image is biased toward the reference versus the GT. This framework enables systematic assessment of the trade-off between identity fidelity and copy-paste artifacts.

WithAnyone: Model Architecture and Training Paradigm

WithAnyone is a diffusion-based model built on the FLUX architecture, designed for controllable, high-fidelity multi-identity generation. The architecture encodes each reference face using both a face-recognition network (ArcFace) and a general image encoder, yielding identity-discriminative and mid-level features. Face embeddings are injected via cross-attention, restricted to attend only to their corresponding face regions (Figure 3).

Figure 3: Model Architecture. Reference images are encoded by both face-recognition and general image encoders, with cross-attention restricted to face regions.

Training Objectives

The training objective is a weighted sum of three losses:

• Diffusion Loss: Standard flow-matching loss for denoising diffusion models.
• Ground-Truth-Aligned ID Loss: Cosine distance between ArcFace embeddings of the generated and GT images, using GT landmarks for alignment. This enables supervision at all noise levels and provides more accurate identity measurement.
• ID Contrastive Loss (InfoNCE): Pulls the generated image closer to its reference in embedding space while pushing it away from negatives (other identities), leveraging the large negative pool enabled by the dataset.

The overall loss is:

$$\mathcal{L} = \mathcal{L}_{\mathrm{diff}} + \lambda_{\text{ID}} \mathcal{L}_{\text{ID}} + \lambda_{\text{CL}} \mathcal{L}_{\text{CL}}$$

with $\lambda_{\text{ID}}$ and $\lambda_{\text{CL}}$ set to 0.1.

Training Pipeline

Training proceeds in four phases:

1. Reconstruction Pre-training (Fixed Prompt): Initialize the backbone using all available data with a dummy prompt.
2. Reconstruction Pre-training (Full Captions): Align identity learning with text-conditioned generation.
3. Paired Tuning: Replace 50% of samples with paired (reference, GT) instances, breaking the shortcut of direct duplication and compelling the model to rely on high-level identity embeddings.
4. Quality Tuning: Fine-tune on a curated high-quality subset to enhance perceptual fidelity and style robustness.

Experimental Results

Quantitative Evaluation

WithAnyone is evaluated against state-of-the-art general customization and face customization models on MultiID-Bench. Most existing methods exhibit a clear trade-off: higher face similarity to the reference coincides with stronger copy-paste artifacts. WithAnyone breaks this trade-off, achieving the highest GT similarity while maintaining a markedly lower copy-paste score (Figure 4).

Figure 4: Trade-off between face similarity and copy-paste. WithAnyone achieves high GT similarity with low copy-paste, deviating from the trade-off curve of other models.

On the single-person subset, WithAnyone achieves a Sim(GT) of 0.460 and a copy-paste score of 0.144, outperforming all baselines in identity fidelity while suppressing copy-paste. On the multi-person subset, WithAnyone maintains superior performance in both identity and generation quality metrics.

Qualitative Results

Qualitative comparisons demonstrate that WithAnyone generates flexible, controllable faces that faithfully preserve identity, while general customization models and VAE-based approaches either exhibit copy-paste artifacts or deviate from the target identity (Figure 5).

Figure 5: Qualitative results of different generation methods. WithAnyone achieves both high identity fidelity and controllability.

Ablation Studies and User Studies

Ablation studies confirm the necessity of each component: paired-data fine-tuning reduces copy-paste without diminishing GT similarity; GT-aligned ID loss provides more accurate and informative gradients; extended negatives in InfoNCE accelerate convergence and improve ID similarity. User studies with 10 participants show that WithAnyone consistently achieves the highest average ranking across identity similarity, copy-paste, prompt adherence, and aesthetics (Figure 6).

Figure 6: User paper results. Larger bubbles indicate higher ranking; WithAnyone consistently ranks highest across all criteria.

Implementation Considerations

• Computational Requirements: Training WithAnyone requires 8 NVIDIA H100 GPUs, with a global batch size of 32. The full training pipeline spans approximately 100k steps.
• Data Requirements: The effectiveness of the model is contingent on access to large-scale, paired multi-ID data. The MultiID-2M dataset is critical for both training and evaluation.
• Deployment: The model architecture is compatible with standard diffusion backbones (e.g., DiT, FLUX) and can be integrated into existing pipelines for face customization, editing, and multi-identity generation.
• Limitations: The model's performance may degrade for identities with insufficient reference diversity or in out-of-distribution scenarios. The reliance on ArcFace for identity supervision may introduce bias toward the embedding space.

Implications and Future Directions

WithAnyone demonstrates that the long-standing trade-off between identity fidelity and copy-paste artifacts is not fundamental, but rather a consequence of data and training limitations. By leveraging large-scale paired data and contrastive objectives, it is possible to achieve both high identity consistency and controllable variation. This has direct implications for applications in personalized content creation, entertainment, and digital avatars, where both fidelity and flexibility are required.

Future work may explore:

• Extending the approach to video and 3D avatar generation.
• Incorporating more robust and disentangled identity representations.
• Reducing data requirements via semi-supervised or synthetic data augmentation.
• Further improving controllability over fine-grained attributes (e.g., accessories, lighting) without sacrificing identity.

Conclusion

WithAnyone provides a principled solution to the copy-paste artifact in identity-consistent image generation by introducing a large-scale paired dataset, a comprehensive benchmark, and a novel training paradigm with contrastive identity supervision. Empirical results demonstrate that WithAnyone achieves state-of-the-art identity fidelity while enabling expressive, controllable generation, establishing a new standard for multi-identity image synthesis.