Domain Expansion of Image Generators

Abstract: Can one inject new concepts into an already trained generative model, while respecting its existing structure and knowledge? We propose a new task - domain expansion - to address this. Given a pretrained generator and novel (but related) domains, we expand the generator to jointly model all domains, old and new, harmoniously. First, we note the generator contains a meaningful, pretrained latent space. Is it possible to minimally perturb this hard-earned representation, while maximally representing the new domains? Interestingly, we find that the latent space offers unused, "dormant" directions, which do not affect the output. This provides an opportunity: By "repurposing" these directions, we can represent new domains without perturbing the original representation. In fact, we find that pretrained generators have the capacity to add several - even hundreds - of new domains! Using our expansion method, one "expanded" model can supersede numerous domain-specific models, without expanding the model size. Additionally, a single expanded generator natively supports smooth transitions between domains, as well as composition of domains. Code and project page available at https://yotamnitzan.github.io/domain-expansion/.

Analysis of "Domain Expansion of Image Generators"

The paper "Domain Expansion of Image Generators" introduces a task termed "domain expansion," aimed at expanding the capabilities of pretrained generative models. Unlike domain adaptation methods, which often overwrite the existing generator to accommodate a new domain, this approach seeks to minimally perturb the latent space while adding new domains. Crucially, it retains the original domain representations thus, allowing a single model to generate across multiple domains seamlessly. This task is operationalized by leveraging redundant, unused directional paths in the latent space known as "dormant" directions.

The authors posit that pretrained generative models house a latent space, which is both linear and interpretable. Within these spaces, the paper identifies dormant directions and proposes their utilization for domain expansion without sacrificing the original generative capabilities. This expands the local dominion of styles, shown to incorporate upwards of hundreds of novel domains without increasing the network size. As a consequence, multiple specialized, task-specific models become redundant, with a single expanded model offering equivalent performance.

In the method section, the paper elucidates the strategy by employing an orthogonal decomposition of the latent space to isolate dormant directions. Once identified, these directions can be transformed to instantiate new dynamics without altering the base model's fundamental behavior. It then generalizes training procedures, conventional to domain adaptation (such as domain-specific tasks), to enact domain expansion. The regularization of the base latent space is critical, leveraging a loss similar to the original loss and memory replay to ensure the domain's persistence.

The experiments conducted verify the efficacy and flexibility of the proposed method. Utilizing StyleGAN2 and Diffusion Autoencoder architectures, the authors successfully expand models over several datasets, capturing diverse domains like caricatures and cartoons, without degrading the image quality of the base domain. Quantitative metrics such as FID scores and qualitative evaluations underscore this preservation.

Notably, the paper evaluates the potential for composing domains, highlighting that disentangled directions allow compound traversal—exemplifying the opportunities this synergy offers over interpolative techniques employed by extant methodologies like StyleGAN-NADA and DiffusionCLIP. Artifacts such as color contamination and feature leakage, often resultant in class-conditioned generative adversarial networks, are curtailed by the disciplined, orthogonal subspaces framing the domain expansion.

The implications of this research span practical and theoretical lines. Practically, the generator's versatility, without bloating latent space or model size, makes it particularly suited to applications demanding multiple stylistic transformations concurrently. Theoretically, this also posits a latent space organization that predisposes toward such expansive exercises without compromising depth or discernibility, reinforcing the growing understanding of latent spaces as underpinning cerebra for sophisticated image-generation tasks.

Future exploration into complex domain relationships, probabilistic conditioning, or to inform model architecture optimization holds the promise of refining these methodologies. The paper's framework delineates clear steps for potential expansion into fields like real-time video generative models or style transfer for novel art styles, demonstrating the inherent growth potential of pretrained generative models and the latent capacity of their unused directions.

In conclusion, "Domain Expansion of Image Generators" lucidly identifies and addresses the problem of accommodating multiple domains in pretrained models without disruptive trade-offs. The research contributes significantly to the field by decoupling domain capacity from an augmented network scale, a step forward in the field of AI and generative modeling paradigms.