AnimalBooth: Personalized Animal Image Generation

• AnimalBooth is a personalized animal image generation framework that integrates multimodal identity features with a tuning-free diffusion model.
• It employs a trainable Animal-Net combined with adaptive attention and frequency-controlled DCT filtering to progressively refine both coarse and fine details.
• Empirical evaluations on the high-resolution AnimalBench dataset demonstrate significant improvements in identity fidelity and perceptual quality over prior methods.

Searching arXiv for the cited AnimalBooth and related animal-avatar papers to ground the article in current preprints. arXiv search query: AnimalBooth multimodal feature enhancement for animal subject personalization AnimalBooth is a framework for personalized animal image generation that addresses rich appearance cues, large morphological variability, and identity drift in encoder-based, tuning-free methods. It strengthens identity preservation with an Animal Net and an adaptive attention module, and it introduces a frequency controlled feature integration module that applies Discrete Cosine Transform filtering in the latent space to guide the diffusion process from coarse structure to detailed texture. The framework is paired with AnimalBench, a high-resolution dataset for animal personalization, and is evaluated against BLIP-Diffusion, Omnigen, and IP-Adapter on LPIPS, DINO, CLIP-T, and CLIP-I metrics (Liu et al., 20 Sep 2025).

1. Problem setting and scope

AnimalBooth is situated in the problem of animal subject personalization from a single reference image. The paper characterizes this setting as more challenging than personalization for rigid objects or human faces for three reasons: animals exhibit subtle fur, feather or scale textures and fine-grained patterns; even within a single species, pose, lighting and viewpoint can dramatically change appearance; and existing encoder-based, tuning-free methods suffer from cross-domain feature misalignment, causing the generated subject to forget distinctive identity cues and drift toward the generic diffusion prior (Liu et al., 20 Sep 2025).

Within this formulation, identity preservation is treated as the central failure mode. The framework therefore focuses on extracting and reinjecting subject-specific information without discarding the generative prior of the base diffusion model. The architectural consequence is a separation between a trainable identity branch and a largely frozen image-generation backbone. This design is consistent with the paper’s emphasis on mitigating cross-domain alignment errors while maintaining perceptual quality (Liu et al., 20 Sep 2025).

2. Animal-Net and multimodal identity representation

AnimalBooth introduces a small, trainable “Animal-Net” branch whose sole purpose is to extract multi-modal identity features from a single reference image. The input modalities are the reference image $\mathcal{X}_a\in\mathbb{R}^{3\times H\times W}$ and the implicit text prompt $\mathcal{C}_t$ fed into the frozen Photography-Net. The reference image is first encoded by the stable-diffusion VAE encoder into a latent map $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$, and a frozen CLIP image encoder produces $K$ image-token embeddings. A small trainable Q-Former, adopted from BLIP-2, maps CLIP tokens into a fixed set of query vectors, after which several transformer blocks fuse $L_a$ as key/value with the Q-Former queries as query (Liu et al., 20 Sep 2025).

The output of this branch is the feature set $\mathfrak{F}_a$, later injected into the main Photography-Net via adaptive attention. The paper states that these fused features encode both coarse structure, such as silhouette and limb positions, and high-frequency detail, such as fur stripes, and remain fixed during diffusion inference; Animal-Net is never noised or finetuned at test time (Liu et al., 20 Sep 2025).

This representation strategy is important because it explicitly combines latent spatial structure with CLIP-derived semantic content. A plausible implication is that Animal-Net is intended to preserve both global morphology and appearance-specific cues under prompt-conditioned synthesis, rather than relying on a single feature space to carry all identity information.

3. Adaptive attention and identity injection

To inject $\mathfrak{F}_a$ into the frozen UNet, referred to as “Photography-Net,” each self-attention block is replaced by an adaptive attention block that combines a frozen self-attention sub-layer and a trainable cross-attention sub-layer. Let $Z_n\in\mathbb{R}^{m\times d}$ denote the features from the UNet at a given layer and $\mathfrak{F}_a\in\mathbb{R}^{k\times d}$ the identity tokens from Animal-Net. The queries, keys, and values are formed as

$Q=Z_nW_q,\qquad K=Z_nW_k,\qquad V=Z_nW_v,$

with identity-conditioned terms

$\mathcal{C}_t$0

The adaptive attention output is

$\mathcal{C}_t$1

Here $\mathcal{C}_t$2 balances identity injection versus the generic prior; the reported best setting is $\mathcal{C}_t$3 (Liu et al., 20 Sep 2025).

The mechanism is architecturally conservative. The frozen self-attention preserves the generative prior of the base model, whereas the trainable cross-attention injects subject identity. This suggests that AnimalBooth is designed to reduce identity drift without fully re-optimizing the generative backbone. In the paper’s terminology, the adaptive attention module is the point at which multi-modal identity features are made operational inside the denoising network (Liu et al., 20 Sep 2025).

4. Frequency-controlled feature integration

AnimalBooth further introduces a frequency controlled feature integration module that applies Discrete Cosine Transform filtering in latent space and feeds back the filtered signal during diffusion. For the $\mathcal{C}_t$4-th channel of the clean VAE latent, $\mathcal{C}_t$5, the 2D DCT is defined as

$\mathcal{C}_t$6

where $\mathcal{C}_t$7 and $\mathcal{C}_t$8 for $\mathcal{C}_t$9 (Liu et al., 20 Sep 2025).

The framework defines four binary masks in the $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$0 plane. The mini mask is active when $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$1, the low mask when $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$2, the mid mask when $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$3, and the high mask when $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$4. Multiplying $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$5 by a mask yields a filtered spectrum, which is mapped back to the spatial domain by IDCT to obtain a control signal $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$6. Each pass, corresponding to mini, low, mid, or high frequency content, produces a separate $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$7 that is concatenated to the UNet input at each diffusion step (Liu et al., 20 Sep 2025).

The paper interprets the four passes as a coarse-to-fine guidance schedule. The mini-pass preserves only the very lowest frequencies and therefore coarse layout; the low-pass recovers overall structure plus smooth texture; the mid-pass re-emphasizes intermediate edges and patterns; and the high-pass sharpens fine contours and high-frequency details. By scheduling these bands during training and inference, the model learns to generate global shape first, then medium-scale patterns, then fine textures (Liu et al., 20 Sep 2025).

5. AnimalBench, training protocol, and evaluation design

To train and evaluate animal personalization, the paper curates AnimalBench, sometimes referred to as Animal-HD. The dataset contains 10,958 high-resolution training images and 1,000 test images. Each image is $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$8 pixels, single-subject, paired with a detailed caption, and accompanied by a pixel-level semantic segmentation mask of the primary subject and masked-subject crops for visual verification. Preprocessing includes subject cropping, background removal via the segmentation mask, caption normalization, and VAE-encoding to $L_a\in\mathbb{R}^{4\times(H/8)\times(W/8)}$9 latents during training (Liu et al., 20 Sep 2025).

The pretrained base model is Stable Diffusion v1.5. The trainable modules are Animal-Net, the Q-Former projection layers, the cross-attention weights, and the frequency-control convolution. The VAE encoder and decoder and the Photography-Net UNet weights are frozen, with the paper specifying that the self-attention is frozen. Training uses AdamW with learning rate $K$0, batch size $K$1, 50 DDIM steps, and eight NVIDIA A100 GPUs (Liu et al., 20 Sep 2025).

The evaluation protocol compares AnimalBooth with BLIP-Diffusion, Omnigen, and IP-Adapter. The reported metrics are LPIPS, described as $K$2-based perceptual distance with lower values better; DINO, a self-supervised feature similarity metric with higher values better; CLIP-T for text-image consistency; and CLIP-I for identity preservation via image-image CLIP similarity (Liu et al., 20 Sep 2025).

6. Empirical results, limitations, and adjacent research directions

The quantitative comparison reported in the paper is summarized below (Liu et al., 20 Sep 2025).

Method	LPIPS	DINO	CLIP-T	CLIP-I
AnimalBooth	49.08	75.66	20.73	90.00
IP-Adapter	62.91	72.88	19.39	89.75
BLIP-Diffusion	68.21	62.96	19.68	82.38
Omnigen	71.68	50.05	19.41	72.95

The paper states that AnimalBooth improves identity fidelity, measured by DINO and CLIP-I, by approximately 3–17 points and perceptual quality, measured by LPIPS, by approximately 14–22 points versus prior methods. In the ablation on frequency masks, the low-pass mask yields the best combined gains, with LPIPS $K$3, DINO $K$4, and CLIP-I $K$5, which the paper interprets as evidence that preserving low frequencies, combining structure and smooth texture, is most critical for identity. Qualitatively, the reported examples on cheetahs, reindeer, Highland cows, and zebras show retention of spot and stripe patterns, reconstruction of antler branching and fur shagginess, and preservation of natural poses and lighting, whereas other methods blur or distort distinctive markings (Liu et al., 20 Sep 2025).

The limitations and future directions are explicit. The framework is currently limited to single, static images; extending it to video consistency is described as non-trivial. It relies on pixel-level segmentation masks for preprocessing, and moving to mask-free or weaker supervision would lower annotation cost. The number of animal species and the amount of in-the-wild variability could be further expanded. The paper also identifies cross-species identity transfer and 3D-aware animal generation as promising next steps (Liu et al., 20 Sep 2025).

These future directions intersect with adjacent lines of animal-centric generative research. “Animal Avatars: Reconstructing Animatable 3D Animals from Casual Videos” develops a method to build animatable dog avatars from monocular videos by linking frames via a 4D solution, equipping the SMAL parametric model with Continuous Surface Embeddings, and learning an implicit duplex-mesh texture; on CoP3D and APTv2 it reports superior results to BARC, BITE, and RAC (Sabathier et al., 2024). “SMAL-pets: SMAL Based Avatars of Pets from Single Image” targets high-quality, editable animal avatars from a single input image by integrating 3D Gaussian Splatting with the SMAL parametric model, adding a multimodal editing suite with text-driven control, and reporting PSNR improvement from approximately 28 dB in Stage I to approximately 42 dB in Stage II (Borycki et al., 17 Mar 2026). This suggests that AnimalBooth occupies the 2D personalization end of a broader research continuum that also includes video-based and 3D animatable animal reconstruction. In a different but related preprocessing domain, “LAD-RCNN: A Powerful Tool for Livestock Face Detection and Normalization” addresses animal face detection and direction normalization in one stage, with 72.74 FPS on a single GeForce RTX 2080 Ti GPU and AP above 95% on all test datasets (Sun et al., 2022).