Appearance-Guided Controllable Modeling

• Appearance-guided controllable modeling is a framework leveraging explicit appearance cues to decouple and integrate features like texture, color, and lighting with geometric or motion controls.
• It employs techniques such as cross-attention, latent space disentanglement, and direct modulation in generative backbones to achieve fine-grained and independent control over visual outputs.
• The approach underpins diverse applications including multi-object tracking, 3D reconstruction, image/video generation, and material design, driving state-of-the-art results in controllable visual synthesis.

Appearance-guided controllable modeling is a general framework in computer vision and graphics wherein explicit appearance cues, such as reference images or semantically disentangled appearance embeddings, are leveraged to control or modulate the synthesis, editing, or tracking of scene entities. This modeling paradigm aims to decouple and integrate appearance features—texture, color, style, lighting—into controllable generative or discriminative visual systems, often alongside geometry, motion, or structural control. Contemporary instantiations span domains including multi-object tracking (Ma et al., 3 Aug 2025), 3D urban reconstruction (Kang et al., 14 Nov 2025), material design (Serrano et al., 2018), image/video generation (Fang et al., 17 Jun 2025, Shen, 2020), real-time rendering (Zhu et al., 2023), and more.

1. Foundational Principles: Appearance-Guided Control Mechanisms

The cornerstone of appearance-guided controllable modeling is the explicit injection and disentanglement of appearance information from other semantic or geometric factors. This is achieved through several key mechanisms:

• Cross-attention and feature fusion: Appearance features derived from reference exemplars are fused (often through cross-attention or concatenation) with geometry, layout, or motion representations to steer image or 3D content generation to be consistent with user-specified styles or textures (Kang et al., 14 Nov 2025, Qu et al., 2024, Zhang et al., 3 Mar 2025).
• Latent space disentanglement: Models (e.g., VAEs, GANs) are trained or designed so that controllable latent codes independently modulate appearance and structure, allowing users or downstream systems to vary texture or color without affecting pose or shape and vice versa (Qu et al., 2024, Wang et al., 2024, Jimenez-Navarro et al., 21 Apr 2025, Serrano et al., 2018).
• Direct modulation in generative backbones: Diffusion and GAN-based architectures condition the generative process at every step (via, e.g., cross-attention, FiLM, prompt-token concatenation) with appearance encodings, so the resultant outputs faithfully reflect both low-level and high-level design (Chen et al., 4 Aug 2025, Deng et al., 2024).

Mathematically, the mapping $G(x_\text{geom}, x_\text{app})$ realizes a target $y$ such that its structural aspects follow $x_\text{geom}$ and its appearance aspects match or interpolate $x_\text{app}$, where $x_\text{app}$ may be an image, a compact descriptor, a semantic vector, or a filtered prompt.

2. Algorithmic Architectures for Appearance-Guided Modeling

Architectures for appearance-guided controllable modeling are highly domain-dependent but share common design patterns:

Domain	Architecture Highlights
Multi-object tracking	Appearance-guided affinity matrices (AMC) + motion fusion (Ma et al., 3 Aug 2025)
Diffusion models	Appearance adapters/IP-Adapter, region-specific cross-attention, pixel-space decomposition (Kang et al., 14 Nov 2025, Deng et al., 2024, Wang et al., 2024, Jimenez-Navarro et al., 21 Apr 2025)
GANs for image/video	Conditioned generators: concat reference image + structure, dual-branch discriminators, appearance-specific losses (Shen, 2020, Tang et al., 2019, Wei et al., 2019)
3D representation	Tri-plane, NeRF, mesh fields, appearance-guided radiance/texture fields, dual-branch cross-attention (Zhu et al., 2023, Athar et al., 2023, Mei et al., 2024, Kang et al., 14 Nov 2025)
Semantic/attribute control	Value encoders for fine-grained attributes, semantic controllers (Chen et al., 4 Aug 2025, Chen et al., 2023)

For example, in LocRef-Diffusion (Deng et al., 2024), a frozen diffusion U-Net is augmented with two adapters: a "Layout-net" fuses explicit layout masks, and an "Appearance-net" injects instance-specific reference features via masked cross-attention, allowing independent per-object appearance control conditioned on bounding boxes.

In Sat2RealCity (Kang et al., 14 Nov 2025), each transformer block contains parallel cross-attention paths for appearance (from a frontal-view crop) and structure (from a top-view image); outputs are fused, enabling disentangled and tokenwise modulation in 3D city entity generation.

3. Training Objectives and Disentanglement Strategies

Achieving controllable appearance manipulation demands specific training regimes and disentanglement strategies:

• Conditional or contrastive objectives: Models are optimized to reconstruct targets consistent with both the supplied geometry and appearance, often via reconstruction, adversarial, or contrastive losses. Cycle consistency and structure-guided identity preservation also feature prominently in GAN-based approaches (Tang et al., 2019).
• Regularization for disentanglement: FactorVAE or similar losses target independence between appearance and structure latents (Jimenez-Navarro et al., 21 Apr 2025, Serrano et al., 2018). Total correlation penalties and $l_n$-norms on latent KL divergences stabilize and balance the information content of appearance codes (Jimenez-Navarro et al., 21 Apr 2025).
• Supervised/unsupervised attribute mapping: For interpretable semantic controls, attribute-to-latent mappings are realized via RBF networks (material attributes to BRDF PCA coefficients (Serrano et al., 2018)) or lightweight value encoders mapping intensity scalars to embeddings (for aesthetic trait control in diffusion models (Chen et al., 4 Aug 2025)).

The joint loss often reflects multiple desiderata, e.g.: $$\mathcal{L}_\text{total} = \mathcal{L}_\text{content} + \mathcal{L}_\text{appearance} + \mathcal{L}_\text{cycle} + \mathcal{L}_\text{adv} + \mathcal{L}_\text{perceptual} + ...$$ where each term targets a specific aspect of appearance/structure fidelity or controllability.

4. Practical Applications and Domain-Specific Implementations

Appearance-guided controllable modeling underpins a broad range of applications:

• Multi-object tracking in complex motion scenarios: AMOT (Ma et al., 3 Aug 2025) couples dense, appearance-guided spatial response maps (AMC matrix) with motion prediction (Kalman filter, MTC module) to robustly associate object identities under abrupt UAV motion. Affinity matrices blend ReID features and motion cues, with tunable hyperparameters allowing practitioners to favor appearance or motion as context dictates.
• 3D controllable human synthesis: TriHuman (Zhu et al., 2023) enables real-time, pose- and appearance-controlled rendering via tri-plane feature fields, warping, and dynamic decoders.
• Controllable image/video generation: GAC-GAN (Wei et al., 2019), Correspondence Learning (Shen, 2020), and Sketch2Human (Qu et al., 2024) exemplify architectures that take explicit appearance inputs (e.g. part-wise images, full reference shots, or reference attribute sketches) and synthesize outputs under various geometric, temporal, or pose constraints by feature warping, spatial compositing, and adversarial training.
• Material editing and transfer: Disentangled latent spaces (learned via self-supervised or rated attribute datasets) enable interpretable sliders for gloss, hue, and lighting, powering artistic editing tools (Jimenez-Navarro et al., 21 Apr 2025, Serrano et al., 2018).
• Fine-grained attribute control: Frameworks like AttriCtrl (Chen et al., 4 Aug 2025) allow users to continuously modulate realism, brightness, detail, and safety, mapping intensity scalars to embeddings that steer the (frozen) diffusion U-Net to synthesize images reflecting precise attribute blends.

5. Quantitative Evaluation and Impact

Quantitative evaluation relies on both standard and application-specific metrics:

• Fidelity and realism: FID, Inception Score, LPIPS, SSIM, and user studies; CLIP-based similarity for alignment between generated and reference styles, especially for appearance (Kang et al., 14 Nov 2025, Deng et al., 2024, Chen et al., 4 Aug 2025).
• Controllability: Attribute alignment errors, interpolation experiments, and value encoder calibration (mapping intensity to achieved output traits) (Chen et al., 4 Aug 2025).
• Downstream task metrics: For feature extractors (e.g., SOLIDER (Chen et al., 2023)), task performance—person re-identification vs. human parsing/AP—is evaluated under varying semantic/appearance trade-offs controlled by tunable semantic controllers.

Across multiple domains, appearance-guided controllable approaches set new state-of-the-art performance, drive improvements in artifact-free editing, class-consistent rendering, and object identity preservation, and exhibit high compatibility with modular, plug-and-play integration into large pre-trained backbones.

6. Open Challenges and Future Directions

Despite substantial progress, several key challenges remain:

• Disentanglement at scale: In real-world data, perfect orthogonality between geometry and appearance can be elusive; improved objectives and large-scale attribute curation are areas of active research (Jimenez-Navarro et al., 21 Apr 2025, Kang et al., 14 Nov 2025).
• Automatic filter/feature selection: Manual specification of appearance channels or pixel-space filters (as in FilterPrompt (Wang et al., 2024)) can be laborious or suboptimal.
• Generalization and out-of-distribution robustness: Ensuring effective control under out-of-sample sketches, textures, or complex styles without over-regularization is unresolved, motivating more adaptive architectures or joint fine-tuning (Qu et al., 2024).
• Real-time and scalable inference: Bridging the gap between generative quality and latency, especially for geometrically complex or combinatorial appearance-geometry controls (as in real-time 3DGS stylization (Mei et al., 2024)), is critical for interactive and production applications.
• Deeper semantic control: Fine-grained, hierarchical, or multimodal attribute control (combining text, image, part, and latent prompts) remains a frontier, with ongoing work in multi-branch fusion strategies and semantic disentangling (Kang et al., 14 Nov 2025, Chen et al., 4 Aug 2025, Chen et al., 2023).

As architectures and training paradigms continue to evolve, appearance-guided controllable modeling is expected to become foundational for next-generation controllable synthesis, editing, and understanding in computer vision, graphics, robotics, and digital content creation.