Pose-Adapter: Versatile Pose Control in Vision

• Pose-Adapter is a design pattern that explicitly injects and disentangles pose information via dedicated modules, auxiliary losses, or transfer layers.
• It is applied across tasks—from cross-domain animal pose estimation using WS-CDA and PPLO to pose-guided text-to-image generation with masked attention and latent injection.
• Recent approaches integrate pose adapters into transformer architectures to enhance structural fidelity, domain adaptation, and personalization without full model retraining.

to=arxiv_search.search 彩神争霸破解: maybe tool name not known. Let's inspect? to=arxiv_search.search 天天中彩票为什么=json {"query":"Pose-Adapter arXiv pose adapter diffusion control", "max_results": 10} to=search_arxiv ฝ่ายขายรายการ=json {"query":"Pose-Adapter", "max_results": 5} Pose-Adapter denotes a family of adapter-style mechanisms for injecting, transferring, or disentangling pose information within larger vision systems. In the surveyed literature, the term is explicit in cross-domain animal pose estimation, where the domain adaptation and pseudo-labeling pipeline is described as a pose “Adapter,” and it is implicit in later work that attaches pose-specific modules to diffusion models, Vision Transformers, and subject-customization pipelines. Across these settings, the recurring objective is to preserve structural fidelity under data scarcity, domain shift, viewpoint variation, or multi-concept conditioning, while avoiding full retraining of the underlying model or limiting the extent of trainable modification (Cao et al., 2019).

1. Terminology and scope

The surveyed literature suggests that Pose-Adapter is not a single canonical architecture but a recurring design pattern. The pattern appears whenever pose is treated as a privileged control signal and routed through a dedicated module, auxiliary loss, sparse transfer layer, or independent modality rather than being left to emerge implicitly from generic RGB conditioning. This includes cross-domain pose estimation, pose-guided text-to-image generation, subject customization, hairstyle transfer, virtual try-on, and pose-aware video representation learning.

A concise way to situate the term is to distinguish the task, the pose representation, and the adapter mechanism. The same high-level idea recurs, but the operational form changes substantially across papers.

Context	Pose signal	Representative mechanism
Animal pose estimation	keypoints, bones, human pose priors	WS-CDA + PPLO
Pose-guided text-to-image generation	skeleton images, pose maps, pose latents	Stable-Pose, SDXL pose adapter, TrioPose
Personalization and subject customization	abstract pose concepts, paired image-pose references, 3D pose	Mod-Adapter, Pose-ICL
Transfer and editing	source-aligned masks, optional pose images	H-Adapter, TOA
Video representation learning	2D keypoint pose maps aligned to patches	PAAB, PAAT

This breadth matters because a common misconception is to equate Pose-Adapter only with ControlNet-style side branches for human keypoints. The literature is broader: pose can be implemented as adversarial domain alignment (Cao et al., 2019), coarse-to-fine masked self-attention in a ViT (Wang et al., 2024), token-level modulation directions in a DiT (Zhong et al., 24 May 2025), a native triple stream in an MM-DiT (Gu et al., 5 Jun 2026), or a 3D-aware positional encoding used in in-context subject customization (Han et al., 9 Jun 2026).

2. Cross-domain pose transfer in estimation

An early explicit formulation appears in "Cross-Domain Adaptation for Animal Pose Estimation" (Cao et al., 2019). The paper addresses the absence of broadly annotated animal pose data by extending pose-labeled data from VOC2011 into an animal pose dataset for five four-legged mammals—dog, cat, horse, sheep, and cow—with 5,517 instances across 3,000+ images, up to 20 keypoints per animal, and 18 bones defined as pairs of keypoints. It also includes additional bounding-box-labeled animal data for seven novel species: otter, antelope, bear, chimpanzee, rhino, bobcat, and hippopotamus. The annotations are aligned with the COCO human pose format, enabling cross-domain learning from human pose corpora.

The core adapter mechanism has two stages. The first is Weakly- and Semi-Supervised Cross-Domain Adaptation (WS-CDA), which uses a CNN feature extractor, a domain discriminator, a keypoint estimator, and a domain adaptation network. The discriminator is trained adversarially against the feature extractor so that the resulting representation becomes domain-invariant across human, labeled animal, and unlabeled animal data. Its integrated objective is

$$\mathcal{L}_{WS-CDA} = \alpha \mathcal{L}_{DDL} + \beta\mathcal{L}_{pose},$$

with $\alpha < 0$ and $\beta > 0$, combining domain discrimination and supervised pose estimation. The second stage is Progressive Pseudo-Label-based Optimization (PPLO), which generates pseudo-labels on unlabeled target animals and incorporates them in a self-paced, alternating training schedule.

This formulation directly addresses three problems named in the paper: data scarcity, domain shift, and cross-species generalization. Empirically, WS-CDA alone achieves up to 65.7 mAP on the animal test set using modest animal-labeled data and large human-labeled data. By contrast, models trained solely on human data almost fail on animal pose estimation, with less than 1% mAP. In leave-one-class-out evaluation on unseen species, WS-CDA + PPLO reports 42.3 mAP for cat, 41.0 for dog, 54.7 for sheep, 57.3 for cow, and 53.1 for horse, outperforming AlphaPose, CycleGAN+PL, domain confusion, and residual transfer. The paper further notes that PPLO enables near-fully unsupervised domain adaptation, achieving 50+ mAP on unseen classes (Cao et al., 2019).

This early usage already contains several elements that later become central in generative pose adapters: transfer from richly labeled source domains, exploitation of structural similarity across species or modalities, and explicit mechanisms for preventing the adapter from collapsing onto domain-specific appearance.

3. Adapter insertion in diffusion backbones

Later work relocates Pose-Adapter from recognition to controllable generation. In "Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation" (Wang et al., 2024), the adapter is a trainable ViT module attached to pre-trained Stable Diffusion. It combines a 6-layer CNN pose encoder $\beta_\theta$ with a pose-masked self-attention ViT $\mathcal{F}_\theta$, and injects the resulting signal into the latent:

$$\upsilon_{\theta}(\mathbf{z}_t, \varphi) = \mathcal{F}_{\theta}(\mathbf{z}_t, \varphi) + \beta_{\theta}(\varphi), \qquad \mathbf{z}'_t = \mathbf{z}_t + \upsilon_{\theta}(\mathbf{z}_t, \varphi).$$

Its distinctive contribution is a coarse-to-fine attention masking strategy. Pose skeletons are binarized and Gaussian-dilated with progressively smaller kernels, so the ViT refines attention from broader pose context to concentrated pose regions. The training loss reweights errors in the pose region via a factor $\alpha \geq 1$, typically set to 5. On Human-Art, Stable-Pose reports AP 48.88 versus 39.52 for ControlNet and 44.57 for HumanSD; on LAION-Human, it reports AP 57.11 versus 44.90 for ControlNet and 50.95 for HumanSD. It also reports AP 29.08 on back-view UBC Fashion and 69.76 on side view, with large gains over the best baseline.

A different adapter configuration is introduced in "From Text to Pose to Image: Improving Diffusion Model Control and Quality" (Bonnet et al., 2024). There the pose adapter retains the ResNet-like architecture of the previous SDXL-Tencent Adapter but expands the input pose map to 18 body keypoints, 42 hand keypoints, and 68 face keypoints. The model is trained on the same 4M image-prompt dataset used for text-to-pose, annotated with DWpose, for 7600 training steps with batch size 256. The motivation is explicit: missing hand and face keypoints reduced pose fidelity in prior adapters. The paper reports that its adapter outperforms the Tencent adapter 70% of the time on aesthetic score and 76% of the time on Human Preference Score v2 on COCO-Pose, while qualitatively improving hand positions, facial expressions, and complex relational poses.

"TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation" (Gu et al., 5 Jun 2026) pushes the adapter concept into MM-DiTs. Rather than concatenating pose with other modalities, TrioPose introduces a Triple-Stream Pose-Aware DiT (TSPA-DiT) in which pose, text, and image are parallel, independent modalities. Pose is active only in the first $L_p$ layers, with ablation indicating $L_p = 12$ as optimal. Pose information is injected twice per block via zero-initialized dual-residual paths, once after attention and once after the MLP, so geometric constraints are enforced gradually without destabilizing the pre-trained latent distribution. The model also introduces a Learnable Relational Bias Mask that classifies token-pair relations into five physical states and maps them to learnable attention biases, as well as a Pose-Guided Spatial Loss Weighting strategy based on heatmap-derived anatomical error maps. On Human-Art, TrioPose reports AP 64.33, a 30% improvement over prior arts, FID 1.65, and CLIP score 34.46; similar trends are reported on CrowdPose and OCHuman.

Taken together, these papers show that pose adapters in diffusion systems have evolved from side-channel conditioning toward deeper architectural integration. The progression moves from masked attention and latent injection, to richer keypoint vocabularies, to native multi-stream transformer formulations that treat pose as an autonomous modality.

4. Personalization, subject customization, and pose-robust transfer

Pose-Adapter also appears in personalization systems where pose is not merely a control condition but an abstract concept to be transferred across subjects. "Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter" (Zhong et al., 24 May 2025) explicitly targets abstract concepts such as pose and lighting, which earlier tuning-free personalization methods handled poorly. The adapter predicts concept-specific modulation directions for concept-related text tokens in a DiT. Its basic operation is

$y' = y + s \Delta_{\text{attribute}},$

where $\alpha < 0$0 is predicted per block from a concept image and concept word. The module combines vision-language cross-attention for aligned concept extraction, Mixture-of-Experts layers with k-means-based routing, and VLM-guided pretraining using captions generated by Qwen2.5-VL. On DreamBench-Abs, which extends DreamBench with 20 abstract concepts including pose, Mod-Adapter reports multi-concept CP 0.70, PF 0.89, CP·PF 0.62, and CLIP-T 0.330, with a +67.6% improvement in CP·PF over the next-best method in the multi-concept setting. Qualitatively, the paper emphasizes correct pose transfer without copying subject identity.

"Pose-ICL: 3D-Aware In-Context Learning for Pose-Controllable Subject Customization" (Han et al., 9 Jun 2026) reframes the problem again. Instead of a pose branch or token modulation module, it uses multiple paired image-pose references and a Surface-Anchored Position Embedding (SAPE) that ties image tokens to normalized 3D surface coordinates of a volumetric bounding box. Attention is then modulated as

$\alpha < 0$1

SAPE is applied only in the first 19 attention layers, uses 21 surface points per patch, and balances the positional embedding dimension across spatial axes. The system is tuning-free at inference and reports much lower pose error than CustomDiffusion360 and SceneDesigner: 10.75° on 3D assets and 13.2° on real-world subjects, while also improving CLIP-I, DINO-I, and FID.

Pose robustness also drives transfer and editing systems. "H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks" (Jeong et al., 24 Jun 2026) uses a region-specific loss that separates hair and non-hair objectives, inducing spatially disentangled cross-attention from which a source-aligned hair edit mask is derived. On the pose-different subset, defined by reference-source yaw gap greater than $\alpha < 0$2, the method reports the best FID, $\alpha < 0$3, and CLIP-I: 12.47, 3.98, and 0.659, respectively. Its VLM-as-a-judge protocol further reports Hair Fidelity Score 3.11, Non-hair Preservation Score 4.23, and Artifact Quality Score 3.73.

"Try-On-Adapter: A Simple and Flexible Try-On Paradigm" (Guo et al., 2024) addresses pose through an outpainting formulation rather than inpainting. It uses a modified latent diffusion pipeline with a Fusion Cross-Attention Block, a Reference U-Net, and optional ControlNet pose conditioning. The paper emphasizes that TOA can preserve the given face and garment, naturally imagine the rest of the image, and provide flexible control with garment properties and human pose. On VITON-HD it reports FID 5.56 for paired and 7.23 for unpaired evaluation.

A useful precursor to these transfer-oriented systems is "Adaptive Appearance Rendering" (Zhai et al., 2021), which disentangles pose and appearance through a pose-guided appearance rendering network. The pose branch encodes a posemap via an encoder-decoder, while the appearance branch produces adaptive appearance filters from a reference image:

$\alpha < 0$4

Although the paper does not use the later adapter vocabulary, it establishes the same structural principle: pose provides geometry, and a separate lightweight mechanism injects appearance-specific detail.

5. Pose-aware representations beyond image synthesis

The adapter concept is not restricted to image generation. "Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers" (Reilly et al., 2023) introduces two generic mechanisms for ViTs: the Pose-aware Attention Block (PAAB) and the Pose-Aware Auxiliary Task (PAAT). Pose priors are represented as 2D keypoints aligned with video patches through binary pose maps $\alpha < 0$5 and patch-keypoint maps $\alpha < 0$6. PAAB is a plug-and-play ViT block that masks self-attention so only pose-containing patches participate. PAAT instead adds an auxiliary patch-wise pose prediction head,

$\alpha < 0$7

and discards the auxiliary head at inference.

This distinction is important for understanding the broader Pose-Adapter landscape. PAAB requires pose maps at inference and explicitly alters attention. PAAT uses pose only during training and produces no extra test-time cost. Empirically, both methods improve downstream performance across seven datasets and three tasks, with PAAT showing a slight edge over PAAB. The paper reports gains of up to 9.8% in real-world action recognition and 21.8% in multi-view robotic video alignment. Specific results include Smarthome CS performance rising from 68.4% for TimeSformer to 71.4% with PAAB and 72.5% with PAAT, and Minecraft alignment error dropping from 0.292 for DeiT+TCN to 0.151 with PAAB and 0.149 with PAAT.

This work broadens the meaning of Pose-Adapter in two ways. First, it shows that pose adaptation can operate at the representation-learning level rather than the image-synthesis level. Second, it demonstrates that the adapter need not be a persistent inference-time module; it can instead be a training-time inductive bias whose only lasting effect is to reshape the learned representation.

6. Design principles, misconceptions, and open problems

Several design principles recur across the surveyed literature. One is that pose distributions often share a structural prior while differing in transformation statistics. "FlexPose: Pose Distribution Adaptation with Limited Guidance" (Wang et al., 2024) makes this assumption explicit by decomposing a pre-trained pose generator into a prior-preserving part $\alpha < 0$8 and a transformation part $\alpha < 0$9, then adapting only a sparse, block-diagonal transfer function $\beta > 0$0:

$\beta > 0$1

Only a few linear layers associated with coarse geometric change are fine-tuned. With few target annotations, the method reports MMD$\beta > 0$2 0.029 and FD 0.80 in COCO $\beta > 0$3 Human3.6M distribution matching, outperforming FreezeD, AdaGAN, and LoRA. The paper also states that adaptation is extremely fast, taking less than 5 minutes on a high-end GPU.

Another recurring principle is disentanglement by explicit routing of nuisance factors. "Preventing Shortcuts in Adapter Training via Providing the Shortcuts" (Goyal et al., 23 Oct 2025) argues that adapters trained by image reconstruction entangle target attributes with confounders such as pose, expression, and lighting. Its Shortcut-Rerouted Adapter Training introduces auxiliary modules $\beta > 0$4—for example ControlNet or LoRA—to explain the confounders during training:

$\beta > 0$5

The shortcut modules are removed at inference. In facial and full-body identity injection, the paper reports improvements in generation quality, diversity, and prompt adherence, along with lower head or body pose copying and better prior preservation.

These results caution against a second misconception: higher pose fidelity does not reduce to stronger conditioning. Several papers instead emphasize calibrated or localized conditioning. Stable-Pose uses coarse-to-fine masked attention and a pose-weighted loss (Wang et al., 2024); TrioPose applies pose only in the first $\beta > 0$6 layers and uses zero-initialized injection (Gu et al., 5 Jun 2026); H-Adapter derives source-aligned masks from disentangled attention rather than directly pasting reference features (Jeong et al., 24 Jun 2026). This suggests that the central technical problem is not merely how to inject more pose signal, but how to inject it without destabilizing pretrained distributions, overfitting appearance, or creating inter-instance crosstalk.

Open problems remain explicit in the source material. The animal pose literature notes that adding even a small number of true labels for the test species still substantially improves mAP, indicating that domain adaptation has not fully closed the gap to supervised learning (Cao et al., 2019). The SDXL pose adapter paper states that image quality, though improved, still lags the base SDXL slightly (Bonnet et al., 2024). Pose-ICL notes occasional limits in foreground-background harmony due to composited training data and sensitivity to extreme pose annotation errors (Han et al., 9 Jun 2026). Together, these observations imply that Pose-Adapter research remains bounded by three unresolved tensions: structural control versus generative prior preservation, sparse guidance versus domain coverage, and pose specificity versus disentangled generalization.