StructInbet: Controllable Inbetweening

• StructInbet is an inbetweening system that employs explicit structural guidance via skeletons and mixed trajectory maps to resolve pose ambiguities.
• It integrates a ControlNet branch with a pre-trained diffusion UNet, injecting both structural and appearance cues through bidirectional temporal attention.
• Empirical evaluations show improved motion alignment, enhanced LPIPS and PSNR metrics, and superior control compared to prior interpolation methods.

Searching arXiv for the StructInbet paper and closely related inbetweening work. StructInbet is an inbetweening system for controllable animation that generates intermediate frames between two keyframes by combining explicit structural guidance with bidirectional temporal attention. The method is designed to address a central ambiguity in inbetweening: sparse point motion does not uniquely determine full-body or full-character motion, so plausible pixel trajectories can correspond to multiple incompatible poses or gestures. StructInbet therefore conditions a pre-trained Stable Diffusion UNet on skeletons and mixed trajectory maps, while separately injecting appearance information from both temporal endpoints to preserve character identity across the transition (Pan et al., 15 Jul 2025).

1. Problem formulation and motivation

Inbetweening is the task of generating plausible intermediate frames given two keyframes. In traditional hand-drawn workflows, these frames are created manually; automatic approaches seek to reduce that burden while maintaining smooth temporal transitions. The paper places existing methods into two broad categories. Uncontrollable interpolation methods can generate smooth transitions but do not let users precisely specify the intended pose or gesture. Interactive point-dragging methods allow some control, but they rely on sparse point inputs that often fail to encode the full structure of a character pose (Pan et al., 15 Jul 2025).

The motivating difficulty is that pixel trajectories are ambiguous. A small number of dragged pixels can be consistent with multiple body configurations, facial states, or limb motions. In the formulation adopted by StructInbet, this ambiguity is not treated as a minor interface issue but as a structural limitation of point-based control. The system’s response is to introduce explicit structural guidance, especially through skeletons and mixed trajectory maps, so that intermediate-frame synthesis is constrained by a clearer geometric specification rather than by sparse motion hints alone.

This suggests that StructInbet is oriented toward controlled frame generation rather than purely automatic interpolation. A plausible implication is that the method is particularly relevant when user intent must be expressed at the level of pose semantics rather than local pixel displacement.

2. System architecture

StructInbet is built on a pre-trained latent diffusion UNet and extends it with three components: a ControlNet branch, a character reference UNet, and a trajectory encoder (Pan et al., 15 Jul 2025).

The ControlNet branch processes mixed structural guidance and injects motion or pose control through zero-conv connections. The character reference UNet is architecturally similar to the diffusion UNet and extracts appearance features from the start frame at time $0$ and the end frame at time $T$. The trajectory encoder encodes mixed trajectory maps and provides embeddings that help align self-attention spatially. Taken together, these components allow structural and appearance constraints to be introduced jointly into intermediate-frame synthesis.

The resulting architecture is organized around two conditioning pathways. The first is explicit structural guidance for motion and pose control. The second is bidirectional temporal attention for appearance preservation. The paper’s stated objective is to make generated inbetween frames both structurally aligned with the desired motion and visually consistent with character appearance across time.

The excerpt does not provide explicit training losses or full optimization details beyond the use of the pre-trained Stable Diffusion UNet and conditioning branches. The exact objective function is therefore not specified. It is possible that training follows a standard diffusion denoising objective, but that is not explicitly stated.

3. Explicit structural guidance

The first core contribution of StructInbet is the introduction of explicit structural guidance into the inbetweening problem. The paper argues that point-based motion control is too sparse for animation settings in which the relevant object is the full body structure rather than a few isolated pixels. Skeletons provide a deterministic pose layout, while pixel trajectories provide flexible fine-grained motion hints. StructInbet combines both into a single conditioning signal (Pan et al., 15 Jul 2025).

The method constructs mixed trajectory maps from skeleton joint correspondences and pixel trajectories. This mixture is intended to encode semantic structure from the skeleton together with local motion detail from pixel-level trajectories. The paper states that the pixel trajectories are rasterized into an integer-labeled guidance map, which is concatenated with the skeleton to form the structural conditioning input.

Structurally, these pose and trajectory maps are processed by a ControlNet encoder. Control features extracted by the ControlNet branch are injected into the diffusion UNet through zero-convolution modules, following the ControlNet paradigm. This gives users direct and explicit control over the motion of the inbetween frame instead of relying on implicit latent interpolation.

The significance of this design lies in how it resolves underdetermination. Sparse drag constraints can encode where a few points move, but they do not fully specify how a limb, torso, or expression should evolve. Skeleton-plus-trajectory conditioning makes the control signal denser and more semantically structured. The paper’s qualitative interpretation is that this reduces ambiguity in pixel trajectories and makes intended motion easier to follow.

4. Bidirectional temporal attention and identity preservation

The second core contribution is bidirectional temporal attention, introduced to preserve character identity and appearance across the interpolated transition. A common failure mode in inbetween generation is visual drift: the synthesized frame aligns with one keyframe more strongly than the other, or loses stable identity cues over time. StructInbet addresses this by incorporating appearance information from both temporal endpoints (Pan et al., 15 Jul 2025).

The character reference UNet extracts features from the two endpoint frames and produces key-value pairs $(K_0, V_0)$ and $(K_T, V_T)$. These replace or augment the keys and values used in the diffusion UNet’s attention blocks. The central attention rule is

$$\mathrm{Attn} = \frac{1}{2} \left( \mathrm{softmax}\left(\frac{\tilde{Q}\tilde{K}_0^\top}{\sqrt{d}}\right)\tilde{V}_0 + \mathrm{softmax}\left(\frac{\tilde{Q}\tilde{K}_T^\top}{\sqrt{d}}\right)\tilde{V}_T \right).$$

The modified query, key, and value tensors are defined by

$$\tilde{Q} = Q + \mathrm{emb}_c,\quad \tilde{K}_0 = K_0 + \mathrm{emb}_0,\quad \tilde{K}_T = K_T + \mathrm{emb}_T,\quad \tilde{V}_0 = V_0 + \mathrm{emb}_0,\quad \tilde{V}_T = V_T + \mathrm{emb}_T,$$

where $\mathrm{emb}_c$ is the embedding for the current control map and $\mathrm{emb}_0,\mathrm{emb}_T$ are embeddings for the endpoint guidance at times $0$ and $T$.

This mechanism computes attention to each endpoint separately and then averages the two outputs. According to the paper’s interpretation, this lets the model preserve identity and appearance from both ends of the transition, avoid overfitting to only one keyframe, improve consistency of character features across the interpolated frame, and better align motion with the intended temporal endpoints.

A useful way to characterize this is that the attention pathway is temporally bidirectional but spatially local in the usual diffusion-attention sense. The paper highlights that it integrates spatial context from two temporal endpoints at a time, improving both alignment and consistency.

5. Inference workflow and empirical evaluation

At inference time, the user provides two keyframes. The system then extracts or receives skeleton structure, pixel trajectories, and mixed trajectory guidance. The structural map is passed through the ControlNet branch, the two endpoint frames are passed through the character reference UNet, bidirectional reference attention merges appearance from both endpoints, and the diffusion UNet synthesizes the intermediate frame under these controls (Pan et al., 15 Jul 2025).

The excerpt does not name the training datasets used for StructInbet. It does, however, specify the evaluation setup and the compared baselines. StructInbet is evaluated against ToonCrafter, DragDiffusion, and Framer. Reported quantitative metrics are FID, LPIPS, and PSNR. A user preference study is also reported with 6 independent participants, a 5-point Likert scale, and three criteria: image quality (IQ), character consistency (CC), and motion alignment (MA).

The reported quantitative results are as follows. ToonCrafter obtains FID $T$0, LPIPS $T$1, and PSNR $T$2. DragDiffusion obtains FID $T$3, LPIPS $T$4, and PSNR $T$5. Framer obtains FID $T$6, LPIPS $T$7, and PSNR $T$8. StructInbet obtains FID $T$9, LPIPS $(K_0, V_0)$0, and PSNR $(K_0, V_0)$1. On these numbers, StructInbet achieves the best LPIPS and the best PSNR among the methods in the table, while not achieving the best FID.

The reported user-study scores are also differentiated by criterion. ToonCrafter receives IQ $(K_0, V_0)$2, CC $(K_0, V_0)$3, and MA $(K_0, V_0)$4. DragDiffusion receives IQ $(K_0, V_0)$5, CC $(K_0, V_0)$6, and MA $(K_0, V_0)$7. Framer receives IQ $(K_0, V_0)$8, CC $(K_0, V_0)$9, and MA $(K_T, V_T)$0. StructInbet receives IQ $(K_T, V_T)$1, CC $(K_T, V_T)$2, and MA $(K_T, V_T)$3. The most salient result is that StructInbet is strongest on motion alignment by a large margin.

The qualitative findings described in the excerpt are consistent with these numbers. StructInbet is said to better align the generated pose with the intended motion, improve over point-dragging approaches that can miss complex gesture semantics, and provide more user-directed transitions than non-interactive interpolation. The teaser is described as emphasizing improved pose alignment versus ToonCrafter and drag-based editing methods.

6. Relation to prior approaches, scope, and limitations

StructInbet is distinguished from prior inbetweening or interpolation methods in two principal ways. First, it uses explicit structure rather than sparse motion hints. Prior drag-based methods rely on sparse point motion, whereas StructInbet uses skeletons plus mixed trajectories, which encode pose and motion more clearly. Second, it preserves identity from both endpoints through bidirectional reference attention, while prior methods often condition only weakly on keyframes or do so implicitly (Pan et al., 15 Jul 2025).

These differences define the method’s claimed advantages for controllable animation and character-centric video synthesis: more precise pose control, less ambiguity in motion specification, better character consistency, improved alignment between user intent and generated inbetween frames, and a more practical controllable animation workflow. The overall conceptual shift is from point-based editing and implicit interpolation toward explicit structural control combined with temporal identity preservation.

At the same time, the excerpt does not include explicit ablation tables or detailed ablation experiments. It implies the importance of two components—explicit structural guidance and bidirectional temporal attention—but does not provide formal ablation results for either. The provided text also does not list a dedicated limitations section. Several limitations are, however, cautiously inferable from the description: the method depends on structural inputs such as skeletons and trajectory maps, user setup may therefore be more involved than pure point dragging, the demonstrated setting is controlled inbetweening rather than general long-video synthesis, and the excerpt does not explain robustness to highly complex deformations, occlusions, or unreliable skeleton extraction.

A common misconception would be to treat StructInbet as merely a higher-capacity interpolation model. The paper frames it differently. Its central premise is that controllability requires an explicit representation of structure, because otherwise the ambiguity of pixel trajectories remains unresolved. Another possible misconception would be to read the reported FID as the sole indicator of method quality. The evaluation as presented emphasizes motion alignment and controllability in addition to generic perceptual fidelity, and the user study particularly favors StructInbet on motion alignment.

In summary, StructInbet defines controllable inbetweening as a joint problem of structural specification and temporal identity preservation. Its method combines ControlNet-based structural conditioning, mixed trajectory maps derived from skeleton joint correspondences and pixel trajectories, and bidirectional reference attention over two keyframes. Within the evaluation reported in the excerpt, this design is associated with the strongest motion-alignment judgments and with the best LPIPS and PSNR among the compared methods, while its broader significance lies in reframing inbetween frame generation around explicit structural guidance rather than sparse local edits.