OSF: Occlusion-Robust Stylization in 3D Animation

• The paper introduces OSF to address the stylization pose gap by leveraging flow-depth edge detection for robust contour recovery under self-occlusion.
• It features a single-stage Unified Stylization Network (USNet) that fuses 2D projections with occlusion-robust edge maps for efficient, coherent stylization.
• Empirical results show improved CLIP, SSIM, FID, and LPIPS metrics alongside faster inference and lower GPU usage compared to two-stage stylization methods.

Occlusion-robust Stylization Framework (OSF) is a stylization component for drawing-based 3D animation, defined as generating a 3D animated video from a single hand-drawn image and a target 3D motion sequence while preserving the artist’s characteristic drawing style under heavy self-occlusions. It was introduced to address the stylization pose gap: stylization networks are fine-tuned on a single, typically occlusion-free pose, yet are deployed on dynamic poses with overlapping body parts, viewpoint changes, and foreshortening. OSF addresses this mismatch through Flow-Depth Edge Detection (FDED) and a single-stage Unified Stylization Network (USNet), targeting contour flickering, stroke blurring, and temporal inconsistency that arise during animation (Yoon et al., 1 Aug 2025).

1. Problem setting and the stylization pose gap

In the underlying drawing-based 3D animation pipeline, a single hand-drawn image $X$ serves as the style reference, and a 3D motion sequence $\{P_i\}$ specifies articulated motion. An image-to-3D diffusion model such as Wonder3D reconstructs a 3D mesh from $X$, rigs it to the skeleton, and animates it according to each pose $P_i$. Each posed mesh is rendered from a camera viewpoint as a 2D projection $Z_i$, after which a stylization network produces the final frame $Y_i$. The stylization stage is necessary because the 3D reconstruction and rendering pipeline is not style-preserving: contours become thicker or misaligned, strokes blur, and fine drawing details degrade (Yoon et al., 1 Aug 2025).

The framework is centered on preserving rough and sometimes irregular contours, internal stroke patterns such as hatching or localized shading, and the overall texture and line quality of the input drawing. The paper identifies a distributional mismatch between training and inference. During training, the stylization network is fine-tuned on a single rest pose using the rendered projection $Z$ as input and the original drawing $X$ as target. This training pose is typically occlusion-free. During inference, the same network is applied to projections $Z_i$ from dynamic motions that contain diverse self-occlusions, including arms crossing the torso, legs overlapping, sitting poses, and dancing poses. Because no hand-drawn ground-truth frames exist for those occluded poses, the network generalizes poorly under occlusion; this mismatch is the stylization pose gap (Yoon et al., 1 Aug 2025).

When that gap is not addressed, the characteristic artifacts are contour flickering at overlapping regions, stroke blurring in occluded areas, and temporal inconsistencies in which the same region changes style across frames. The reported qualitative failures include arm textures and adjacent clothing becoming blurry and contours separating overlapping parts intermittently disappearing. Quantitatively, the paper states that temporal consistency and fidelity drop significantly for motions with occlusion, with worse CLIP similarity between frames and worse LPIPS to the original drawing (Yoon et al., 1 Aug 2025).

2. System architecture and single-stage stylization

OSF plugs into the stylization stage of the animation pipeline and has two principal aims: to provide occlusion-robust edge guidance for contour preservation, and to replace earlier two-stage stylization with a single-stage network. At frame $i$, the external 3D animation subsystem outputs the rendered image $\{P_i\}$0 and its depth map $\{P_i\}$1; FDED then constructs an occlusion-robust edge map $\{P_i\}$2; USNet finally maps the projection-edge pair to the stylized drawing (Yoon et al., 1 Aug 2025).

The stylization mapping is written as

$\{P_i\}$3

where $\{P_i\}$4 is implemented as an encoder-decoder U-Net. The rendered projection $\{P_i\}$5 and the edge map $\{P_i\}$6 are concatenated into a $\{P_i\}$7-channel input, and the output is the stylized RGB frame $\{P_i\}$8. The paper notes that more sophisticated fusion mechanisms such as transformers or attention are possible, but the main experiments use simple early fusion by concatenation (Yoon et al., 1 Aug 2025).

This architecture departs from earlier two-stage stylizers such as the DrawingSpinUp stylizer (DSU), which separated texture stylization from contour stylization. In that older formulation, a texture stylization network first produced a coarse drawing-like image $\{P_i\}$9, and a second contour network refined silhouettes and lines. OSF replaces this with the Unified Stylization Network, which directly stylizes textures and contours in one run. The significance is both algorithmic and systems-level: there is no intermediate label $X$0, no separate contour branch, and no second forward pass. Contour fidelity is instead induced by the edge prior $X$1 and by edge-guided contrastive learning inside USNet (Yoon et al., 1 Aug 2025).

3. Flow-Depth Edge Detection (FDED)

FDED is the mechanism that supplies occlusion-robust structural guidance. Its edge map is decomposed as

$X$2

where $X$3 denotes the depth-based edge for unoccluded regions and $X$4 denotes the flow-based edge for occluded regions. The design premise is that per-frame edge detection is useful as a stylization prior, but becomes unreliable when self-occlusions occur, especially when overlapping parts have similar colors or depths (Yoon et al., 1 Aug 2025).

For unoccluded edges, OSF uses depth rather than color. Let $X$5 be the depth at pixel $X$6. A local adaptive threshold is computed with Gaussian weighting over a window:

$X$7

with Gaussian kernel

$X$8

and normalization factor

$X$9

The depth-based edge map is then

$P_i$0

This captures silhouette-like and depth-discontinuity contours, but can miss boundaries between body parts at similar depths (Yoon et al., 1 Aug 2025).

For occluded edges, OSF propagates contour evidence temporally with optical flow. Given consecutive projections $P_i$1 and $P_i$2, RAFT estimates the flow field $P_i$3. Edge points $P_i$4 on the previous depth edge $P_i$5 are propagated as

$P_i$6

A filtering function $P_i$7 keeps propagated points that lie inside the region enclosed by the current depth edge $P_i$8, focusing reconstruction on internal occluded contours. A continuous edge map is then produced by

$P_i$9

where the interpolation operator $Z_i$0 is implemented as morphological dilation rather than B-spline interpolation for efficiency. The final edge prior is therefore the union of current depth-based boundaries and flow-propagated internal contours (Yoon et al., 1 Aug 2025).

The practical effect is that a contour missing from the current depth edge can still be inferred from its previous-frame support. Filtering through the current depth outline constrains the propagated structure to the object interior, and dilation converts sparse correspondences into a usable edge curve. This is the paper’s primary mechanism for reducing contour disappearance and frame-to-frame structural drift under self-occlusion.

4. Unified Stylization Network and training objective

USNet is a U-Net-like encoder-decoder with skip connections and no explicit temporal modules such as 3D convolutions or recurrent units. Temporal stability is not imposed through video-specific architecture; it is induced through consistent per-frame conditioning on occlusion-robust edges and through deterministic mapping from $Z_i$1 to $Z_i$2. The network is fine-tuned per character using the original drawing $Z_i$3 as ground truth, with the rest-pose projection serving as the training input (Yoon et al., 1 Aug 2025).

The reconstruction term is a patch-wise $Z_i$4 loss. If the image is divided into $Z_i$5 patches, with $Z_i$6 and $Z_i$7 denoting ground-truth and predicted patches, then

$Z_i$8

This constrains the output to match the global appearance of the original drawing. Because contours are sparse and easy to underfit, OSF adds edge-guided contrastive learning. A contour patch index $Z_i$9 is selected from patches overlapping the occlusion-robust edge $Y_i$0; $Y_i$1 is treated as the positive for output patch $Y_i$2, and another edge-containing patch $Y_i$3, with $Y_i$4, is used as a hard negative. The intended ranking is

$Y_i$5

implemented as

$Y_i$6

with margin $Y_i$7. The total objective is

$Y_i$8

The paper explicitly states that there are no additional style Gram losses, adversarial losses, or explicit temporal losses in USNet training (Yoon et al., 1 Aug 2025).

Conceptually, the framework handles the stylization pose gap in two ways. First, it improves input-side robustness: even though training uses only an occlusion-free pose, inference inputs are paired with occlusion-robust edge maps that encode current geometry, including occluded boundaries. Second, it improves contour-aware learning: the contrastive objective concentrates capacity on edge patches, encouraging the network to reproduce contour-specific local style properties such as line thickness and stylized corners. The paper further states that there is no explicit curriculum or data augmentation for occlusions because multiple human-drawn poses are not available; robustness instead arises from flow-based edge propagation, depth-based silhouettes, and patch-wise contrastive learning (Yoon et al., 1 Aug 2025).

5. Empirical performance, ablations, and efficiency

The reported evaluation uses 120 test characters from the Amateur Drawings dataset, following the same split as DrawingSpinUp, together with 80 additional characters for validation and ablation. Training remains per character with a single ground-truth drawing $Y_i$9. For motion, each character is paired with 20 non-occluded motions and 20 occluded motions from Mixamo, for a total of 4800 animation sequences. The 3D animation setup uses Wonder3D and an automatic skeleton such as a Mixamo skeleton with 65 joints. Stylization quality is assessed with CLIP, SSIM, FID, and LPIPS for texture, and with CLIP, SSIM, and FID on extracted contours; CLIP similarity between consecutive frames is used as a temporal consistency measure (Yoon et al., 1 Aug 2025).

Against DSU, OSF’s full configuration, USNet+FDED, improves both texture and contour metrics on non-occlusion and occlusion sets. For texture, CLIP rises from $Z$0 to $Z$1, SSIM from $Z$2 to $Z$3, FID drops from $Z$4 to $Z$5, and LPIPS drops from $Z$6 to $Z$7. For contours, CLIP rises from $Z$8 to $Z$9, SSIM from $X$0 to $X$1, and FID drops from $X$2 to $X$3. Human preference is reported as 61% for OSF versus 35% for DSU in pairwise comparisons. The paper also states that FDED can be used as a drop-in prior for DSU, improving DSU’s metrics as well, which indicates that the occlusion-robust edge prior is useful independently of the single-stage stylizer (Yoon et al., 1 Aug 2025).

Ablation results isolate the role of each component. Without FDED, CLIP and SSIM are lowest. Depth-only edges improve over no FDED, while depth-plus-flow gives the best texture and contour performance, supporting the claim that flow-based edges address occlusion weaknesses. The adaptive-thresholding window size is relatively insensitive between 7 and 13, with $X$4 selected. Dilation performs slightly better than B-spline interpolation for sparse propagated edge points. Removing the contrastive loss slows convergence and yields blurrier textures and contours. Comparing edge strategies shows a clear progression: no edge guidance produces unstable stylization and blurred contours; depth only captures silhouettes but misses internal occluded contours; depth plus flow without interpolation produces scattered contours; depth plus flow plus dilation gives clean sharp contours. The paper also reports that directly using 3D neural edge detection such as Neural Contours as stylization guidance causes confusion, with the stylizer merging texture into detailed 3D edges and producing blurriness; the coarser 2D FDED edges are more suitable (Yoon et al., 1 Aug 2025).

The efficiency gains are central to OSF’s design.

Stylizer	Inference time	GPU memory
DSU	0.276 s/frame	11.62 GB
USNet	0.108 s/frame	4.81 GB
USNet+FDED	0.115 s/frame	5.05 GB

These numbers correspond to the paper’s summary that the single-stage design is about $X$5 faster than DSU and uses about $X$6 less GPU memory, while FDED adds only minimal overhead. The occlusion robustness analysis further defines an occlusion rate as the reduction in visible area compared to the rest pose and reports that, without FDED, texture and contour metrics degrade sharply as occlusion rate increases, whereas with FDED the system remains robust up to about 40% occlusion, after which quality drops under extreme multi-layer overlap (Yoon et al., 1 Aug 2025).

6. Relation to adjacent research, limitations, and future directions

OSF belongs to a broader family of occlusion-aware and consistency-aware stylization or generation methods, but its design point is specific. OSDiffST addresses multi-view consistent style transfer for 3D scenes using a one-step diffusion model, CLIP-based vision conditioning, LoRA adaptation, and structure-preserving losses, while explicitly avoiding 3D information and handling occlusions only implicitly through per-view structure preservation and scene-level training (Zuo et al., 2024). This suggests a complementary relation rather than a direct alternative: OSDiffST emphasizes scene-level multi-view coherence, whereas OSF explicitly targets articulated self-occlusion in drawing-based 3D animation through flow- and depth-conditioned contour guidance.

A second neighboring line appears in occlusion-aware video editing. The framework in “Occlusion-Aware Physics-Semantic Keyframe Selection for Robust Video Editing” selects a reliable anchor frame using structural completeness, cycle-consistent tracking stability, and vision-language-based attribute visibility, then propagates supervision through bidirectional tracking and region-aware diffusion loss (Liu et al., 22 May 2026). A plausible implication is that anchor selection and mask propagation could complement OSF on sequences with prolonged or extreme occlusion, particularly because OSF currently relies on per-frame edge conditioning rather than explicit anchor-based temporal control.

A third related direction is training-free occlusion arbitration in diffusion models. DepthArb formulates occlusion as cross-attention competition and resolves it through Attention Arbitration Modulation, Spatial Compactness Control, and Layout Confinement, all modulated by relative depth (Niu et al., 25 Mar 2026). This suggests another explicit route to occlusion robustness: instead of propagating contours with optical flow, one can suppress background activations in overlapping regions at the attention level. The relation is conceptual rather than architectural, since OSF is a learned stylization module in a drawing-based animation pipeline, whereas DepthArb is a training-free guidance mechanism for depth-ordered image synthesis.

The limitations reported for OSF are specific. First, extreme occlusions above 40% occlusion rate remain difficult; even flow-based reconstruction becomes insufficient when many body parts overlap in complex ways. Second, the method depends on optical flow quality, so fast motion, large viewpoint changes, or heavy rendering noise can degrade edge propagation. Third, only one drawing is available for training, which may constrain robustness for highly complex styles or extreme pose and view changes. Fourth, temporal consistency is enforced implicitly through FDED and deterministic per-frame stylization rather than through explicit temporal training loss (Yoon et al., 1 Aug 2025).

The paper outlines several future directions. Patch-wise quantization of features, for example via vector quantization, is suggested as a way to improve robustness under extreme occlusions by encoding local style primitives more discretely. More advanced fusion of edge and image features through transformers or attention is also suggested. A further stated direction is to combine FDED with diffusion-based acceleration or video modules to improve both speed and temporal stability. Taken together, these directions indicate that OSF is best understood not as a general solution to all occlusion-consistency problems, but as a specialized stylization framework that addresses the stylization pose gap through explicit contour recovery and efficient single-stage rendering-to-drawing translation (Yoon et al., 1 Aug 2025).