Partial-Convolution Style Transfer

• The paper introduces a partial convolution mechanism that integrates masks within the style transfer network to compute style statistics only from the region of interest.
• It employs internal blending techniques such as mask feathering, expansion during convolution, and decoder content blending to minimize boundary artifacts.
• The approach supports multi-mask and multi-style processing, enabling parallel, region-specific stylization using a modified pre-trained style transfer network.

Searching arXiv for the specified paper and the key related works it builds on, to ground the article with citations. A partial-convolution-based style transfer network is a masked style transfer architecture designed for the case where a user wants to stylize only a selected region of an image, rather than the full frame. In this formulation, the mask is moved inside the style-transfer network via partial convolutions, so that stylization is computed only from masked pixels and the style features are applied only to the region of interest. The approach is presented in "Improving Masked Style Transfer using Blended Partial Convolution" (Seyed et al., 7 Aug 2025), which argues that post-hoc masking after full-image stylization is inadequate because the network has already mixed content statistics from the whole image into the stylization process.

1. Problem formulation and motivation

The method targets a common workflow in practical editing tools: a user has a content image, a style image, and a segmented mask from something like SAM, and wants the style applied only to the object or region defined by the mask (Seyed et al., 7 Aug 2025). The paper’s starting point is that most algorithms apply the artistic style transfer to the whole image, but individual users may only need to apply a style transfer to a specific region in the image. The standard practice is to simply mask the image after the stylization.

The paper shows that this approach tends to improperly capture the style features in the region of interest. Standard feed-forward style transfer networks are not naturally aware of a region-of-interest mask, so if one simply stylizes the entire image and then masks the result, the stylization statistics are computed from the full image rather than from the target region. That mismatch can make the stylized object look under-stylized, too dark, too washed out, or otherwise inconsistent with the intended style.

The paper further argues that style transfer methods such as the linear universal style transfer of Li et al. operate on global feature statistics, so if the masked region has a very different color distribution from the full image, the output inside the masked area may not reflect the style well. In other words, the stylization network is implicitly conditioned on pixels that will later be discarded, which leads to a mismatch between the statistics used for stylization and the statistics of the final displayed region. The experiments show that this problem is especially noticeable when the region-of-interest color distribution diverges from the full image’s distribution.

A common misconception addressed by the work is that masked style transfer is merely a compositing problem. The paper’s central claim is that masked style transfer should not be treated as a simple post-processing problem, because the relevant mismatch arises inside the stylization computation rather than only at the final image boundary.

2. Network architecture and partial-convolution formulation

The proposed architecture starts from the linear style transfer network of Li et al. (2019), chosen because it is fully convolutional and can be modified layer by layer (Seyed et al., 7 Aug 2025). The key change is that every convolution in the encoder, decoder, and transformation blocks is replaced with a partial convolution. The paper says they use the same partial convolution code/operator as Liu et al. (2018), but disable the post-convolution normalization used in the original inpainting setting.

The input to the network is both the image and a binary mask. In the encoder, the mask is propagated through the same padding and pooling operations as the image so that the mask remains spatially aligned at each feature level. The matrix multiplication used for the style transformation is also masked at the feature level, using the smallest output mask from the encoder. In the decoder, the mask is resized as needed, and finally the stylized output is alpha-blended back onto the original image. This ensures that only pixels inside the masked region contribute to the stylization computation and only that region is replaced in the final image.

The paper does not re-derive the partial convolution formula explicitly in LaTeX, but it describes the operator as a masked convolution that computes a weighted average only over valid pixels. In standard partial convolution notation, for a convolution window $x$ and binary mask $m$, the output at a location is typically computed as

$$y = \begin{cases} \frac{\sum (W \odot x \odot m)}{\sum m} + b, & \text{if } \sum m > 0 \ 0, & \text{otherwise} \end{cases}$$

where $W$ is the kernel weight tensor, $b$ is the bias, $\odot$ denotes elementwise multiplication, and $m$ zeros out contributions from pixels outside the region of interest. The point is that the convolution “sees” only pixels inside the mask, so the style statistics are shaped by the target region rather than the whole image.

A notable implementation property is that because the style-transfer backbone is still the same pretrained network, the authors can reuse pretrained weights without additional fine-tuning. The paper does not claim to fine-tune or retrain the network for masked transfer; instead, the pretrained linear style transfer model is modified at inference time by inserting partial convolutions and propagating masks through the network.

3. Network-internal blending and boundary handling

A major practical issue addressed by the method is that segmentation masks are never perfect and boundary artifacts are common (Seyed et al., 7 Aug 2025). If the style is only applied inside a hard mask, visible seams can appear where the stylized region meets the original background. To mitigate this, the paper introduces several blending techniques, which it calls internal or network-level blending because they are integrated into the network rather than applied as a final image-space blur.

The three techniques are described as follows:

• Mask Feathering Before Encoding: the binary mask is softened before stylization, gradually transitioning from 1 to 0 at the boundary.
• Mask Expansion During Partial Convolution: the mask is dynamically dilated during convolution layers so boundary pixels can access more context from nearby pixels.
• Content Feathering in the Decoder: the decoder incorporates non-stylized content features from the autoencoder around the masked boundary.

These blending methods are not presented as a new loss function; rather, they are architectural and processing choices that improve boundary coherence. The paper finds that using all three together gives the best-looking and best-scoring border transitions, while decoder content feathering has the strongest individual effect on reducing visible seams. The best overall configuration uses all three blending components and achieves the lowest gradient magnitude and color discontinuity. Numerically, the full combination yields gradient magnitude $82.65$ and color continuity $26.23$, while the no-blending baseline is reported at $157.1$ and $m$0.

This suggests that the method’s contribution is twofold: the partial-convolution mechanism addresses the statistics mismatch inside the region of interest, and the blending refinements address seam formation at the region boundary.

4. Multi-mask and multi-style extension

The paper also presents a multi-mask, multi-style extension enabled by the partial-convolution design (Seyed et al., 7 Aug 2025). Because stylization is computed in a mask-aware way at the feature level, the network can process multiple segmented regions in parallel and assign different styles to different regions in one pass.

In this extension, overlapping areas are merged by weighted summation according to mask values rather than being overwritten sequentially. The paper states that this allows smoother transitions between adjacent stylized regions than a sequential, one-region-at-a-time approach.

A plausible implication is that the architectural choice to keep masking inside the feature pipeline generalizes naturally from single-region editing to parallel region-wise stylization. The paper’s description emphasizes that this is a consequence of the mask-aware computation rather than a separate retraining procedure or a new optimization scheme.

5. Evaluation protocol and reported results

Evaluation is performed on 500 images from the SA-1B dataset, with randomly selected masks of at least 2% of the image area and 11 style images (Seyed et al., 7 Aug 2025). The paper also uses SA-1B for quantitative blending experiments. Since the method is an inference-time modification of a pretrained feed-forward style-transfer model, the implementation is described as relatively simple: partial convolutions are inserted, masks are resized through the network, and the final stylized region is composited back onto the original image.

The experiments compare the proposed method against several baselines: Style-then-mask, Mask-then-style, SAMStyler, StyTR$m$1, and StyleID. Qualitatively, the paper shows that partial convolution produces masked-region stylization that better matches the style image’s colors and texture statistics. The key visual improvement is that the stylized object or region looks like it truly inherited the style, rather than a washed-out or exaggerated subset of a full-image stylization.

The quantitative evaluation uses Earth Mover’s Distance (EMD), sliced EMD, and Perceptual Style Loss. Lower values indicate that the stylized region’s distribution is closer to the style image.

Method	Metrics
Style-then-mask	Gray EMD = 0.121, Sliced EMD = 0.168, Style Loss = 760
PartialConv	Gray EMD = 0.086, Sliced EMD = 0.118, Style Loss = 449

These numbers show a substantial improvement in style-statistics alignment. In the earlier visual study, among 500 examples, users judged partial convolution better than style-then-mask in 212 cases, the methods as the same in 283 cases, and style-then-mask better only 5 times. The paper interprets this as evidence that the benefit of partial convolution is strongest when the region-of-interest distribution is substantially different from the full image distribution. It also notes that if the ROI is large or its color distribution is close to the background, the two methods may look similar.

6. Assumptions, limitations, and significance

The method has several assumptions and limitations (Seyed et al., 7 Aug 2025). First, it relies on a reasonably good segmentation mask; if the mask is inaccurate, the stylized result can still show artifacts, which is why the blending techniques are important. Second, the method is built around a fully convolutional linear style-transfer backbone; the authors specifically choose Li et al.’s model because it can be easily converted to partial convolutions. Third, the paper does not claim to fine-tune or retrain the network for masked transfer, so performance depends on the suitability of the pretrained style-transfer model and the quality of mask propagation through the layers. Finally, there are cases where the benefit is small or absent—especially when the masked region and the full image already have similar color statistics, or when the style itself has a narrow color distribution that limits the usefulness of distribution matching.

Within that scope, the paper’s main message is that masked style transfer should not be treated as a simple post-processing problem. By integrating the mask directly into the style-transfer computation through partial convolutions, the network better preserves the intended style in the region of interest. The blending refinements then make the output more usable in real editing workflows by smoothing seams and handling imperfect masks.

The work is therefore best understood as an inference-time reformulation of masked style transfer rather than a new training paradigm. Its distinguishing properties are the replacement of every convolution in the encoder, decoder, and transformation blocks with a partial convolution, the preservation of pretrained weights without additional fine-tuning, and the addition of network-internal blending mechanisms for boundary coherence. Code is publicly available at https://github.com/davidmhart/StyleTransferMasked.