Parameterized Brushstroke Style Transfer

Abstract: Computer Vision-based Style Transfer techniques have been used for many years to represent artistic style. However, most contemporary methods have been restricted to the pixel domain; in other words, the style transfer approach has been modifying the image pixels to incorporate artistic style. However, real artistic work is made of brush strokes with different colors on a canvas. Pixel-based approaches are unnatural for representing these images. Hence, this paper discusses a style transfer method that represents the image in the brush stroke domain instead of the RGB domain, which has better visual improvement over pixel-based methods.

• The paper introduces a differentiable brushstroke-based style-transfer method that optimizes 12-parameter quadratic Bézier strokes using VGG-19 content and style losses rather than RGB pixels.
• The differentiable renderer replaces hard stroke masks and assignments with sigmoid and softmax relaxations, while restricting computations to nearby strokes to optimize 5,000 strokes in about 138 seconds on an NVIDIA A100.
• The method produces more visibly coherent painterly structure than pixel optimization, but loses fine details such as facial features and lacks quantitative evaluation against stronger modern baselines.

Motivation and positioning

This paper presents a PyTorch implementation of the CVPR 2021 method "Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes" by Kotovenko et al. (Kotovenko et al., 2021). Its central premise is that conventional neural style transfer, following Gatys et al.'s content/style loss formulation [gaty], operates entirely in the RGB pixel domain. Because real paintings are composed of discrete brush strokes with distinct colors, widths, and trajectories on a canvas, pixel-space optimization produces stylizations that lack the structural character of hand-painted work. The authors therefore reformulate style transfer as optimization over an explicit set of parameterized brush strokes rather than over pixels.

The paper situates this within a lineage that includes texture synthesis via image quilting (Efros and Freeman), image analogies (Hertzmann et al.), feed-forward arbitrary style transfer with Adaptive Instance Normalization (Huang and Belongie) (Huang et al., 2017), and style-aware content losses for high-resolution transfer (Sanakoyeu et al.) (Sanakoyeu et al., 2018). On the stroke-based side, it distinguishes its approach from classical painterly rendering — Haeberli's interactive paint-by-numbers and Hertzmann's multi-scale curved-stroke rendering — and from brushstroke extraction methods such as POET and the rhythmic-brushstroke analysis of Li et al., which analyze existing paintings rather than synthesize new ones.

Method

The pipeline follows the iterative optimization paradigm of Gatys et al., but the optimization variables are brush stroke parameters instead of pixel values. Each stroke is a quadratic Bézier curve,

$$B(t) = (1-t)^2 P_0 + 2(1-t)t P_1 + t^2 P_2, \quad t \in [0,1],$$

with twelve parameters per stroke: two for location, six for the three control points $P_0, P_1, P_2$, one for width, and three for RGB color. A network initialized with $N$ stroke parameter vectors is optimized by gradient descent on the standard VGG-19-based content loss, Gram-matrix style loss, and their weighted sum $$\mathcal{L}_{\text{total}} = \alpha \mathcal{L}_{\text{content}} + \beta \mathcal{L}_{\text{style}}$$.

Differentiable renderer

The renderer maps $\mathbb{R}^{N \times 12}$ to an $H \times W \times 3$ canvas. Rendering is built from distance computations: each pixel is assigned to the nearest stroke, and pixels within a stroke's width of its curve are painted with its color. The authors illustrate the mechanism with flat disks before generalizing to Bézier curves, which are rasterized by sampling $S$ equidistant points along the curve and masking by distance to those samples.

Two design choices make this practical:

• Differentiability: hard masking and nearest-neighbor assignment are discontinuous, so they are replaced with sigmoid masking and high-temperature softmax assignment, allowing gradients to flow back through the renderer into the stroke parameters.
• Efficiency: full pairwise distances between all strokes and all pixels are redundant since each stroke affects only a local region; distance computation is restricted to the $K$ nearest strokes per pixel.

After stroke optimization converges, an optional pixel-level refinement step in the style of Gatys et al. blends the strokes and adds fine texture, yielding a more cohesive final canvas.

Results

The implementation runs on NVIDIA A100 GPUs in PyTorch. The headline runtime figure is that optimizing 5,000 brush strokes with 10 samples per curve takes approximately 138 seconds — indicating the differentiable renderer and the K-nearest-stroke approximation keep the iterative optimization tractable at nontrivial stroke counts.

Qualitatively, the reported results show outputs that more closely resemble hand-painted artwork than Gatys-style pixel optimization, with visible, coherent brush strokes in zoomed views. The optional pixel optimization stage blends strokes into a realistic painted-canvas appearance. However, when applied to images of people, the method loses fine facial detail — the authors state plainly that intricate, high-frequency content is not preserved under the stroke representation, and identify this as requiring improved handling of high-frequency information.

Limitations and open questions

The paper is candid about several constraints. First, the fidelity limitation above: the stroke-domain representation sacrifices fine detail, particularly facial features, and the proposed remedy — CNN-based feed-forward architectures with hierarchical feature extraction to preserve rich image attributes — remains unimplemented here. Second, the evaluation is purely qualitative; no quantitative metrics (e.g., LPIPS, user studies) or comparisons beyond Gatys' baseline are reported, so claims of visual superiority rest on visual inspection alone. Third, the renderer's approximations introduce their own assumptions: the softmax assignment and sigmoid masking are smooth surrogates whose temperature settings affect both gradient quality and rendering accuracy, and the K-nearest-stroke restriction trades exactness for speed without analysis of when it fails. Finally, the work leaves open whether CLIP-based conditioning (Radford et al., 2021) could provide language-guided control over stroke placement and style, as the conclusion suggests but does not demonstrate.

Conclusion

This paper demonstrates that style transfer can be performed directly in a parameterized brushstroke domain, using a differentiable renderer built from continuous relaxations of discrete painting operations, and that the resulting outputs exhibit stroke structure absent from pixel-domain baselines at a practical computational cost (~138 s for 5,000 strokes). Its main open problems are preserving high-frequency content detail, providing quantitative evaluation against stronger baselines, and extending the explicit stroke representation toward text-conditioned editing.