Birth of a Painting: Differentiable Brushstroke Reconstruction (2511.13191v1)

Abstract: Painting embodies a unique form of visual storytelling, where the creation process is as significant as the final artwork. Although recent advances in generative models have enabled visually compelling painting synthesis, most existing methods focus solely on final image generation or patch-based process simulation, lacking explicit stroke structure and failing to produce smooth, realistic shading. In this work, we present a differentiable stroke reconstruction framework that unifies painting, stylized texturing, and smudging to faithfully reproduce the human painting-smudging loop. Given an input image, our framework first optimizes single- and dual-color Bezier strokes through a parallel differentiable paint renderer, followed by a style generation module that synthesizes geometry-conditioned textures across diverse painting styles. We further introduce a differentiable smudge operator to enable natural color blending and shading. Coupled with a coarse-to-fine optimization strategy, our method jointly optimizes stroke geometry, color, and texture under geometric and semantic guidance. Extensive experiments on oil, watercolor, ink, and digital paintings demonstrate that our approach produces realistic and expressive stroke reconstructions, smooth tonal transitions, and richly stylized appearances, offering a unified model for expressive digital painting creation. See our project page for more demos: https://yingjiang96.github.io/DiffPaintWebsite/.

• The paper introduces a unified differentiable framework for reconstructing painterly strokes with semantic layer-awareness and high fidelity.
• It employs a multi-phase, coarse-to-fine optimization scheme combining Bézier curve geometry, StyleGAN-based texture synthesis, and smudge simulation.
• Experimental results demonstrate up to 17.7% LPIPS reduction and improved perceptual metrics, outperforming traditional stroke-based methods.

Differentiable Brushstroke Reconstruction: Technical Summary and Analysis

Introduction and Motivation

"Birth of a Painting: Differentiable Brushstroke Reconstruction" (2511.13191) presents a unified differentiable framework for reconstructing human-like painting processes. The approach overcomes key limitations observed in previous painterly generation systems: absence of explicit stroke structure, inadequate texture modeling, and unrealistic color blending. The method targets high fidelity, layer-aware stroke-based reconstructions with smooth shading and stylistic detail across diverse media—oil, watercolor, ink, digital. By modeling the full paint–smudge loop and optimizing over geometry, color, and texture in a coarse-to-fine regime, the paper advances procedural digital painting synthesis.

Framework Overview

The proposed pipeline encapsulates three optimization phases applied in a hierarchical (coarse-to-fine) grid decomposition of the input canvas:

• Paint Stroke Reconstruction: Parameter optimization over Bézier curve geometry, dual-endpoint color, and transparency produces dense stamp-based strokes in parallel via an efficient differentiable renderer.
• Stroke Texture Stylization: A conditional StyleGAN synthesizes latent style codes conditioned on stroke geometry for visually rich, style-flexible textures.
• Smudge Stroke Reconstruction: Differentiable smudge operations simulate blending and shading via one-shot initialization, enabling color continuity with stable forward and backward passes.

  Figure 1: Pipeline: segmentation produces supervision masks for paint-stroke parameter optimization, parallel differentiable rendering rasterizes geometry, StyleGAN synthesizes textures, and differentiable smudge rendering performs color blending and shading.

Explicit parametric representation underpins all stages, enabling joint optimization of geometry $\mathbf{x}_{B}$, color $\mathbf{c}$, radius $\mathbf{r}$, transparency $\alpha$, and style latent $\mathbf{w}$ components:

$$\{\mathbf{x}_{B}, \mathbf{r}, \mathbf{c}, \alpha, \mathbf{w}\}$$

This unified representation for both paint and smudge strokes yields flexible compositionality, essential for differentiable optimization and gradient propagation throughout paint–smudge cycles.

Differentiable Stroke and Smudge Rendering

Prior stroke-based approaches relied on sequential rasterization, limiting parallelizability and efficiency. The paper introduces a pixel assignment scheme based on distance to nearest stamp in Bézier interpolated space, allowing full parallelization while preserving color continuity. For smudge rendering, one-shot initialization leverages a length-weighted averaging kernel:

$$\mathcal{K}_{k,i} = \frac{t_i^{\,a-1}(1-t_i)^{\,b-1}}{\sum_{m=0}^{k} t_m^{\,a-1}(1-t_m)^{\,b-1}}$$

This dispenses with Markovian recursions and ensures differentiability for brush and canvas state updates.

Figure 2: Comparison of sequential (a) and parallel (b) differentiable paint rendering—parallel achieves comparable accuracy with significant speedup; difference visualization (c) shows minor discrepancies (bright blue: larger error regions).

Figure 3: Traditional (a) versus differentiable smudge rendering (b)—the one-shot scheme in (b) is quantitatively accurate, and (c) highlights difference regions (blue: more deviation).

Optimization and Structural Guidance

Each phase employs multi-term loss functions:

• Pixel L1 loss and perceptual (VGG) losses for color and semantic alignment.
• Gradient-based alignment enforces stroke orientation and magnitude to conform to local edges and contours.
• Segmentation losses (using SAM) guarantee layer-aware, semantically segmented stroke placement, preventing cross-object blending.
• Regularization via entropy-optimal transport and area terms discourages degenerate (vanishing/fragmenting) stroke masks and stabilizes the backward pass.

This configuration lifts stroke optimization from simple color filling to a semantically, structurally, and perceptually informed interpolation that scales with grid granularity.

Experimental Evaluation

Extensive evaluation on 80 paintings (WikiArt and digital) across four styles demonstrates substantial gains in both quantitative and qualitative metrics relative to established baselines: Stylized Neural Painting, Paint Transformer, Learning to Paint, and Optimize{content}Reduce.

Numerical Results (see Table, Section 5):

• LPIPS reduction: up to 17.7% improvement at 128 strokes; lower perceptual difference relative to ground truth
• FD metric: up to 32% decrease, indicating improved feature-level realism
• SSIM/PSNR: consistently highest across varying stroke densities

User paper confirms robust perceptual preferences for color alignment (up to 82%), texture/shading accuracy (up to 85%), and overall quality (up to 84%) versus all baselines.

Figure 4: Qualitative comparison with leading baselines—proposed method displays more coherent shading, faithful textural reproduction, and balanced painterly structure.

Figure 5: Sparse stroke settings (128–512 strokes): framework retains geometric contour and natural shading even with limited stroke budget, outperforming all baselines in structure consistency.

Figure 6: Ablation paper—quality improves across pipeline phases and with inclusion of regularization and structural guidance (segmentation and gradient).

Figure 7: Stroke number ablation across styles—adequate structure and shading preserved from 128 to 1024 strokes.

Applications

Layer-Aware Generation uses segmentation-based decomposition for editable and semantically structured paintings, facilitating composite manipulation or object-wise stylization.

Figure 8: Layered painting generation—high-quality semantic separation and layer-aware synthesis.

Style Transfer adapts the style latent to a reference painting, decoupling geometry from texture/color for structure-preserving, high-fidelity style substitute.

Figure 9: Style transfer—content structure is preserved while color/texture and artistic characteristics are adapted to the target style.

Theoretical and Practical Implications

Technically, this work establishes differentiable programmable painting as a unified framework for human-like image synthesis. The parametric open-curve strategy outperforms closed-region SVG primitives in color blending and painterly effect, and the coarse-to-fine optimization offers scalable reconstruction without exponential stroke increase.

Practically, the method enables explicit, editable vectorization of complex paintings for downstream digital manipulation, transfer, and restoration. The pipeline's modularity supports direct integration into professional painting software, robotic painting platforms, and style transfer APIs.

One strong claim is the robust generalization: the framework reconstructs images accurately across media and with sparse stroke counts, suggesting practical feasibility in resource-constrained scenarios (e.g., real-time digital painting, robot-driven physical painting).

Limitations and Future Directions

Main constraints are:

• Fixed paint–smudge cycle order due to grid partition rigidity; dynamic alternation remains unaddressed.
• Open Bézier curve representation may miss highly irregular, fluid-diffusion effects (notably in ink/watercolor).
• No explicit modeling of temporal evolution for learning-based, sequential brushwork.

Future research can expand stroke parameterization (e.g., using learned displacement fields, hybrid edge primitives) and explore reinforcement learning for dynamic stroke scheduling. The framework's differentiability could be leveraged for end-to-end learning of personalized painting agents and inverting detailed human painting trajectories.

Conclusion

"Birth of a Painting" (2511.13191) defines a technically rigorous framework for physically, semantically, and perceptually plausible brushstroke reconstruction. The differentiable, unified pipeline—integrating parallel stroke rendering, StyleGAN-based texture synthesis, and efficient smudge operators—advances automated painting synthesis. Its strong reconstruction metrics, perceptual fidelity, and extensible design position it as a central reference for AI painting and digital art reconstruction research.