Differentiable Stroke Reconstruction Framework

• Differentiable Stroke Reconstruction Framework is a computational approach that models brush strokes using parametric curves and differentiable rendering to enable gradient-based optimization.
• It combines spline and Bézier formulations with neural style modules to achieve texture synthesis and style transfer for refined digital artwork.
• Staged optimization and semantic guidance mechanisms ensure robust reconstruction in applications ranging from automated digital painting to robotic art execution.

Differentiable stroke reconstruction frameworks mathematically and algorithmically model the process of reconstructing, optimizing, and rendering brush strokes in digital and robotic painting. These frameworks systematically unify stroke geometry, appearance, style, and compositing in a fully differentiable pipeline, enabling the use of gradient-based optimization for expressive painting, stylization, and semantic planning. This article surveys foundational models and contemporary research advances, emphasizing stroke parameterization, differentiable rendering, style modeling, optimization strategies, and integration in downstream tasks.

1. Mathematical Formulations of Stroke Parameterization

Contemporary frameworks define strokes through parametric models optimized for both geometric flexibility and differentiability. Spline-FRIDA (Chen et al., 30 Nov 2024) transitions from quadratic Bézier curves to polyline splines with $n=32$ control points: $$\left\{ (x_i, y_i, h_i) \right\}_{i=1}^n, \quad x_i, y_i \in \mathbb{R},\; h_i \in \mathbb{R}$$ where $(x_i, y_i)$ are planar displacements, and $h_i$ is brush height.

In “Birth of a Painting” (Jiang et al., 17 Nov 2025), each brush stroke is parameterized by: $$\Theta = \{\, \mathbf{x}_B,\, \mathbf{r},\, \mathbf{c},\, \alpha,\, \mathbf{w} \}$$ where $\mathbf{x}_B$ contains Bézier start, control, and end points; $\mathbf{r}$ are endpoint radii; $\mathbf{c}$ are endpoint colors; $\alpha$ is transparency, and $\mathbf{w}$ is a latent style vector.

Piecewise-linear splines and Bézier formulations allow explicit control over shape, width, and stylistic appearance. Parameters may be further warped by learned affine transformations and rigid offsets to accommodate canvas-specific geometries.

2. Differentiable Rendering of Brush Strokes

Differentiable renderers convert parametric stroke representations into pixel-level masks and images, supporting direct gradient flow to stroke parameters. In Spline-FRIDA (Chen et al., 30 Nov 2024), the renderer (Traj2Stroke) computes, for each line segment between successive spline points, an explicit mask $M_k(u,v)$: $$M_k(u,v) = \left[ \mathrm{clamp}\left(1 - \frac{D_k(u,v)}{W_k(u,v)}, 0, 1\right) \right]^c$$ with $D_k(u,v)$ as the distance-to-segment map and $W_k(u,v)$ an affine function of brush height. The composite stroke mask $M(u,v)$ is a pixelwise maximum across segments.

In “Birth of a Painting” (Jiang et al., 17 Nov 2025), the paint renderer places $N+1$ “stamps” at Bézier-interpolated points $x_k(t_k)$, generating the final painted image $\tau_p(\mathbf{x})$ via alpha compositing or nearest-stamp selection: $p(\mathbf{x}) = \alpha\, \mathbf{c}_{\,k^*}$ with gradients backpropagating through each parameter by analytic chain rule.

All frameworks enforce differentiability of min, clamp, exponentiation, and dot‐product operations via smooth or subgradient-compatible routines, ensuring optimization compatibility in both analytic and neural network-based renderers.

3. Style Modeling and Texture Synthesis

Expressive digital painting demands texture and style control beyond flat color interpolation. “Birth of a Painting” (Jiang et al., 17 Nov 2025) introduces an explicit style generation module: a conditional StyleGAN generator $G$, producing geometrically-conditioned texture for each stroke segment: $$\tau_{st} = G\left(\Theta^{\rm app}; \mathbf{w} \right)$$ where $\Theta^{\rm app}$ is fixed geometry and color, and $\mathbf{w}$ is a style latent optimized via perceptual, pixel, and gradient-based losses. StyleGAN’s style-modulated convolutions receive affine transformations of $\mathbf{w}$ for global and local appearance modulation.

Such approaches decouple geometric shape from appearance, allowing efficient fine-tuning or style transfer on small datasets (often $< 20$ strokes). The modularity suggests extensibility to other GAN architectures or multimodal style objectives.

4. Training and Optimization Strategies

Differentiable stroke frameworks employ staged or coarse-to-fine optimization, targeting appearance, geometry, style, and smudge effects. Spline-FRIDA (Chen et al., 30 Nov 2024) combines VAE pretraining on aggregate human stroke datasets with rapid user-specific fine-tuning. Its Traj2Stroke dynamics module is trained via weighted $L_1$ losses comparing real and synthetic “after” images at the pixel level: $$\mathcal{L}_{\rm stroke} = \sum_{u,v} w(u,v)\, \left| \left[I_{\rm before}(u,v) + \left(M \cdot \Delta I\right)(u,v) \right] - I_{\rm after}(u,v) \right|$$ All parameters—including spline points, affine warps, and renderer thickness/softness—receive gradients for direct optimization.

“Birth of a Painting” (Jiang et al., 17 Nov 2025) executes a sequential three-phase approach—geometry and color optimization (Phase I), style latent fine-tuning (Phase II), and smudge dynamics (Phase III)—at successively finer canvas partitions. Optimization leverages pixel, perceptual, gradient alignment, segmentation, optimal transport, and area regularization objectives, with RMSprop and Adam optimizers.

5. Semantic and Geometric Guidance Mechanisms

Advanced frameworks integrate geometric and semantic guidance into stroke optimization. Gradient-alignment losses steer stroke orientation and magnitude to image edges: $$\mathcal{L}_{\rm grad} = \alpha\, \|\nabla I_r - \nabla I_t\|_1 + \beta\, \|\angle \nabla I_r - \angle \nabla I_t\|_1$$ Segmentation losses restrict strokes to target regions, penalizing cross-boundary placement using segment masks from external methods (e.g., SAM). Optimal transport and area losses enforce coverage regularization and prevent empty or collapsed strokes.

Such guidance schemes yield reconstructions faithful to both the local image structure and overall semantic content, critical for artistic control and compositional consistency.

6. Integration into Painting and Robotics Systems

Differentiable stroke reconstruction is central to optimization-based painting planners such as FRIDA (Chen et al., 30 Nov 2024), where the full loop is defined: $$(z,\Delta) \xrightarrow{\text{TrajVAE decoder}} \{ (x_i, y_i, h_i) \} \xrightarrow{\text{warp+Traj2Stroke}} M \xrightarrow{\text{stamp\;onto\;canvas}} \hat{I} \xrightarrow{\text{objective}} \mathcal{L} \xrightarrow{\text{backprop}} \frac{\partial \mathcal{L}}{\partial z},\, \frac{\partial \mathcal{L}}{\partial \Delta}$$ Optimized results are directly deployed on robotic systems (e.g., UR5 manipulators) for execution. This workflow unifies simulation and real-world deployment, leveraging the sample efficiency and generalization of both VAE-based representations and the segment-based differentiable renderer.

7. Comparative Frameworks and Practical Applications

Multiple frameworks and architectures extend differentiable stroke reconstruction to character recognition (Huang et al., 2018), handwriting trajectory recovery (Archibald et al., 2021), image stylization (Hu et al., 2023, Jiang et al., 17 Nov 2025), robot painting (Chen et al., 30 Nov 2024), and multi-view 3D representation (Duan et al., 2023). Parameterizations range from weighted quadratic Bézier curves, polylines, and splines to region-specific stroke templates and 3D curve primitives.

Applications include generative painting, automated artistic stylization, handwriting synthesis, robust character segmentation, and robot-physical rendering. The frameworks support modular integration into downstream objectives via end-to-end differentiable loss, enabling new capabilities in semantic planning and expressive human-like output.

Table: Stroke Parameterizations in Major Frameworks

Framework	Stroke Model	Differentiability Method
Spline-FRIDA (Chen et al., 30 Nov 2024)	Polyline/spline ($n=32$)	VAE + segment mask renderer
Birth of a Painting (Jiang et al., 17 Nov 2025)	Bézier path + style latent	Parallel/stamp-based alpha-comp.
SCR (Huang et al., 2018)	Weighted quadratic Bézier	Neural net decoder $D(p)$
TRACE (Archibald et al., 2021)	Sequential ($\Delta x,\Delta y$)	CRNN + DTW loss
Neural 3D Strokes (Duan et al., 2023)	3D SDF splines/primitives	Volumetric compositing, SDF/soft-CDF

Each framework emphasizes differentiable operations throughout the rendering pipeline, supporting robust optimization and backpropagation from outputs to parametric stroke space.

---

Differentiable stroke reconstruction constitutes a core methodology in modern computational painting, robotic art, and handwriting modeling. By precisely controlling stroke geometry, style, and compositing via gradients, such frameworks offer highly expressive and sample-efficient solutions deployed across simulation and real-world systems.