Neural Artistic Tracing (NeAT) Method

• NeAT is a feed-forward style transfer method that predicts an RGB residual to edit a pre-processed content image, ensuring semantic preservation while fusing style.
• The approach integrates style, content, adversarial, contrastive, and patch co-occurrence losses to achieve realistic, halo-free stylizations.
• It leverages the large-scale BBST-4M dataset and Sobel-guided patch sampling to curate data and correct style halos during training for fast high-resolution inference.

NeAT, short for Neural Artistic Tracing, is a feed-forward style transfer method that re-formulates style transfer as image editing rather than image generation. In this formulation, the model constructs a stylized image by predicting an RGB residual to be added to a pre-processed content prior, with the stated aims of preserving the semantic content of the source image, matching the artistic style of a target image, eliminating “style halos,” and enabling fast inference at high resolution. The same work introduces BBST-4M, a large-scale high-resolution dataset, and a stylistic image classifier used to curate style and content data at scale (Ruta et al., 2023).

1. Architectural formulation

NeAT operates on a content image $I_c$ and a style image $I_s$ through a pipeline that begins with content-prior pre-processing. The input content image is first blurred with Gaussian blur of radius 7px and bilateral-filtered with kernel 25 and $\sigma = 100$ to remove fine detail. The blurred image is then re-coloured so that its first two moments, specifically mean and covariance in RGB, match those of the style image. This pre-processed prior is down-weighed by $0.5$, yielding $I_c'$.

The core network follows an encoder–attention–decoder structure. A frozen VGG-19 extracts multi-layer features $\{\phi_i(\cdot)\}_{i=1}^L$, four PAMA style-attention blocks fuse content and style features, and a small convolutional decoder predicts an RGB “delta” image $\Delta \in \mathbb{R}^{H \times W \times 3}$. The final stylized output is defined as

$I_{sc} = I_c' + \Delta.$

This residual construction is central to the paper’s image-editing interpretation of feed-forward style transfer (Ruta et al., 2023).

NeAT also includes adversarial and contrastive components. A domain-level image discriminator $\mathcal D$ drives realism. Two contrastive projection heads, $\ell_s$ and $I_s$0, encourage stylizations sharing the same style to be close in style-embedding space and stylizations sharing the same content to be close in content-embedding space. In addition, a patch co-occurrence discriminator $I_s$1 is split into “simple” and “complex” variants, corresponding respectively to low-frequency and high-frequency regions.

2. Objective function and loss design

The optimization objective combines style, content, identity, adversarial, contrastive, and patch co-occurrence terms. The style loss $I_s$2 matches the spatial mean and standard deviation of VGG-19 feature maps between the stylized output $I_s$3 and the style image $I_s$4 across multiple layers. The content loss $I_s$5 is defined at $I_s$6 and constrains preservation of source content. The identity loss $I_s$7 enforces exact reconstruction when style and content coincide, using both pixel-space and feature-space terms.

The adversarial loss $I_s$8 uses the image discriminator $I_s$9. The style-contrastive loss $\sigma = 100$0 pulls together stylizations that share the same style, while the content-contrastive loss $\sigma = 100$1 pulls together stylizations that share the same content. The patch co-occurrence losses $\sigma = 100$2 and $\sigma = 100$3 are applied to simple and complex patch sets selected by Sobel-guided sampling.

The full objective is

$\sigma = 100$4

The hyperparameter defaults reported for the patch terms are $\sigma = 100$5 and $\sigma = 100$6, and the contrastive temperature is $\sigma = 100$7. The remaining $\sigma = 100$8 are tuned on a small validation set. This loss design makes explicit that NeAT is not optimized solely for perceptual style matching; it is jointly optimized for content retention, realism, embedding consistency, and local texture behavior (Ruta et al., 2023).

3. “Style halos” and their correction

A central contribution of NeAT is the identification and correction of “style halos,” described as overly-complex texture rings around strong edges. The paper attributes this artifact to the behavior of global style statistics in entirely low-frequency regions: far from edges, there is no local guidance, so global statistics push texture complexity upward.

To address this, NeAT introduces Sobel-guided patch sampling. Patch sampling is divided into two buckets: simple for low-frequency patches and complex for high-frequency patches. The procedure samples patch centers uniformly, crops identically located patches from the stylized image and the style image, computes the average Sobel intensity in each crop, and assigns the patch to the simple or complex bucket according to whether the average is below or above the global median. Separate patch-co-occurrence losses are then computed for the two sets (Ruta et al., 2023).

This separation is important because it changes what the discriminator learns about local structure. In the formulation given by the paper, forcing the patch discriminator to examine low-frequency regions separately allows it to learn that such regions should remain low-complexity, thereby suppressing halos. Within NeAT, halo correction is therefore not a post-processing step but a training-time mechanism embedded in the adversarial objective.

4. BBST-4M and stylistic data curation

NeAT is trained and evaluated not only on standard datasets but also on BBST-4M, introduced as a large-scale high-resolution dataset of 4.2 million images, all at least 1024 px on the shorter side. The reported split consists of 2.2 million “style” images from Behance.net, including digital art, oil paint, and vector images, and 2.0 million “content” images from Flickr, described as photographs with no strong style (Ruta et al., 2023).

A stylistic classifier is used to curate these data automatically. The classifier has two branches: a Vision Transformer pretrained on ImageNet to provide content features, and an ALADIN-ViT pretrained on style-similarity to provide style features. The resulting feature vectors are concatenated and passed to a small MLP that predicts the probability of an image being “stylistic.” Training is described as iterative and human-in-the-loop, using StyleBabel image labels until near-perfect accuracy.

The classifier is then used to accept only “non-stylistic” Flickr images into the content split and only “stylistic” Behance images into the style split. A plausible implication is that the paper treats large-scale style transfer performance as dependent not only on model capacity but also on dataset curation quality, especially on the separation between source-content distributions and target-style distributions.

5. Training procedure

The reported training pipeline begins by sampling a content–style pair $\sigma = 100$9, applying blur, bilateral filtering, and re-colouring to produce $0.5$0, and forwarding $0.5$1 and $0.5$2 through the encoder–attention–decoder stack to obtain $0.5$3 and $0.5$4. VGG features and Sobel maps are then extracted, after which the style, content, identity, adversarial, contrastive, and patch losses are computed and summed into $0.5$5 before backpropagation.

Optimization uses AdamW with learning rate $0.5$6 and $0.5$7. The batch size is 16, accumulated to an effective batch size of 64 for contrastive learning. Training is reported to take approximately 3 days on a single A100. The paper also lists three stability and speed mechanisms: logit accumulation for large-batch contrastive learning, a two-stage Sobel discriminator for halo correction, and the content prior, which reduces decoder work and speeds convergence (Ruta et al., 2023).

The training design reflects a hybrid objective characteristic of recent feed-forward stylization systems: perceptual supervision is combined with adversarial supervision, representation-level contrastive supervision, and local patch discrimination. In NeAT, these are explicitly coordinated through the image-editing residual pathway rather than through full image synthesis.

6. Inference behavior and empirical results

At inference time, NeAT is single-pass and feed-forward. It supports up to 2 K resolution in approximately 0.22 s on an A100. The reported runtimes are 0.049 s per image at $0.5$8, 0.068 s at $0.5$9, and 0.226 s at $I_c'$0. The model size is reported as approximately 45 MB for the E–T–D weights together with the discriminator weights. The system also exposes two inference-time controls: adjustable content-prior blur, which trades off detail, and adjustable style strength $I_c'$1 through interpolation in decoder feature-space,

$I_c'$2

with $I_c'$3 (Ruta et al., 2023).

Evaluation uses LPIPS for content preservation, SIFID for style consistency, Chamfer-color-distance for color transfer, and runtime, with lower being better on all reported quantitative metrics. In the quantitative comparison, NeAT trained on WikiArt+MSCOCO reports LPIPS 0.655, SIFID 1.171, and Chamfer 78.78; NeAT without prior blur reports 0.596, 2.343, and 96.54; NeAT trained on BBST-4M reports 0.687, 1.053, and 41.74; and NeAT trained on BBST-4M without blur reports 0.635, 1.306, and 54.23. The listed baselines are ContraAST, PAMA, SANet, CAST, and AdaIN.

The user study on AMT reports overall preference over eight strong baselines, with NeAT preferred over all of them; content-only and style-only preference are also reported as above 50% for the method. The BBST-4M variant is reported as preferred 56% over the WikiArt model on fine-art comparisons and 60% on out-of-distribution modern styles. Qualitative evidence includes a main-paper figure showing zoomed stylizations free of halos and a supplement containing dozens of high-resolution examples together with a video stylization showcase.

The reported numbers do not indicate a uniform improvement on every metric for every variant. The BBST-4M model achieves the best SIFID and Chamfer-color-distance among the reported entries, whereas LPIPS is lower for some baselines and for NeAT variants without prior blur. This suggests that, within the reported evaluation, NeAT’s strongest gains are concentrated in style consistency, color transfer, generalization across a wide variety of styles, and the suppression of halo artifacts, while content preservation depends more sensitively on the use of the blur prior and the chosen training variant.