The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation

Abstract: Generative text-to-image models are typically trained on large-scale web-scraped datasets that include diverse visual content such as copyrighted and stylistically distinctive artworks, raising concerns about ownership, attribution, and the unintended reuse of protected visual expressions. A key issue is that models can learn stylistic patterns from this data and reproduce them in generated outputs without any explicit reference in the prompt. We refer to this phenomenon as The Silent Brush, where such learned styles reappear even when they are not requested. Existing evaluation methods mainly focus on near-duplicate retrieval or membership inference and do not account for this form of unintended stylistic resurfacing across prompts. To address these gaps, we first formulate guiding principles for evaluation of The Silent Brush. We then introduce Art Arena, an evaluation protocol that measures how strongly artworks are encoded, how they interact, and how frequently their stylistic traits reappear in generated outputs without explicit mention in prompts. We evaluate Art Arena on widely used text-to-image diffusion models, including Stable Diffusion v1.5, Stable Diffusion XL (SDXL), and SANA-1.5, and design it to generalize across text-to-image generative systems. Our results show that The Silent Brush arises from differences in representational strength and interaction dynamics between artworks, leading to asymmetric blending in model generations. Code and evaluation resources are available at: https://anonymous.4open.science/r/ArtArena-EBE4.

• The paper introduces Art Arena, a many-to-one evaluation protocol using Entry Trials, Motif Duels, and an Influence Ledger to measure how artistic styles compete and combine in generated images.
• Experiments across 8,737 artworks, three models, and three style metrics show that imitation strength does not predict blending dominance, with SD v1.5 producing the strongest and most consistent stylistic recall.
• Fine-tuning can elevate weakly represented styles and shift influence rankings, but strongly embedded styles resist displacement, highlighting the need for interaction-based audits of AI art models.

The Silent Brush phenomenon

The paper, "The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation" (2605.17500), addresses a gap in the evaluation of text-to-image diffusion models: the tendency of learned artistic styles to reappear in generated outputs without any explicit reference in the prompt. The authors term this phenomenon the Silent Brush, and argue that existing evaluation paradigms—near-duplicate retrieval, style-similarity scoring, and membership inference—are structurally incapable of detecting it, because each is instance-level and one-to-one. Retrieval-based audits detect near-duplicates but cannot measure the relative influence of multiple training sources within a single generation; membership inference scores indicate persistence of training signals but do not predict whether a style will surface during sampling; single-image style similarity conflates style with shared semantic content. The Silent Brush, by contrast, is a many-to-one, interaction-dependent phenomenon: stylistic traces from multiple artworks compete and combine within a single output.

The paper grounds this concern in the documented lineage of training corpora such as LAION-5B, whose subsets (LAION-2B-en, LAION-high-resolution, LAION-Aesthetics) are confirmed training data for Stable Diffusion v1.x, and whose contents include copyrighted and stylistically distinctive artworks. The authors also connect the phenomenon to theoretical accounts of diffusion models as associative memories, in which high-capacity models form attractor states reflecting stylistic priors that can be activated without explicit conditioning. The motivating example is concrete: a prompt describing an androgynous figure in a nocturnal street scene, naming neither Café Terrace at Night nor The Scream, produces an output visibly incorporating elements of both.

Guiding principles and the Art Arena protocol

The evaluation is organized around four questions: (Q1) whether models imitate artworks under explicit attribution, (Q2) how models blend imitated artworks, (Q3) whether imitated artworks exhibit proactive blending, and (Q4) whether proactive blending can be steered. Art Arena operationalizes these questions in three stages.

Entry Trials (fitness test). Each artwork $w$ is prompted as "<title(w)> in the style of <artist(w)>", $K$ samples are generated, and a fitness score is computed as the mean proximity between generated samples and the reference image. Artworks exceeding a threshold $\tau_f$ enter the FitSet. High fitness under explicit attribution is interpreted as evidence of stored stylistic traces rather than diffuse influence.

Motif Duels. FitSet artworks compete in a round-robin tournament. For each ordered challenger–defender pair, the challenger contributes motifs (content-only elements extracted from the artwork) that are blended with the defender's template "<title(d)> in the style of <artist(d)>". Across $R$ rounds per match, proximity is computed against both references, and a round is awarded to whichever artwork is closer by more than a margin $\delta$. The motif extraction pipeline is deliberately constrained: motifs are derived from art-historical sources (museum and curatorial references) via GPT-4o, restricted to visible content with explicit prohibitions on style, mood, medium, and artist vocabulary, and all $2^N - 1$ non-empty motif combinations are enumerated to ensure full combinatorial coverage of content without stylistic interference.

Influence Ledger. Cumulative match wins are aggregated into a ranked leaderboard, which the authors treat as a structural map of stylistic dominance and, by implication, of leakage potential.

Proximity is measured with three complementary metrics: CLIP cosine similarity (Semantics), LPIPS (Aesthetics), and Contrastive Style Descriptors (Fidelity). The authors acknowledge that each has known limitations—CLIP's sensitivity to prompt wording and dataset bias, LPIPS's vulnerability under distribution shift and adversarial perturbation, and CSD's dependence on curated training data—and use their conjunction as a partial mitigation rather than a solution.

Experimental setup

The corpus comprises 8,737 artworks by the twenty most popular artists on WikiArt, chosen for likely presence in LAION-derived training data. Three generators spanning both dominant architectures are evaluated: SD v1.5 and SDXL (U-Net latent diffusion) and SANA-1.5 (DiT-based). The fitness threshold selects the top twenty artworks per metric per model, giving a FitSet of twenty for each of nine model–metric combinations. Motif duels run with $R = 5$ rounds per pair and $\delta = 0$. For Q4, the bottom five artworks per model undergo LoRA fine-tuning (rank 64, 1024×1024, 10 epochs, learning rate $10^{-4}$) and the tournament is repeated.

Results

Imitation is uneven and architecture-dependent. No single artwork qualifies for the FitSet under all three proximity metrics, indicating that no artwork is uniformly well-represented across stylistic dimensions. Fitness distributions show substantial variability with prominent outliers, confirming that imitation is artwork-dependent rather than uniform. A notable and somewhat counterintuitive finding is that SD v1.5, the smallest model, achieves the strongest and most consistent imitation across all three metrics, while SDXL falls behind on Aesthetics and Fidelity despite sharing the U-Net architecture, and SANA-1.5 is weakest and most variable across all dimensions. This directly contradicts any assumption that model scale uniformly improves stylistic recall.

Blending is asymmetric and selective. Outputs sharing the same challenger motifs but differing defender templates retain different aspects of the challenger—one preserving texture cues, another color composition—showing that the model resolves competing stylistic cues rather than averaging them. A striking instance of within-artist leakage is documented: when SANA-1.5 is prompted with Alpilles with Olive Trees in the Foreground by van Gogh, the output exhibits elements of The Starry Night, indicating that a more strongly encoded work by the same artist overrides the explicitly named one. Blending dominance is dimension-specific per model: SDXL shows concentrated dominance (van Gogh leading Semantics and Fidelity, O'Keeffe leading Aesthetics), SANA-1.5 clusters around Warhol and Pollock, and SD v1.5 distributes influence more evenly.

Proactive blending is role-asymmetric. Across models and metrics, defender wins generally span more artworks than challenger wins, indicating that defender influence is broadly distributed while challenger influence is concentrated among fewer artworks. The exception is SANA-1.5 under Aesthetics, the only configuration where challenger influence spans more artworks. Critically, imitation rank predicts defender dominance but not challenger performance: in the SD v1.5 Semantics consistency matrix, artwork A18 records two challenger wins despite ranking below A5, A10, and A13, which achieve none. This is the paper's central methodological claim—imitation strength measures how reproducibly a style is encoded but not how styles interact, so interaction-based evaluation such as Motif Duels is necessary to characterize leakage.

Steering is partial. Targeted LoRA fine-tuning shifts Influence Ledger rankings, but with model-dependent magnitude: SANA-1.5 shows frequent and large rank changes (e.g., a 10-position gain for Sunny Lawn in a Public Park under Aesthetics, and a 17-position gain for Picasso's Weeping Woman under Fidelity for SD v1.5), SDXL shows moderate shifts, and SD v1.5 is comparatively stable. However, artworks that rank highly before fine-tuning resist displacement, indicating that strongly embedded styles are only partially controllable. Fine-tuning can elevate weakly represented styles, but does not readily dislodge dominant ones.

Limitations and open questions

The authors state several limitations explicitly. The proximity metrics are proxies that capture only partial aspects of style, so Influence Ledger rankings are approximations of stylistic influence rather than evidence of memorization or infringement. Motif extraction depends on GPT-4o and curatorial sources that may carry bias, and prompt-design effects may influence generations independently of latent representations. The corpus covers only highly popular WikiArt artists, limiting coverage of underrepresented traditions, and the evaluation is restricted to pairwise interactions, whereas real generations involve many simultaneous stylistic sources. The framework measures relative dominance but not causal attribution: a high ranking does not distinguish memorization from distributed generalization, dataset frequency, or architectural bias, and for models such as SANA-1.5 that do not disclose training data, the presence of specific artworks in training corpora cannot be verified. The authors also note that influence scores could be misused to rank styles competitively or reinforce cultural hierarchies, and position Art Arena strictly as a diagnostic tool for transparency and governance rather than enforcement; no legal claims are made. An open question the paper leaves is whether the observed role asymmetry (broad defender influence, concentrated challenger influence) generalizes beyond pairwise settings to many-source blending.

Conclusion

Art Arena reframes stylistic influence in generative models as a measurable, interaction-driven phenomenon rather than a static similarity problem. Its principal empirical findings are that imitation strength does not predict blending dominance, that stylistic influence during blending is asymmetric and architecture- and dimension-dependent, and that fine-tuning can elevate but not readily displace strongly encoded styles. The protocol provides a basis for auditing unprompted stylistic persistence, though its reliance on proxy metrics, popular-artist corpora, and pairwise interactions bounds the strength of any conclusions drawn from it.