VLMaterial: Inverse Procedural Material Synthesis

Updated 13 July 2026

VLMaterial is a framework for inverse procedural material modeling that converts a material image into an editable Blender node-graph program using Python code.
It leverages fine-tuned large vision-language models, program-level augmentation, and MCMC-based parameter search to synthesize compact and semantically interpretable shader graphs.
Evaluations demonstrate improved style loss, program correctness, and user preference over baselines, enhancing material editing workflows in tools like Blender.

Searching arXiv for the target paper and closely related material-generation/selection work to ground citations. arXiv search: id:([2501.18623](/papers/2501.18623)) OR ti:"VLMaterial" OR ti:"Procedural Material Generation with Large Vision-LLMs" VLMaterial is a framework for inverse procedural material modeling that takes a single input image of a material and synthesizes a fully editable procedural-material definition as a Blender node-graph program. In this formulation, the target is not a bitmap texture or a fixed latent code, but executable Python that constructs a shader graph whose rendered appearance matches the image. The system combines a fine-tuned large vision-LLM with a dataset of procedural materials transpiled from Blender, program-level augmentation driven by another LLM, and a post hoc parameter search that compensates for the non-differentiability of Blender’s node evaluation. Reported results show improved performance over prior baselines on unseen Blender materials, synthetic Substance 3D materials, and real smartphone photographs (Li et al., 27 Jan 2025).

1. Problem Setting and Conceptual Scope

VLMaterial addresses the problem of converting an image of a material on a flat surface into an editable procedural representation. The paper defines the output representation as a Blender node-graph program rather than a raster texture, emphasizing editability, resolution independence, and direct compatibility with authoring workflows used in tools such as Blender and Adobe Substance 3D (Li et al., 27 Jan 2025).

This framing places VLMaterial within inverse procedural graphics rather than within classical material classification or pixelwise segmentation. The target object is a functional graph composed of texture generators, filters, shader nodes, and links organized as a directed acyclic graph. A plausible implication is that the method treats material authoring as visual program synthesis: the image conditions a code-generation process whose execution reconstructs the material in an editable form rather than merely approximating its appearance in image space.

The system’s stated goal is to leverage large vision-LLMs to generate executable Python code that constructs a shader node graph in Blender. This differs from systems that estimate physical maps such as Albedo, Metallicity, and Roughness for reconstruction pipelines, as in PBR3DGen (Wei et al., 14 Mar 2025), and from systems that produce click-conditioned similarity masks for material selection in editing workflows, as in "Fine-Grained Spatially Varying Material Selection in Images" (Guerrero-Viu et al., 10 Jun 2025).

2. Procedural Materials as Executable Programs

A central design choice in VLMaterial is the representation of procedural materials as standard Python programs using the Blender Python API. In Blender, a material created in the node editor can be transpiled into code that creates nodes, establishes links, and sets parameters. VLMaterial therefore treats each material as a function operating on a bpy.types.Material object and generates the function body token by token (Li et al., 27 Jan 2025).

At inference time, execution of the generated Python code instantiates the node graph directly inside Blender. This has two consequences. First, the output is immediately renderable. Second, the result remains user-editable, preserving one of the principal advantages of procedural materials over image textures.

A minimal structural pattern of the representation is:

$[0,1]$ 5

The paper’s example for a “simple wood” material includes ShaderNodeTexNoise, ShaderNodeValToRGB, ShaderNodeBsdfPrincipled, and ShaderNodeOutputMaterial, with parameter assignments for scale, detail, color-ramp values, roughness, and metallicity, followed by explicit graph connectivity. The example illustrates that the model does not emit an abstract symbolic description; it emits Blender-executable shader construction code.

A common misconception is that VLMaterial must recover the exact original graph used to generate a reference image. The reported objective is instead appearance matching subject to graph compactness and editability. The paper notes that resulting node graphs remain compact, at most 30 nodes, and semantically interpretable (Li et al., 27 Jan 2025).

3. Dataset Construction and Program-Level Augmentation

VLMaterial’s training data originate from 3,663 free Blender procedural materials collected from BlenderKit, Infinigen, and individual material packs. The preprocessing pipeline performs depth-first search from the Material Output node to prune unused subgraphs, then discards materials with more than 30 nodes or invalid outputs. Additional filtering removes transpiled code longer than 2,048 tokens and materials that produce empty or low-complexity renders, operationalized as JPEG size below 12 KB. After cleaning, 1,640 unique artist-created node graphs remain, spanning categories such as wood, metal, fabrics, and abstract patterns (Li et al., 27 Jan 2025).

Because this corpus is limited for VLM fine-tuning, the paper introduces program-level augmentation in two forms: graph-structure augmentation and parameter augmentation.

Graph-structure augmentation uses GPT-4o-mini to perform what the paper calls LLM “Evolution.” A pool of 870 complex artist materials is selected as parents. Random pairs of parent Python programs are presented to GPT-4o-mini with an instruction to “cross-over” them and produce a new shader_material(material) function with at most 30 nodes. Each pairing yields up to 4 valid offspring after up to 20 trials, and validity is checked by executing the program in Blender. This process produces approximately 50.4 K new graph structures (Li et al., 27 Jan 2025).

Parameter augmentation perturbs node parameters for each program structure. Floats, vectors, colors, and ramp points are uniformly varied within $\pm 25\%$ of original values, with a minimum absolute range of $\pm 0.05$ . Hue is sampled over $[0,1]$ in HSV space; discrete and categorical fields are resampled with 25% probability; default-valued parameters are sampled only 20% of the time to limit code-length growth; and sequences exceeding the 2,048-token limit are pruned. The final result is a training set of approximately 550 K image-program pairs.

The following table summarizes the reported data pipeline.

Stage	Reported quantity	Constraint or note
Downloaded free Blender materials	3,663	BlenderKit, Infinigen, individual packs
Artist-created materials after filtering	1,640	$\leq 30$ nodes, valid outputs, code $\leq 2{,}048$ tokens
Parent materials for LLM evolution	870	Selected as complex artist materials
New graph structures from evolution	$\sim 50.4$ K	Blender-validated offspring
Final training set	$\sim 550$ K	Image-program pairs

This augmentation scheme is notable because the supervision target is code. A plausible implication is that the paper treats diversity in graph topology as at least as important as diversity in rendered appearance, since the model must learn to generate executable structures rather than only regress perceptual statistics.

4. Vision-LLM and Optimization Pipeline

The base model is LLaVA-NeXT, composed of a CLIP ViT/L-14 vision encoder, a LLaMA 3 8B language decoder, and an MLP projector. The model takes as input a material image and the textual prefix prompt: “Write a Python function with Blender API to create a material node graph for this image.” The output is a sequence of Python tokens encoded with the LLaMA tokenizer (Li et al., 27 Jan 2025).

VLMaterial fine-tunes the MLP projector together with LoRA adapters inserted in all LLaMA attention layers. The LoRA configuration is rank $r=8$ , $\alpha=32$ , and dropout $=0.05$ , for approximately 40 M trainable parameters. Optimization uses AdamW with learning rate $\pm 0.05$ 0, cosine annealing, and 3% warmup. Training is performed in BF16 with batch size 32 on 8 NVIDIA H100 GPUs using DeepSpeed ZeRO-3 for 5 epochs, taking about 3 days (Li et al., 27 Jan 2025).

The training objective is standard token-level cross-entropy:

$\pm 0.05$ 1

where $\pm 0.05$ 2 are the ground-truth program tokens.

This objective optimizes syntactic and structural fidelity at the token sequence level, but the paper explicitly adds a second-stage post-optimization because Blender’s node evaluation is not differentiable. The local search is MCMC-based and refines node parameters rather than graph topology. The reported procedure initializes with the generated material $\pm 0.05$ 3 and perceptual loss $\pm 0.05$ 4, then iteratively proposes $\pm 0.05$ 5 by resampling 10% of node parameters within $\pm 0.05$ 6, accepts improvements or accepts with small probability $\pm 0.05$ 7, and returns the best material seen (Li et al., 27 Jan 2025).

The perceptual objective is:

$\pm 0.05$ 8

The paper describes this as Gram-matrix style loss from VGG features plus an $\pm 0.05$ 9 term on $[0,1]$ 0 downsampled renders. A common misconception is that the entire inverse problem is solved by end-to-end differentiable rendering; in fact, the explicit use of gradient-free local search is a direct response to the non-differentiability of Blender evaluation.

5. Evaluation Protocol and Empirical Results

The evaluation covers three test regimes: 44 unseen Blender materials for in-distribution testing, 64 synthetic Substance 3D materials for out-of-distribution evaluation, and 64 smartphone-captured real material images (Li et al., 27 Jan 2025). The baselines are GPT-4o-mini zero-shot prompting, nearest-neighbor retrieval, Conditional MatFormer pretrained on Substance Source, and BlenderAlchemy iterative editing.

The paper reports four quantitative criteria: style loss, sliced Wasserstein distance (SWD), CLIP cosine similarity, and program correctness. Program correctness is defined as the fraction of runs that produce a valid, executable material whose rendered JPEG size exceeds 12 KB.

Reported performance for VLMaterial before post-optimization is:

Dataset	Style loss $[0,1]$ 1	SWD $[0,1]$ 2	CLIP $[0,1]$ 3	Prog. corr. $[0,1]$ 4
Blender	0.019	1.760	0.856	0.911
Substance	0.026	2.283	0.762	0.890
Real Images	0.025	2.417	0.722	0.870

The paper states that VLMaterial outperforms all baselines across both synthetic and real inputs, and that post-optimization further reduces style loss by approximately 20% (Li et al., 27 Jan 2025). Qualitatively, side-by-side comparisons show close matching on wood grain, brushed metal, and stone tiling, while preserving compact and semantically interpretable node graphs.

The user study includes 16 participants, split evenly between technical artists and researchers, each comparing VLMaterial against BlenderAlchemy on 12 test cases. The reported outcomes are 91% preference for visual match, 94% preference for graph editability, and usability ratings of 6.8 for VLMaterial versus 4.2 for BlenderAlchemy on a 1–10 scale.

These results characterize two separate success criteria. The first is visual similarity, captured by style loss, SWD, and CLIP similarity. The second is operational validity, captured by executable program correctness and editability preference. This suggests that the paper treats procedural material generation as a joint graphics-and-program-synthesis problem rather than as pure image translation.

6. Practical Behavior, Failure Modes, and Technical Constraints

VLMaterial’s practical output is an executable Blender shader function. Once run, the node graph becomes available for rendering and manual modification. This makes the method suitable for workflows in which a generated material is a starting point for further authoring rather than a terminal output (Li et al., 27 Jan 2025).

The system, however, is bounded by several explicit constraints. The first is representational capacity: very intricate natural textures may exceed the expressiveness of a graph with at most 30 nodes and the available node types. The second is sequence length: Python programs longer than roughly 2,048 tokens cannot be reliably generated. The third is syntax safety: the method does not use an inference-time grammar mask, so invalid token sequences can still occur. The fourth is search variance: sampling-based inference and MCMC refinement introduce exploration–exploitation tradeoffs. The fifth is augmentation quality: the diversity of the training set depends in part on GPT-4o-mini-generated structural offspring (Li et al., 27 Jan 2025).

These limitations clarify what VLMaterial does not claim. It does not guarantee exact syntax correctness at generation time, exact recovery of a ground-truth graph, or faithful modeling of highly irregular textures beyond the expressiveness of compact Blender graphs. It also does not rely on differentiable shader execution. Instead, the architecture couples token-level program generation with an external search stage for parameter refinement.

A plausible implication is that future progress could arise along three orthogonal axes already named in the paper: a more compact DSL than raw Python, grammar-constrained decoding or syntax checking for guaranteed correctness, and reward-driven fine-tuning that directly optimizes rendered similarity.

7. Position in the Literature and Terminological Disambiguation

VLMaterial belongs to a broader wave of vision-language methods for material-related tasks, but its task definition is unusually specific. It is not a material selector, a material classifier, or a physics-based identifier. In "Fine-Grained Spatially Varying Material Selection in Images" (Guerrero-Viu et al., 10 Jun 2025), the goal is a per-pixel similarity map conditioned on a user click, with texture-level and subtexture-level masks for downstream editing. In PBR3DGen (Wei et al., 14 Mar 2025), the objective is multi-view estimation of PBR attributes such as Albedo, Metallicity, and Roughness for 3D mesh reconstruction. In "VLMaterial: Vision-LLM-Based Camera-Radar Fusion for Physics-Grounded Material Identification" (Zhu et al., 13 Apr 2026), the task is training-free camera–radar fusion using dielectric-constant extraction for material recognition. VLMaterial in (Li et al., 27 Jan 2025), by contrast, generates Blender-executable node-graph programs from images.

This distinction matters because the generated artifact determines both the learning problem and the evaluation protocol. A segmentation system is judged by mask quality; a PBR-estimation system by decomposition fidelity and relighting behavior; a physics-grounded recognition system by identification accuracy; VLMaterial is judged by rendered similarity, executable correctness, and graph editability.

Within that landscape, VLMaterial demonstrates that open vision-LLMs can be fine-tuned on image–program pairs to synthesize editable procedural materials directly from visual input (Li et al., 27 Jan 2025). The paper states that it advances inverse procedural graphics and provides a public dataset and codebase to seed further research in visual program synthesis for material design.