VaseVLM: Adaptive VLM for 3D Greek Pottery

• VaseVLM is a domain-adaptive vision-language model designed for multimodal question answering and analysis of 3D ancient Greek pottery artifacts.
• It employs LoRA adapters in a Qwen2.5-VL backbone and trains with supervised fine-tuning followed by reinforcement learning with dimension-specific rewards.
• The model achieves significant gains in retrieval performance and lexical fidelity over previous systems, setting a new standard in digital heritage analysis.

VaseVLM is a domain-adaptive vision–LLM (VLM) specifically developed for question answering and multimodal analysis of 3D ancient Greek pottery artifacts. Designed to address the limitations of general-purpose VLMs in cultural heritage settings, VaseVLM leverages LoRA adaptation, 3D-to-2D video input encoding, and a reinforcement learning paradigm with verifiable, dimension-specific rewards. The approach demonstrates significant improvements over previous state-of-the-art systems in both retrieval performance and specialist lexical fidelity, establishing a new technical standard for digital heritage artifact analysis (Zhang et al., 6 Oct 2025).

1. Model Architecture and Input Representation

The VaseVLM architecture is based on the open-source Qwen2.5-VL backbone, available in both 3 billion and 7 billion parameter configurations. The architecture consists of three principal components:

• Vision Transformer (ViT) Visual Encoder: Used for frame-level feature extraction. The encoder is frozen during downstream fine-tuning, focusing adaptation on subsequent layers.
• LLM Decoder/Encoder: Generates textual answers conditioned on visual input.
• Cross-Modal Fusion: Attends jointly to visual and textual token streams by cross-modal fusion layers.

LoRA Adapters are inserted into the query/key/value and feed-forward sublayers of both ViT and LLM. These low-rank matrices are the only components updated during fine-tuning, leaving the remaining pre-trained backbone parameters unchanged.

Input Representation is video-based. Each 3D vase model (GLB format) is rendered as a synthetic 360° video using Blender 3.6, producing 16 frames at 512×512 px and 2 fps. The resulting images are concatenated into a sequence of tokens for the ViT encoder. There is no explicit use of mesh or point-cloud features at inference; the 3D model is only used offline to generate training frames.

2. Domain-Adaptive Training Methodology

VaseVLM employs a two-stage training regime comprising supervised fine-tuning followed by reinforcement learning with verifiable rewards (RLVR):

1. Supervised Fine-Tuning (SFT)
   • Objective: next-token cross-entropy loss over caption and QA tokens,

   $$\mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^T \sum_{v\in V} y_{t,v}\,\log p_\theta(v\mid x,\,y_{<t}),$$

   with $x$ as video input and $y$ as token sequence. - Only LoRA weights are updated.

2. Reinforcement Learning with Verifiable Rewards (RLVR)
   • Policy $\pi_\theta$ generates sequence $\hat{y}$. Ground-truth archaeological descriptions are decomposed into six semantic dimensions: Fabric, Technique, Shape, Date, Decoration, Attribution.
   • For each $i$:

   $$r_i = \begin{cases} \cos(f_i(\hat{y}),\,g_i), &\text{if }\cos(f_i(\hat{y}),g_i)\ge 0.7 \ 0, &\text{otherwise} \end{cases}$$

   where $f_i(\hat{y})$ and $g_i$ are semantic embedding vectors. - Quality-control penalties are applied:

   $$P = \alpha_\ell P_{\mathrm{length}} + \alpha_r P_{\mathrm{repetition}} + \alpha_i P_{\mathrm{irrelevant}},\quad (\alpha_\ell=0.1,\,\alpha_r=0.1,\,\alpha_i=0.15)$$

• A bonus $x$0 is provided based on ROUGE-L sequence similarity.
• Final reward:

  $x$1

  with dimension weights $x$2.

• Training maximizes expected reward via policy-gradient:

  $x$3

This two-stage curriculum first brings VaseVLM in line with domain-validated label distributions via SFT, then sharpens multi-attribute reasoning and description granularity through RLVR targeting semantically structured, expert-aligned rewards.

3. VaseVQA-3D Dataset Construction

VaseVQA-3D is the primary data resource for 3D pottery VQA, extending the earlier 2D VaseVQA set (Ge et al., 21 Sep 2025). Its creation involves three stages:

• Stage 1: Filtering — Low-quality images are filtered with a ResNet-50 classifier. — CLIP ViT-B/32 distinguishes fragments from intact vases and performs optimal view selection. — Results in 3,880 high-quality 2D images.
• Stage 2: 2D-to-3D Generation — Model selection between TripoSG and Hunyuan3D is performed using metrics such as PSNR, SSIM, LPIPS, Chamfer Distance, and Normal Consistency. — TripoSG is selected for mesh and semantic fidelity, yielding 664 validated GLB 3D vase models.
• Stage 3: Annotation — Each GLB is rendered into a 16-frame video. — Attribute QAs adapted from the original VaseVQA, with descriptions further augmented using GPT-4o. — A total of 4,460 structured QA pairs are produced, with a 70/15/15 train/val/test split.

4. Evaluation Metrics and Experimental Protocol

• 3D Generation Quality (VaseEval) — PSNR, SSIM, LPIPS, Chamfer Distance, Normal Consistency, CLIP-Image, CLIP-Text similarity.
• VQA and Retrieval Performance (VaseVQA-3D) — Recall@k for image-to-text retrieval:

  $x$4

— Lexical Similarity (ROUGE-L F-measure).

• Human Evaluation — Ten archaeology experts score VQA outputs (0–5) for accuracy and cultural appropriateness.

5. Quantitative Results and Comparative Performance

VaseVLM demonstrates substantial improvements over previous approaches, specifically VaseVL (Ge et al., 21 Sep 2025). The following table summarizes salient results on VaseVQA-3D:

Method	R@1	Lexical Similarity
VaseVL	2.08%	0.255
VaseVLM-7B-SFT	3.12%	0.272
VaseVLM-7B-RL	3.52%	0.276

• VaseVLM-7B-RL achieves a +1.44 percentage point absolute R@1 increase over VaseVL, a 69% relative gain.
• Lexical similarity (ROUGE-L) rises by +0.021, an 8.2% relative gain.
• Reinforcement learning over SFT leads to a +12.8% R@1 and +1.5% lexical similarity gain.

Qualitative comparisons indicate enhanced taxonomic discrimination (“kylix” vs. “lekythos”), richer motif description (e.g., “meander key-pattern frieze with palmette accents”), and accurate functional attributions.

Common failure cases include incomplete reconstructions (embedding similarity below reward threshold) and attribution errors involving rare workshops, while strengths include robust technical vocabulary recognition and accurate, multi-dimensional descriptive reasoning.

6. Context, Limitations, and Ongoing Directions

VaseVLM substantially advances 3D cultural heritage VQA, integrating domain-targeted adaptation, video tokenization of 3D artifacts, and semantically decomposed reward modeling for reinforcement learning (Zhang et al., 6 Oct 2025). In comparison to earlier systems, it establishes higher benchmarks for both retrieval and text fidelity in applied heritage object analysis.

Notable limitations include sensitivity to worn or incomplete 3D reconstructions (i.e., systematic reward attenuation where visual semantics are ambiguous) and residual errors on highly infrequent attribution terms (e.g., rare workshop or painter names).

A plausible implication is that reward decomposition and strict domain input modeling, as exemplified by VaseVLM, enable specialist VLMs to surpass generic models on highly structured, multi-faceted heritage analysis tasks. Ongoing research trajectories include scaling to larger artifact typologies, explicit integration of mesh features, continual domain-adaptive RL, and knowledge graph augmentation for expanded interpretability and generalization.