ArtMUSE: Multimodal Art Research

Updated 9 July 2026

ArtMUSE is a multifaceted term referring to systems and datasets that integrate museum metadata, creative writing, image synthesis, and audio generation using multimodal approaches.
It leverages heterogeneous modalities like text, images, and audio to improve retrieval, aesthetics assessment, and creative expression through techniques such as property graphs and token-based scoring.
The research spans museum informatics, narrative creative writing, generative visual art, and artwork-to-music systems, demonstrating enhanced efficiency, semantic alignment, and interpretability.

Searching arXiv for the provided identifiers and topic variants to ground the article. arxiv_search query="ArtMUSE OR ArtiMuse OR MuseKG OR Art2Mus OR FlexMUSE OR Art2Music" max_results=10

ArtMUSE is an overloaded designation in recent arXiv literature. It appears as a search and exploration platform for art museum collections that can be powered by MuseKG, as a manually curated dataset for multimodal creative writing introduced with FlexMUSE, and, under the closely related name ArtiMuse, as a multimodal LLM for image aesthetics assessment (Li et al., 20 Nov 2025, Chen et al., 22 Aug 2025, Cao et al., 19 Jul 2025). Across these usages, the literature repeatedly emphasizes integration of heterogeneous modalities—museum metadata, text, images, audio, and expert judgments—while preserving semantic consistency, controllability, or interpretability.

1. Terminological scope

In the current literature, the term does not denote a single canonical architecture. Instead, it names several art-centered multimodal resources and systems.

Referent	Role	Primary source
ArtMUSE via MuseKG	Search/exploration platform for art museum collections	(Li et al., 20 Nov 2025)
ArtMUSE dataset	Around 3,000 calibrated text–image pairs for MMCW	(Chen et al., 22 Aug 2025)
ArtiMuse / ArtMUSE	MLLM-based image aesthetics assessment system and dataset	(Cao et al., 19 Jul 2025)

The designation also appears as a task framing for artistic image synthesis and artwork-conditioned music generation. In this broader usage, Muse is positioned as a model for artistic image synthesis and editing, while Art2Mus, ArtToMus, and Art2Music are presented as components or foundations for an “ArtMUSE” system that maps artworks to music through cross-modal generation (Chang et al., 2023, Rinaldi et al., 2024, Rinaldi et al., 19 Feb 2026, Hong et al., 27 Nov 2025).

This multiplicity is not merely terminological. It reflects distinct research programs: cultural-heritage knowledge representation, multimodal creative writing, generative image synthesis, artwork-to-music generation, and expert-level aesthetic assessment. The shared emphasis is on art-domain multimodality rather than on a single implementation stack.

2. Museum collections, symbolic grounding, and natural-language access

In the museum informatics setting, ArtMUSE is directly informed by MuseKG, an end-to-end knowledge-graph framework for integrating heterogeneous cultural heritage data and supporting natural-language interaction over museum collections (Li et al., 20 Nov 2025). MuseKG addresses “vast but fragmented artefact information spread across structured catalogues, unstructured texts, and multimedia (images)” by combining symbolic representations with neural methods in a retrieval-augmented generation pipeline. Its core representation is a typed property graph,

$G = (V, E, \tau, \rho, A),$

where nodes represent entities in museum collections, edges encode directed relationships, $\tau$ maps nodes to types, $\rho$ assigns relation labels, and $A$ maps nodes to attribute dictionaries. The permissible node types are {object, person, organisation, image_label, image, place, concept}, and relation labels are normalized to a fixed vocabulary of 7 relations, including examples such as has_primary_producer, has_secondary_producer, and has_related_object.

The system architecture has two modules: a KG constructor and an NL query interface. The KG constructor performs normalization, entity identification and node creation, node type assignment, relation labeling and edge creation, deduplication, and schema validation. Source data come from JSON records from The University of Melbourne’s MDHS Museums collections and include object metadata, relational fields, images, and image labels. A running example maps the object “Long Scale Galvanometer” (OBJ123), its attributes, its primary producer “Walden Precision Apparatus Limited,” and associated images into the graph. Named entities detected from textual fields are linked as place, person, or concept nodes, and image-derived labels are explicitly represented as image_label nodes.

The NL interface uses KG-RAG rather than translation from natural language to SPARQL or Cypher. An LLM identifies key entities in the question, retrieves all node attributes and one-hop neighbors for each identified entity, and is prompted to answer using only the KG-derived context. Supported query categories are attribute lookup (C1), relation-neighbour retrieval (C2), and attribute-of-related-entity retrieval (C3). The paper’s examples include retrieving the measurements of the “Long Scale Galvanometer,” where the answer is “14.0 x 29.0 x 22.0 cm,” and traversing has_entity from “Certificate of Passing First Year of Bachelor of Laws” to recover the accession number “MHM06682.”

Evaluation uses a 150-question benchmark, with 50 questions per category, and compares MuseKG against zero-shot, few-shot, and SPARQL-prompt baselines across Gemma-3-4B, Gemma-3-12B, and GPT-OSS-20B. MuseKG achieves the highest accuracy across all backbones and categories; with Gemma-3-12B, its latency is reported as approximately 0.36 s per query, compared with 0.37 s for few-shot, 3.38 s for SPARQL-prompt, and 8.96 s for zero-shot inference. The reported interpretation is that symbolic grounding improves interpretability, reduces hallucinations, and keeps LLM computation confined to brief synthesis over pre-filtered graph context (Li et al., 20 Nov 2025).

3. Multimodal creative writing and the ArtMUSE dataset

In FlexMUSE, ArtMUSE denotes a manually curated dataset of approximately 3,000 text–image pairs for multimodal creative writing (MMCW), a task in which textual and visual contexts “need not be strictly aligned” in the captioning or visual-storytelling sense (Chen et al., 22 Aug 2025). The dataset was created to support illustrated-article generation with flexible interaction patterns: text-only inputs that optionally trigger text-to-image generation, or joint text–image inputs. Its distinctive property is loose semantic relation. Text and image are calibrated to be thematically coherent and creatively resonant rather than literally co-descriptive.

The dataset is sourced from Chinese social media content in architecture, design, and advertising. Images are curated at 1024×1024 resolution. Calibration combines automated filtering and expert review: obvious advertisements and low-quality images at or below 128×128 are removed, and 30 experts in art, architecture, and landscape architecture verify that each pair is thematically coherent, creatively strong, and suitable for illustrated content. The paper does not report official train, validation, or test splits, detailed category distributions, or exact text-length statistics, and it states that public release will follow peer review for data privacy.

ArtMUSE is tied to a specific methodological agenda. FlexMUSE uses pre-trained CLIP to obtain text and vision features $Z^T$ and $Z^V$ , an optional T2I module to create visual anchors when only text is given, a modality semantic alignment gating module (msaGate) to restrict textual input when the two modalities are already semantically close, attention-based cross-modality fusion for semantic enhancement, and modality semantic creative direct preference optimization (mscDPO) to encourage creativity without breaking topic coherence. The evaluation protocol combines ROUGE-1/2/L and BertScore with LLM-based judgments on Style Consistency, Context Consistency, Creativity, Richness, and Coherence, each in both reference-free and reference-aware variants.

On ArtMUSE, FlexMUSE reports ROUGE-1 0.55, ROUGE-2 0.34, ROUGE-L 0.43, and BertScore 0.72, surpassing baselines such as GPT-4o, GLM-4V, Qwen2.5-vl, DOC, LaDiC, mm-cot, and mPLUG-Owl. Its LLM-based scores are reported as SC^f = 4.50, SC^a = 3.98, CC^f = 4.31, CC^a = 3.90, CV^f = 4.40, CV^a = 3.48, RN^f = 4.47, RN^a = 3.70, CO^f = 4.63, and CO^a = 3.93. Human evaluation by 50 experts over 100 article pairs is summarized as generally preferring FlexMUSE over GPT-4o. The paper reports at least 9.091% improvement in BertScore, 12.27% in creativity, and 2.61% in coherence over baselines on ArtMUSE (Chen et al., 22 Aug 2025).

4. Artistic image synthesis in the Muse lineage

In another usage, ArtMUSE designates the application of Muse to artistic image synthesis and editing (Chang et al., 2023). Muse is a text-to-image Transformer trained in discrete visual token space with masked token prediction. Rather than regressing pixels or decoding autoregressively token by token, it predicts randomly masked image tokens in parallel, conditioned on embeddings from a frozen T5-XXL encoder with 4.6B parameters. The image tokenizer is based on two VQGANs with codebook size 8192: a low-resolution tokenizer at 256×256 with downsampling ratio $f=16$ , producing a 16×16 latent token grid, and a high-resolution tokenizer at 512×512 with downsampling ratio $f=8$ , producing a 64×64 latent token grid.

The model consists of a masked image transformer for the low-resolution stage and a super-resolution transformer that maps 16×16 low-resolution tokens to 64×64 high-resolution tokens in latent space, conditioned on text and low-resolution tokens. Inference uses iterative parallel decoding with confidence-based progressive unmasking: 24 iterations for the 16×16 base tokens and 8 iterations for the 64×64 super-resolution tokens. The paper attributes Muse’s efficiency to the use of discrete tokens, classification-style losses, and parallel decoding.

Quantitatively, the 632M Muse base plus super-resolution model achieves FID 6.06 and CLIP 0.26 on CC3M, while the 3B parameter model reaches FID 7.88 and CLIP 0.32 on zero-shot COCO evaluation. On TPUv4 hardware, Muse-3B is reported at approximately 0.5 s for 256×256 generation and approximately 1.3 s for 512×512 generation, more than 10× faster than Imagen-3B or Parti-3B and about 3× faster than Stable Diffusion v1.4.

The same masked-token formulation enables zero-shot inpainting, outpainting, and mask-free editing without fine-tuning or inversion. For inpainting and outpainting, the input image is tokenized, masked regions are decoded in low and high resolution, and the final 512×512 result is recovered through the VQGAN decoder. For mask-free editing, a small random subset of tokens is iteratively resampled, with the paper recommending guidance scale around 4 and top- $k$ sampling with $k=3$ for edits that preserve pose and composition while altering identity, style, or content (Chang et al., 2023).

5. Artwork-to-music generation as an ArtMUSE direction

Artwork-to-music generation is a major cross-modal branch of the ArtMUSE idea. Three closely related systems define this trajectory: Art2Mus, ArtToMus, and Art2Music (Rinaldi et al., 2024, Rinaldi et al., 19 Feb 2026, Hong et al., 27 Nov 2025).

Art2Mus extends AudioLDM 2 to support image-conditioned music generation from digitized artworks (Rinaldi et al., 2024). Its pipeline uses ImageBind as an artwork encoder, a trainable Image Projection Layer to align image embeddings to GPT-2 token space, FLAN-T5 for conditioning text, GPT-2 as a “Language of Audio” translator, and AudioLDM 2’s VAE plus latent diffusion backbone for waveform synthesis. The base AudioLDM 2 backbone and GPT-2 remain frozen; only the Image Projection Layer is trained. Dataset construction proceeds by pairing a 10,000-artwork subset from ArtGraph with 10,000 tracks from FMA Large using cosine similarity in ImageBind embedding space, plus a complementary 10,000 text–music dataset from LLaVA-generated artwork descriptions. Objective results show that the 4-token variant improves over the 1-token variant in ImageBind Score, but text-conditioned AudioLDM 2 baselines remain stronger in FAD and cross-modal alignment. Subjective results also tilt toward text-conditioned baselines.

ArtToMus, introduced later as the redesigned direct artwork-to-music framework, removes the image-to-text intermediate and treats visual-to-audio generation as a distinct problem (Rinaldi et al., 19 Feb 2026). It is trained on ArtSound, a dataset of 105,884 artwork–music pairs derived from ArtGraph and FMA Large, enriched with dual-modality captions and alignment metadata. Pairing uses cosine similarity between ImageBind embeddings of raw images and raw audio, with mean Image–Audio similarity reported as 0.1864. The model conditions AudioLDM 2 through a learned visual-to-LoA interface: CLIP or ImageBind provides a visual embedding, an Image Aligner maps it to $\tau$ 0 tokens in GPT-2’s 768-dimensional token space, GPT-2 produces visual LoA conditioning vectors, and the diffusion prior generates audio latents that are decoded through a VAE and HiFiGAN vocoder. Only the Image Aligner is trained. With ImageBind and 16 tokens per image, ArtToMus reports FAD 13, KL-Div 0.00296, IBSc Artw–GeMus 0.04671, and IBSc GtMus–GeMus 0.20368. The paper explicitly states that lower absolute alignment than text-conditioned systems is expected because linguistic supervision has been removed.

Art2Music takes a different route by focusing on “multi-modal feeling alignment” rather than direct visual-only conditioning (Hong et al., 27 Nov 2025). It introduces ArtiCaps, a pseudo-aligned image–text–audio dataset of 443,662 triplets built by matching image–comment pairs from ArtEmis to music–text pairs from MusicCaps using TinyBERT text embeddings. The generation model uses frozen OpenCLIP encoders for image and text, a gated residual fusion module, a 4-layer bidirectional LSTM to predict 80-bin mel-spectrograms of length $\tau$ 1, a frequency-weighted L1 loss to emphasize high-frequency fidelity, and a fine-tuned HiFi-GAN vocoder. Under full multimodal conditioning, Art2Music reports MCD 11.36, FAD 0.70, LSD 9.64, and cosine similarity 0.56, improving over random-input, text-only, and image-only variants. The paper emphasizes lightweight deployment, single-RTX-4060 training, and applications in interactive art, personalized soundscapes, and digital art exhibitions.

Taken together, these systems define three progressively different positions. Art2Mus retains a text-guided AudioLDM 2 pipeline with image projection; ArtToMus prioritizes direct visual conditioning at larger scale; Art2Music focuses on image-plus-comment feeling alignment with lightweight components. A plausible implication is that “ArtMUSE” in the music-generation literature names a research space rather than a single model family.

6. ArtiMuse / ArtMUSE as fine-grained image aesthetics assessment

ArtiMuse, also referred to as ArtMUSE in the paper’s details, is a multimodal LLM for comprehensive Image Aesthetics Assessment (IAA) (Cao et al., 19 Jul 2025). It is designed to deliver both a holistic aesthetic score on a 0–100 scale and expert-level textual analysis across eight attributes: Composition & Design, Visual Elements & Structure, Technical Execution, Originality & Creativity, Theme & Communication, Emotion & Viewer Response, Overall Gestalt, and Comprehensive Evaluation. The model is built on InternVL-3-8B, using InternViT-300M-448px-V2.5 as visual encoder, Qwen2.5-7B as LLM, and an MLP projector for multimodal fusion. All images are resized to a fixed 448×448 resolution; the paper reports that this fixed-resolution strategy yields approximately 0.3 SRCC/PLCC gains and more than doubles efficiency relative to dynamic tiling.

A central contribution is the “Token As Score” paradigm. Rather than predicting a coarse rating level or emitting a free-form numeric string, the model maps existing discrete LLM tokens—specifically ordered “twin-letter” tokens such as “aa, ab, …” mapped to 0 through 100—to integer scores, and then computes the final score as the expectation over the token probability distribution. The paper contrasts this with “Level As Score,” which partitions the score range into five intervals, and reports that Token As Score performs better.

ArtiMuse is supported by ArtiMuse-10K, described as the first expert-curated image aesthetic dataset with 10,000 images spanning 5 main categories and 15 subcategories, each annotated with eight attribute analyses and an overall score. The category totals are Photography 4111, Painting & Calligraphy 3095, AIGC 1453, 3D Design 823, and Graphic Design 518. The dataset split is 9,000 train and 1,000 test. Training uses a two-stage recipe: text pretraining with LoRA rank 16 and score finetuning with LoRA rank 128, both on 4× A100 80GB GPUs.

Quantitatively, ArtiMuse reports SRCC/PLCC of 0.827/0.826 on AVA, 0.936/0.958 on PARA, 0.510/0.543 on TAD66K, 0.814/0.837 on FLICKR-AES, and 0.614/0.627 on ArtiMuse-10K, outperforming Q-Align and most other baselines on these benchmarks. In cross-dataset generalization when trained only on AVA, it continues to exceed Q-Align on PARA, TAD66K, FLICKR-AES, and ArtiMuse-10K. For expert-level understanding, an MLLM-based comparison against human expert references reports that ArtiMuse wins on average 71.1% across the eight attributes, and a human user study reports 67.8% preference for ArtiMuse’s analyses. The paper’s stated limitations are that the model currently focuses on understanding and analysis rather than professional enhancement recommendations, and that inter-rater reliability and licensing specifics are not reported (Cao et al., 19 Jul 2025).

7. Recurrent design principles, limitations, and outlook

Several recurrent principles unify the disparate ArtMUSE usages. Museum-oriented work emphasizes symbolic grounding, traceability, and efficient one-hop reasoning over typed property graphs (Li et al., 20 Nov 2025). The multimodal creative-writing line emphasizes loose semantic relation, expert calibration, and evaluation criteria that privilege creativity and topic coherence over literal visual grounding (Chen et al., 22 Aug 2025). Artwork-to-music systems explore whether semantic resonance can be learned directly from visual features, with later work treating the removal of text supervision as a deliberate increase in task difficulty rather than as a regression (Rinaldi et al., 19 Feb 2026). ArtiMuse, in aesthetics assessment, addresses modality bias by combining expert curation, fine-grained taxonomies, and token-based numeric scoring (Cao et al., 19 Jul 2025).

The limitations are equally systematic. MuseKG currently evaluates 150 questions from a single institution’s collections, focuses on one-hop reasoning, and does not describe alignment to CIDOC-CRM, Europeana, or Wikidata (Li et al., 20 Nov 2025). The ArtMUSE dataset for MMCW does not report official splits, license, or public URL, and its Chinese social-media origin may constrain generalization across domains and languages (Chen et al., 22 Aug 2025). Art2Mus reports weaker perceptual quality and artwork alignment than text-conditioned baselines, reflecting the difficulty of synthetic pairing and sparse trainable parameters (Rinaldi et al., 2024). ArtToMus reports a nontrivial modality gap in image–audio pairing and lower absolute alignment than text-conditioned systems, while Art2Music explicitly accepts only weak pseudo feeling alignment in ArtiCaps (Rinaldi et al., 19 Feb 2026, Hong et al., 27 Nov 2025). ArtiMuse does not yet provide actionable enhancement recommendations and does not report inter-rater reliability (Cao et al., 19 Jul 2025).

This suggests that ArtMUSE is best understood not as a single platform, but as a family of art-centered multimodal research programs. In that family, the technical frontier lies in balancing semantic flexibility with grounding: typed graphs versus neural generation, loose thematic resonance versus strict alignment, direct visual conditioning versus language-mediated supervision, and expert interpretability versus end-to-end efficiency.