FlexMUSE: Multi-Modal Creative Writing Framework

• FlexMUSE is a framework for multi-modal creative writing that integrates flexible text and image interactions without strict alignment requirements.
• It employs a shared CLIP-style encoder, probabilistic gating, and attention-based fusion to ensure semantic consistency and enhanced creativity.
• The system optimizes illustrated article generation using modality semantic creative direct preference optimization and efficient single-GPU training.

Searching arXiv for the target paper and any directly relevant companion context. FlexMUSE is a framework for multi-modal creative writing (MMCW), a task that aims to produce illustrated articles and is presented as distinct from common multi-modal generative tasks such as storytelling or caption generation because textual and visual contexts are not strictly related to each other. The framework is introduced as a method for economically performing MMCW with flexible interactive patterns while improving semantic alignment between output modalities. It combines optional text-to-image generation, a shared CLIP-style encoding stage, a modality semantic alignment mechanism, an attention-based fusion module, and a preference-optimization procedure designed to enhance creativity and coherence (Chen et al., 22 Aug 2025).

1. Task formulation and problem setting

MMCW is described as an “entirely new and more abstract challenge” in which the relation between textual and visual contexts is not strictly constrained. This differentiates it from more conventional multi-modal generative settings, where cross-modal correspondence is often tighter and more directly supervised. Within this formulation, the central difficulty is not only generating text and images jointly, but doing so under flexible interaction patterns while preserving semantic consistency across modalities (Chen et al., 22 Aug 2025).

The framework is explicitly motivated by limitations of existing methods for related tasks. Those methods can be forcibly migrated to MMCW, but they require specific modality inputs or costly training, and often suffer from semantic inconsistencies between modalities. FlexMUSE is proposed against that background as a system that permits both text-only and text+image workflows, while emphasizing “the unification between modalities” and “semantic enhancement” (Chen et al., 22 Aug 2025).

A plausible implication is that FlexMUSE treats MMCW as a coordination problem between partially aligned modalities rather than as a strict cross-modal grounding task. That interpretation is consistent with the framework’s use of probabilistic gating and fusion rather than a deterministic one-to-one alignment scheme.

2. Architectural organization and flexible interaction

The architecture supports two input modes. In the text-only setting, given a topic or title $T$, FlexMUSE invokes a pre-trained text-to-image diffusion model, such as Stable Diffusion, to produce $n$ diverse but semantically related images. In the text+image setting, the text-to-image step is skipped and supplied image(s) are used directly. This is the basis for the framework’s “flexible interaction” at inference time (Chen et al., 22 Aug 2025).

Its pipeline can be summarized as $T$ (with or without images) $\rightarrow$ CLIP encoder $\rightarrow$ msaGate $\rightarrow$ cross-modal fusion $\rightarrow$ LLM $\rightarrow$ illustrated creative text. Both the text $T$ and each image $V_i$ are embedded by the same CLIP-style encoder $n$0, yielding $n$1 and $n$2. The decoder is a T5-style LLM head initialized with supervised fine-tuning on ArtMUSE and then further tuned with modality-semantic creative direct preference optimization (mscDPO) (Chen et al., 22 Aug 2025).

The encoder-decoder instantiation reported for training uses a pre-trained Chinese CLIP as the multi-modal encoder and Mengzi-T5-base (Langboat) as the decoder. Batch size, weight decay, and other settings are kept in line with standard T5 fine-tuning, and regularization is implemented through probabilistic masking in msaGate and dropout in fusion. These choices indicate an emphasis on a lightweight system that can be trained on a single NVIDIA RTX 4090 rather than on a large distributed configuration (Chen et al., 22 Aug 2025).

3. Modality Semantic Alignment Gating and cross-modality fusion

The Modality Semantic Alignment Gate (msaGate) is introduced to suppress redundant or conflicting textual features when text and image are already highly similar. The semantic similarity between text and image is defined as

$n$3

A random threshold is sampled as $n$4, and the gate mask $n$5 is defined by

$n$6

and

$n$7

The gated text feature is then

$n$8

The stated intuition is that when text and image are already closely aligned, the model can zero out the entire text embedding with some probability, forcing greater reliance on the visual anchor and reducing inter-modal conflict (Chen et al., 22 Aug 2025).

After gating, FlexMUSE performs attention-based cross-modality fusion. The projected representations are

$n$9

with $T$0. Cross-modal attention is computed as

$T$1

which is described as emphasizing textual tokens most relevant to the visual query. An adaptive fusion weight is then defined by

$T$2

and the final fused feature is

$T$3

This fusion is presented as preserving modality-specific cues while anchoring both modalities in a shared semantic space (Chen et al., 22 Aug 2025).

Taken together, msaGate and cross-modality fusion form the core semantic-control mechanism of the system. The paper attributes semantic consistency primarily to this combination, stating that msaGate plus fusion reduce text-image conflict. This suggests that FlexMUSE does not attempt to maximize raw cross-modal overlap; rather, it selectively attenuates textual dominance and then reconstructs a fused signal under visual guidance.

4. Modality Semantic Creative Direct Preference Optimization

FlexMUSE extends direct preference optimization through modality semantic creative direct preference optimization (mscDPO). Standard DPO pairs each prompt $T$4 with one chosen output $T$5 and one rejected output $T$6. By contrast, mscDPO leverages all non-chosen chunks from the same reference as additional negatives, with the aim of enriching the contrastive signal and boosting creative divergence across paragraphs (Chen et al., 22 Aug 2025).

The text is chunked into $T$7 segments $T$8 aligned to an image $T$9. The chosen segment $\rightarrow$0 is the segment corresponding to $\rightarrow$1, and all other segments in the same reference serve as negatives $\rightarrow$2. The DPO log-odds ratio is defined as

$\rightarrow$3

and the mscDPO objective is

$\rightarrow$4

where $\rightarrow$5 is a temperature constant (Chen et al., 22 Aug 2025).

The reported differences from standard DPO are twofold. First, multiple negatives per positive reinforce the model’s ability to distinguish the correct chunk from semantically related but non-matching content. Second, the method is said to encourage divergent, creative coverage of the topic while still aligning each paragraph to its illustration. The framework’s discussion identifies creativity as a direct strength of mscDPO because it leverages richer negative sampling to promote novel descriptions (Chen et al., 22 Aug 2025).

A plausible implication is that the optimization objective is tailored to paragraph-level illustrated composition rather than to a single global response preference. This is consistent with the paper’s emphasis on article-level coherence together with local paragraph-to-image correspondence.

5. ArtMUSE dataset

To support MMCW, the work introduces ArtMUSE, a dataset containing around 3,000 calibrated text-image pairs. The reported source material consists of approximately 3,000 Chinese social-media posts from architecture, design, and advertising. Each item includes text and high-resolution images at $\rightarrow$6 pixels (Chen et al., 22 Aug 2025).

The curation pipeline combines automated filtering and human calibration. Automated filtering removes advertisements and detects and discards images smaller than $\rightarrow$7 pixels. Human calibration is performed by 30 domain experts to ensure tight text-image alignment and artistic quality. The resulting corpus is characterized as a domain-specific but challenging testbed for multimodal creative writing (Chen et al., 22 Aug 2025).

The paper also notes a limitation directly tied to the dataset: the ArtMUSE scale, at roughly 3K pairs, limits general-domain robustness, and larger corpora are described as desirable. Comparisons are also said to be primarily on Chinese content, with cross-lingual extension still pending. These constraints are central to interpreting the reported performance: ArtMUSE is valuable as a calibrated benchmark, but it does not yet establish broad multilingual or general-domain coverage (Chen et al., 22 Aug 2025).

6. Training configuration, evaluation, and reported performance

Training proceeds in two stages. Supervised fine-tuning is conducted for 500K steps with a learning rate of $\rightarrow$8, swept in $\rightarrow$9. The subsequent mscDPO tuning stage runs for 200K steps with a learning rate of $\rightarrow$0, swept in $\rightarrow$1. The full training setup runs on a single NVIDIA RTX 4090, and peak VRAM is reported as lower than DOC and ChatVLM in both training and inference. Stability is also reported in the form of stable BertScore and ROUGE-L across learning rates in $\rightarrow$2 (Chen et al., 22 Aug 2025).

On automatic metrics over ArtMUSE, FlexMUSE reports ROUGE-1 $\rightarrow$3, ROUGE-2 $\rightarrow$4, ROUGE-L $\rightarrow$5, and BertScore $\rightarrow$6. The best baseline, GLM-4V, is reported with ROUGE-1 $\rightarrow$7 and BertScore $\rightarrow$8. In LLM-based evaluation on a 0–5 scale across five criteria, FlexMUSE achieves top scores in Creativity, with $\rightarrow$9 and $\rightarrow$0, and in Coherence, with $\rightarrow$1 and $\rightarrow$2. Reference-free Context Consistency is reported as $\rightarrow$3, and Reference-aware Context Consistency as $\rightarrow$4 (Chen et al., 22 Aug 2025).

Human expert evaluation is reported for $\rightarrow$5 experts over 100 articles against GPT-4o. In that comparison, FlexMUSE is preferred on Style Consistency, Context Consistency, Creativity, Richness, and Coherence, with win rates greater than 60%. Ablation results are also reported. With supervised fine-tuning only, the system reaches $\rightarrow$6 and $\rightarrow$7. Adding mscDPO increases these to $\rightarrow$8, $\rightarrow$9, $\rightarrow$0, and $\rightarrow$1. Adding msaGate yields $\rightarrow$2, with a slight drop in $\rightarrow$3 but improved overall robustness. The full FlexMUSE system reports $\rightarrow$4, $\rightarrow$5, $\rightarrow$6, $\rightarrow$7, and $\rightarrow$8 (Chen et al., 22 Aug 2025).

Evaluation setting	Reported result
Automatic metrics on ArtMUSE	ROUGE-1 $\rightarrow$9, ROUGE-2 $\rightarrow$0, ROUGE-L $\rightarrow$1, BertScore $\rightarrow$2
Best baseline named in automatic metrics	GLM-4V with ROUGE-1 $\rightarrow$3, BertScore $\rightarrow$4
LLM-based evaluation highlights	$\rightarrow$5, $\rightarrow$6, $\rightarrow$7, $\rightarrow$8, $\rightarrow$9, $T$0
Human expert comparison	Preferred over GPT-4o on Style Consistency, Context Consistency, Creativity, Richness, and Coherence with win rates $T$1

The combined evidence is used in the paper to support three claims: consistency, creativity, and coherence. Within the ablations, mscDPO appears most directly linked to gains in creativity and coherence, while msaGate is associated with style consistency and robustness. This suggests a modular decomposition in which preference optimization and semantic gating contribute differently to overall MMCW quality.

7. Significance, limitations, and prospective directions

The framework’s stated strengths are semantic consistency, creativity, coherence, flexibility, and efficiency. Semantic consistency is attributed to the combination of msaGate and fusion, creativity to richer negative sampling in mscDPO, coherence to fused features plus language-model fine-tuning, flexibility to the optional text-to-image module, and efficiency to single-GPU compatibility and modest VRAM usage (Chen et al., 22 Aug 2025).

The reported limitations are equally explicit. ArtMUSE remains small, with about 3K pairs, which limits general-domain robustness. The reported comparisons focus primarily on Chinese content, so cross-lingual extension remains pending. The paper also identifies several directions for future work: integration with fully open-source multimodal LLMs, exploration of more advanced fusion such as multi-head cross-attention or dynamic gating, and additional human-in-the-loop controls including user-driven style sliders and motif templates (Chen et al., 22 Aug 2025).

One possible misconception is to treat FlexMUSE as merely a standard multimodal generation pipeline with an optional image generator. The paper’s formulation is narrower and more specific: it is designed for MMCW, where textual and visual contexts are not strictly related and where paragraph-level illustrated composition requires balancing semantic alignment with creative divergence. Another possible misconception is to interpret the gating mechanism as simple feature dropout. The description instead frames msaGate as a semantic-similarity-conditioned probabilistic suppression mechanism, intended to reduce inter-modal conflict rather than to regularize representations in a modality-agnostic way (Chen et al., 22 Aug 2025).

In that sense, FlexMUSE is best understood as a framework for multimodal unification and semantics enhancement under flexible interaction, specialized to illustrated creative writing. Its contribution lies not in a single component, but in the coordination of optional visual generation, probabilistic semantic gating, lightweight cross-modal attention fusion, and chunk-level preference optimization within the MMCW setting (Chen et al., 22 Aug 2025).