VisuCraft: Modular Visual Creativity

• VisuCraft is a modular framework for creative LVLM generation that extracts structured visual details and dynamically synthesizes prompts for nuanced long-form outputs.
• It improves key deficits in traditional LVLMs—visual grounding, sustained creativity, and precise instruction adherence—across storytelling, poetry, and advertising.
• The framework’s architecture integrates a visual extractor, a dynamic prompt generator, and existing LVLMs to enhance creative performance without retraining base models.

VisuCraft is a modular framework for complex visual-guided creative content generation with large vision-LLMs (LVLMs). It was introduced to improve long-form outputs such as stories, poetry, and advertising copy by inserting two intermediate components between the input image and the downstream LVLM: a multimodal structured information extractor, which converts the image into an explicit structured representation, and a dynamic prompt generation module, which combines that representation with a user instruction to form an optimized prompt for the LVLM. The resulting design targets three deficits identified in prior LVLMs for creative generation—weak visual grounding, limited sustained creativity, and poor adherence to nuanced instructions—without retraining the underlying LVLM end to end (Jiang et al., 4 Aug 2025).

1. Problem setting and design rationale

VisuCraft is motivated by the increasing use of LVLMs for creative tasks that require deep visual grounding, genuine creativity, and precise instruction adherence over long-form text. The motivating applications include long-form story generation, poetry composition, and ad copy. In this setting, the central difficulty is not merely recognizing image content, but using visually grounded evidence to sustain style, tone, atmosphere, and narrative consistency under explicit user constraints (Jiang et al., 4 Aug 2025).

The framework is presented against a diagnosis of current LVLM limitations. These models often produce generic or repetitive outputs, exhibit weak correlation between generated text and fine-grained visual details, and struggle with nuanced instructions such as tone, style, or content constraints. The paper also identifies hallucination, superficial associations when visual signals are underspecified, and difficulty handling long-context reasoning with complex visual dependencies. Its stated root cause is that existing LVLMs do not extract sufficiently structured, semantically rich visual information; instead, they often rely on generic captions or flat embeddings that are inadequate for nuanced creative generation (Jiang et al., 4 Aug 2025).

This framing positions VisuCraft as an enhancement layer rather than a replacement architecture. A common misconception is to treat it as a new base LVLM. In fact, the framework is explicitly model-agnostic and is designed to wrap off-the-shelf systems such as LLaVA and InstructBLIP. Its novelty lies in restructuring the information path into the model rather than modifying the underlying generative engine.

2. Modular architecture and formal data flow

The VisuCraft pipeline is organized around three modules: the extractor $\mathcal{E}$, the prompt generator $\mathcal{G}$, and the underlying LVLM $\mathcal{M}$. Given an input image $I$ and user instruction $U$, the framework first derives a structured representation $V$, then constructs an optimized prompt $P$, and finally generates long-form creative text $T$ (Jiang et al., 4 Aug 2025).

The system equations are given as:

$V = \mathcal{E}(I)$

$P = \mathcal{G}(V, U)$

$\mathcal{G}$0

$\mathcal{G}$1

The extractor $\mathcal{G}$2 distills fine-grained, interpretable visual details into a structured textual or JSON representation. The prompt module $\mathcal{G}$3 then merges this representation with the user instruction, prioritizing the most relevant visual details for the intended creative task. The LVLM $\mathcal{G}$4 receives the resulting prompt rather than raw visual features. The paper argues that this reduces input ambiguity and limits hallucination by anchoring generation to structured visual facts and explicit instruction guidance (Jiang et al., 4 Aug 2025).

The architecture is deliberately modular. $\mathcal{G}$5 is not retrained end to end, which makes the framework operationally distinct from approaches that improve performance primarily through full-model fine-tuning. This suggests a deployment model in which existing LVLM infrastructures can be upgraded through intermediate structured reasoning rather than through costly retraining.

3. Structured visual extraction

The extractor $\mathcal{G}$6 takes an image $\mathcal{G}$7 and produces a structured representation $\mathcal{G}$8 that captures objects, attributes, relations, lighting, textures, color palettes, spatial layout, atmosphere or emotions, and implied narrative cues. The emphasis is on explicitness and interpretability. Instead of producing a generic caption or latent embedding, the extractor aims to surface concrete visual facts such as “a person in a contemplative pose, wearing rough-textured clothing, illuminated by soft evening light, conveying a sense of solitude” (Jiang et al., 4 Aug 2025).

The schema described for $\mathcal{G}$9 includes several classes of information. At the object level, it records category, pose, clothing or material properties, age descriptors, condition, textures, and colors. At the relational level, it includes spatial relations such as left, right, or behind, as well as interactions such as a character gazing at the sea. It also records lighting and atmosphere, including light type, dominant light sources, time of day, weather, emotional atmosphere, and implied narrative cues. Additional fields cover style hints, texture descriptors, color palettes, and background-foreground depth organization (Jiang et al., 4 Aug 2025).

The paper does not provide a formal ontology for these categories. Instead, it states that the extractor is trained or fine-tuned using ImageNet, COCO, and OpenImages augmented with scene graphs, attribute labels, and sentiment tags, and optimized via a combination of contrastive learning and multi-task learning to capture diverse semantics and interrelations. This is important because the system’s downstream behavior depends on the granularity and organization of the extracted structure rather than on a simple captioning objective (Jiang et al., 4 Aug 2025).

Interpretability is a central design criterion. The structured text or JSON is intended to be directly actionable for prompt construction, so that the downstream module can explicitly enumerate “visual facts” and “style cues.” A plausible implication is that VisuCraft treats structured representation not only as an internal latent scaffold but also as a human-inspectable interface between perception and generation.

4. Task-aware prompt synthesis

The dynamic prompt generation module $\mathcal{M}$0 receives the structured visual representation $\mathcal{M}$1 and the user instruction $\mathcal{M}$2, and returns an optimized prompt $\mathcal{M}$3 for the LVLM. Its operations are described in terms of integration, prioritization, and contextualization. Integration converts elements from $\mathcal{M}$4 into fluent text fragments that mesh with $\mathcal{M}$5. Prioritization emphasizes visual aspects or instruction cues depending on task type; poetry favors atmosphere and tone, while story generation emphasizes characters and interactions. Contextualization adds guidance on style, length, constraints, and audience (Jiang et al., 4 Aug 2025).

The framework allows multiple implementation strategies for $\mathcal{M}$6. The paper lists a smaller fine-tuned LLM, a rule-based system with semantic parsing, and retrieval-augmented generation as implementation options. It does not provide a formal optimization objective for $\mathcal{M}$7, nor does it specify an explicit scoring mechanism or reinforcement-learning criterion. This absence matters: VisuCraft’s reported gains come from a task-aware prompt-construction layer whose internal optimization remains underspecified in the paper (Jiang et al., 4 Aug 2025).

A representative qualitative example is a melancholic poem about isolation and the search for light, grounded in a scene containing a lone figure on a desolate rocky cliff, a stormy grey sea, a dramatic twilight sky, and a faint lighthouse beam. In such a case, $\mathcal{M}$8 is described as prioritizing atmosphere, metaphoric affordances, and explicit constraints such as somber tone, line count, and the requirement to weave the lighthouse into the poem as a fragile-hope motif. For stories, by contrast, the module emphasizes characters, actions, relationships, and coherent narrative progression; for ad copy, it emphasizes product-like elements, salient features, audience cues, and concision (Jiang et al., 4 Aug 2025).

The conceptual role of $\mathcal{M}$9 is therefore narrower and more technical than generic “prompt engineering.” It is not merely concatenating an instruction with image content. It is selecting, ranking, and reformulating structured visual evidence so that the downstream LVLM receives an instruction-conditioned, task-specific abstraction of the scene.

5. Benchmarking, dataset construction, and metrics

VisuCraft is evaluated on ImageStoryGen-500K, a self-constructed benchmark containing 500,000 diverse images paired with complex creative generation instructions. The paper describes the dataset as covering three task categories: StoryGen, Poetry, and AdCopyGen. Instructions include nuanced prompts such as “Write a poem about loneliness” and “Compose a short story describing internal struggles.” The text does not specify dataset splits, diversity statistics, or quality-control procedures (Jiang et al., 4 Aug 2025).

Evaluation relies on the VisuGen Metrics, which comprise Visual Grounding (VG.), Creativity (C.), Instruction Adherence (IA.), and their arithmetic mean:

$I$0

Visual Grounding measures alignment of the generated text with image details; Creativity measures uniqueness, imagination, and novelty; Instruction Adherence measures compliance with user instructions. The paper provides no explicit equations for these metrics beyond the system equations above. Secondary analyses describe possible inferred formulations using pretrained cross-modal similarity models and evaluator models, but these are not formalized as part of the paper itself and should therefore be treated as interpretive reconstructions rather than canonical definitions (Jiang et al., 4 Aug 2025).

The evaluation also includes human judgments on 1–5 Likert scales using PVR, HCS, HIA, and OQ. These measures are described as complementary to the automatic metrics, and the paper does not provide mathematical formalizations for them. Statistical significance is not reported (Jiang et al., 4 Aug 2025).

6. Empirical performance and ablation evidence

Across the reported tasks, VisuCraft consistently outperforms both a standard pre-trained LVLM baseline and an enhanced LVLM baseline that lacks VisuCraft’s structured extraction and dynamic prompting. On StoryGen, VisuCraft obtains VG. 0.825, C. 0.810, IA. 0.830, and Mean 0.822, compared with LVLM-Enhanced at 0.812, 0.795, 0.825, Mean 0.811, and LVLM-Base at 0.789, 0.752, 0.801, Mean 0.781. On Poetry, VisuCraft reaches VG. 0.810, C. 0.805, IA. 0.815, Mean 0.810, compared with LVLM-Enhanced at 0.798, 0.780, 0.805, Mean 0.794, and LVLM-Base at 0.771, 0.765, 0.780, Mean 0.772. For AdCopyGen, VisuCraft records VG. 0.799, C. 0.790, IA. 0.820, Mean 0.803 (Jiang et al., 4 Aug 2025).

Task	VisuCraft scores
StoryGen	VG. 0.825, C. 0.810, IA. 0.830, Mean 0.822
Poetry	VG. 0.810, C. 0.805, IA. 0.815, Mean 0.810
AdCopyGen	VG. 0.799, C. 0.790, IA. 0.820, Mean 0.803

The ablation results indicate that both intermediate modules matter. On StoryGen, removing the extractor and reverting to generic visual features yields VG. 0.801, C. 0.785, IA. 0.805, Mean 0.797. Removing the dynamic prompt module and using simple prompting yields VG. 0.815, C. 0.798, IA. 0.820, Mean 0.811. The full model restores VG. 0.825, C. 0.810, IA. 0.830, Mean 0.822. A further granularity analysis shows that performance rises as the visual representation becomes richer: Level 1, defined as basic object detection, reaches VG. 0.775, C. 0.748, IA. 0.785, Mean 0.769; Level 2, objects plus attributes, reaches 0.803, 0.787, 0.810, Mean 0.800; Level 3, full structured information, reaches 0.825, 0.810, 0.830, Mean 0.822 (Jiang et al., 4 Aug 2025).

Human evaluation follows the same pattern. For StoryGen, LVLM-Base scores PVR 3.52, HCS 3.38, HIA 3.65, OQ 3.50; LVLM-Enhanced scores 3.80, 3.75, 3.90, 3.82; VisuCraft scores 4.25, 4.18, 4.30, 4.24. For Poetry, LVLM-Base scores 3.45, 3.40, 3.55, 3.47; LVLM-Enhanced scores 3.70, 3.68, 3.80, 3.73; VisuCraft scores 4.10, 4.05, 4.15, 4.10. The qualitative example in the paper attributes these gains to deeper visual integration, stronger melancholic tone control, more distinctive metaphors, and more explicit use of the lighthouse as a motif of fragile hope (Jiang et al., 4 Aug 2025).

The reported computational trade-off is modest. The extractor requires multi-task training or fine-tuning and adds inference overhead to produce $I$1, while the prompt generator adds a lightweight language-model or rule-based step whose cost is described as negligible relative to LVLM inference. Since the underlying LVLM is not retrained, the framework has lower computational cost than retraining large models end to end. A plausible implication is that the architecture is particularly attractive in settings where model weights are fixed or expensive to adapt.

7. Limitations, adjacent formulations, and practical significance

VisuCraft’s limitations are largely in specification and reproducibility rather than in the headline architecture. The paper does not provide training schedules, preprocessing details, hardware configuration, decoding parameters, or code availability. It also does not formalize an optimization objective for the prompt generator, does not report statistical significance, and does not specify a formal ontology for the extracted representation. These omissions do not negate the reported results, but they constrain direct replication and make some implementation choices underdetermined (Jiang et al., 4 Aug 2025).

The framework is situated within a broader methodological trend toward explicit structured intermediates in multimodal generation. The paper contrasts itself with grounding approaches such as Kosmos-2, which improve grounding via location-aware links, whereas VisuCraft distills structured attributes, layout, and atmosphere to guide generation. Later work also uses the term “VisuCraft-style systems” more generically to describe visual crafting pipelines based on segmentation, retrieval, pose specification, and structured assembly, but that usage differs from the specific LVLM enhancement framework named VisuCraft in (Jiang et al., 4 Aug 2025). Prompt2Craft, for example, uses the phrase to denote a distinct class of visual crafting systems for functional assembly from primitives, emphasizing physics validation rather than long-form creative text generation (Isume et al., 4 Dec 2025). A related but separate trajectory appears in World Craft, where a structured intermediate variable $I$2 is used to decouple intent parsing from geometric layout generation in executable scene synthesis; this suggests a broader convergence around explicit intermediate representations as a control mechanism in generative systems (Sun et al., 14 Jan 2026).

The practical applications named for VisuCraft include storytelling assistants, poetry generators, marketing copywriters, and scene-description tools for creative industries. The paper also identifies several ethical considerations: extractor bias arising from sentiment or atmosphere tags and attribute extraction, hallucination control through grounding to structured visual facts, and instruction safety through screening or rephrasing unsafe requests. Copyright is mentioned only peripherally, but the paper notes that prompt-engineering policies could incorporate safety and attribution guidelines (Jiang et al., 4 Aug 2025).

Taken together, VisuCraft presents a specific thesis about LVLM-based creativity: high-quality visual-guided long-form generation depends less on exposing the model to more raw multimodal input than on converting that input into structured, task-relevant, interpretable constraints. Within the evidence reported, the framework’s strongest contribution is not a new base model, but a disciplined interface between visual understanding and creative generation.