CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

Abstract: While post-training improves the capabilities of LLMs, it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as well as those that require it implicitly, e.g., reinforcement learning (RL). We instead propose CreativeInstruct, a scalable instruction-tuning method that teaches LLMs to balance creative, base-model-like generations with the quality of post-trained models, by learning to inject special [StartCreativity] spans that bias generation toward creativity. Furthermore, we introduce a structural diversity metric based on graph edit distance, which captures narrative level variation missed by purely lexical and semantic metrics. On narrative generation, CreativeInstruct matches or exceeds the diversity of both multi-model baselines and distilled variants of their outputs, without sacrificing quality or requiring multiple models at inference time. These results are mirrored in our human evaluation, where we find that annotators rate CreativeInstruct generations as more creative than the post-trained LLMs' generations in 70.3% of cases. We also show the benefits of creative models as a substrate for RL: GRPO applied to a CreativeInstruct checkpoint improves by ~4% on AMC and ~5% points on MATH over the same training applied to the post-trained checkpoint.

• The paper introduces span-level creativity tags that distill base-versus-aligned model routing into one model, improving semantic and structural diversity while reducing inference compute by half.
• CreativeInstruct outperforms instruction-tuning, test-time BACo, and untagged distillation across multiple models, with up to 63% higher structural diversity and 70.3% human preference for creativity.
• The paper proposes LLM-GED for measuring narrative-structure diversity and shows that CreativeInstruct’s more varied rollouts improve GRPO results on MATH and AMC reasoning benchmarks.

Motivation and problem statement

Post-training improves instruction-following, reasoning, and safety in LLMs, but it systematically reduces output diversity and creativity. This homogeneity persists across model families and scales, and it harms both explicitly creative tasks (e.g., story generation) and implicitly diversity-dependent ones such as reinforcement learning (RL), where diverse rollouts are needed for exploration. Existing remedies that mix base-model and aligned-model outputs at inference time (e.g., BACo) recover diversity but require loading two models, doubling test-time compute, and presuppose access to a released base model — an assumption that fails for checkpoints like Qwen3 32B.

CreativeInstruct addresses this by distilling the routing behavior of a dual-model decoding framework into a single aligned model. The key mechanism is span-level creativity tagging: training data is generated by token-level routing between the base and aligned variants of each model family (using BACo's prob+punc variant, which sends punctuation/formatting tokens to the aligned model and high-entropy tokens to the base model). Contiguous spans produced by the base model are wrapped in special [StartCreativity]/[EndCreativity] tokens, and the aligned model is fine-tuned with LoRA on this tagged corpus. At inference time, the model self-injects the creativity tokens, learning where in a response to switch between base-like diversity and aligned-like quality. The authors note the method is agnostic to the particular routing scheme used for data generation.

Training data is drawn from 4,000 English writing-related prompts from Tülu V3 SFT with three outputs per prompt (12,000 samples), making the approach scalable with general-purpose instruction-tuning data rather than requiring curated creative corpora. For Qwen3 32B, which lacks a public base variant, data generated from Qwen2.5 32B is used, demonstrating cross-family transfer.

The LLM-GED metric

A second contribution is LLM Graph Edit Distance (LLM-GED), a structural diversity metric. An LLM judge converts each story into a canonicalized event graph $G = (V,E)$ — entities abstracted to Character1, Location1, etc., events labeled from a shared vocabulary, edges capturing agent/affected/causes relations and directed next_event temporal ordering — then estimates pairwise normalized graph edit distances:

$$\text{nGED}(G_A, G_B) = \frac{\text{GED}(G_A, G_B)}{\max(|G_A|, |G_B|)}$$

averaged over all pairs. Validation against a reference ranking of controlled narrative variations (identical stories < lexical shifts < temporal reordering < unrelated stories) shows LLM-GED correlates best among all tested metrics (0.889), exceeding cosine dissimilarity with all-MiniLM embeddings (0.852) and all lexical metrics (Dist-$n$, Self-BLEU, Self-ROUGE-L, etc.). A deterministic two-pass pipeline (graph extraction followed by exact branch-and-bound GED) yields statistically equivalent scores (mean difference −0.012, paired t-test $p = 0.56$), supporting the single-pass LLM formulation as a cheaper substitute.

Main results

Evaluations on the Narrative Discourse dataset across five models (LLaMA-3.1 8B, Qwen2.5 7B/32B, Qwen3 8B/32B) compare CreativeInstruct against the Instruct baseline, test-time BACo (which uses twice the compute), a Distill baseline trained on the same BACo-generated corpus without tags, and CrPO for LLaMA-3.1 8B.

Model	Method	Cos-D (M) ↑	Vendi ↑	LLM-GED ↑	WQRM ↑
LLaMA-3.1 8B	Instruct	0.309	2.913	0.366	5.93
	BACo	0.255	3.499	0.374	5.73
	Distill	0.444	3.975	0.523	5.85
	CreativeInstruct	0.458	4.749	0.545	6.65
Qwen2.5 7B	Instruct	0.290	2.681	0.442	6.17
	BACo	0.348	3.204	0.476	6.47
	CreativeInstruct	0.395	4.167	0.523	6.42
Qwen3 8B	Instruct	0.399	3.257	0.453	6.56
	BACo	0.265	2.974	0.395	6.86
	CreativeInstruct	0.369	3.926	0.498	6.90

On LLaMA-3.1 8B, CreativeInstruct yields roughly 48% relative gains in semantic diversity and 63% in structural diversity over the instruct baseline. Notably, despite being distilled from BACo's outputs, CreativeInstruct consistently outperforms BACo at test time (average gains of ~29% semantic and ~28% structural diversity), indicating that learning the routing signal enables generalization beyond imitation of the teacher's per-token decisions while halving inference cost. Against the Distill ablation, CreativeInstruct achieves higher LLM-GED in every setting, and the tags also improve quality: Distill generally underperforms on coherence, fluency, relevance, and WQRM, so the explicit markers help preserve quality during tuning rather than trading it away.

Entity-level repetition is quantified via proper noun uniqueness: CreativeInstruct reaches 37.1% at the prompt-group level versus 26.6% for the strongest baseline (Distill) and 18.1% for Instruct, more than doubling Instruct's uniqueness at both granularities ($p < 0.001$, Mann–Whitney U).

Scaling behavior is favorable: diversity metrics improve monotonically up to 12,000 training samples without plateauing, suggesting further gains from additional data. Data composition matters: a variant trained on in-domain narrative data underperforms one trained on general-purpose Tülu data even at matched sample counts, indicating that diverse training prompts support creative generalization better than domain-matched ones.

Human evaluation

Three annotators performed anonymized pairwise comparisons over 50 prompts (five generations per system). Inter-annotator agreement was high for creativity (Cohen's $\kappa = 0.720$), moderate for diversity ($\kappa = 0.417$), and negative for quality ($\kappa = -0.167$); the authors attribute the quality result to small sample size and exclude human quality judgments accordingly. CreativeInstruct outputs were preferred for creativity in 70.3% of comparisons (significant by two-sided binomial test) and for diversity in 57.4%. Qualitative examples show the instruct baseline repeatedly opening with near-identical phrasing ("I woke up to an eerie silence") across generations, whereas CreativeInstruct varies framing and narrative structure.

Creativity as a substrate for RL

The paper tests whether improved diversity benefits downstream RL by applying identical GRPO training (1,000 steps, 8 rollouts, MATH training split) to Qwen3 8B Instruct and Qwen3 8B CreativeInstruct checkpoints, averaged over three seeds:

Model	MATH	AMC
Instruct baseline	0.374	0.432
Instruct + RL	0.409	0.438
CreativeInstruct baseline	0.424	0.428
CreativeInstruct + RL	0.459	0.478

Two observations stand out. First, the CreativeInstruct checkpoint alone improves in-domain MATH accuracy (+3.5%) but slightly degrades out-of-domain AMC accuracy relative to the instruct baseline — creative instruction-tuning does not by itself improve math ability. Second, after RL the ordering reverses decisively: CreativeInstruct+RL exceeds Instruct+RL by ~4 points on AMC and ~5 points on MATH, with gains concentrated at higher difficulty levels. The interpretation offered is that diversity improves rollout exploration, providing a better initialization for policy optimization and stronger OOD generalization; this positions creativity as relevant beyond generative writing tasks.

Limitations and open questions

Several caveats bear directly on the results. The Qwen3 32B case is an exception: the Distill baseline beats CreativeInstruct on most automatic diversity metrics there, though CreativeInstruct retains higher LLM-GED — the tag mechanism's advantage may shrink at larger scale or with certain model families. Human evaluation covers only LLaMA-3.1 8B, uses three annotators, and produces unreliable quality agreement, leaving human validation of quality and of other model families unaddressed. The RL experiment is limited to one model size, one algorithm (GRPO), and mathematical reasoning; whether the effect extends to other RL settings or reward structures is not established. The method also inherits dependence on the availability of some base-aligned pair for data generation (mitigated, but not eliminated, by demonstrated cross-family transfer), and the scaling curves have not plateaued, so the ceiling of the approach at larger data volumes remains open. Finally, LLM-GED relies on an LLM judge's internal estimation of edit distances rather than exact computation; its robustness across judge models is validated only against one reference ranking and one deterministic pipeline.

Conclusion

CreativeInstruct converts test-time base/aligned model routing into a learned, self-triggered capability via span-tagged instruction tuning, recovering much of the base model's diversity in a single checkpoint without sacrificing quality. The accompanying LLM-GED metric captures narrative-structural variation that lexical and embedding-based metrics miss. Results across five models show consistent diversity gains over both multi-model decoding and distillation baselines, strong human preference for creativity (70.3%), and improved downstream GRPO performance when used as an RL starting point — evidence that diversity-oriented instruction tuning has value well outside creative writing.