POLARIS: Guiding Small Models to Write Long Stories

Abstract: Small open-weight models struggle at long-form creative writing: their generated stories either fall far short of the requested length, or their quality significantly degrades as length increases, especially when compared to frontier models. We present POLARIS (Policy Optimization with LLM-as-a-judge rewards and Anchored-Reference Injection for Storywriting), a lower-compute GRPO recipe with two key ingredients: a frontier LLM judge with a structured Story Quality rubric as the online reward, and human-reference injection (HRI), where a teacher-forced human-written story serves as a high-reward anchor within each GRPO group. By applying our training recipe to Qwen3.5-9B, using a dataset of approximately 1.4K prompt-story pairs derived from 100 short-story anthologies and 4 A100 GPUs, we obtain POLARIS-9B. Across five benchmarks spanning in-distribution and out-of-distribution prompts and rubrics, POLARIS-9B is competitive with much larger open-weight models while following length instructions more closely. A blinded human evaluation confirms that POLARIS-9B is preferred to the base Qwen3.5-9B and on par with Qwen3.5-27B. Despite training only on stories up to 4k words, POLARIS-9B preserves quality on prompts requesting stories up to 3 times the training length, a regime where most open-weight models degrade substantially in quality, length adherence, or both. More broadly, our results suggest that length generalization is a meaningful stress test for creative-writing models and a useful lens for distinguishing otherwise close models.

• The paper introduces POLARIS, a novel training recipe that leverages rubric-driven rewards and human-reference injection to enable small models to produce long coherent stories.
• It employs a structured LLM-as-judge system that assesses story quality across 16 dimensions and stabilizes training via warmup-weighted human references.
• POLARIS achieves robust length generalization up to 12k words, outperforming baselines in maintaining prompt adherence and overall narrative quality.

POLARIS: Guiding Small Models to Write Long Stories

Motivation and Problem Definition

Conventional RLHF and SFT approaches for long-form creative text suffer from severe length and quality degradation as the requested output length increases, with small open-weight models dramatically underperforming compared to frontier-scale LLMs in story-writing tasks. This work introduces the POLARIS training recipe, which specifically targets small models writing stories at lengths far beyond their SFT regime, without reliance on custom reward model finetuning, massive compute, or proprietary datasets.

The key difficulties addressed include (1) reward signal design that remains fine-grained and diagnostic in the creative-writing regime and (2) avoidance of stagnation and reward homogenization during open-ended RL, where policy rollouts rapidly become similar, causing training signal collapse. POLARIS employs a structured LLM-as-judge rubric-driven reward and introduces human-reference injection (HRI) during GRPO to supply a fixed high-reward anchor in each group, stabilizing gradient pressure for story quality.

Training Approach: Structured LLM-as-Judge Reward and HRI

POLARIS replaces pairwise or scalar-trained reward models with a live, per-iteration LLM-as-judge system utilising a detailed Story Quality rubric—anchored in both narrative theory and empirical discriminants for human vs. synthetic storytelling—enabling reward signals across 16 dimensions (6 positive, 10 negative, plus catch-all terms) and textual justifications.

Human-reference injection is applied by teacher-forcing a human-written story as an additional member for each GRPO group during optimization. This trajectory is excluded from within-group statistics and only contributes via a warmup-weighted advantage, differentiating this from prior off-policy demonstration mechanisms. The overall composite reward is a weighted, clipped sum of rubric score, repetition penalty, length penalty, and emptiness penalty, sublinearly scaled for short outputs to resist reward hacking.

The complete method is summarized schematically as:

Figure 1: Validation curve of Story Quality and composite reward for Qwen3.5-9B+GRPO and POLARIS, showing consistent gain of HRI-enabled training.

Experimental Design

POLARIS is trained on Qwen3.5-9B over 1,388 story-prompt pairs derived from short-story anthologies, with batch size 8 on 4 × A100 GPUs, and no story exceeding 4k words in the training references. Benchmarks use an extended set (EQ-Bench LongForm, EQ-Bench Creative, WritingBench, LongBench-Write) and assess both in-distribution (1–4k) and OOD prompt lengths (up to 12k), with evaluations by multiple LLM-judges (GPT-5.4, Gemini 3.1 Pro, Gemini 3 Flash) and human annotators.

Key ablations include SFT and plain-GRPO-trained variants, measuring the incremental effect of HRI.

Results: Length Generalization and Comparative Analysis

POLARIS demonstrates robust length generalization. Despite never being exposed to stories longer than 4k at training time, it retains quality and length adherence up to 12k requested words, whereas baseline models display one or more failure pathologies: (a) abrupt collapse in story quality at long lengths (Qwen3.5-9B), (b) degenerate overlength output with high self-repetition (LongWriter-Zero-32B), or (c) undergeneration producing shorter stories than requested with realistic surface quality but failing adherence (Gemma 4 31B).

In out-of-distribution benchmarks (WritingBench, EQ-Bench Creative), POLARIS achieves performance competitive with open models of 3× parameter size (e.g., Gemma 4 31B, Qwen3.5-27B) and is only strictly outscored by proprietary frontier LLMs (GPT-5.4, Claude Opus 4.6).

For prompt lengths out to 12k words, POLARIS maintains a mean length adherence ratio of 0.72, contrasting with Gemma 4 31B at 0.36 or LongWriter-Zero-32B at 2.21 (massive overrun via repetition).

Figure 2: POLARIS outperforms other open-weight models in length-adjusted Story Quality and length adherence up to OOD requested lengths.

Figure 3: Distribution of length-adjusted Story Quality scores by length grouping, highlighting robust far-transfer for POLARIS.

Dimension-Level Effects and Per-Cluster Analysis

Dimension-wise training curves indicate steady improvement in voice, character depth, world realization, and negative-side reduction for drift, bloat, and generic language. HRI is consistently associated with better generic-language and predictability control and stronger positive-arc recovery, especially at far OOD lengths.

Figure 4: Positive-dimension training trajectories for Qwen3.5-9B+GRPO, showing systematic improvement on core narrative qualities.

Hierarchical clustering of 18 evaluated models by per-dimension rubric score shows POLARIS forming a distinct “strong / distinctive” cluster, intermediate between mid-tier open-weight LLMs and proprietary frontier models, driven particularly by improvements in stylistic and narrative structure dimensions.

Figure 5: Hierarchical clustering reveals POLARIS's distinct profile and proximity to frontier models in per-dimension Story Quality.

Figure 6: Per-dimension Story Quality comparison: POLARIS outperforms other 9B/27B open models in voice, scene, and generic-language controls while retaining length adherence.

Human Evaluation

POLARIS is preferred over Qwen3.5-9B in 67.5% of pairwise human-annotated ratings, and is statistically indistinguishable from Qwen3.5-27B (51.2% win rate). Annotators note strong gains for POLARIS in atmosphere, voice, and scene realization, but residual issues in local coherence and stylistic overloading persist.

Theoretical and Practical Implications

The primary theoretical contribution is the demonstration that structured, open-ended LLM-as-judge rewards anchored in a diagnostic rubric, coupled with minimal HRI, suffice for robust OOD length generalization in small models trained on realistic compute budgets. The observed efficacy of per-example HRI suggests that even shallow off-policy demonstration can offset reward stagnation in RL for open-ended generation. The results also establish length generalization as a decisive stress test for the creative-writing regime, revealing sharp discontinuities in model behavior not exposed by aggregate benchmarks.

Practically, POLARIS offers a competitive low-compute recipe for open-weight, long-form creative-writing LMs, directly targeting deployment settings where compute or data access precludes the use of proprietary or multi-run reward model pipelines. Moreover, the methodology reduces the requirement for heavy preference model curation or the risks associated with overfitting to narrow scalar objectives.

Limitations and Future Directions

Core limitations concern judge validity (systematic LLM-judge biases are only partially mitigated by cross-family and rubric separation), limited corpus and domain scope (narrative short fiction only), and undershooting of requested lengths at the longest buckets. Further research should extend HRI-based RL to other creative-writing domains (screenwriting, essays), address judge-anchored reward shifts, and test synthetic reference trajectories. Moreover, the implications for non-narrative open-ended generation and integration with modular reward specifications (e.g., token-level or span-level feedback (Xu et al., 3 Apr 2026)) remain to be elucidated.

Conclusion

POLARIS demonstrates that small open models can be robustly aligned for long-form story generation via minimal, rubric-driven RL and a single human-reference injection per batch, preserving qualitative and quantitative quality at 3× the training length regime. This approach bridges the quality gap with large open-weight LLMs while maintaining strict prompt adherence and plausible human preference—enabling competitive, scalable creative-writing systems in resource-constrained open-source workflows (2606.04095).