On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility

Abstract: LLMs excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volatility of the output. This volatility not only leads to significant computational costs but also severely impacts the models' reliable application. To address this gap, our work unfolds in three stages: benchmarking, probing, and mitigation. We first propose the VOlatility in Long-form Text Benchmark (VOLTBench), a novel heterogeneous-task benchmark designed to systematically quantify the length volatility of long-form generation. Subsequently, by analyzing attention traces, we conduct an in-depth probe to identify several common internal patterns that cause this volatility. Finally, to mitigate long-form output volatility, we propose Stable Generation via Logits Boosting (GLoBo), a lightweight decoding-stage optimization strategy, designed to significantly enhance both the length accuracy and stability of long-form generation without additional training. Extensive experiments on VOLTBench provide the first systematic confirmation of severe long-form output instability in mainstream models and validate that our proposed method successfully improves the mean output length of the base model by 148% and reduces the length volatility by 69%, while maintaining high generation quality.

• The paper introduces VOLTBench to quantify output volatility by comparing required versus actual output lengths in long-form generation.
• The paper analyzes attention collapse and instability, linking these internal failures to premature termination and content degradation.
• The paper proposes GLoBo, a decoding strategy that dynamically adjusts logits to enhance length control, structural adherence, and overall quality.

Stable Long-Form Generation in LLMs: Benchmarking and Mitigating Length Volatility

Motivation and Problem Formulation

While state-of-the-art LLMs exhibit remarkable competency in long-context understanding, their generative capabilities are beset by significant volatility in both output length and structural stability. Empirical evidence reveals a pronounced gap between the required and actual output lengths across all popular architectures—even those tuned for ultra-long generation—accompanied by unpredictable fluctuations and frequent task failures such as premature stopping or structural shortcutting.

Figure 1: Model performance on VOLTBENCH showing systematic undershooting and high volatility in output length as requirements scale.

This volatility imposes substantial computational inefficiency and erodes reliability for applications demanding long-form, structured content. Existing benchmarks focus on single-generation quality or unstructured tasks, lacking objective, scalable metrics for volatility and structure adherence.

VOLTBench: Multi-Dimensional Evaluation Framework

The paper introduces VOLTBench, the first benchmark specifically designed to quantify output volatility in long-form generation. Its architecture leverages a hierarchical chapter-based format to scale instructions from 5 to 500 sections, spanning both structured (code, formula, profiles) and unstructured (story, diary, architecture) tasks in English and Chinese, and varying complexity levels. Performance is systematically assessed by repeated sampling, evaluating metrics such as Length Standard Deviation (LSD), Length Variation Coefficient (LVC), Mean Length Accuracy (MLA), Format Standard Deviation (FSD), and structured/unstructured content accuracy (SCA/UCA).

Figure 2: The VOLTBench framework covers diverse task types and dimensions, evaluating both generation quality and volatility.

Empirical analysis across VOLTBench demonstrates consistent failures: models seldom exceed 50 sections or 10,000 words, with mean lengths routinely falling short and instability exacerbated as requirements rise. Structured tasks provide stronger guidance, mitigating volatility in comparison to free-form outputs.

Attention Failures and Internal Mechanisms

To move beyond external phenomena, the paper probes models' internal attention dynamics. Attention traces are extracted and averaged across layers, focusing on constraint-encoding tokens in the prompt. Analysis reveals two principal failure signatures:

• Attention Collapse: Periodic attention spikes align with successful section generation. Collapse to near-zero attention signals loss of focus, yielding premature termination and content degeneration.
• Attention Instability: Aberrant large spikes presage section skipping or shortcutting. The model identifies start/end structural anchors but omits intermediate content, prioritizing pattern completion over genuine creation.

  Figure 3: Attention traces for Qwen2.5-7B and Qwen2.5-3B during long-form diary generation; collapse and instability correspond directly to output failures.

These patterns are diagnostic rather than fully causal; architectural limits, dataset length bias, and memory pressure are compounding factors. The primary insight is that output volatility tightly correlates with measurable failures in constraint-focused attention.

Mitigation via Stable Generation and Logits Boosting (GLoBo)

The paper proposes GLoBo—Stable Generation via Logits Boosting—a lightweight, training-free decoding strategy to enforce structural and constraint adherence in long-form generation. GLoBo operates by dynamically adjusting output logits:

• Hybrid Structural Enforcement: Soft boosting of section title tokens upon natural interruption (punctuation/newline) once section length target is reached; hard boosting after a grace period to circumvent infinite loops.
• Proactive Failure Prevention: Suppression of known failure tokens (conversational fillers, premature EOS) before the final section, ensuring continued generation throughout structural milestones.

  Figure 4: Logits boosting prevents attention decay and maintains sharp section initiation peaks, producing stable generation across 6,000 tokens.

This single-pass solution generalizes across both sectioned and free-form tasks, requiring minimal runtime overhead and architecture-agnostic applicability.

Empirical Results and Analysis

GLoBo establishes substantial improvements in length control, volatility reduction, and content quality across all VOLTBench dimensions:

• Mean output length increased by 148% versus the base model, with LVC reduced by 69% and MLA exceeding 78% (a >2x improvement over LongWriter-8B).
• Section count volatility is dramatically suppressed, with stable scaling as requirements grow.

  Figure 5: GLoBo achieves precise length matching and coherent content, in contrast to baselines that exhibit repetitive inflation or early collapse.

Lexical diversity improves sharply (TTR up to 0.4570, n-gram repetition rates <4%), avoiding neural degeneration and repetitive loops characteristic of standard strategies.

• Structured Content Accuracy (SCA): GLoBo attains perfect scores even at large scales, outperforming competitive frameworks in both code and LaTeX formula tasks.
• Unstructured Content Accuracy (UCA): Human evaluations confirm superior fluency, coherence, and instruction-adherence.

Representational stability analysis (CKA) shows that GLoBo mitigates drift in hidden states, maintaining semantic cohesion throughout long-generation horizons.

Figure 6: CKA analysis demonstrates reduced representational drift in GLoBo, preventing semantic collapse over extended generation.

External validation on LongBench-Write and WritingBench confirms strong performance on length and quality metrics, outcompeting proprietary baselines and specialized models in the longest-range categories.

Constraint-Following and Generalization

Fine-grained constraint benchmarks further illustrate systemic collapse: all mainstream models fail to track and execute localized instructions (character pattern, keyword presence, theme adherence) beyond 100 sections, with performance deteriorating at large scales. GLoBo consistently employs dynamic resynchronization to preserve section boundaries and constraint fulfillment, with negligible runtime cost compared to stepwise decoding alternatives.

Figure 7: Constraint satisfaction rates for Character-level Pattern, with ideal performance only achieved by GLoBo as section counts grow.

Implications and Future Directions

The findings expose critical limitations in current LLMs for long-form applications, with attention collapse and volatility constituting major deployment barriers. GLoBo offers a practical, architecture-agnostic mitigation, enabling reliable content generation for ultra-long tasks in creative writing, business documentation, codebase generation, and agentic workflow orchestration.

Theoretically, this work motivates further research in stable attention anchoring, representational drift management, and learning dynamics under extreme-length supervision. Practically, adoption of logit-based stabilization opens avenues for efficient, scalable decoding interventions in inference pipelines, shifting the emphasis from model re-training to robust posthoc control.

Conclusion

This paper provides a comprehensive framework for benchmarking, probing, and mitigating length volatility in LLM long-form generation. Through VOLTBench and rigorous attention analysis, the authors systematically diagnose internal instability patterns and introduce GLoBo, a decoding-stage optimization that substantially improves length accuracy, output stability, and content quality. The proposed solution marks significant progress toward controlled, reliable long-form generation, with implications for both model architecture design and practical deployment strategies in high-value domains.