Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation

Abstract: Generating diverse, pedagogically valid stories for Arabic early-grade reading assessments requires balancing tight constraints on vocabulary, reading level, and narrative structure against the need to avoid repetitive plots that undermine assessment validity. We investigate noise steering, injecting calibrated Gaussian perturbations into the internal representations of transformer models at inference time, as a training-free diversity method evaluated across five small Arabic-centric LLMs (7-9B parameters). We compare four injection strategies against high-temperature sampling baselines, measuring diversity, quality, constraint adherence, and reading grade level. Residual stream noise consistently improves narrative diversity with minimal quality or constraint cost and preserves early-grade reading level across all models. Attention entropy noise injection (AENI) stabilizes the otherwise unreliable attention-logit noise while recovering quality. High-temperature sampling inflates reading grade level and causes catastrophic collapse on several models. We find internal representation-level perturbation to be a more suitable diversity strategy than output-level stochasticity for constrained educational content generation.

• The paper introduces noise steering by injecting calibrated Gaussian noise into internal transformer layers to enhance narrative diversity while preserving EGRA constraints.
• It evaluates four noise injection methods, finding that Residual Stream Noise (L-Res) and AENI significantly improve Vendi Scores with minimal modal collapse and reading level fluctuations.
• Experimental results demonstrate that the proposed training-free approach balances diversity and quality, offering scalable enhancements for Arabic educational text generation.

Noise Steering for Controlled Arabic Educational Text Generation

Introduction and Motivation

The paper "Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation" (2604.03380) addresses a critical trade-off in generative approaches for Arabic early-grade reading assessment (EGRA) story generation: achieving diversity without sacrificing strict pedagogical and linguistic constraints. Manual curation of EGRA passages is resource-intensive, especially in morphologically rich and diglossic languages such as Arabic, motivating the development of automatic solutions using small, Arabic-centric LLMs (SLMs) in the 7–9B parameter regime.

While stochastic output-level decoding strategies (e.g., temperature, top-$k$, top-$p$ sampling) can superficially increase diversity, they frequently destabilize constraint adherence and inflate story reading levels beyond target grades. The core research question is whether injecting calibrated Gaussian noise into targeted internal representations of transformer models at inference can improve narrative diversity and maintain EGRA constraint fidelity, bypassing the limitations of output-level approaches.

Noise Steering Methodology

The authors systematically analyze four methods of injecting Gaussian noise during decoding, each targeting a distinct component:

1. Embedding Noise: Applied directly to the embedding vectors of sampled tokens immediately after lookup. Detected as highly destabilizing, particularly in models with fragile Arabic embedding spaces.
2. Attention-Logit Noise: Applied after the self-attention output projection and before the MLP, introducing variance in contextualization, but shown to result in high modal collapse rates if not adaptively modulated.
3. Residual Stream Noise (L-Res): Injected post block (after both attention and MLP sublayers), modulating the entire residual information channel of each transformer block. This method produces robust improvements in narrative diversity across models with minimal quality degradation.
4. Attention Entropy Noise Injection (AENI): An entropy-aware, adaptive perturbation of attention outputs, where noise strength is proportional to the peakedness of attention head distributions. Designed to intervene during high attention concentration events indicative of impending degeneracy or repetition.

To prevent pervasive constraint violation, noise magnitude is decayed via a cosine schedule as decoding proceeds, focusing early-story creativity without undermining ending structure and constraint satisfaction.

A model-specific root mean square (RMS) calibration for activation scaling ensures that perturbation is appropriately normalized, given inherent differences in representation scale across architectures.

Experimental Protocol

The empirical evaluation covers five prominent Arabic-centric SLMs (ALLaM 7B, AceGPT 8B, Fanar 9B, Jais 8B, Phi-4-mini), all prompted to generate EGRA-aligned stories under a battery of baseline and noise-injection configurations. Each configuration generates 50 independent stories per model, seeding sampling for variance control.

Evaluation encompasses three axes:

• Diversity: Measured via the Vendi Score (semantic diversity based on BAAI/bge-m3 embeddings) and Self-BLEU (lexical diversity).
• Quality: Mean across four LLM-judge (GPT5.3 Chat) dimensions, including readability, logicality, grammatical correctness, and linguistic quality.
• Constraint Adherence: Programmatic and LLM-based checks for story length, presence of required characters and structure, gender balance, tense, and stereotype avoidance.

Grade level fidelity is separately assessed via Osman Readability and LLM-judge reading level estimation.

Results

Diversity–Quality–Constraint Trade-off Analysis

The interplay among diversity, quality, and constraint adherence is summarised in Figure 1, plotting Vendi Score versus LLM-judged quality. Marker area encodes the number of EGRA constraint violations:

Figure 1: Story quality versus output diversity (Vendi Score) for all evaluated conditions, marker shape denoting model, color denoting decoding method, and size indicating constraint violation rate.

L-Res and AENI notably occupy regions of the trade-off space that improve or maintain baseline diversity without significant quality or constraint adherence costs. In contrast, purely output-level strategies (high-temperature sampling) can only access high-diversity regions via increased mode collapses, diminished quality, and pronounced escalation in reading level, particularly outside the Jais model.

Key quantitative findings:

• Residual Stream Noise (L-Res) achieves consistent diversity gains (e.g., Vendi Score increases of +1.28-2.38 across strong models) at a 0% modal collapse rate. It preserves constraint adherence (max Δ violations < 0.78/story) and maintains reading grade level except for the notably weaker Phi-4-mini model.
• AENI mitigates collapse associated with raw attention noise (reducing collapse rates to 0–2%) and preserves both quality and reading level. Although it induces a slight Vendi Score reduction compared to raw attention noise, this is counterbalanced by much-improved reliability.
• High-Temperature Sampling: Delivers improved diversity and quality only for Jais 8B, which is over-conservative at baseline. Other models exhibit catastrophic collapse rates (up to 84%) and substantial grade inflation, with outputs shifting into unsuitable complexity for early-grade assessment.

Readability and Grade-Level Fidelity

Osman readability and Vendi Scores (Figure 2) corroborate the overall pattern:

Figure 2: Osman readability and Vendi Score for Baseline, AENI, and L-Res across all evaluated models; plus marker denotes improvement over baseline.

L-Res and AENI sustain or exceed baseline readability (e.g., ALLaM: baseline 70.42, L-Res 76.81), confirming that diversity gains are not achieved by elevating syntactic or lexical complexity. High-temperature settings, by contrast, consistently raise grade level out of the EGRA target zone.

Statistical Significance

ANOVA and post-hoc significance tests show that only L-Res and AENI yield quality and constraint adherence metrics statistically indistinguishable from the baseline across all tested models. Output-level and non-adaptive noise approaches are significantly degraded in both respects.

Implications and Future Directions

The results validate internal, representation-level noise steering as a practical, training-free method for enhancing generative diversity in tightly constrained, educational text generation applications. The efficacy of residual stream noise and entropy-based attention noise—especially the ability of AENI to adaptively inject perturbations at critical degeneracy points—suggests broad applicability for safeguarding controllability and pedagogical validity.

Practically, these findings enable resource-constrained educational contexts to leverage SLMs for reliable, diverse story generation without recourse to large frontier models or computationally expensive fine-tuning, while ensuring outputs remain suitable for young learners.

Theoretically, the granularity of injection site and adaptive perturbation magnitude surfaces new axes for understanding internal representations and their influence on narrative exploration versus adherence. It supports the notion that unlocking latent model diversity is contingent on perturbing well-formed intermediate features without destabilizing foundational task alignment.

Future work should address:

• Layer- and head-specific noise targeting to further increase diversity while preserving fidelity.
• Extension to other low-resource and diglossic languages.
• Human-in-the-loop evaluation by domain expert educators to complement automated LLM judgements and calibrate metrics to ground-truth pedagogical standards.

Conclusion

Noise steering via residual stream and entropy-scaled attention perturbations provides a robust, fine-tuning-free mechanism to enhance narrative diversity in constrained educational text generation for Arabic SLMs. L-Res and AENI offer complementary strategies, with L-Res delivering strong, model-agnostic performance and AENI supplying adaptive stabilization. These approaches supersede standard output-level diversification techniques, maintaining early-grade reading level fidelity and constraint adherence even under high-diversity demands. This methodological advance enhances the practical deployment of SLMs for scalable, valid, and diverse educational assessment content creation (2604.03380).