SafeCtrl-RL: Inference-Time Adaptive Behaviour Control for LLM Dialogue via RL-Driven Prompt Optimisation

Abstract: Ensuring safe and contextually appropriate behaviour in LLMs remains a critical challenge for real-world deployment. We present \textbf{SafeCtrl-RL}, an inference-time behavioural control framework that enables adaptive safety regulation without model retraining or parameter modification. The method formulates dialogue generation as a sequential decision process, where a reinforcement learning agent dynamically selects prompt adjustment strategies based on contextual feedback. This allows unsafe behaviours to be suppressed through iterative refinement, which we conceptualise as inference-time behavioural unlearning. Evaluated across multiple LLMs and unsafe dialogue scenarios, SafeCtrl-RL consistently improves safety and response quality, outperforms existing prompt-based optimisation methods, and achieves favourable performance--efficiency trade-offs. **Warning: This paper may contain examples of harmful language, and reader discretion is recommended.

• The paper introduces a model-agnostic, inference-time framework that uses RL-driven sequential prompt optimisation to ensure safe LLM dialogue.
• It casts dialogue generation as a closed-loop decision process, employing a 36-dimensional state embedding and 11 discrete prompt adjustment strategies.
• Empirical results demonstrate significant safety-quality improvements and enhanced efficiency across diverse LLM architectures compared to static methods.

SafeCtrl-RL: Inference-Time Adaptive Behaviour Control for LLM Dialogue via RL-Driven Prompt Optimisation

---

Motivation and Contributions

The challenge of ensuring safe and contextually appropriate behaviour in LLMs during interactive dialogue is central for reliable deployment, particularly in dynamic contexts where unsafe or undesirable responses must be suppressed at inference time. Existing approaches—parameter-based editing and prompt-based controls—are either reliant on offline retraining or lack adaptability during ongoing interaction. SafeCtrl-RL introduces a model-agnostic, inference-time framework for adaptive behavioural control that operates through sequential prompt optimisation driven by a reinforcement learning (RL) agent. Dialogue generation is cast as a closed-loop decision process, where the RL agent selects among discrete prompt adjustment strategies to iteratively steer the model towards safety and quality.

Figure 1: SafeCtrl-RL safeguarding loop, illustrating iterative RL-driven prompt refinement and evaluation for inference-time suppression of unsafe responses.

---

Framework Architecture and Safeguarding Formulation

SafeCtrl-RL formalises the safeguarding task as trajectory-level control over prompt-conditioned generation. For a fixed LLM $M$, each response to user input is generated under the current system prompt and evaluated by a composite safety-quality scoring function. Responses below threshold are iteratively refined via prompt adjustment, forming a generate–evaluate–refine loop. The RL agent's state embedding aggregates meta-learning dynamics, score statistics, prompt features, risk and safety signals, exploration factors, and interaction hashes into a 36-dimensional vector, enabling policy conditioning on both immediate and longitudinal feedback. The action space comprises 11 distinct prompt construction strategies that span minimal/no-history, AI summarisation, performance-tiered, and contrast-based mechanisms. The reward is an exponential weighted product of safety and quality scores, sharply penalising deficits in either dimension and enforcing a hard safety constraint.

---

Experimental Methodology

Baseline and Corpus

Evaluation is performed across uncensored and safety-aligned LLMs (e.g., BlackSheep-Llama3.2-3B, Evil-Alpaca-3B, DialoGPT-Large, DeepSeek-R1-Distill), using an unsafe prompt corpus categorised into 12 harm types and comprising over one million prompts. Static and dynamic prompt strategies from literature, as well as state-of-the-art prompt optimisers like OPRO and GRIPS, serve as baselines.

Figure 2: Distribution of unsafe prompt categories within the evaluation corpus.

Metrics

Performance is measured along four axes: final safety-quality score, improvement over unsafeguarded baseline, performance-efficiency ratio (gain per refinement iteration), and post-safeguarding retention (sustained behavioural correctness after removal of safeguard).

---

Empirical Results

Comparative Effectiveness

SafeCtrl-RL achieves the highest macro safety-quality scores across all tested models, with scores ranging from $0.647$ (DialoGPT) to $0.898$ (DeepSeek-R1), and a mean of $0.818$. Notably, $\Delta P$ improvements are consistently positive and substantial for SafeCtrl-RL in all settings, outperforming static hand-crafted strategies and adaptive search/gradient-based optimisers, the latter displaying negative or negligible gains in safety-critical scenarios. The method further delivers robust generalisation with stable efficacy across diverse LLM architectures, whereas individual strategies exhibit model sensitivity and reduced transferability.

Performance-Efficiency Trade-Off

SafeCtrl-RL ranks superior in efficiency, achieving high improvement per refinement step ($R_{\text{perf}}$), with significantly fewer iterations required compared to fixed baselines (see Figure 3).

Figure 3: Performance efficiency ratio—the improvement in safety-quality per refinement iteration—demonstrates rapid convergence and low computational overhead for SafeCtrl-RL versus static strategies.

Retention and Behavioural Stability

Induced safe behaviours partially persist post-safeguarding. Retention rates exceed 90% on Evil-Alpaca and BlackSheep and remain above baseline in all models, indicating that inference-time adaptive control can effect lasting alignment in response patterns. However, degree of retention is modulated by intrinsic model steerability and prior alignment.

---

Theoretical and Practical Implications

SafeCtrl-RL demonstrates the practicality and scalability of inference-time adaptive control for LLM safety without retraining or parameter access. The RL-driven framework substantiates the paradigm of dynamic prompt refinement as sequential decision-making, outperforming static and offline optimisers on both effectiveness and efficiency. The closed-loop control approach offers a new axis of intervention for black-box deployments, enabling continuous, context-sensitive regulation of behaviour and real-time unlearning of unsafe patterns.

The partial retention of safe behaviour raises questions about the internalisation of externally induced alignment and suggests opportunities for hybrid offline–online adaptation, especially via integration with soft prompting or semi-supervised instruction tuning. The robust performance across diverse architectures further signals generalisability potential. Latency bottlenecks are dominated by evaluator overhead; lightweight, local evaluators could improve real-time applicability.

---

Future Directions

Extending SafeCtrl-RL to multi-turn, multimodal LLM dialogue and domain-specific safety regimes, as well as developing evaluator-agnostic or self-supervised feedback mechanisms, will be critical for broader adoption. Hybrid frameworks that combine prompt-level RL optimisation with soft prompt internalisation may facilitate faster convergence and persistent behaviour modulation, further reducing inference and adaptation costs. Systematic analysis of distributional retention effects and large-scale evaluation across foundation models will clarify generality and inform advances in adaptive safety-alignment.

---

Conclusion

SafeCtrl-RL enables inference-time, RL-driven adaptive behavioural control for LLMs, dynamically suppressing unsafe or undesired outputs with strong and stable improvements in both safety and quality metrics relative to established baselines. The framework achieves superior performance–efficiency trade-offs and demonstrates partial persistence of induced behaviour after safeguard removal. The results provide evidence for the utility of sequential, policy-driven prompt refinement over static or offline approaches for real-time safety alignment in interactive AI systems. Future work should focus on evaluator integration and scaling to richer dialogue and multimodal contexts for reliable, responsive safeguarding.