Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI (2511.01689v1)

Abstract: The character of the "AI assistant" persona generated by modern chatbot LLMs influences both surface-level behavior and apparent values, beliefs, and ethics. These all affect interaction quality, perceived intelligence, and alignment with both developer and user intentions. The shaping of this persona, known as character training, is a critical component of industry post-training, yet remains effectively unstudied in the academic literature. We introduce the first open implementation of character training, leveraging Constitutional AI and a new data pipeline using synthetic introspective data to shape the assistant persona in a more effective and controlled manner than alternatives such as constraining system prompts or activation steering. Specifically, we fine-tune three popular open-weights models using 11 example personas, such as humorous, deeply caring, or even malevolent. To track the effects of our approach, we introduce a method which analyzes revealed preferences, uncovering clear and holistic changes in character. We find these changes are more robust to adversarial prompting than the above two alternatives, while also leading to more coherent and realistic generations. Finally, we demonstrate this fine-tuning has little to no effect on general capabilities as measured by common benchmarks. We describe and open-source our full post-training method, the implementation of which can be found at https://github.com/maiush/OpenCharacterTraining.

• The paper introduces an open-source character training method that leverages constitutional AI and synthetic introspection to shape the persona of language models.
• The methodology employs hand-written constitutions, direct preference optimization, and introspective data to induce robust, coherent trait expression, as evidenced by significant shifts in trait preferences.
• The work demonstrates that character training preserves general capabilities while enhancing coherence, realism, and resistance to adversarial prompts.

Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI

Introduction

The paper presents the first open-source implementation of character training for LLMs, leveraging Constitutional AI and synthetic introspective data to systematically shape the persona of AI assistants. The approach is designed to move beyond superficial role-play or prompt engineering, enabling robust, coherent, and realistic expression of desired character traits. The authors fine-tune three open-weight models (Llama 3.1 8B, Qwen 2.5 7B, Gemma 3 4B) across 11 distinct personas, including pro-social (e.g., loving, flourishing) and adversarial (misaligned) cases. The work introduces a novel evaluation methodology based on revealed preferences, demonstrating that character training induces deep, holistic changes in model behavior, with minimal impact on general capabilities.

Methodology

Character Training Pipeline

The character training pipeline consists of three sequential stages:

1. Hand-Written Constitutions: Each persona is defined by a constitution—a set of first-person assertions specifying desired character traits and behavioral policies. These constitutions differ from prior work by focusing on role-play and manner rather than content.
2. Distillation via Direct Preference Optimization (DPO): A teacher model (GLM 4.5 Air) is prompted with the constitution and generates responses to a diverse prompt set. The student model (one of the open-weight LLMs) generates responses without explicit persona instructions. DPO is used to fine-tune the student, optimizing for responses that embody the desired character traits.
3. Introspection with Synthetic Data: The post-distillation model generates introspective data via self-reflection and self-interaction. This includes biographies, diary entries, and multi-turn dialogues between two instances of the model, all designed to surface nuanced trait expression. The model is further fine-tuned on this data via supervised learning.

   Figure 1: The character training pipeline: (1) hand-writing constitutions, (2) DPO-based distillation, and (3) synthetic introspective data generation for SFT.

Persona Definitions

Eleven personas are implemented, ranging from sarcastic and humorous to mathematical, poetic, flourishing, loving, and misaligned. The constitutions for each are carefully crafted and iteratively refined based on empirical results.

Evaluation: Revealed Preferences

The authors introduce a robust evaluation framework based on revealed preferences, inspired by economic and value alignment literature. Models are prompted to choose between pairs of traits (from a set of ~150) and generate responses accordingly. An LLM-as-a-Judge (GLM 4.5 Air) is used to infer which trait was expressed, and Elo scores are computed to quantify relative preferences.

Figure 2: Visualization of trait preference shifts (Elo scores) for Llama 3.1 8B before and after character training, showing global and fine-grained changes across constitutions.

Key findings include:

• Character training induces strong, holistic shifts in trait preferences, boosting desired traits and suppressing opposing ones.
• Different models converge to similar personas post-training, as measured by increased Spearman correlation of trait rankings (from 0.44 to 0.87).
• The standard deviation of trait preferences increases, indicating more "opinionated" and distinct personas.

  Figure 3: Distribution of trait Elo scores before and after character training, showing increased standard deviation and convergence across models.

Robustness to Adversarial Prompting

To assess the depth of character integration, the authors conduct adversarial prompting experiments. Models are instructed to "break character" or respond "naturally," and a classifier is trained to predict the expressed persona. Character-trained models maintain trait expression with higher accuracy than those shaped by system prompts or activation steering.

Figure 4: Classifier accuracy for persona prediction under adversarial prompting, demonstrating superior robustness of character-trained models.

Multi-turn "prefill attack" experiments further show that character-trained models are more likely to remain in-character across turns, compared to distillation-only baselines.

Coherence and Realism

Character training yields more coherent and realistic trait expression than activation steering or prompt-based methods. LLM-as-a-Judge evaluations show that character-trained responses are preferred for coherence and naturalness, avoiding exaggerated or performative behavior.

(Figure 2, Figure 3, Figure 4)

Figures 3–5: Empirical evidence for improved coherence, realism, and robustness in character-trained models.

General Capabilities

Benchmarking on TruthfulQA, WinoGrande, HellaSwag, ARC Challenge, and MMLU reveals that character training does not degrade general capabilities, except in cases where the constitution explicitly encourages misaligned or incorrect behavior. LoRA-based fine-tuning enforces minimal deviation from the base model, preserving task performance.

Implementation Details

• Training: LoRA adapters (rank 64, $\alpha=128$), batch size 32, learning rate $5 \times 10^{-5}$, DPO $\beta=0.1$, per-token KL and NLL penalties.
• Data: ~6M tokens for DPO, ~8M tokens for introspection per model/persona pair.
• Models: Llama 3.1 8B, Qwen 2.5 7B, Gemma 3 4B (instruction-tuned).
• Open Source: All code, data, and model weights are released at https://github.com/maiush/OpenCharacterTraining and HuggingFace.

Trade-offs and Limitations

• Activation Steering vs. Character Training: Activation steering can induce trait expression but is brittle, difficult to tune, and prone to incoherence. Character training is more robust, scalable, and yields naturalistic behavior.
• Scale: Experiments are limited to models <10B parameters; scaling to larger models may require additional engineering.
• Evaluation: Reliance on model-based judges and classifiers may introduce bias; human evaluation and cross-judge replication are recommended for future work.

Implications and Future Directions

The open character training framework enables controlled, reproducible research on persona shaping in LLMs, bridging the gap between academic and industrial post-training practices. The methodology facilitates red-teaming, safety analysis, and the development of assistants with richer, more human-like values and mannerisms. Future work may explore reinforcement learning variants, scaling to larger models, and deeper integration of human feedback.

Conclusion

Open Character Training provides a principled, open-source approach to shaping the persona of AI assistants via Constitutional AI and introspective data. The method yields robust, coherent, and realistic trait expression, outperforming prompt engineering and activation steering in both depth and reliability. The framework and released models establish a foundation for further research on alignment, safety, and the social impact of AI assistants.