Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using Large Language Models (2406.17287v1)

Abstract: Accurate assessment of personality traits is crucial for effective psycho-counseling, yet traditional methods like self-report questionnaires are time-consuming and biased. This study exams whether LLMs can predict the Big Five personality traits directly from counseling dialogues and introduces an innovative framework to perform the task. Our framework applies role-play and questionnaire-based prompting to condition LLMs on counseling sessions, simulating client responses to the Big Five Inventory. We evaluated our framework on 853 real-world counseling sessions, finding a significant correlation between LLM-predicted and actual Big Five traits, proving the validity of framework. Moreover, ablation studies highlight the importance of role-play simulations and task simplification via questionnaires in enhancing prediction accuracy. Meanwhile, our fine-tuned Llama3-8B model, utilizing Direct Preference Optimization with Supervised Fine-Tuning, achieves a 130.95\% improvement, surpassing the state-of-the-art Qwen1.5-110B by 36.94\% in personality prediction validity. In conclusion, LLMs can predict personality based on counseling dialogues. Our code and model are publicly available at \url{https://github.com/kuri-leo/BigFive-LLM-Predictor}, providing a valuable tool for future research in computational psychometrics.

The paper "Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using LLMs" investigates the use of LLMs to predict personality traits from counseling dialogues, aiming to provide a more efficient and less biased alternative to traditional self-report questionnaires. This paper addresses critical aspects of psycho-counseling by proposing a novel framework that leverages role-play and questionnaire-based prompting techniques.

Key Contributions and Methodology

1. Framework Introduction:
   • The framework specifically designed for this paper applies role-play simulations and questionnaire-based prompting to condition the LLMs on counseling sessions.
   • Role-play simulations help in generating more realistic and contextually rich dialogues, while the questionnaire prompts align the model outputs with responses typical of personality trait assessments.
2. Evaluation:
   • The framework was evaluated on a dataset comprising 853 real-world counseling sessions in Chinese.
   • The researchers found a significant correlation between LLM-predicted Big Five traits and actual traits measured using traditional methods, thereby validating the effectiveness of their approach.
3. Improvement Techniques:
   • Ablation studies were conducted to identify the importance of different components of the framework, such as role-play simulations and task simplification using questionnaires. These studies highlighted that both components are crucial for enhancing prediction accuracy.
4. Model Performance:
   • A fine-tuned Llama3-8B model, optimized with Direct Preference Optimization and Supervised Fine-Tuning, was developed.
   • This model showed a substantial improvement of 130.95% over baseline methods and outperformed the state-of-the-art Qwen1.5-110B model by 36.94% in terms of personality prediction validity.

Results and Implications

• Correlation and Validity:
  • The paper demonstrates that LLMs can indeed predict the Big Five personality traits from counseling dialogues with a high degree of validity.
  • The framework’s significant correlation with actual personality measures suggests its potential utility in clinical settings.
• Components' Impact:
  • The ablation studies underscore the importance of incorporating role-playing elements and simplifying tasks with questionnaires. These elements not only improve the realism of the scenario but also assist the LLM in generating more accurate personality predictions.
• Public Accessibility:
  • The authors have made their code and model publicly accessible, facilitating future research and development in the field of computational psychometrics. The repository can be found at the provided GitHub link.

Conclusion

This paper presents a promising approach to predicting personality traits using LLMs, addressing the limitations of traditional methods. By integrating role-play and questionnaire-based prompting, the proposed framework significantly enhances prediction accuracy. The findings suggest that such advanced computational techniques could be valuable tools in psycho-counseling, opening up new avenues for research and practical application in personality assessment.