Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CultureLLM: Incorporating Cultural Differences into Large Language Models (2402.10946v3)

Published 9 Feb 2024 in cs.CL, cs.AI, and cs.LG

Abstract: LLMs are reported to be partial to certain cultures owing to the training data dominance from the English corpora. Since multilingual cultural data are often expensive to collect, existing efforts handle this by prompt engineering or culture-specific pre-training. However, they might overlook the knowledge deficiency of low-resource culture and require extensive computing resources. In this paper, we propose CultureLLM, a cost-effective solution to incorporate cultural differences into LLMs. CultureLLM adopts World Value Survey (WVS) as seed data and generates semantically equivalent training data via the proposed semantic data augmentation. Using only 50 seed samples from WVS with augmented data, we fine-tune culture-specific LLMs and one unified model (CultureLLM-One) for 9 cultures covering rich and low-resource languages. Extensive experiments on 60 culture-related datasets demonstrate that CultureLLM significantly outperforms various counterparts such as GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) with comparable performance to GPT-4 or even better. Our human study shows that the generated samples are semantically equivalent to the original samples, providing an effective solution for LLMs augmentation. Code is released at https://github.com/Scarelette/CultureLLM.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Turkish Spam V01. UCI Machine Learning Repository, 2019. DOI: https://doi.org/10.24432/C5WG7F.
  2. Persianllama: Towards building first persian large language model. arXiv preprint arXiv:2312.15713, 2023.
  3. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528, 2023.
  4. aimansnigdha. Bangla-abusive-comment-dataset. https://github.com/aimansnigdha/Bangla-Abusive-Comment-Dataset, 2018.
  5. Overview of mex-a3t at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets. In Notebook papers of 3rd sepln workshop on evaluation of human language technologies for iberian languages (ibereval), seville, spain, volume 6, 2018.
  6. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop on semantic evaluation, pages 54–63, 2019.
  7. Developing a multilingual annotated corpus of misogyny and aggression. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 158–168, Marseille, France, May 2020. European Language Resources Association (ELRA). URL https://aclanthology.org/2020.trac-1.25/.
  8. Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00132, 2022.
  9. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arxiv. Preprint posted online on March, 31, 2023.
  10. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6193–6202, 2020.
  11. Çağrı Çöltekin. A corpus of turkish offensive language on social media. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174–6184, Marseille, France, 2020. URL https://www.aclweb.org/anthology/2020.lrec-1.758.
  12. Harmonizing global voices: Culturally-aware models for enhanced content moderation. arXiv preprint arXiv:2312.02401, 2023.
  13. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
  14. A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6203–6212, 2020.
  15. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  16. daanVeer. Korean hatespeech dataset. https://github.com/daanVeer/HateSpeech_dataset, 2020.
  17. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17, pages 512–515, 2017.
  18. Angel Felipe Magnossao de Paula and Ipek Baris Schlicht. Ai-upv at iberlef-2021 detoxis task: Toxicity detection in immigration-related web news comments using transformers and statistical models. arXiv preprint arXiv:2111.04530, 2021.
  19. Offensive comments in the brazilian web: a dataset and baseline results. 2017.
  20. Werner Delanoy. What Is Culture?, page 17–34. Cambridge Handbooks in Language and Linguistics. Cambridge University Press, 2020. doi: 10.1017/9781108555067.003.
  21. Direct experience and attitude-behavior consistency. In Advances in experimental social psychology, volume 14, pages 161–202. Elsevier, 1981.
  22. Overview of the task on automatic misogyny identification at ibereval 2018. Ibereval@ sepln, 2150:214–228, 2018.
  23. Normsage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. arXiv preprint arXiv:2210.08604, 2022.
  24. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
  25. Google. Gemini. https://deepmind.google/technologies/gemini/#introduction, 2023.
  26. HASOC. Hasoc2020. https://hasocfire.github.io/hasoc/2020/index.html, 2020.
  27. Geert Hofstede. Culture’s consequences: International differences in work-related values, volume 5. sage, 1984.
  28. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  29. F Husain. Osact4 shared task on offensive language detection: Intensive preprocessing-based approach. arxiv 2020. arXiv preprint arXiv:2005.07297, 2020.
  30. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023.
  31. Detect camouflaged spam content via stoneskipping: Graph and text joint embedding for chinese character variation representation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP2019). ACM, 2019.
  32. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. arXiv e-prints, pages arXiv–2310, 2023.
  33. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785, 2022.
  34. Dataset of arabic spam and ham tweets. Data in Brief, 52(10990):4, 2024.
  35. Kaggle. Jigsaw-multilingual-toxicity. https://www.kaggle.com/code/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models, 2019.
  36. Kaggle. 5k turkish tweets with incivil content. https://www.kaggle.com/datasets/kbulutozler/5k-turkish-tweets-with-incivil-content, 2021.
  37. Kaggle. turkish offensive language detection. https://www.kaggle.com/datasets/toygarr/turkish-offensive-language-detection, 2022.
  38. Detecting abusive instagram comments in turkish using convolutional neural network and machine learning methods. Expert Systems with Applications, 174:114802, 2021.
  39. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870, 2023.
  40. K-MHaS: A multi-label hate speech detection dataset in Korean online news comment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3530–3538, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclanthology.org/2022.coling-1.311.
  41. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. arXiv preprint arXiv:2010.04543, 2020.
  42. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  43. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023.
  44. Mala-500: Massive language adaptation of large language models, 2024.
  45. Taiwan llm: Bridging the linguistic divide with a culturally aligned language model. arXiv preprint arXiv:2311.17487, 2023.
  46. Tinygsm: achieving> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241, 2023a.
  47. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591, 2023b.
  48. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
  49. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023.
  50. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342, 2023.
  51. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 25–31, Online, July 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.socialnlp-1.4.
  52. Social influence and the collective dynamics of opinion formation. PloS one, 8(11):e78433, 2013.
  53. Overview of osact5 shared task on arabic offensive language and hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 162–166, 2022.
  54. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456, 2023.
  55. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907–1917, 2023a.
  56. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023b.
  57. Large language models can replicate cross-cultural differences in personality. arXiv preprint arXiv:2310.10679, 2023.
  58. OpenAI. Chatgpt. https://chat.openai.com/, 2023a.
  59. OpenAI. Gpt-4 technical report, 2023b.
  60. Multilingual and multi-aspect hate speech analysis. In Proceedings of EMNLP. Association for Computational Linguistics, 2019.
  61. Detecting and monitoring hate speech in twitter. Sensors, 19(21):4654, 2019.
  62. Typhoon: Thai large language models. arXiv preprint arXiv:2312.13951, 2023.
  63. Offendes: A new corpus in spanish for offensive language research. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1096–1108, 2021.
  64. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in llms. arXiv preprint arXiv:2310.07251, 2023.
  65. Hate speech detection in the bengali language: A dataset and its baseline evaluation. In Proceedings of International Joint Conference on Advances in Computational Intelligence: IJCACI 2020, pages 457–468. Springer, 2021.
  66. Solid: A large-scale semi-supervised dataset for offensive language identification. arXiv preprint arXiv:2004.14454, 2020.
  67. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In Michael Beißwenger, Michael Wojatzki, and Torsten Zesch, editors, Proceedings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, volume 17 of Bochumer Linguistische Arbeitsberichte, pages 6–9, Bochum, sep 2016.
  68. Multilingual hatecheck: Functional tests for multilingual hate speech detection models. arXiv preprint arXiv:2206.09917, 2022.
  69. Tackling cyber-aggression: Identification and fine-grained categorization of aggressive texts on social media using weighted ensemble of transformers. Neurocomputing, 490:462–481, 2022.
  70. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025, 2023.
  71. A large-scale comprehensive abusiveness detection dataset with multifaceted labels from reddit. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 552–561, 2021.
  72. World Values Survey. World values survey. https://www.worldvaluessurvey.org/wvs.jsp, 2022.
  73. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  74. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv. org/abs/2307.09288, 2023.
  75. HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7174–7183, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.777.
  76. Fabrication and errors in the bibliographic citations generated by chatgpt. Scientific Reports, 13(1):14045, 2023.
  77. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023a.
  78. Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.02144, 2023b.
  79. Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models. arXiv preprint arXiv:2310.13671, 2023c.
  80. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. arXiv preprint arXiv:2310.12481, 2023d.
  81. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93, 2016.
  82. Overview of the germeval 2018 shared task on the identification of offensive language. 2018.
  83. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv 2307.09705, 2023.
  84. From instructions to intrinsic human values–a survey of alignment goals for big models. arXiv preprint arXiv:2308.12014, 2023.
  85. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.
  86. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235, 2020.
  87. Towards identifying social bias in dialog systems: Frame, datasets, and benchmarks. arXiv preprint arXiv:2202.08011, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Cheng Li (1094 papers)
  2. Mengzhou Chen (1 paper)
  3. Jindong Wang (150 papers)
  4. Sunayana Sitaram (54 papers)
  5. Xing Xie (220 papers)
Citations (12)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube