Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Toward Informal Language Processing: Knowledge of Slang in Large Language Models (2404.02323v2)

Published 2 Apr 2024 in cs.CL

Abstract: Recent advancement in LLMs has offered a strong potential for natural language systems to process informal language. A representative form of informal language is slang, used commonly in daily conversations and online social media. To date, slang has not been comprehensively evaluated in LLMs due partly to the absence of a carefully designed and publicly accessible benchmark. Using movie subtitles, we construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications: 1) slang detection, and 2) identification of regional and historical sources of slang from natural sentences. We also show how our dataset can be used to probe the output distributions of LLMs for interpretive insights. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance. Furthermore, we show that our dataset enables finetuning of LLMs such as GPT-3.5 that achieve substantially better performance than strong zero-shot baselines. Our work offers a comprehensive evaluation and a high-quality benchmark on English slang based on the OpenSubtitles corpus, serving both as a publicly accessible resource and a platform for applying tools for informal language processing.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Metaphors in pre-trained language models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037--2050, Dublin, Ireland. Association for Computational Linguistics.
  2. ParaCotta: Synthetic multilingual paraphrase corpora from the most diverse translation sample pair. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, pages 533--542, Shanghai, China. Association for Computational Lingustics.
  3. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207--219.
  4. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49--72.
  5. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214--217, Barcelona, Spain. Association for Computational Linguistics.
  6. Language (technology) is power: A critical survey of ‘‘bias’’ in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454--5476, Online. Association for Computational Linguistics.
  7. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. In Proceedings of the Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML), Halifax, Canada.
  8. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135--146.
  9. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877--1901. Curran Associates, Inc.
  10. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77--91. PMLR.
  11. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171--4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  12. A paraphrase and semantic similarity detection system for user generated short-text content on microblogs. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2880--2890, Osaka, Japan. The COLING 2016 Organizing Committee.
  13. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492--2501, Online. Association for Computational Linguistics.
  14. Connie C Eble. 2012. Slang & Sociability: In-group Language among College Students. University of North Carolina Press, Chapel Hill, NC.
  15. Jonathan Green. 2010. Green’s Dictionary of Slang. Chambers, London.
  16. Dirk Hovy. 2015. Demographic factors improve classification performance. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 752--762, Beijing, China. Association for Computational Linguistics.
  17. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 483--488, Beijing, China. Association for Computational Linguistics.
  18. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591--598, Berlin, Germany. Association for Computational Linguistics.
  19. Slangvolution: A causal analysis of semantic change and frequency dynamics in slang. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1422--1442, Dublin, Ireland. Association for Computational Linguistics.
  20. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  21. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684--7689.
  22. Vivek Kulkarni and William Yang Wang. 2018. Simple models for word formation in slang. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1424--1434, New Orleans, Louisiana. Association for Computational Linguistics.
  23. William Labov. 1972. Language in the inner city: Studies in the Black English vernacular. University of Pennsylvania Press.
  24. William Labov. 2006. The social stratification of English in New York City. Cambridge University Press.
  25. How is BERT surprised? layerwise detection of linguistic anomalies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4215--4228, Online. Association for Computational Linguistics.
  26. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521--535.
  27. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923--929, Portorož, Slovenia. European Language Resources Association (ELRA).
  28. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437--4452, Seattle, United States. Association for Computational Linguistics.
  29. Roberta: A robustly optimized bert pretraining approach. arXiv.
  30. Characterizing English variation across social media communities with BERT. Transactions of the Association for Computational Linguistics, 9:538--556.
  31. Elisa Mattiello. 2005. The pervasiveness of slang in standard and non-standard english. Mots Palabras Words, 6:7--41.
  32. Learning to explain non-standard English words and phrases. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 413--417, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  33. OpenAI. 2023. GPT-4 technical report. arXiv.
  34. Slang detection and identification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 881--889, Hong Kong, China. Association for Computational Linguistics.
  35. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982--3992, Hong Kong, China. Association for Computational Linguistics.
  36. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842--866.
  37. Zhewei Sun and Yang Xu. 2022. Tracing semantic variation in slang. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1299--1313, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  38. Slang generation as categorization. In Proceedings of the 41st Annual Conference of the Cognitive Science Society, pages 2898--2904. Cognitive Science Society.
  39. A computational framework for slang generation. Transactions of the Association for Computational Linguistics, 9:462--478.
  40. Semantically informed slang interpretation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5213--5231, Seattle, United States. Association for Computational Linguistics.
  41. Bradley A. Swerdfeger. 2012. Assessing the viability of the urban dictionary as a resource for slang.
  42. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53--59, Valencia, Spain. Association for Computational Linguistics.
  43. What do you learn from context? probing for sentence structure in contextualized word representations. In Proceedings of 7th International Conference on Learning Representations, ICLR 2019.
  44. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214--2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  45. Exploring demographic language variations to improve multilingual sentiment analysis in social media. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1815--1827, Seattle, Washington, USA. Association for Computational Linguistics.
  46. IndoCollex: A testbed for morphological transformation of Indonesian colloquial words. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3170--3183, Online. Association for Computational Linguistics.
  47. Semi-supervised low-resource style transfer of Indonesian informal to formal language with iterative forward-translation. In 2020 International Conference on Asian Language Processing (IALP), pages 310--315. IEEE.
  48. Philipp Wicke. 2023. LMs stand their ground: Investigating the effect of embodiment in figurative language interpretation by language models.
  49. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38--45, Online. Association for Computational Linguistics.
  50. Gathering and generating paraphrases from Twitter with application to normalization. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora, pages 121--128, Sofia, Bulgaria. Association for Computational Linguistics.
  51. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now