Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SoMeLVLM: A Large Vision Language Model for Social Media Processing (2402.13022v1)

Published 20 Feb 2024 in cs.CL and cs.MM

Abstract: The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision LLMs make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision LLM for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, volume 35, pages 23716–23736. Curran Associates, Inc.
  2. Emily Allaway and Kathleen McKeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
  3. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982–4991, Online. Association for Computational Linguistics.
  4. Disc-medllm: Bridging general large language models and real-world medical consultation.
  5. Benjamin S. Bloom and David R. Krathwohl. 1956. Taxonomy of educational objectives; the classification of educational goals by a committee of college and university examiners. Handbook I: Cognitive Domain. Longmans, Green, New York, NY.
  6. Modeling empathy and distress in reaction to news stories. In Conference on Empirical Methods in Natural Language Processing.
  7. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  8. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning.
  9. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  10. Unifying vision-and-language tasks via text generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1931–1942. PMLR.
  11. Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark.
  12. Scaling instruction-finetuned language models.
  13. Toxic comment classification challenge.
  14. Instructblip: Towards general-purpose vision-language models with instruction tuning.
  15. Qlora: Efficient finetuning of quantized llms.
  16. Computational social science and sociology. Annual Review of Sociology, 46(1):61–81.
  17. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345–363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  18. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19358–19369.
  19. Facilitating the communication of politeness through fine-grained paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5127–5140, Online. Association for Computational Linguistics.
  20. Misinfo reaction frames: Reasoning about readers’ reactions to news headlines. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3108–3127, Dublin, Ireland. Association for Computational Linguistics.
  21. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009.
  22. Scott A. Golder and Michael W. Macy. 2014. Digital footprints: Opportunities and challenges for online social research. Annual Review of Sociology, 40(1):129–152.
  23. Exploring hate speech detection in multimodal publications. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1470–1478.
  24. Felipe González-Pizarro and Savvas Zannettou. 2022. Understanding and detecting hateful content using contrastive learning.
  25. Testing the etch-a-sketch hypothesis: a computational analysis of mitt romney’s ideological makeover during the 2012 primary vs. general elections. In APSA 2013 Annual Meeting Paper, American Political Science Association 2013 Annual Meeting.
  26. Does BERT learn as humans perceive? understanding linguistic styles through lexica. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6323–6331, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  27. Avery Holton and Seth Lewis. 2011. Journalists, social media, and the use of humor on twitter. Electronic Journal of Communication, 21.
  28. SemEval-2020 task 7: Assessing humor in edited news headlines. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 746–758, Barcelona (online). International Committee for Computational Linguistics.
  29. Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
  30. Analytical sociology and computational social science. Journal of Computational Science.
  31. A large self-annotated corpus for sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  32. The hateful memes challenge: Detecting hate speech in multimodal memes.
  33. Multimodal post attentive profiling for influencer marketing. In Proceedings of The Web Conference 2020, pages 2878–2884.
  34. Life in the network: The coming age of computational social science. 323.
  35. Computational social science: Obstacles and opportunities. Science, 369(6507):1060–1062.
  36. LAVIS: A one-stop library for language-vision intelligence. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 31–41, Toronto, Canada. Association for Computational Linguistics.
  37. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
  38. Mvptr: Multi-level semantic alignment for vision-language pre-training via multi-stage learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4395–4405.
  39. Reform-eval: Evaluating large vision language models via unified re-formulation of task-oriented benchmarks.
  40. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  41. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207.
  42. Gpt-4v (ision) as a social media analysis engine. arXiv preprint arXiv:2311.07547.
  43. SemEval 2021 task 7: HaHackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105–119, Online. Association for Computational Linguistics.
  44. Stance and sentiment in tweets.
  45. A measurement study of hate speech in social media. In Proceedings of the 28th ACM Conference on Hypertext and Social Media, HT ’17, page 85–94, New York, NY, USA. Association for Computing Machinery.
  46. Unifying local and global knowledge: Empowering large language models as political experts with knowledge graphs. International World Wide Web Conference.
  47. Align voting behavior with public statements for legislator representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1236–1246, Online. Association for Computational Linguistics.
  48. UPPAM: A unified pre-training architecture for political actor modeling based on language. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11996–12012, Toronto, Canada. Association for Computational Linguistics.
  49. Sentiment analysis on social media. In 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pages 919–926.
  50. OpenAI. 2023. ChatGPT. https://chat.openai.com/. Accessed: 2024-02-03.
  51. Overview of the 6th author profiling task at pan 2018: Multimodal gender identification in twitter. In Conference and Labs of the Evaluation Forum.
  52. Jiaxin Pei and David Jurgens. 2020. Quantifying intimacy in language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307–5326, Online. Association for Computational Linguistics.
  53. It takes two to lie: One to lie, and one to listen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3811–3854, Online. Association for Computational Linguistics.
  54. Automatically identifying complaints in social media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5008–5019, Florence, Italy. Association for Computational Linguistics.
  55. Automatically neutralizing subjective bias in text. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):480–489.
  56. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.
  57. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. Applications of Natural Language to Information Systems.
  58. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium. Association for Computational Linguistics.
  59. Big data, digital media, and computational social science: Possibilities and perils. The ANNALS of the American Academy of Political and Social Science, 659(1):6–13.
  60. Fakenewsnet: A data repository with news content, social context and dynamic information for studying fake news on social media. arXiv preprint arXiv:1809.01286.
  61. Fake news detection on social media: A data mining perspective. SIGKDD Explor. Newsl., 19(1):22–36.
  62. Llama: Open and efficient foundation language models.
  63. Llama 2: Open foundation and fine-tuned chat models.
  64. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Processing Systems, volume 34, pages 200–212. Curran Associates, Inc.
  65. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics.
  66. Introducing CAD: the contextual abuse dataset. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2289–2303, Online. Association for Computational Linguistics.
  67. Finetuned language models are zero-shot learners.
  68. Orion Weller and Kevin Seppi. 2019. Humor detection: A transformer gets the last laugh. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3621–3625, Hong Kong, China. Association for Computational Linguistics.
  69. Most users do not follow political elites on twitter; those who do show overwhelming preferences for ideological congruity. Science Advances, 8(39):eabn9418.
  70. Image-text multimodal emotion classification via multi-view attentional network. IEEE Transactions on Multimedia, 23:4014–4026.
  71. Disc-lawllm: Fine-tuning large language models for intelligent legal services.
  72. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  73. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  74. Chatglm-6b fine-tuning for cultural and creative products advertising words. pages 291–295.
  75. Yunxiang Zhang and Xiaojun Wan. 2022. Mover: Mask, over-generate and rank for hyperbole generation.
  76. Judging llm-as-a-judge with mt-bench and chatbot arena.
  77. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
  78. Can large language models transform computational social science?
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now