Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 162 tok/s
Gemini 2.5 Pro 51 tok/s Pro
GPT-5 Medium 31 tok/s Pro
GPT-5 High 28 tok/s Pro
GPT-4o 95 tok/s Pro
Kimi K2 202 tok/s Pro
GPT OSS 120B 425 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

Semi-supervised News Discourse Profiling with Contrastive Learning (2309.11692v1)

Published 20 Sep 2023 in cs.CL

Abstract: News Discourse Profiling seeks to scrutinize the event-related role of each sentence in a news article and has been proven useful across various downstream applications. Specifically, within the context of a given news discourse, each sentence is assigned to a pre-defined category contingent upon its depiction of the news event structure. However, existing approaches suffer from an inadequacy of available human-annotated data, due to the laborious and time-intensive nature of generating discourse-level annotations. In this paper, we present a novel approach, denoted as Intra-document Contrastive Learning with Distillation (ICLD), for addressing the news discourse profiling task, capitalizing on its unique structural characteristics. Notably, we are the first to apply a semi-supervised methodology within this task paradigm, and evaluation demonstrates the effectiveness of the presented approach.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Saleh Albelwi. 2022. Survey on self-supervised learning: Auxiliary pretext tasks and contrastive learning methods in imaging. Entropy, 24(4):551.
  2. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031.
  3. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  4. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  5. Sachin Chanchani and Ruihong Huang. 2023. Composition-contrastive learning for sentence embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15836–15848, Toronto, Canada. Association for Computational Linguistics.
  6. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  7. Prafulla Kumar Choubey and Ruihong Huang. 2021. Profiling news discourse structure using explicit subtopic structures guided critics. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1594–1605, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  8. Prafulla Kumar Choubey and Ruihong Huang. 2022. Modeling document-level temporal structures for building temporal dependency graphs. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 357–365, Online only. Association for Computational Linguistics.
  9. Discourse as a function of event: Profiling discourse structure in news articles around the main event. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5374–5386, Online. Association for Computational Linguistics.
  10. DiffCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207–4218, Seattle, United States. Association for Computational Linguistics.
  11. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  12. HongSeok Choi Dongha Choi and Hyunju Lee. 2022. Domain knowledge transferring for pre-trained language model via calibrated activation boundary distillation. Annual Meeting of the Association for Computational Linguistics.
  13. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  14. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659.
  15. Distilling the knowledge in a neural network. arXiv:1503.02531.
  16. Sparse progressive distillation: Resolving overfitting under pretrain-and-finetune paradigm. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 190–200, Dublin, Ireland. Association for Computational Linguistics.
  17. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations, 2021.
  18. Momentum contrast for unsupervised visual representation learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  19. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  20. Contrastive representation learning: A framework and review. IEEE Access, 8:193907–193934.
  21. Sentence-level media bias analysis informed by discourse structures. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10040–10050.
  22. A survey of discourse parsing. Frontiers of Computer Science, 16(5):1–12.
  23. Ming Li and Ruihong Huang. 2023. Rst-style discourse parsing guided by document-level content structures. arXiv preprint arXiv:2309.04141.
  24. Less is more: Simplifying feature extractors prevents overfitting for neural discourse parsing models. arXiv preprint arXiv:2210.09537.
  25. Multi-granularity structural knowledge distillation for language model compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1001–1011, Dublin, Ireland. Association for Computational Linguistics.
  26. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
  27. EASE: Entity-aware contrastive learning of sentence embedding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3870–3885, Seattle, United States. Association for Computational Linguistics.
  28. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  29. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.
  30. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897.
  31. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  32. Multitask semi-supervised learning for class-imbalanced discourse classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 498–517, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  33. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958.
  34. Danqi Chen Tianyu Gao, Xingcheng Yao. 2021. Simcse: Simple contrastive learning of sentence embeddings. EMNLP.
  35. Mohammad Norouzi Ting Chen, Simon Kornblith and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. International conference on machine learning.
  36. Representation learning with contrastive predictive coding. arXiv:1807.03748.
  37. Teun A Van Dijk. 1985. Structures of news in the press. Discourse and communication: New approaches to the analysis of mass media discourse and communication, 10:69.
  38. Teun A Van Dijk. 1988. News analysis. Case Studies of International and National News in the Press. New Jersey: Lawrence.
  39. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  40. InfoCSE: Information-aggregated contrastive learning of sentence embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3060–3070, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  41. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
  42. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 87–94, Online. Association for Computational Linguistics.
  43. Universal-kd: Attention-based output-grounded intermediate layer knowledge distillation. Conference on Empirical Methods in Natural Language Processing.
  44. Predicting sentence deletions for text simplification using a functional discourse structure. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 255–261, Dublin, Ireland. Association for Computational Linguistics.
  45. Supporting clustering with contrastive learning. arXiv preprint arXiv:2103.12953.
  46. Contrastive data and learning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, pages 39–47, Seattle, United States. Association for Computational Linguistics.
  47. Learning to perturb for contrastive learning of unsupervised sentence representations.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (2)

  1. Ming Li
  2. Ruihong Huang
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up