Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsing (2312.14966v1)

Published 18 Dec 2023 in cs.CL

Abstract: The intricate hierarchical structure of syntax is fundamental to the intricate and systematic nature of human language. This study investigates the premise that LLMs, specifically their attention distributions, can encapsulate syntactic dependencies. We introduce Dynamic Syntax Mapping (DSM), an innovative approach for the agnostic induction of these structures. Our method diverges from traditional syntax models which rely on predefined annotation schemata. Instead, we focus on a core characteristic inherent in dependency relations: syntactic substitutability. This concept refers to the interchangeability of words within the same syntactic category at either end of a dependency. By leveraging this property, we generate a collection of syntactically invariant sentences, which serve as the foundation for our parsing framework. Our findings reveal that the use of an increasing array of substitutions notably enhances parsing precision on natural language data. Specifically, in the context of long-distance subject-verb agreement, DSM exhibits a remarkable advancement over prior methodologies. Furthermore, DSM's adaptability is demonstrated through its successful application in varied parsing scenarios, underscoring its broad applicability.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
  2. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi:10.18653/v1/D18-1151. URL https://aclanthology.org/D18-1151.
  3. Retrofitting structure-aware transformer language model for end tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2151–2161, 2020a.
  4. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195–1205, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi:10.18653/v1/N18-1108. URL https://aclanthology.org/N18-1108.
  5. Yoav Goldberg. Assessing BERT’s syntactic abilities. CoRR, abs/1901.05287, 2019. URL http://arxiv.org/abs/1901.05287.
  6. MRN: A locally and globally mention-based reasoning network for document-level relation extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1359–1370, 2021.
  7. Matching structure for dual learning. In Proceedings of the International Conference on Machine Learning, ICML, pages 6373–6391, 2022a.
  8. Software defect prediction via convolutional neural network. In Proceedings of the 2017 International Conference on Software Quality, Reliability and Security, pages 318–328. IEEE, 2017.
  9. Boundaries and edges rethinking: An end-to-end neural model for overlapping entity relation extraction. Information Processing & Management, 57(6):102311, 2020b.
  10. Unified named entity recognition as word-word relation classification. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10965–10973, 2022.
  11. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:10.18653/v1/N19-1419. URL https://aclanthology.org/N19-1419.
  12. Do attention heads in BERT track syntactic dependencies? CoRR, abs/1911.12246, 2019. URL http://arxiv.org/abs/1911.12246.
  13. Universal Dependencies According to BERT: Both More Specific and More General. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2710–2722, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.findings-emnlp.245. URL https://aclanthology.org/2020.findings-emnlp.245.
  14. Mastering the explicit opinion-role interaction: Syntax-aided neural transition system for unified opinion role labeling. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, pages 11513–11521, 2022.
  15. Effective token graph modeling using a novel labeling strategy for structured sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4232–4241, 2022.
  16. Latent emotion memory for multi-label emotion classification. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7692–7699, 2020c.
  17. Entity-centered cross-document relation extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9871–9881, 2022.
  18. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, March 2022. doi:10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7.
  19. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi:10.18653/v1/W18-5431. URL https://aclanthology.org/W18-5431.
  20. Better combine them together! integrating syntactic constituency and dependency representations for semantic role labeling. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, pages 549–559, 2021a.
  21. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy, August 2019. Association for Computational Linguistics. doi:10.18653/v1/W19-4828. URL https://aclanthology.org/W19-4828.
  22. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 478–485, Barcelona, Spain, July 2004. doi:10.3115/1218955.1219016. URL https://aclanthology.org/P04-1061.
  23. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2369–2385, Florence, Italy, July 2019. Association for Computational Linguistics. doi:10.18653/v1/P19-1228. URL https://aclanthology.org/P19-1228.
  24. Next-gpt: Any-to-any multimodal llm, 2023.
  25. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  26. Learn from syntax: Improving pair-wise aspect and opinion terms extraction with rich syntactic knowledge. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 3957–3963, 2021.
  27. Encoder-decoder based unified semantic role labeling with label-aware syntax. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12794–12802, 2021b.
  28. Lasuie: Unifying information extraction with latent adaptive structure-aware generative language model. In Proceedings of the Advances in Neural Information Processing Systems, NeurIPS 2022, pages 15460–15475, 2022b.
  29. Enriching contextualized language model from knowledge graph for biomedical information extraction. Briefings in Bioinformatics, 22(3), 2021c.
  30. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1xPR3NtPB.
  31. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166–4176, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.383. URL https://aclanthology.org/2020.acl-main.383.
  32. Scene graph as pivoting: Inference-time image-free unsupervised multimodal machine translation with visual scene hallucination. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5980–5994, 2023.
  33. Linguistic dependencies and statistical dependence. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2941–2963, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.emnlp-main.234. URL https://aclanthology.org/2021.emnlp-main.234.
  34. On the inductive bias of masked language modeling: From statistical to syntactic dependencies. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5131–5146, Online, June 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-main.404. URL https://aclanthology.org/2021.naacl-main.404.
  35. Cross-lingual semantic role labeling with model transfer. IEEE ACM Trans. Audio Speech Lang. Process., 28:2427–2437, 2020d.
  36. Syntactic dependencies correspond to word pairs with high mutual information. In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pages 3–13, Paris, France, August 2019. Association for Computational Linguistics. doi:10.18653/v1/W19-7703. URL https://aclanthology.org/W19-7703.
  37. When classifying grammatical role, BERT doesn’t care about word order… except when it matters. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 636–643, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.acl-short.71. URL https://aclanthology.org/2022.acl-short.71.
  38. Tim Hunter. The chomsky hierarchy. In Nicholas Allott, Terje Lohndal, and Georges Rey, editors, A Companion to Chomsky, chapter 5, pages 74–95. John Wiley & Sons, Ltd, 2021. ISBN 9781119598732. doi:https://doi.org/10.1002/9781119598732.ch5. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/9781119598732.ch5.
  39. Igor Mel’čuk. Dependency in natural language. In Alain Polguère and Igor Mel’čuk, editors, Dependency in linguistic description, volume 111 of Studies in language companion series. John Benjamins Pub. Co, Amsterdam; Philadelphia., 2009.
  40. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020. URL https://nlp.stanford.edu/pubs/qi2020stanza.pdf.
  41. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330, 1993. URL https://aclanthology.org/J93-2004.
  42. Generating typed dependency parses from phrase structure parses. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy, May 2006. European Language Resources Association (ELRA). URL http://www.lrec-conf.org/proceedings/lrec2006/pdf/440_pdf.pdf.
  43. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.497.
  44. SUD or surface-syntactic Universal Dependencies: An annotation scheme near-isomorphic to UD. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 66–74, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi:10.18653/v1/W18-6008. URL https://aclanthology.org/W18-6008.
  45. R. C. Prim. Shortest connection networks and some generalizations. The Bell System Technical Journal, 36(6):1389–1401, 1957. doi:10.1002/j.1538-7305.1957.tb01515.x.
  46. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi:10.18653/v1/P19-1452. URL https://aclanthology.org/P19-1452.
  47. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4077–4091, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.375. URL https://aclanthology.org/2020.acl-main.375.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now