Papers
Topics
Authors
Recent
Search
2000 character limit reached

Replication in Requirements Engineering: the NLP for RE Case

Published 20 Apr 2023 in cs.SE | (2304.10265v2)

Abstract: [Context]} Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks' inherent hairiness, and, in turn, the heterogeneous reporting structure. [Solution] To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. [Results] In this paper: (i) we report on hands-on experiences of replication, (ii) we review the state-of-the-art and extract replication-relevant information, (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction, and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. [Contribution] This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication, but can also be used in other contexts, e.g., for educational purposes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Online Annex: Replication in Requirements Engineering: the NLP for RE Case. https://doi.org/10.6084/m9.figshare.21824481
  2. Automatic Transformation of Natural to Unified Modeling Language: A Systematic Review. In 2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA). 112–119. https://doi.org/10.1109/SERA54885.2022.9806783
  3. Carlos E. Anchundia and Efraín R. Fonseca C. 2020. Resources for Reproducibility of Experiments in Empirical Software Engineering: Topics Derived From a Secondary Study. IEEE Access 8 (2020), 8992–9004. https://doi.org/10.1109/ACCESS.2020.2964587
  4. Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533, 7604 (2016).
  5. From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity, A Handbook. http://se.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf
  6. Rosanna L Breen. 2006. A practical guide to focus-group research. Journal of geography in higher education 30, 3 (2006), 463–475.
  7. Jeffrey C Carver. 2010. Towards reporting guidelines for experimental replications: A proposal. In 1st international workshop on replication in empirical software engineering, Vol. 1. 1–4.
  8. Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 18th Conference on Empirical Methods in Natural Language Processing (EMNLP’14).
  9. The detection and classification of non-functional requirements with application to early aspects. In 14th IEEE International Requirements Engineering Conference. 39–48.
  10. Automated classification of non-functional requirements. Requirements engineering 12, 2 (2007), 103–120.
  11. Replication of studies in empirical software engineering: A systematic mapping study, from 2013 to 2018. IEEE Access 8 (2019), 26773–26791.
  12. Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineering 19, 3 (2014), 501–557.
  13. Requirements Classification with Interpretable Machine Learning and Dependency Parsing. In Proceedings of the 27th IEEE International Requirements Engineering Conference, RE 2019. 142–152. https://doi.org/10.1109/RE.2019.00025
  14. explainable-re/RE-2019-Materials v0. https://doi.org/10.5281/zenodo.3309669
  15. Natural language processing for requirements engineering: The best is yet to come. IEEE software 35, 5 (2018), 115–119.
  16. Pinpointing Ambiguity and Incompleteness in Requirements Engineering via Information Visualization and NLP. In Proceedings of the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ’18).
  17. Fred D. Davis. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 13, 3 (1989), 319–340. http://www.jstor.org/stable/249008
  18. F. De Bruijn and H. Dekkers. 2010. Ambiguity in Natural Language Software Requirements: A Case Study. In Proceedings of the International Workshop on Requirements Engineering: Foundation of Software Quality (REFSQ) (Essen, DE). 233–247.
  19. Investigations about replication of empirical studies in software engineering: A systematic mapping study. Information and Software Technology 64 (2015), 76–101.
  20. Mining and searching app reviews for requirements engineering: Evaluation and replication studies. Information Systems 114 (2023), 102181. https://doi.org/10.1016/j.is.2023.102181
  21. taphsir v6. https://doi.org/10.5281/zenodo.6009790
  22. Automated Handling of Anaphoric Ambiguity in Requirements: A Multi-solution Study. In 2022 IEEE/ACM 44th International Conference on Software Engineering.
  23. Using domain-specific corpora for improved handling of ambiguity in requirements. In 2021 IEEE/ACM 43rd International Conference on Software Engineering.
  24. Naming the pain in requirements engineering. Empirical software engineering 22, 5 (2017), 2298–2338.
  25. Natural Language Requirements Processing: A 4D Vision. IEEE Softw. 34, 6 (2017), 28–35.
  26. PURE: A Dataset of Public Requirements Documents. In 2017 IEEE 25th International Requirements Engineering Conference. 502–505. https://doi.org/10.1109/RE.2017.29
  27. The state-of-practice in requirements specification: an extended interview study at 12 companies. Requirements Engineering (2023). https://doi.org/10.1007/s00766-023-00399-7
  28. Ambiguity in Requirements Engineering: Towards a Unifying Framework. In From Software Engineering to Formal Methods and Tools, and Back. Springer.
  29. Martin Glinz. 2007. On non-functional requirements. In 15th IEEE international requirements engineering conference (RE 2007). IEEE, 21–26.
  30. Jesús M González-Barahona and Gregorio Robles. 2012. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering 17 (2012), 75–89.
  31. Jesus M Gonzalez-Barahona and Gregorio Robles. 2023. Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Information and Software Technology 164 (2023), 107318.
  32. Community expectations for research artifacts and evaluation processes. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 469–480.
  33. David W Johnson and Frank P Johnson. 1991. Joining together: Group theory and group skills. Prentice-Hall, Inc.
  34. Natalia Juristo and Sira Vegas. 2011. The role of non-exact replications in software engineering experiments. Empirical Software Engineering 16, 3 (2011), 295–324.
  35. Erik Kamsties and Barbara Peach. 2000. Taming ambiguity in natural language requirements. In Proceedings of the 13th International Conference on Software and Systems Engineering and Applications (ICSSEA’00).
  36. Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, 2676–2686. https://doi.org/10.18653/v1/P18-1249
  37. Requirements for tools for ambiguity identification and measurement in natural language requirements specifications. Requirements Engineering 13, 3 (2008).
  38. Zijad Kurtanović and Walid Maalej. 2017. Automatically classifying functional and non-functional requirements using supervised machine learning. In 2017 IEEE 25th International Requirements Engineering Conference (RE). Ieee, 490–495.
  39. Zijad Kurtanović and Walid Maalej. 2018. On user rationale in software engineering. Requirements Engineering 23, 3 (2018), 357–379.
  40. J. Richard Landis and Gary G. Koch. 1977. An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers. Biometrics 33, 2 (1977).
  41. Non-functional requirements as qualities, with a spice of ontology. In 2014 IEEE 22nd International Requirements Engineering Conference (RE). IEEE, 293–302.
  42. Open science in software engineering. Contemporary empirical methods in software engineering (2020), 477–501.
  43. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24 (2019), 1057–1060.
  44. Empirical research on requirements quality: a systematic mapping study. Requirements Engineering (2022), 1–27.
  45. Requirement Ambiguity Not as Important as Expected — Results of an Empirical Evaluation. In Proceedings of the International Workshop on Requirements Engineering: Foundation of Software Quality (REFSQ) (Essen, DE). 65–79.
  46. Requirement Formalisation Using Natural Language Processing and Machine Learning: A Systematic Review. In International conference on Model-Based Software and Systems Engineering. SCITEPRESS Digital Library, 1–8.
  47. Cristina Ribeiro and Daniel Berry. 2020. The Prevalence and Severity of Persistent Ambiguity in Software Requirements Specifications: Is a Special Effort Needed to Find Them? Science of Computer Programming 195 (2020), 102472.
  48. Is the SAFE Approach Too Simple for App Feature Extraction? A Replication Study. In Requirements Engineering: Foundation for Software Quality, Eric Knauss and Michael Goedicke (Eds.). Springer International Publishing, Cham, 21–36.
  49. The role and value of replication in empirical software engineering results. Information and Software Technology 99 (2018), 120–132.
  50. The role of replications in empirical software engineering. Empirical software engineering 13, 2 (2008), 211–218.
  51. Roel J Wieringa. 2014. Design science methodology for information systems and software engineering. Springer.
  52. Jonas Winkler and Andreas Vogelsang. 2016. Automatic Classification of Requirements Based on Convolutional Neural Networks. In 24th IEEE International Requirements Engineering Conference, RE 2016, Beijing, China, September 12-16, 2016. IEEE Computer Society, 39–45. https://doi.org/10.1109/REW.2016.021
  53. A retrospective study of one decade of artifact evaluations. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 145–156.
  54. Experimentation in software engineering. Springer Science & Business Media.
  55. Analysing anaphoric ambiguity in natural language requirements. Requirements Engineering 16, 3 (may 2011), 163. https://doi.org/10.1007/s00766-011-0119-y
  56. A Methodology for Automatic Identification of Nocuous Ambiguity. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010). Coling 2010 Organizing Committee, Beijing, China, 1218–1226. https://aclanthology.org/C10-1137
  57. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study. ACM Computing Surveys (CSUR) 54, 3 (2021), 1–41.
  58. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free