Insight Into SEER (2311.01164v2)
Abstract: Developing test oracles can be inefficient: developer generative oracles are time-intensive and thus costly while automatic oracle generation in the form of regression or exception oracles assumes that the underlying code is correct. To mitigate the high cost of testing oracles, the SEER tool was developed to predict test outcomes without needing assertion statements. The creators of SEER introduced the tool with an overall accuracy of 93%, precision of 86%, recall of 94%, and an F1 score of 90%. If these results are replicable on new data with perturbations, i.e. SEER is generalizable and robust, the model would represent a significant advancement in the field of automated testing. Consequently, we conducted a comprehensive reproduction of SEER and attempted to verify the model's results on a new dataset.
- TOGA: a neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering, pages 2130–2141, Pittsburgh Pennsylvania, May 2022. ACM. ISBN 978-1-4503-9221-1. doi: 10.1145/3510003.3510141. URL https://dl.acm.org/doi/10.1145/3510003.3510141.
- Scaling static analyses at Facebook. Communications of the ACM, 62(8):62–70, July 2019. ISSN 0001-0782, 1557-7317. doi: 10.1145/3338112. URL https://dl.acm.org/doi/10.1145/3338112.
- Neural-based test oracle generation: A large-scale evaluation and lessons learned. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, page 120–132, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400703270. doi: 10.1145/3611643.3616265. URL https://doi.org/10.1145/3611643.3616265.
- Perfect is the enemy of test oracle. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 70–81, Singapore Singapore, Nov. 2022. ACM. ISBN 978-1-4503-9413-0. doi: 10.1145/3540250.3549086. URL https://dl.acm.org/doi/10.1145/3540250.3549086.
- Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437–440, San Jose CA USA, July 2014. ACM. ISBN 978-1-4503-2645-2. doi: 10.1145/2610384.2628055. URL https://dl.acm.org/doi/10.1145/2610384.2628055.
- Adversarial Examples for Models of Code, Oct. 2020. URL http://arxiv.org/abs/1910.07517. arXiv:1910.07517 [cs].
- Kasra Lekan (1 paper)
- Nicki Choquette (1 paper)