ReproBreak: A Dataset of Reproducible Web Locator Breaks

Abstract: Automated GUI testing frameworks such as Cypress and Playwright rely on locators to find and interact with web elements. A locator break occurs when a structural change in the application under test causes a locator to no longer find its target element, resulting in test breakages even when the underlying functionality remains unchanged. Despite its impact on test maintenance, no dataset exists to evaluate locator fragility in Cypress and Playwright at scale. In this paper, we present ReproBreak, a dataset of reproducible locator breaks in web application GUI tests. We analyzed 359 open-source repositories to identify commits that contain locator changes. To confirm whether these changes are indeed locator breaks, we reproduced them in the top 4 projects with the largest number of locator changes and found 449 locator breaks, which are provided in the dataset along with scripts for automated reproduction. We believe ReproBreak serves as a valuable artifact to support research on locator fragility, repair techniques, and test robustness. The video is available at: https://youtu.be/mZByS_TnCvE. The dataset is at https://github.com/rub-sq/ReproBreak.

• The paper introduces ReproBreak, a dataset capturing 449 reproducible locator breaks from over 9,500 web locator changes in modern GUI tests.
• The paper details a methodology combining automated static analysis and Docker-based replay to validate locator breakage in Cypress and Playwright.
• The paper demonstrates practical implications for benchmarking locator repair techniques and designing more robust GUI test automation frameworks.

ReproBreak: A Dataset of Reproducible Web Locator Breaks

Motivation and Context

Automated GUI testing frameworks such as Cypress and Playwright are fundamental in continuous integration pipelines for modern web applications, facilitating regression detection at the interface level. These frameworks depend critically on locators—element identifiers such as IDs, CSS selectors, or XPath—to accurately interact with web elements during test execution. Structural changes to the Application Under Test (AUT), such as renaming element classes or restructuring the DOM, frequently invalidate locators, resulting in test failures that impede effective development and maintenance workflows. This phenomenon, known as locator fragility or locator breakage, remains a preeminent challenge in large-scale GUI test automation—up to 75% of test files in Selenium-based setups require locator updates within nine commits, and fragile locators account for over 70% of observed test failures [christophe2014prevalence, hammoudi2016record, nass2021many]. Despite significant research aimed at improving locator robustness and repair methodologies, empirical evaluation has been constrained to Selenium; until now, there has been a lack of scalable datasets targeting Cypress and Playwright.

Dataset Construction and Methodology

ReproBreak establishes a systematic foundation for empirical evaluation of locator fragility in Cypress and Playwright. The dataset was constructed by mining 359 open-source repositories (189 Playwright, 154 Cypress, 16 both) and analyzing their evolution for explicit locator modifications. The data collection pipeline integrates automated static analysis and manual environment provisioning steps, highlighted in the following workflow:

• Step 1: Locator Change Identification Test files modified across commits were detected using regex heuristics for framework-specific APIs (e.g., cy.get for Cypress, page.locator for Playwright). Each commit pair was diffed to enumerate locator changes ($L_c$), capturing both prophylactic (non-breaking) and break-inducing modifications.
• Step 2: Docker-Based Replay Environment Reproducibility was ensured by encapsulating each application and test in a Docker environment. Manual effort was required for complex dependency setups, focusing on the four projects with highest locator change frequency.

  Figure 1: Data Collection Pipeline. Dashed boxes represent manual steps required to create reproducible environments.

• Step 3: Automated Locator Break Validation For each $L_c$, the test was rerun with the original locator on the updated codebase. Failure indicated a locator break ($L_b$), while success yielded a non-breaking locator change ($L_{nb}$). In total, 449 locator breaks were validated across four targeted projects (angluar-slickgrid, openmct, koenig, playwright).

Dataset Structure and Coverage

ReproBreak is stored as a relational SQLite database, centered on the locator_change table. Each entry includes metadata linking projects, test files, commits, and reproduction artifacts (Dockerfiles and execution scripts). The relational schema is designed for extensibility, enabling cross-framework comparisons and custom filtering.

Figure 2: Entity-relationship model of the dataset, showing links between locator breaks, projects, test files, commits, and reproduction artifacts.

A total of 9,572 locator changes were identified across 211 repositories. Of these, 449 were classified as reproducible locator breaks and validated in real execution contexts, providing strong empirical evidence for test fragility in Playwright and Cypress. Automated scripts facilitate benchmark-oriented experimentation, enabling reproduction of locator breaks, verification of repaired locators, and active modification within the test environment.

Application Scenarios and Research Implications

ReproBreak enables diverse empirical and algorithmic research directions in automated GUI test maintenance:

• Evaluation of Robust Locator Generation Existing algorithms like Robula+ and Sidereal can now be quantitatively benchmarked on Playwright and Cypress, rather than only Selenium, highlighting generalizability and framework-specific nuances.
• Fragility Score Development The dataset allows adaptation and empirical validation of fragility scoring metrics, identifying locator types most susceptible to breakage and informing the design of less fragile locators for each framework.
• Locator Repair Techniques Repair approaches, including similarity-based strategies (Similo, Vista, Color) and future LLM-driven repair agents, can be rigorously assessed in realistic settings provided by ReproBreak.
• NL-Based Testing Robustness ReproBreak facilitates evaluation of natural-language-based (NL) testing abstractions [leotta2024empirical], determining the degree to which higher-level test representations mitigate locator fragility and adapting findings from Selenium contexts to broader frameworks.

Practical and Theoretical Implications

The introduction of ReproBreak marks a substantial advance in reproducible empirical software testing research by bridging previous gaps in Playwright and Cypress-oriented evaluations. Practically, researchers and industry practitioners can now benchmark locator robustness and repair methods in realistic, reproducible environments, accelerating progress towards robust GUI test maintenance. Theoretically, ReproBreak lays the groundwork for cross-framework generalization of fragility and repair metrics, enabling the systematic study of locator types and repair strategies under diverse execution semantics.

Anticipated future directions include expansion of coverage to additional frameworks (e.g., Puppeteer), reduction of manual environment setup via agent-based automation (potentially LLM-driven), and exploration of hybrid repair solutions integrating NL-based abstractions, statistical locator generation, and visual heuristics.

Conclusion

ReproBreak provides a curated, reproducible dataset of locator breaks for Cypress and Playwright, supporting benchmark-driven research and empirical evaluation of test fragility, repair methodologies, and robustness metrics. With 449 validated locator breaks across 9,572 changes, the dataset fills a critical gap in GUI test maintenance research, offering practical scripts for active experimentation. The extensible schema and automation capabilities promise broad impact on future AI-driven software testing tools and theoretical understanding of locator stability (2605.12158).