TestMigrationsInPy: unittest to pytest
- TestMigrationsInPy is a curated dataset featuring 923 isolated Python test migrations with before/after code pairs and migration-type labels from unittest to pytest.
- It supports evaluation of automated migration tools by providing clearly classified examples of assertion, fixture, import, skip, and expected failure migrations.
- The dataset offers practical insights for benchmarking test refactoring techniques, LLM-assisted migration, and empirical studies on test framework evolution in Python.
TestMigrationsInPy is a curated dataset of real-world Python test migrations from unittest to pytest, designed to support research on automated framework migration and test-refactoring tools (Alves et al., 4 Feb 2026). It addresses the problem of how to build, evaluate, and compare automated tools that migrate Python tests from unittest to pytest using real developer changes as ground truth. The dataset contains 923 real-world migrations performed by developers, collected from 100 highly popular Python software systems, found in 37 projects, and derived from 690 migration commits. Its distinctive features are before/after code pairs, migration-type labels, and manual curation of isolated migrations, which together make it suitable for evaluating migration systems ranging from simpler assertion migrations to more complex fixture migrations (Alves et al., 4 Feb 2026).
1. Research context and motivation
The motivation for TestMigrationsInPy is that unittest and pytest are the most popular testing frameworks in Python, while migration from unittest to pytest is often time-consuming, incremental, and error-prone (Alves et al., 4 Feb 2026). pytest is presented as attractive because it offers less verbose tests, simpler assertions, fixture support, and interoperability with existing unittest tests, allowing gradual migration. In the formulation used for the dataset, unittest tests are usually written inside classes that inherit from TestCase, whereas pytest tests can be plain functions; unittest uses specialized methods such as self.assertEqual and self.assertIn, whereas pytest allows ordinary Python assert statements; and pytest fixtures often replace setUp, tearDown, and setUpClass (Alves et al., 4 Feb 2026).
The ecosystem significance of this migration pattern is reinforced by prior evidence that 27% of the top-100 most popular Python projects had migrated or were migrating to pytest (Alves et al., 4 Feb 2026). Related work on large-scale migration in Python also shows that migration is rarely a trivial substitution problem. In the LSST modernization effort, migration testing exposed issues involving global state, randomness, shared filesystem paths, and process-local behavior, illustrating that framework migration is not only about language syntax changes (Jenness, 2017). This suggests that a ground-truth dataset of real test migrations is useful not merely for rewriting assertions, but for studying broader structural and behavioral transformations.
A common misconception is that migration from unittest to pytest is just syntax replacement. The source material explicitly rejects that view. Test framework migration can involve converting class-based tests to function-based tests, replacing self.assert* methods with plain assert, and moving setUp and tearDown logic into @pytest.fixture functions; the last of these is described as much harder because there is no direct one-to-one mapping between unittest setup/teardown and Pytest fixtures (Alves et al., 3 Feb 2026).
2. Dataset composition and curation model
The headline structure of TestMigrationsInPy is:
| Item | Value |
|---|---|
| Real-world migrations | 923 |
| Highly popular Python software systems | 100 |
| Projects | 37 |
| Migration commits | 690 |
The relationship is summarized as (Alves et al., 4 Feb 2026). The 100 systems were selected from the authors’ earlier empirical study on Python test migration.
The dataset was built in two major steps. First, the authors automatically detected migration commits by reusing the migration-detection tool from Barbosa and Hora. This tool mines the version history using PyDriller, scans added and removed lines per commit, and identifies commits as migration commits if they contain at least one migration type from the dataset’s taxonomy (Alves et al., 4 Feb 2026). Second, the authors manually identified isolated migrations. Because a migration commit may contain multiple migrations plus unrelated changes, they manually inspected commits and extracted isolated migrations, defined as migrations that simply replace unittest by pytest and nothing else unrelated. Commits modifying more than 5 test files were excluded in order to reduce noise (Alves et al., 4 Feb 2026).
This curation procedure is central to the dataset’s use as ground truth. The reported rationale is that verified inputs and outputs are necessary for automated migration evaluation, and manually isolating clean transformations makes the examples more suitable for benchmarking than tangled repository diffs (Alves et al., 4 Feb 2026). A plausible implication is that the dataset is oriented toward evaluating transformation correctness under controlled conditions rather than reconstructing the full socio-technical process of repository-wide migration.
3. Migration taxonomy and transformation granularity
TestMigrationsInPy explicitly defines five migration types used by the detection tool and the dataset (Alves et al., 4 Feb 2026).
| Migration type | Definition |
|---|---|
| Assert migration | removes unittest self.assert*; adds pytest assert statements |
| Fixture migration | removes unittest setups/teardowns; adds pytest fixtures |
| Import migration | removes import unittest; adds import pytest |
| Skip migration | removes unittest skip usage; adds pytest skip usage |
| Expected failure migration | removes unittest expected-failure usage; adds pytest expected-failure usage |
These labels provide what the source describes as a taxonomy of migration difficulty. Assert migrations are often simpler and more local, while fixture migrations are harder because there is no direct one-to-one mapping between unittest lifecycle methods and pytest fixtures (Alves et al., 4 Feb 2026). Import, skip, and expected-failure migrations capture other framework-specific constructs. The dataset is therefore not limited to assertion rewriting; it also includes test lifecycle and control-flow conventions that are framework-specific.
This taxonomy aligns with a broader empirical view of migration complexity in Python. The study on Python library migrations introduces PyMigTax, a taxonomy in which migration-related code changes may involve function call, function reference, type, exception, attribute, decorator, and import, together with properties such as argument transformation and output transformation (Islam et al., 2022). Although PyMigTax concerns Python library migrations rather than test framework migration, it provides relevant context: real migrations frequently combine several kinds of edits, and many cannot be reduced to one-to-one function-call replacement. TestMigrationsInPy adopts a narrower but operationally useful taxonomy tailored to unittest-to-pytest transformation (Alves et al., 4 Feb 2026).
An important interpretive point is that fixture migration is treated as structurally harder than assertion migration. That distinction is not merely descriptive; it supports benchmarking tools at different difficulty levels, from local assertion replacement to transformations involving setUp, tearDown, fixture injection, and test restructuring (Alves et al., 4 Feb 2026).
4. Repository organization and representative examples
TestMigrationsInPy is organized as a GitHub repository. For each project, there is a list of migration commits; each migration commit contains a list of isolated migrations; and each isolated migration includes the before code in unittest form and the after code in pytest form (Alves et al., 4 Feb 2026). The repository structure includes diff/, which stores the migration code with before and after test files, and output.info, a summary file containing commit hash, number of changed files, number of migrations in the commit, and migration type (Alves et al., 4 Feb 2026).
The paper reports representative examples that illustrate the dataset’s range. A simple assertion migration comes from Saleor: one migration commit containing four migrations. The example includes changes such as self.assertEquals(func, oauth_callback) becoming assert func is oauth_callback, self.assertIn(a, b) becoming assert a in b, and self.assertIsInstance(a, b) becoming assert isinstance(a, b) (Alves et al., 4 Feb 2026). These examples show that even assertion migration may require expression restructuring rather than method-name substitution alone.
A more complex example comes from Dash: four migration commits, with the first commit containing nine migrations. That example includes removal of a TestCase class, replacement of a setUp method with a pytest fixture, conversion of a test method into a plain test function, and use of fixture injection by function parameters (Alves et al., 4 Feb 2026). The paper also highlights a case from pyvim in which a setUp method was split into four separate pytest fixtures: prompt_buffer, editor_buffer, window, and tab_page. This demonstrates that fixture migration may require structural decomposition (Alves et al., 4 Feb 2026).
These examples help clarify what the dataset captures as “isolated migrations.” The emphasis is on transformations where unittest is replaced by pytest without unrelated code changes. That makes the before/after pairs well suited to supervised learning, rule inference, or direct evaluation of automated migration systems.
5. Use in automated migration research
The primary intended role of TestMigrationsInPy is as a ground-truth resource for future research proposing novel solutions to migrate frameworks in Python (Alves et al., 4 Feb 2026). Because it provides real developer-produced migrations, before/after pairs, migration-type labels, and manual curation to isolate clean transformations, it is suitable for training or evaluating machine learning and LLM-based migration tools, studying which migration types are easier or harder, and measuring accuracy on assertion-only migrations versus more complex fixture migrations (Alves et al., 4 Feb 2026).
The paper “Testing Framework Migration with LLMs” operationalizes this use case directly. It introduces a curated dataset of real-world migrations extracted from the top 100 Python open-source projects and then executes LLM-generated test migrations in their respective test suites (Alves et al., 3 Feb 2026). From the 923 real-world migrations, additional filtering yielded 40 migrations suitable for execution-based evaluation, drawn from seven projects and classified into 30 simple and 10 complex cases (Alves et al., 3 Feb 2026). Across 480 LLM migration requests, 233 passed and 247 failed, corresponding to 48.5% passed and 51.5% failed (Alves et al., 3 Feb 2026). The reported outcome is that LLMs can accelerate migration, but human review is still essential.
The same study also found that coverage remained exactly the same for all successful migrations (Alves et al., 3 Feb 2026). This is significant because it suggests that, when execution succeeds, the migrated tests usually preserve intended test behavior rather than merely producing syntactically acceptable code. The reported failure categories—AssertionError, Structural Mismatch, Missing Fixtures, TypeError, Signature Drift, and SyntaxError—indicate that errors are concentrated in dependency handling, fixture adaptation, and setup inconsistencies (Alves et al., 3 Feb 2026). These findings are consistent with the taxonomy in TestMigrationsInPy, especially the distinction between simpler assert migration and harder fixture migration.
The dataset paper also reports an initial LLM-based usage: GPT-4o was used with TestMigrationsInPy to migrate tests from unittest to pytest, and the initial results suggest GPT-4o can accelerate migration, but it may produce minor incorrect updates, especially for fixture migrations (Alves et al., 4 Feb 2026). This suggests that the dataset is already functioning both as a benchmark and as a diagnostic instrument for error analysis.
6. Relation to broader migration benchmarks and methodologies
TestMigrationsInPy belongs to a broader class of migration benchmarks, but its scope is distinctive. CODEMENV benchmarks LLMs on code migration across software environments, with 922 examples spanning 19 Python and Java packages and three core tasks: identifying functions incompatible with specific versions, detecting changes in function definitions, and adapting code to target environments (Cheng et al., 1 Jun 2025). Its logic is organized around detection, diagnosis, and repair. By contrast, TestMigrationsInPy focuses specifically on framework migration from unittest to pytest, grounded in real developer-performed transformations rather than version-specific API evolution (Alves et al., 4 Feb 2026).
This contrast is methodologically important. CODEMENV evaluates migration with exact-match function localization, structured change classification, and unit-test-backed migration validation (Cheng et al., 1 Jun 2025). TestMigrationsInPy instead provides before/after test code and migration-type labels, enabling evaluation of framework-specific refactorings such as assertion conversion, fixture introduction, and skip or expected-failure replacement (Alves et al., 4 Feb 2026). A plausible implication is that TestMigrationsInPy is especially suitable for studying source-to-source refactoring and test modernization, whereas CODEMENV addresses version-aware environment migration.
Another relevant comparison is IntentTester, an intent-driven framework for cross-library and cross-language test reuse. IntentTester abstracts tests into a language-agnostic Test Description Language, aligns them with semantically related entities in a repository graph, and synthesizes executable tests through LLM-guided reasoning and iterative validation (Gao et al., 24 Jun 2026). That work concerns cross-library and cross-language migration, including Python, and reports 2,776 syntactically correct tests with 85% correctness and 74% effectiveness (Gao et al., 24 Jun 2026). TestMigrationsInPy is narrower in scope but more tightly grounded in one concrete migration axis: unittest to pytest. This suggests a complementary relationship: TestMigrationsInPy provides a focused corpus of framework transformations, while broader systems such as IntentTester explore semantic transfer beyond framework boundaries.
Historical migration experience in LSST provides further context. There, modernization included a move from exit-status-based script execution to pytest, together with JUnit-compatible reporting, pytest-xdist, and pytest-flake8 (Jenness, 2017). That example shows that adoption of pytest in large systems can be part of a broader modernization pass rather than an isolated framework switch. TestMigrationsInPy captures the code-level manifestations of such shifts in a form usable for comparative research.
7. Limitations, validity considerations, and prospective extensions
The limitations discussed for TestMigrationsInPy are explicit. The dataset is drawn from 100 highly popular projects, which do not represent the entire Python ecosystem, and less popular projects may have different migration practices (Alves et al., 4 Feb 2026). This introduces popularity bias. The authors therefore suggest expanding the dataset to include more projects for better coverage (Alves et al., 4 Feb 2026).
A second limitation concerns taxonomy granularity. The current classification could be made more specific, for example by distinguishing setUp, setUpClass, and tearDown on the unittest side, and pytest features such as parametrized tests on the pytest side (Alves et al., 4 Feb 2026). This indicates that the current five-type taxonomy is useful but not exhaustive. A plausible implication is that future versions could support more fine-grained evaluation of migration difficulty and idiomatic pytest adoption.
A third concern is possible residual noise. Although the dataset filters tangled changes and manually curates isolated migrations, any manual curation process can still miss edge cases (Alves et al., 4 Feb 2026). The construction also relies on a migration-detection tool that reportedly has 100% precision and recall for detecting migrations (Alves et al., 4 Feb 2026). Within the confines of the reported methodology, this supports confidence in the mining stage, but it does not eliminate all downstream curation risks.
More broadly, empirical work on Python migration warns against overly narrow assumptions. The characterization of Python library migrations shows that real migrations often involve non-function program elements, higher-cardinality mappings, and combinations of properties such as argument transformation and output transformation (Islam et al., 2022). While that result concerns library migration rather than test framework migration, it strengthens the view that migration benchmarks should evolve toward richer transformation taxonomies. In the case of TestMigrationsInPy, future extensions that distinguish finer-grained fixture patterns, parametrization, and other pytest idioms would be consistent with that broader empirical lesson.
In summary, TestMigrationsInPy is significant because it combines a real-world corpus, explicit migration-type labels, manual isolation of clean transformations, and code-level before/after pairs (Alves et al., 4 Feb 2026). Its main limitation is representativeness: it currently reflects a subset of the ecosystem, primarily popular projects. Even so, it provides a concrete and reusable benchmark for research on automated migration, LLM-based test refactoring, and comparative evaluation of framework-migration tools.