Replication Study ID-Card
- Replication Study ID-Card is a standardized artifact that captures 47 replication-relevant items across 7 topical areas to enhance reproducibility in NLP for requirements engineering.
- It details critical aspects such as dataset provenance, annotation protocols, tool implementation, and evaluation metrics to reduce ambiguities in empirical studies.
- Developed through an iterative design-science process, the ID-Card guides both prospective planning and retrospective auditing of replication readiness.
Searching arXiv for recent and foundational papers on replication study ID-cards and related replication frameworks. arXiv search query: "replication study ID-card replication requirements engineering design space replication-related studies"
The replication study ID-Card is a structured artifact for recording replication-relevant properties of a research study in a compact, standardized form. In the NLP for requirements engineering literature, it was introduced as a lightweight “identity card” for a paper that collects 47 items in seven topics, with the explicit goal of making it possible to re-produce datasets, re-implement tools, assess replication readiness, and reduce the guesswork that often accompanies empirical reconstruction. Subsequent work has used the template to audit concrete replications, while broader replication-methodology research has situated ID-card-like reporting within a design space that characterizes experiment, data, participant, and analysis choices through explicit comparison levels (Abualhaija et al., 2023, Bhatt et al., 29 Jul 2025, Liang et al., 5 Mar 2026).
1. Origins and Motivation
The ID-Card emerged from a specific replication problem in requirements engineering, where natural language processing had become widely used for tasks such as classification and ambiguity detection, but replication activity remained limited. The motivating diagnosis was that replication in NLP4RE is hampered by context specificity, heterogeneity of tasks, the tasks’ inherent hairiness, and a heterogeneous reporting structure. The proposed response was not a new learner or benchmark, but a reporting artifact designed to surface the information most often missing when researchers attempt to reproduce datasets or reconstruct tools (Abualhaija et al., 2023).
Within that formulation, the ID-Card serves several purposes simultaneously. It is intended to make explicit every piece of information needed to re-produce datasets, including annotation protocols, context, and inter-rater agreement, and to re-implement tools, including algorithms, parameters, libraries, and documentation. It is also positioned as a support instrument for newcomers surveying the state of the art, for reviewers and readers assessing replicability, and for educators using papers as structured teaching material. The underlying premise is that many replication failures are not attributable to theoretical disagreement, but to incomplete operational reporting. This suggests that the ID-Card is best understood as a transparency mechanism rather than a methodological substitute (Abualhaija et al., 2023).
2. Information Structure and Coverage
In its NLP4RE form, the ID-Card is a questionnaire-style template comprising 47 replication-relevant items organized into seven topical sections. The instrument was refined to 32 closed and 15 open items, and all questions permit free-text elaboration where needed. Its structure separates the substantive task from the computational realization, and both from the evaluation design, which is central to its replication function (Abualhaija et al., 2023).
| Section | Main focus | Typical contents |
|---|---|---|
| I. RE Task | Problem scope | Requirements Classification, Tracing, Defect Detection, App Reviews, Requirements IE |
| II. NLP Task(s) | NLP operation | Classification, Translation, Information Extraction, Information Retrieval, Other |
| III. NLP Task Details | Task instantiation | Input granularity, binary vs. multi-class, labels, output type, cardinality |
| IV. Data and Dataset | Data provenance and form | Size, source, abstraction level, format, language, domain, availability, license |
| V. Annotators and Annotation Process | Annotation protocol | Annotator count, background, guidelines, context, fatigue mitigation, agreement metrics |
| VI. Tool (Implementation) | Executable artifact | Solution type, algorithms, release mode, install/run steps, documentation, dependencies |
| VII. Evaluation | Empirical assessment | Metrics, validation strategy, baseline type and details |
Several sections were explicitly designed to map onto known replication bottlenecks. Sections IV and V address dataset-annotation challenges by requiring disclosure of theory source, annotator expertise, protocol details, training materials, benchmark linkage, class balance, context provision, and conflict resolution. Section VI targets tool-reconstruction challenges by capturing missing implementation details, data provenance, library versions, maintenance status, and hosting information. Sections I–III and VII reduce ambiguity about task boundaries, input/output granularity, and evaluation repeatability. The recommendation is to fill one card per distinct RE sub-task, even when a paper contains multiple NLP steps (Abualhaija et al., 2023).
3. Design-Science Construction and Validation
The ID-Card was developed through a structured, iterative design-science process following Wieringa’s design-science cycle. The process began with hands-on problem investigation in two concrete replication settings: re-annotation of discourse-ambiguity data and reconstruction of a functional versus non-functional requirements classifier. These exercises grounded the artifact in failure modes encountered during actual replication attempts rather than in abstract reporting desiderata alone (Abualhaija et al., 2023).
The next stages combined focus-group elicitation with iterative schema refinement. One focus group identified 10 annotation-related challenges, while another identified 6 tool-reconstruction challenges. On that basis, pairs of researchers drafted an initial 56-question schema over seven topics. An internal assessment on 46 representative NLP4RE papers then led to consolidation and refinement into the 47-item version. External assessment followed with 15 original-paper authors, who independently filled the ID-Card for their own paper and completed a short TAM-inspired survey; a plenary discussion was then used to resolve ambiguities and tune question phrasing. The final focus group validated completeness and usability, and mapped remaining gaps to real-world replication issues. In methodological terms, the ID-Card is therefore both an artifact and a codified record of what repeatedly proved missing in practice (Abualhaija et al., 2023).
A notable feature of this construction is that the artifact is deliberately lightweight. The paper does not introduce a formal modeling language or data-schema formalism for the card itself; it remains a questionnaire, not an ontology. That design decision preserves portability across heterogeneous studies, although it also means that comparability depends on disciplined completion rather than machine-enforced schema compliance. This suggests a trade-off between expressive flexibility and normalization (Abualhaija et al., 2023).
4. Relation to Replication Design Spaces
A broader methodological generalization appears in the later design-space framework for replication-related studies, which treats replication experimental design as a pairwise comparison problem. In that framework, a replication is represented by four practical dimensions—Experiment, Data, Participant, and Analysis—each compared to the reference study at one of three levels: identical, similar, or different. The notation is given as
The framework is intended for both retrospective characterization and prospective planning, and includes operationalization checklists, examples, and an ID-Card template that records study pair, component levels, rationale, overall design tuple, collapse rules, and additional notes (Liang et al., 5 Mar 2026).
The four dimensions correspond to stages of the evidence pipeline. Experiment covers tasks, stimuli, materials, environment, timing, and instructions. Data covers whether one reuses the original dataset, collects new measurements of the same variables, or collects a different kind of evidence. Participant records whether the sampled population is identical, similar, different, or not applicable. Analysis records whether the inferential or qualitative procedure is identical, commensurate, or fundamentally different. Two collapse rules are central: if , then Experiment and Participant collapse to NA because no new experiment or participants are needed; if the study is purely computational, then (Liang et al., 5 Mar 2026).
The relation between this framework and the NLP4RE ID-Card is complementary rather than redundant. The earlier ID-Card concentrates on replication-enabling metadata for datasets, annotation, tools, and evaluation; the later design space provides a higher-level language for expressing how a replication differs from its reference study. A plausible implication is that the two artifacts address different granularity levels: one is optimized for operational reproducibility, the other for comparative characterization of replication scope (Liang et al., 5 Mar 2026).
5. Use as a Replication-Readiness Instrument
The ID-Card has also been used as an assessment instrument in later replication work. In the replication of Mekala et al.’s study on automatic classification of user requirements from online feedback, one research question explicitly asked how complete and detailed the baseline package was when measured by the Abualhaija et al. replication ID-card template. The resulting assessment found that dataset description, annotation process, tool release, and evaluation metrics were well documented, while environment specification, specific library versions, and reproducibility scripts were missing. The overall readiness judgment was “Almost replication-ready,” and the replication package added a requirements.txt and a Conda environment.yml so that others could install dependencies in one step (Bhatt et al., 29 Jul 2025).
That case illustrates several features of the ID-Card in operation. First, the card functions as an audit checklist rather than merely as a literature-summary form. Second, it can expose omissions that do not invalidate a paper’s results but materially affect end-to-end reproducibility, such as missing environment files and version pinning. Third, it can be extended into the replication package itself: the same study reports a replication study ID-card for the baseline study and a further card for the replication, thereby turning the artifact into part of the reproducibility deliverable rather than an after-the-fact commentary. This suggests that the ID-Card is particularly useful when public code and data already exist, but the execution context remains underspecified (Bhatt et al., 29 Jul 2025).
The same study also demonstrates that the ID-Card can coexist with conventional empirical reporting. The replication reports concrete datasets, preprocessing, environment setup, dependencies, metrics, and results, while the card provides a normalized summary of those details. In this sense, the artifact does not replace the method section; it indexes it for replicators (Bhatt et al., 29 Jul 2025).
6. Scope, Uses, and Limits
The proposed uses of the ID-Card extend beyond one subfield. In the NLP4RE formulation, it can be used in systematic literature reviews to tag candidate papers with replication attributes, in graduate courses as a scaffold for reading papers, in artifact-evaluation tracks to standardize badge-awarding decisions, and as a first step toward a public registry of artifacts. The guidance also recommends that, when authorship permits, original paper authors complete the ID-Card at submission time and archive it alongside the paper in a public repository or as supplementary material. The stated rationale is that earlier completion minimizes guesswork and increases consistency of reporting (Abualhaija et al., 2023).
At the same time, the literature is explicit about limits. The ID-Card is not a guarantee of reproducibility; it is a mechanism for exposing what would otherwise remain implicit. The broader replication-design framework likewise emphasizes that replication involves design decisions about which components remain the same and which are altered, and that these decisions must be categorized rather than hidden. The recommendation to use the card both prospectively and retrospectively reflects that dual role: it can guide planning before the replication begins and characterize scope once it is complete (Liang et al., 5 Mar 2026).
A recurring misconception is to treat the ID-Card as either a bibliographic metadata sheet or a formal replication taxonomy. The literature supports neither reduction. In the NLP4RE work, it is a field-tested questionnaire centered on practical barriers to re-production and reconstruction; in the broader design-space work, it becomes a compact representation of comparison choices across Experiment, Data, Participant, and Analysis. Taken together, these formulations show the replication study ID-Card as a family of structured reporting devices whose common function is to make replication-relevant design information explicit, inspectable, and reusable across studies (Abualhaija et al., 2023, Liang et al., 5 Mar 2026).