CC30k: Citation Contexts for Reproducibility
- CC30k is a curated citation-context dataset that defines reproducibility-oriented sentiment by labeling contexts as positive, negative, or neutral.
- It comprises 30,734 annotated contexts from ML literature, enabling large-scale analysis of reproducibility discourse and citation practices.
- A controlled negative augmentation pipeline addresses class imbalance, ensuring robust labeling and improved baseline performance in reproducibility studies.
CC30k is a citation-context dataset for reproducibility-oriented sentiment analysis in machine learning literature. It comprises 30,734 citation contexts, each labeled as Positive, Negative, or Neutral according to whether the citing text suggests successful reproducibility or replicability, indicates irreproducibility or failed reproduction, or provides no reproducibility signal. The dataset was introduced to support large-scale computational study of reproducibility discourse in ML/AI papers, where conventional sentiment resources and citation-function corpora are not aligned with the scientific semantics of reproducibility claims (Obadage et al., 11 Nov 2025).
1. Concept and label semantics
CC30k is defined around citation contexts rather than full papers or generic opinion text. A citation context is a sentence or fragment surrounding a citation, and the task is to infer a reproducibility-oriented sentiment (ROS) from that local discourse. The three labels are explicitly defined as follows: Positive denotes that the context suggests successful reproducibility or replicability, such as reuse of the data, code, methods, or concepts from the cited paper; Negative denotes that the context hints at irreproducibility or irreplicability, such as unavailable code or data or failed attempts to obtain results; Neutral denotes that the paper is cited without any reproducibility implication (Obadage et al., 11 Nov 2025).
This framing differentiates CC30k from ordinary polarity corpora. The labels are not intended to capture approval, affect, or rhetorical stance in the broad sense used in review or social-media sentiment analysis. Instead, they operationalize a domain-specific scientific signal: whether the citing language communicates evidence about the cited work’s reproducibility or replicability. This suggests that ROS is closer to a structured discourse-analytic variable than to generic lexical sentiment.
The dataset size and final class composition are reported explicitly. Of the 30,734 citation contexts, 25,829 were crowdsourced and 4,905 were introduced through a controlled negative augmentation pipeline. The final label distribution is as follows.
| Label group | Count |
|---|---|
| Positive (C) | 5,102 |
| Neutral (C) | 20,448 |
| Negative (C) | 279 |
| Negative (AHV) | 1,055 |
| Negative (AML) | 3,850 |
The total negative count is therefore 5,184. The dataset further reports an average text length of 36.59 words and 1.29 sentences per context. A plausible implication is that the task requires extracting reproducibility cues from short, locally constrained text spans rather than extended argumentative passages.
2. Research motivation and problem setting
CC30k was created to address two problems identified in prior reproducibility-oriented sentiment work: the lack of a large labeled dataset for ROS and the extreme scarcity of negative examples. The earlier manually labeled set cited by the authors contained only 1,937 citation contexts, including just 23 negative examples, which was described as insufficient for training reliable models (Obadage et al., 11 Nov 2025).
The dataset is situated within a broader concern that reproducibility in AI/ML is difficult to assess at scale. Citation contexts had already been identified as a promising signal of downstream community judgments about reproducibility, but available resources were too small and imbalanced for systematic modeling. CC30k therefore treats citation language as an observable proxy for whether a cited result, method, dataset, or code artifact appears reproducible in later literature.
The distinction from traditional sentiment datasets is central to the dataset’s design. Resources such as IMDb, SemEval Twitter sentiment, Twitter US Airline Sentiment, Sentiment140, and SentiGrad measure generic polarity in informal or evaluative text domains. By contrast, CC30k targets a scientific discourse phenomenon. The relevant linguistic cues may include references to reusing code, inability to replicate reported performance, or mere citation without reproducibility content. This suggests that transfer from conventional sentiment classifiers is structurally limited because the label space is defined by scientific practice rather than general affect.
The paper also distinguishes CC30k from citation-analysis datasets such as SciCite or ACL-ARC. Those datasets focus on citation function or intent, whereas CC30k focuses on reproducibility-oriented sentiment. The difference is not merely terminological: citation function asks why a paper is cited, while ROS asks what the citing language implies about the cited work’s reproducibility status.
3. Corpus construction and data cleansing
The corpus creation pipeline begins with source collection. The authors collected metadata and PDF files for 145 reproducibility studies and 130 original studies, drawn mostly from ML reproducibility challenge sources and related repositories. From these materials they identified 13,314 citing papers, yielding 41,244 citation contexts in January 2024 (Obadage et al., 11 Nov 2025).
The source breakdown reported in the paper includes the following venues and collections: ICLR 2019, NeurIPS 2019, MLRC 2020, MLRC 2021, MLRC 2022, and TSR 2023. The counts are given as 4 reproducibility studies and 4 original studies for ICLR 2019 with 651 citations and 2,102 contexts; 10 and 10 for NeurIPS 2019 with 3,364 citations and 9,908 contexts; 23 and 22 for MLRC 2020 with 3,224 citations and 10,798 contexts; 47 and 41 for MLRC 2021 with 2,596 citations and 6,958 contexts; 45 and 37 for MLRC 2022 with 2,881 citations and 9,869 contexts; and 16 and 16 for TSR 2023 with 598 citations and 1,609 contexts.
Citation contexts were extracted using the Semantic Scholar Graph API. Only explicit citation contexts were retained, meaning the text had to contain visible citation marks. The paper then emphasizes a substantial cleansing stage before annotation. Contexts with ambiguous or mixed citation structures were excluded because they complicated attribution of the context to a uniquely identifiable cited paper. The filtering relied on regular-expression parsing and manual rules.
A key quantitative effect of this cleansing stage is that the dataset size decreased from 41,244 raw citation contexts to 25,829 clean contexts for crowdsourcing. The retained filtered set includes three scenario types: 20,830 single citation mark contexts, 4,871 contexts with multiple APA-like citation marks, and 128 contexts containing only the first author’s name. Certain formats were excluded explicitly, including mixed structures such as [1,2,3] text text text [4], because they could not uniquely identify the cited paper.
This filtering strategy is methodologically important because ROS annotation depends on unambiguous linkage between the cited discourse and the cited paper. A plausible implication is that the authors treated citation disambiguation as part of the label-validity problem rather than as a downstream preprocessing convenience.
4. Annotation protocol and quality control
The crowdsourcing pipeline was designed to constrain annotator quality before large-scale labeling. Rather than exposing the task to unrestricted workers, the authors created a pilot dataset of 20 citation contexts with expert ground-truth labels, consisting of 10 real contexts from the eventual dataset and 10 dummy contexts whose labels were obvious (Obadage et al., 11 Nov 2025).
Workers were required to be Mechanical Turk Masters. The pilot was run repeatedly in batches, each time admitting only workers who had not participated before, with the aim of expanding a vetted worker pool. This process produced a pool of 138 workers. From that pool, 16 workers were selected for the main task. The selection criteria were explicit: each selected worker had to label at least 15 pilot contexts, achieve over 90% accuracy, and correctly label all dummy contexts.
The 25,829 cleaned citation contexts were then annotated on Amazon Mechanical Turk. Each context received labels from three independent annotators, and the final label was determined by majority vote, either 2/3 agreement or 3/3 agreement. The job was posted in 26 batches of about 1,000 contexts, and the interface used MTurk Crowd HTML Elements.
For label verification, the authors manually validated about 1% of the crowdsourced portion: 244 citation contexts. The sample was selected by stratified sampling to preserve label proportions and agreement patterns, and it deliberately included contexts with both 2/3 and 3/3 agreement. In that validation subset, the comparison between majority-vote labels and human ground truth was reported as Negative: 55 vs 55, Positive: 102 vs 99, Neutral: 87 vs 75, with 244 ground-truth items versus 229 majority-voted items after removing 15 mislabels.
From this validation, the reported overall labeling accuracy is 93.85%, described as 94%. The paper also reports Macro F1 = 0.94 and Weighted F1 = 0.94 on the validation subset, with per-class precision, recall, and F1 values of 0.89, 1.00, and 0.94 for Negative; 0.96, 0.86, and 0.91 for Neutral; and 0.95, 0.97, and 0.96 for Positive. Two additional observations are emphasized: all mislabels occurred only in 2/3 agreement cases, and no mislabels were found in 3/3 agreement cases.
The reported agreement statistics are unusual in combination: overall percent agreement among annotators is 99.35%, whereas Krippendorff’s (ordinal) is 0.29. The paper interprets the low as an artifact of extreme class imbalance, especially the dominance of Neutral labels, rather than as evidence of poor annotation quality. This is a standard type of tension in imbalanced annotation settings, where raw agreement can be very high while chance-corrected agreement remains modest.
5. Negative augmentation and class-imbalance handling
Class imbalance, especially the scarcity of negative examples, is addressed through a controlled augmentation pipeline. The authors first harvested 692,604 citation contexts from 21,757 computer science papers published after 2017. These were classified using an ensemble of five transformer models fine-tuned on the crowdsourced data: SPECTER, SciBERT, DistilBERT, BioBERT, and BlueBERT (Obadage et al., 11 Nov 2025).
The ensemble reached a weighted average F1 of 0.81 and identified 43,790 negative candidates. From this candidate set, a random sample of 5,578 was manually checked, producing 1,055 true negative citation contexts. From the remaining verified non-negative contexts, 1,055 were randomly selected as additional ground-truth-like samples. These 2,110 samples were then used to fine-tune several binary classifiers: SciBERT, RoBERTa, DistilBERT, DeBERTa, and GPT-4o in zero-shot and few-shot settings. The best binary classifier was RoBERTa, with F1 = 0.67.
That RoBERTa model was then used to filter the remaining 38,212 candidate negatives. It produced 3,850 high-confidence candidates with probability greater than 0.99. Combined with the manually verified negatives, this yielded 4,905 new negative contexts added to the dataset. The paper distinguishes two augmented negative subsets: AHV, meaning augmented and human-validated, and AML, meaning augmented and machine-labeled. It also introduces a label_type field with the values crowdsourced, augmented_human_validated, and augmented_machine_labeled.
This pipeline is the dataset’s principal response to the negative-class bottleneck. It does not eliminate imbalance, and the authors explicitly acknowledge that part of the negative class remains machine-labeled. The resulting dataset should therefore be understood as combining directly crowdsourced labels with controlled augmentation rather than as a purely manual gold standard. A plausible implication is that model-development work on CC30k may need to account explicitly for label provenance.
6. Baselines, downstream utility, and nomenclature
The paper reports that generic sentiment models perform poorly on ROS classification. On the 25,829 crowdsourced labels, several open-source HuggingFace sentiment classifiers obtain macro F1 scores only in the 0.30–0.40 range: a RoBERTa-based model reaches 0.37, a BERT-based model 0.36, BERTweet 0.40, BERT AutoTrain 0.37, and BERT sbcBI 0.30 (Obadage et al., 11 Nov 2025). This result supports the claim that ordinary sentiment analysis models do not capture reproducibility-oriented sentiment effectively.
The authors then evaluate three LLM setups under zero-shot, few-shot, and fine-tuned settings using 3k, 9k, and 15k labeled examples from CC30k. For Qwen1.5-7B, the reported weighted average F1 values are 0.436 for base zero-shot, 0.539 for base few-shot, 0.625 for fine-tuned few-shot with 3k examples, 0.695 for fine-tuned few-shot with 9k examples, and 0.428 for fine-tuned few-shot with 15k examples. For LLaMA-3-8B, the best reported result is 0.671 with 3k training samples under few-shot prompting, with 0.658 at 9k few-shot and lower performance at 15k. For GPT-4o with retrieval-augmented generation, the best result is 0.786 with 3k training samples under zero-shot RAG, and the second best is 0.767 with 9k few-shot RAG.
The paper characterizes the overall improvement from fine-tuning on CC30k as roughly 5% to 27% over base direct inference. It also notes that improvements are not strictly monotonic with additional training data and attributes this partly to label noise and task difficulty. This suggests that CC30k is useful both as a benchmark and as evidence that ROS is a distinct, nontrivial classification problem.
The intended uses listed for CC30k include training classifiers for reproducibility-oriented sentiment detection, large-scale analysis of reproducibility discourse in ML/AI literature, placing ROS-labeled citation contexts into citation graphs, identifying patterns in reproducibility practices, and supporting reproducibility-aware data or software recommendation. The dataset and Jupyter notebooks are publicly available at https://github.com/lamps-lab/CC30k, and the dataset is distributed as a structured CSV file with 37 columns that include the citation context, worker labels, majority vote, agreement level, bibliographic metadata, and label source.
A nomenclature issue merits clarification. CC30k refers to the citation-context dataset introduced in “CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis” (Obadage et al., 11 Nov 2025). It should not be conflated with “ccHarmony,” a color-checker-based image harmonization dataset derived from the phrase “color checker (cc)” (Huang et al., 2022). The provided record for ccHarmony explicitly states that there is no mention there of “CC30k” or a 30k-scale variant.