---
title: SCIDOCA 2025 Shared Task
url: https://www.emergentmind.com/topics/scidoca-2025-shared-task
type: topic
---

# SCIDOCA 2025 Shared Task

Searching arXiv for the specified SCIDOCA 2025 papers to ground the article in the latest available sources.
SCIDOCA 2025 Shared Task is a benchmark initiative for citation prediction, discovery, and placement in scientific documents under constrained input conditions, using paragraph-level contexts together with only the titles and abstracts of candidate references rather than full-text candidate papers [2509.24283]. It is positioned at the intersection of citation recommendation, citation-context analysis, and scientific-text understanding, and is organized into three subtasks of increasing granularity: Citation Discovery, Masked Citation Prediction, and Citation Sentence Prediction [2509.24283]. The task was designed to support scientific writing and document understanding by establishing a large, rigorously curated, publicly released dataset and evaluation protocol, while also exposing the distinctive difficulty of identifying relevant citations from contextually plausible distractor abstracts [2509.24283].

## 1. Origin, motivation, and scope

SCIDOCA stands for Scientific Document Citation and was introduced to address a specific limitation in prior citation-recommendation and context-analysis systems: many assume full-text access to candidate papers, including body text, figures, and full reference lists, whereas real-world settings such as arXiv, preprint servers, open-access repositories, and web-based search engines often provide only metadata and abstracts [2509.24283]. The shared task therefore benchmarks citation discovery and placement in an information-sparse regime defined by paragraph contexts and title-plus-abstract candidate representations [2509.24283].

Its stated objectives are fourfold: to help researchers rapidly surface and attribute the most relevant prior work when drafting or reviewing a scientific paragraph; to lay groundwork for writing and editing tools that can automatically insert or repair citation markers; to advance understanding of citation-intent modeling and fine-grained context analysis; and to establish a standard benchmark in scientific document understanding [2509.24283]. Within this framing, SCIDOCA 2025 is not only an evaluation exercise but also a dataset release and protocol standardization effort.

A plausible implication is that the benchmark explicitly tests whether citation models can remain effective when deprived of the rich structural signals available in full-text corpora. This makes the task particularly relevant for retrieval systems deployed over heterogeneous or incomplete scholarly repositories.

## 2. Task design and formal evaluation

The shared task is divided into three subtasks, each defined over a paragraph or sentence context and a candidate pool of references [2509.24283].

**Subtask 1: Citation Discovery** takes as input a citation-free paragraph plus a pool of candidate references, each represented by title and abstract, and requires systems to output the subset of candidates that truly support or are cited by the paragraph [2509.24283]. Evaluation is based on paragraph-level Precision, Recall, and harmonic-mean F1. For paragraph \(i\), with \(TP_i\), \(FP_i\), and \(FN_i\) denoting true positives, false positives, and false negatives, the task defines:
\[
\mathrm{Precision}_i=\frac{TP_i}{TP_i+FP_i},
\qquad
\mathrm{Recall}_i=\frac{TP_i}{TP_i+FN_i},
\qquad
\mathrm{F1}_i=2\frac{\mathrm{Precision}_i\cdot \mathrm{Recall}_i}{\mathrm{Precision}_i+\mathrm{Recall}_i}.
\]
The overall score is the gold-weighted average:
\[
\mathrm{F1}=\frac{\sum_i GT_i \times F1_i}{\sum_i GT_i},
\qquad
GT_i=\text{number of gold citations in } i.
\]
[2509.24283]

**Subtask 2: Masked Citation Prediction** provides a paragraph in which each original citation marker has been replaced by a unique placeholder such as \([MASK\ 1]\), \([MASK\ 2]\), together with the same candidate pool, and requires a mapping from each mask to exactly one candidate reference [2509.24283]. Evaluation computes F1 per mask as a standard classification label, macro-averages across masked slots, and then weights by the number of masks \(M_i\) in each paragraph [2509.24283].

**Subtask 3: Citation Sentence Prediction** takes a sequence of sentences with citation markers removed and asks systems to determine, for each sentence, either an empty list or one or more citations from the pool [2509.24283]. For sentences that originally had citations, sentence-level Precision, Recall, and F1 are computed as in Subtask 1; for sentences without citations, the benchmark reports no-citation accuracy
\[
\mathrm{Acc}_{\mathrm{no\usc cit}}=\frac{1}{|S_{nc}|}\sum_{i\in S_{nc}} \mathbb{1}\bigl(\hat y_i=\emptyset\bigr),
\]
and a Micro-F1 over all sentences is also reported to address class imbalance [2509.24283].

The overview paper also notes standard retrieval metrics for completeness, including Recall@K, Mean Reciprocal Rank, and Mean Average Precision, although SCIDOCA’s official scoring emphasizes precision/recall/F1 formulations [2509.24283].

## 3. Dataset construction and benchmark characteristics

All data are drawn from the Semantic Scholar Open Research Corpus (S2ORC) [2509.24283]. The construction pipeline begins with 8.1 million full-text papers and applies multi-stage filtering and curation to produce three resource layers: a Primary Paper Set of 10,000 papers for context generation, a Curated Reference Set of 10.15 million target papers from which correct citations are drawn, and a Comprehensive Reference Set of 75.56 million papers to enlarge distractor coverage [2509.24283].

After automated alignment of citation spans to paragraphs in the primary set, the organizers obtained a train split of 61,556 paragraphs and a test split of 1,000 paragraphs [2509.24283]. The train split has an average of 5.62 sentences per paragraph, 2.11 gold citations, and 18.04 candidate references per paragraph, with paragraph lengths ranging from 1 to 86 sentences, gold citations from 1 to 36, and candidate pools up to 151 [2509.24283]. The test split contains 1,000 paragraphs, each sampled from a different paper to avoid cross-context leakage, and distractors are taken from other references of the same paper outside the target paragraph so that all candidates are contextually plausible [2509.24283].

No manual annotation was used; labels and distractors are derived from original S2ORC citation spans and reference lists [2509.24283]. This design choice has two consequences evident from the benchmark description. First, it enables scale. Second, it makes negative candidates comparatively difficult, because they originate from the same paper’s broader reference environment rather than from arbitrary unrelated documents.

The task-specific description provided by Team LA for Subtask 1 presents the same problem from the system perspective: a single long paragraph \(P\), typically a self-contained abstract or excerpt of 100–300 words, is paired with a candidate pool \(C=\{c_1,c_2,\dots,c_N\}\) of distractor abstracts, and the goal is to predict the single ID \(i^\ast\) such that \(c_{i^\ast}\) is the true source for \(P\) [2506.18316]. Their summary reports that most paragraphs span 5–8 sentences, approximately 150–300 tokens, and that candidate-pool size varies per query but is typically on the order of 20–50 abstracts [2506.18316]. This narrower description reflects the operational conditions encountered by one participating system on Citation Discovery.

## 4. Participating systems and methodological patterns

Seven teams registered for SCIDOCA 2025, and three teams—LA, VN25, and ZLB—submitted results [2509.24283]. The submitted systems illustrate three distinct methodological families.

**Team LA** participated in Task 1 only [2509.24283]. The overview describes their approach as combining multi-stage filtering through TF–IDF lexical overlap, dense retrieval for semantic matches, and an LLM-driven relation extractor to reduce the candidate set, followed by an LLM in a two-step prompt chain—prediction with rationale, then rationale simplification—to decide citation relevance [2509.24283]. The team paper characterizes the method as a two-step, relation-centred, zero-shot pipeline: document-level triples are extracted via an LLM, top-\(k\) abstracts are retrieved by triple similarity, and the same LLM reranks those \(k\) candidates to produce the final citation [2506.18316].

In Team LA’s formulation, both the query paragraph \(P\) and each candidate abstract \(c_j\) are converted into sets of relation triples \((\text{head}, \text{relation}, \text{tail})\) using a “relation-extracting prompt” with Mistral-7B-Instruct-v0.3 [2506.18316]. The triples are vectorized as binary feature vectors over the union of observed triples:
\[
\mathbf{r}_p \in \{0,1\}^d,\qquad
[\mathbf{r}_p]_i = 1 \text{ if } t_i \text{ appears in } P,
\]
\[
\mathbf{r}_{c_j} \in \{0,1\}^d,\qquad
[\mathbf{r}_{c_j}]_i = 1 \text{ if } t_i \text{ appears in } c_j.
\]
The summary notes that the paper does not give an explicit similarity formula, but identifies cosine similarity as the natural choice and also mentions Jaccard or simple dot product as alternatives [2506.18316]. The retrieval stage uses only these relational features, with no fine-tuning, and experiments consider \(k \in \{10,15,20\}\) [2506.18316].

The same Mistral-7B-Instruct-v0.3 model is then used unchanged in a second pass for final ranking [2506.18316]. The prompt receives the original paragraph plus the retrieved abstracts and asks which abstract ID best supports or describes the paragraph’s claims, returning a single best ID [2506.18316].

**Team VN25** adopted a Longformer-based encoder with up to 4,096 tokens to embed paragraphs and candidate titles and abstracts, then concatenated text and candidate representations into a binary classification head [2509.24283]. They extended this architecture to all three subtasks by re-pairing inputs, such as single-mask contexts for Subtask 2 and sentence candidates for Subtask 3 [2509.24283].

**Team ZLB** framed discovery and sentence prediction as retrieval-and-rerank tasks using pre-built transformer encoders, specifically bge-large-en-v1.5 and gte-qwen2-1.5b-instruct, for initial semantic similarity scoring, and then a cross-encoder reranker, bge-reranker-large, fine-tuned on SCIDOCA to refine top candidates [2509.24283]. The overview reports that the cross-encoder was especially effective in sentence-level placement [2509.24283].

Taken together, the three systems instantiate a spectrum from zero-shot LLM-centric relation extraction, to long-context supervised encoding, to retrieval-plus-cross-encoder reranking. This suggests that SCIDOCA 2025 functions not only as a benchmark for citation modeling but also as a comparative testbed for retrieval architectures under abstract-only evidence constraints.

## 5. Empirical performance

The official overview reports the following results across subtasks [2509.24283].

| Subtask | Team | Reported result |
|---|---|---|
| Citation Discovery | LA | Precision 42.55%, Recall 81.65%, F1 50.75% |
| Citation Discovery | ZLB | Precision 49.26%, Recall 35.92%, F1 36.87% |
| Citation Discovery | VN25 | Precision 28.90%, Recall 15.54%, F1 16.68% |
| Masked Citation Prediction | ZLB | Precision = Recall = F1 28.86% |
| Masked Citation Prediction | VN25 | F1 = 6.60% |
| Citation Sentence Prediction | ZLB | Precision 21.13%, Recall 34.24%, F1 23.27%, No-citation Acc 67.00% |
| Citation Sentence Prediction | VN25 | Precision 7.93%, Recall 21.62%, F1 10.40%, No-citation Acc 49.21% |

For Subtask 1, the overview concludes that Team LA ranked first with very high recall, whereas ZLB achieved the best balance in Task 1 and dominated the more structured Masked Citation Prediction and Citation Sentence Prediction settings through cross-encoder reranking [2509.24283]. VN25’s results are interpreted in the overview as showing that long-range attention alone was insufficient to achieve strong precision without additional reranking or candidate filtering [2509.24283].

Team LA’s system paper provides a more granular analysis of their own retrieval variants on 1,000 held-out queries [2506.18316]. For retrieval alone, the reported table includes TF-IDF-10 with Recall 0.8259, Precision 0.2060, F1 0.3297; dense-10 with Recall 0.8401, Precision 0.2095, F1 0.3354; and relation-10 with Recall 0.8506, Precision 0.2156, F1 0.3440 [2506.18316]. The summary states that relation-based retrieval yields the highest precision, approximately 0.22 at \(k=10\), and the best F1 among pure retrieval methods [2506.18316].

For retrieval plus LLM inference, the same source reports LLM+TF-IDF with Recall 0.4079, Precision 0.2347, F1 0.2980; LLM+Dense Retrieval with Recall 0.3253, Precision 0.2590, F1 0.2884; and LLM+Relation-based Retrieval with Recall 0.4626, Precision 0.2125, F1 0.2912 [2506.18316]. According to that comparison, LLM inference boosts recall substantially, especially when combined with relation retrieval, while the best LLM-augmented F1 in the table is 0.298 with TF-IDF [2506.18316].

There is an apparent difference between the official shared-task leaderboard numbers and the internal figures reported in Team LA’s system summary. The overview presents final shared-task results for Subtask 1, whereas the Team LA paper reports experiments on the training dataset provided by the organizers and on 1,000 held-out queries sampled for evaluation [2506.18316; 2509.24283]. This suggests that the two sources are describing different evaluation setups rather than directly contradictory outcomes.

## 6. Error modes, limitations, and future directions

The shared-task overview identifies several recurrent error sources [2509.24283]. In Task 1, false negatives often arise when key domain-specific terminology is paraphrased far from the candidate abstracts. In Subtask 2, systems struggle to disambiguate multiple citations with the same author names or publication year. In Subtask 3, performance is affected by high class imbalance and by the difficulty of determining whether a methodological or transitional sentence truly requires citation support [2509.24283].

Team LA’s system-specific discussion emphasizes a different but compatible set of limitations [2506.18316]. Their strengths include higher precision than keyword or dense-vector retrieval through relation-based filtering, and zero-shot use of a 7B-parameter LLM for both relation extraction and final ranking without task-specific training [2506.18316]. Their limitations include recall trade-offs when \(k\) is small or relations are incomplete, dependence on LLM quality because incorrect triple extraction can mislead retrieval, and the absence of end-to-end fine-tuning, which limits adaptability if the relation taxonomy shifts [2506.18316]. No ablation on the number of extracted triples or on alternative similarity functions was reported [2506.18316].

The overview paper lists several future directions: tighter integration of rough retrieval with strong cross-encoders or LLM-based rerankers, knowledge-augmented embeddings leveraging citation graphs or topical facets, end-to-end meta-learning for rare or unseen citation styles, and adversarial distractors and domain adaptation for robustness across disciplines [2509.24283]. Team LA’s future directions are more specific to their architecture: hybrid retrieval that dynamically fuses dense or dictionary features with relation filtering based on paragraph complexity, graph-based relational reasoning such as multi-hop traversal over a global triple graph, and fine-tuning the LLM in a retrieval-augmented generation setup to improve extraction and ranking jointly [2506.18316].

A plausible synthesis is that SCIDOCA 2025 exposes a central systems trade-off: broad candidate coverage tends to favor recall-oriented retrieval, whereas precise attribution under dense distractor sets benefits from stronger reranking and richer relational or cross-encoding mechanisms. The shared task’s design makes this trade-off measurable under conditions that are close to real scholarly search environments.

## 7. Significance for citation modeling and scientific document understanding

SCIDOCA 2025 establishes a benchmark for citation modeling in settings where only partial evidence about candidate references is available [2509.24283]. By using paragraph contexts and candidate title-plus-abstract representations, it directly targets deployment scenarios faced by digital libraries, open-access search engines, and scholarly writing assistants [2509.24283]. The dataset, evaluation scripts, sample code, and leaderboard interface are publicly available through the shared task repository cited in the overview paper [2509.24283].

Its benchmark structure is notable for spanning three related but non-identical problems: identifying all relevant citations for a paragraph, assigning specific references to masked citation slots, and predicting sentence-level citation needs and assignments [2509.24283]. This subdivision allows evaluation of broad retrieval ability, slot-level disambiguation, and fine-grained attribution behavior within a unified framework.

The participating systems further indicate that citation modeling under abstract-only conditions remains an open technical problem. Team LA’s relation-based zero-shot retrieval demonstrates the utility of extracting structured scientific relations from long paragraphs and candidate abstracts [2506.18316]. ZLB’s strong results on the more structured subtasks indicate the effectiveness of retrieval-and-rerank pipelines with a fine-tuned cross-encoder [2509.24283]. VN25’s Longformer approach highlights that long-range contextual modeling is useful but not sufficient by itself for precise citation assignment [2509.24283].

In that sense, SCIDOCA 2025 functions simultaneously as a benchmark, a data resource, and an empirical map of current methodological trade-offs in scholarly citation prediction. Its focus on constrained evidence, plausible distractors, and multiple citation granularities makes it a reference point for future work on citation recommendation, context-sensitive retrieval, and automated scientific writing support [2509.24283].

Source: https://www.emergentmind.com/topics/scidoca-2025-shared-task