Sentence-Level Citation Extraction

• Sentence-level citation extraction is the process of automatically identifying citation-worthy sentences using sequence labeling methods and context-aware features.
• Hierarchical models like BiLSTM with attention and transformer-based schemes (e.g., SelfCite) significantly boost citation F1 scores by integrating local and section-level context.
• This extraction technique enhances downstream applications such as fact-checking, scholarly writing assistance, and citation function analysis.

Sentence-level citation extraction is the automatic identification of sentences within scientific texts that require, or are supported by, specific references to other scholarly sources. In contemporary research and retrieval-augmented generation (RAG) applications, accurate sentence-level citation extraction is essential for both factual verification and the support of claims in machine-generated or human-written content. The task is formalized as a sequence labeling or binary classification problem: for each sentence, the system predicts whether it is “citation-worthy,” i.e., whether it should be attributed to an external source.

1. Problem Formulation and Definitions

Let a scientific document $d$ be decomposed into an ordered sequence of $n$ sentences $S = [s_1, s_2, \ldots, s_n]$. The objective is to infer a binary sequence of labels $y = [y_1, y_2, \ldots, y_n]$ where $y_i = 1$ denotes that sentence $s_i$ requires a citation, and $y_i = 0$ otherwise. The model’s predicted probability for $y_i$ depends on both the target sentence and a context window, typically including neighboring sentences or section headers. The learning objective is to minimize the total cross-entropy or regularized cross-entropy loss over all sentences:

$$\mathcal{L}(\theta) = -\sum_{i=1}^n [y_i \log p_\theta(y_i = 1 \mid S) + (1 - y_i) \log (1 - p_\theta(y_i = 1 \mid S))]$$

Citation extraction supports downstream tasks, including automatic fact-checking, scholarly writing assistants, and citation function analysis (Gosangi et al., 2021, Zeng et al., 2024).

2. Model Architectures and Methodologies

Two principal classes of models have achieved state-of-the-art results: hierarchical BiLSTM-based sentence classifiers with attention and hybrid transformer-hierarchical schemes.

BiLSTM with Attention

The canonical model encodes each sentence $s_i$ into a vector with pretrained embeddings (e.g., GloVe, RoBERTa). Contextualization is achieved through a bidirectional LSTM operating over both the local sentence window and, optionally, section identity features. Attention mechanisms aggregate token-level or sentence-level representations, enhancing the model’s focus on citation-relevant phrases (Zeng et al., 2024).

Hierarchical Sequence Models

More recent architectures use a hierarchical pipeline: sentence encodings using transformers (e.g., RoBERTa) are passed to a document-level BiLSTM, forming context-aware representations. A feed-forward classifier or sigmoid layer outputs the citation-worthiness probability for each sentence. Context size matters: models using ±2 neighboring sentences and section headers realize significant F1 gains compared to sentence-only baselines (Gosangi et al., 2021).

Context Ablation and Self-Supervised Preference Alignment

Emerging approaches employ LLMs and self-supervision. In SelfCite, a context ablation strategy is used: for each proposed citation, the model measures the sensitivity of the generated statement to the cited context using conditional log-probability differences—quantifying both necessity and sufficiency. This reward signal guides best-of-N citation selection at inference and preference optimization during fine-tuning (Chuang et al., 13 Feb 2025).

Model	Principal Feature	Citation F1 (best)	Data Used
Att-BiLSTM	Attention + GloVe + Context	0.507 / 0.856	ACL-ARC / PMOA-CITE
SSM + RoBERTa	Transformer + BiLSTM (window=16)	0.723	ACL-cite
SelfCite	Self-supervised LLM, context ablation	0.791	LongBench-Cite

3. Input Representations and Features

Citation extraction models utilize both neural and engineered features:

• Word and Character Embeddings: 300-dim GloVe or transformer-based word vectors; character-level BiLSTM representations are concatenated (Zeng et al., 2024).
• Section Title Embeddings: Section names are encoded to capture disciplinary norms (e.g., “Introduction” typically needs more citations).
• Contextual Windowing: Surrounding sentences ($n$0, $n$1) and section headers are explicitly integrated.
• Handcrafted Features: Binary indicators for citation presence in neighboring sentences, sentence length, similarity measures between adjacent sentences.

4. Datasets and Evaluation Protocols

Multiple large-scale datasets enable robust evaluation:

• SEPID-cite: 1.23M sentences, 7.0% citation-worthy.
• PMOA-cite: 1.01M sentences, 19.5% citation-worthy, extracted from biomedical full texts (Zeng et al., 2024).
• ACL-cite: 2.7M sentences (11.3% citation-worthy), with document-level context, supports long-range dependency modeling (Gosangi et al., 2021).
• LongBench-Cite: Multiple QA and summarization datasets annotated for fine-grained, statement-level citations (Chuang et al., 13 Feb 2025).

Metrics are standard: precision, recall, and F1 score for the citation-worthy class. Bootstrapped significance testing is employed to verify improvements.

5. Empirical Results and Error Analysis

In comparative studies:

• Contextual attention-based BiLSTM achieves $n$2 on ACL-ARC, and $n$3 on PMOA-CITE with section/neighbor context (Zeng et al., 2024).
• Hierarchical SSM + RoBERTa models push $n$4 to 0.730 on ACL-cite (Gosangi et al., 2021).
• SelfCite, integrating self-supervised reward signals and BoN inference, improves citation F1 by up to +5.3 points over strong LLM baselines—reaching $n$5 on LongBench-Cite (Chuang et al., 13 Feb 2025).

Significant gains are attributed to wider context modeling (±16 sentences yields +5.6 points F1 over ±2), integration of section headers (+0.7 F1), and utilization of transformer-based embeddings.

Error analysis reveals high-impact categories:

• Ambiguous citation needs (multi-clause references disambiguated by neighbors)
• Section disambiguation (section type shifts citation norms)
• Topic continuity (first technical mention requires citation; elaborations do not)

Some predicted “errors” are genuine annotation or citation omissions, indicating the system can surpass noisy human-derived gold standards (Zeng et al., 2024).

6. Interpretability, Transfer, and Practical Utility

Interpretable models reveal that section titles (“introduction,” “background”) and lexical indicators (“studies,” “described previously”) strongly predict citation-worthiness, while references to figures, tables, or standard methods negatively correlate (Zeng et al., 2024). Random forest analyses confirm “neighbor-citation” flags as dominant features.

Cross-domain transfer performance shows that citation-worthiness models trained on biomedical or computational linguistics corpora generalize imperfectly but exhibit promising robustness. Public tools expose model predictions for author- or editor-facing review, supporting integration into manuscript workflows.

7. Future Directions and Open Challenges

Current bottlenecks include:

• Annotation quality and coverage for fine-grained (sub-sentence or statement) citation extraction
• Scalability: BiLSTM architectures are memory- and time-intensive for long sequences; transformer-based long-context architectures (e.g., Longformer, BigBird) offer a promising alternative (Gosangi et al., 2021)
• Limited “what to cite” modeling: present systems mostly flag “where” a citation belongs rather than recommending specific sources
• Generalization beyond scientific writing: adaptations for legal, news, or Wikipedia genres require domain-specific retraining or adaptation (Zeng et al., 2024)

Recent advances, particularly self-supervised context ablation and LLM preference optimization, expand the potential for citation extraction where human-annotated data are limited (Chuang et al., 13 Feb 2025). Ongoing integration with factual consistency checking and RAG systems continues to drive the field.

---

References:

• (Gosangi et al., 2021) On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles
• (Zeng et al., 2024) Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models
• (Chuang et al., 13 Feb 2025) SelfCite: Self-Supervised Alignment for Context Attribution in LLMs