---
title: 'SciCap Project: Advancing Figure Captioning'
url: https://www.emergentmind.com/topics/scicap-project
type: topic
---

# SciCap Project: Advancing Figure Captioning

The SciCap Project is a multiyear initiative focused on automating the generation of high-quality, contextually informative captions for scientific figures, tables, equations, and related visual objects in scholarly papers. Building on large-scale dataset creation, rigorous modeling, and evaluation—including annual open competitions—the project has shaped technical progress at the intersection of natural language processing, computer vision, and human-centered scientific communication.

## 1. Origins, Goals, and Evolution

The SciCap Project was conceived to address the prevalence of low-quality, often generic or uninformative captions in scientific literature, which diminish both accessibility (notably for visually impaired readers) and the overall interpretability of scientific findings [2110.11624][2512.21789]. Its foundational ambitions were to:

- Release the first large-scale, real-world dataset of figure–caption pairs, primarily culled from arXiv preprints in computer science and related fields.
- Test the hypothesis that domain-specific, context-aware models—beyond standard vision-language models—can more accurately capture the semantics and communicative function of scientific captions.
- Create automated tools to support scientists, with generated captions approaching or exceeding the quality of those written by domain experts.

The project rapidly evolved. SciCap v1.0 (2021) released over 133,000 single-panel graph plots with captions, followed by the incorporation of contextual data (mentioning paragraphs, OCR of in-figure text), culminating in datasets such as SciCap+ (2023) and multimodal LaMP-Cap. By 2023–2025, annual SciCap Challenges attracted the global community, and the corpus grew to include over 476,000 figures spanning eight scientific domains, five principal figure types, and diverse metadata [2512.21789][2501.19353].

## 2. Dataset Construction and Annotation

The SciCap family of datasets is defined by scale, real-world heterogeneity, and multimodal annotation.

- **Source and Filtering**: Initial datasets used arXiv bulk PDF downloads, filtering computer science and machine learning papers (2010–2020), with PDFFigures 2.0 extracting over two million figures. Compound figures were separated using rule-based cues and CNN-based tools (e.g., Compound FigureSeparator).
- **Figure Typing**: Classifiers (notably FigureSeer) assigned figures to categories: graph plots, tables, equations, flowcharts, scatter plots, bar charts, and others [2110.11624].
- **Caption Selection**: Captions were normalized (number/bracket substitutions) and split into "first-sentence," "single-sentence," or "≤100-word" sets to facilitate training and benchmarking.
- **Multimodal Context**: SciCap+ [2306.03491] and subsequent datasets paired figures not only with captions, but also:
  - "Mention paragraphs"—first running-text occurrences citing the figure.
  - OCR tokens and bounding boxes for all in-figure text.
  - Paper- and domain-level metadata (e.g., scientific subfield, figure category, profile figures for style transfer).

Dataset statistics (SciCap+) include~414k train figures, ~10k validation and test each, with caption lengths typically ≤66 words and mention-paragraphs capped at 192 subwords [2306.03491]. The LaMP-Cap corpus (2025) features 110,828 articles, each with one target figure and up to three profile figures with eight modalities of context [2510.07993].

## 3. Modeling Approaches and System Architectures

The project has served as a testbed for diverse captioning methods, ranging from classic vision–language models to large-scale transformer and LLM-based approaches.

- **Vision and Text Baselines**: Early work benchmarked ResNet–LSTM architectures with attention, using either image features, OCR text, or both [2110.11624]. These approaches achieved low BLEU-4 scores (<0.03), indicating difficulty in capturing caption semantics from visual or shallow textual cues alone.
- **Multimodal Transformers**: M4C-Captioner [2306.03491] introduced multimodal, pointer-equipped transformers encoding images (ResNet-152), text (SciBERT for mention paragraphs), and OCR tokens (FastText+PHOC embeddings, Faster R-CNN features, and spatial coords). Generation used a weighted sum of vocabulary and copy distributions.
- **Summarization-Based Models**: The major insight of the SciCap Project is that captions are most often concise contextual summaries of figure-mentioning paragraphs—approximately 75% of caption words appear in the associated text [2501.19353]. Fine-tuned summarizers (Pegasus, Flan-T5-XL, LLaMA) proved highly effective [2512.21789][2407.01897].
- **Auxiliary Information Integration**: Leading entries in SciCap 2024 [2407.01897] incorporated:
  - Accurate OCR (via PaddleOCR)
  - Filtered, informativeness-scored text chunks
  - Object mention spans, with fusion by learned gating in the transformer decoder
  - Ensemble checkpoint ranking for increased robustness
- **Prompt Optimization and Personalization**: The 3rd SciCap Challenge [2510.07993] utilized a two-stage approach:
  - Category-specific prompt optimization (DSPy MIPROv2 and SIMBA) for domain fidelity
  - Few-shot author-style prompting (profile figure-caption pairs) for stylistic consistency, mediated via LLMs
- **Large Multimodal Models (LMMs)**: GPT-4V demonstrated state-of-the-art human-judged captions, outperforming all open and proprietary models as well as original author-written captions [2501.19353]. Notably, reference-based automatic metrics (BLEU, ROUGE) did not correlate with these human preferences.

## 4. Evaluation Frameworks and Benchmarking

Captioning systems are assessed across both automatic, reference-based metrics and human evaluation protocols.

### Automatic Metrics

These include:

- **BLEU-N**: N-gram precision with brevity penalty
- **ROUGE-1/2/L**: Recall and precision of unigrams, bigrams, and longest common subsequences. Normalized variants account for length bias
- **METEOR** and **CIDEr**: Additional n-gram similarity and consensus metrics (used in human–vs–machine annotation studies) [2306.03491][2110.11624]

### Human Evaluation

Protocols have included:

- Expert rater (Ph.D.-level) scales of “helpfulness” (1–6) given context [2512.21789].
- Undergraduate and professional editor rankings; editors overwhelmingly preferred GPT-4V outputs to author or model baselines by large margins [2501.19353].
- LLMs (GPT-4) as zero-shot judges with Kendall's τ correlation up to 0.40 with Ph.D. raters, outperforming non-expert human scorers.

Results have revealed:

- Vision-only or even OCR+vision approaches lag far behind context-summarization baselines.
- Summarization models (Pegasus, Flan-T5-XL, LLaMA) fine-tuned on mentioning paragraphs boost BLEU and ROUGE scores by up to 30% [2512.21789].
- GPT-4V (image+paragraph) captions, despite lower reference metric scores, rank first in human preference in 70% of editor trials, challenging the relevance of n-gram metrics for this task [2501.19353].

## 5. Competition Structure, Signal Approaches, and Results

Annual SciCap Challenges (2023–2025) have catalyzed rapid methodological progress by providing common evaluation scaffolds, new datasets, and human-in-the-loop rankings [2501.19353][2510.07993][2407.01897].

### 2023–2024: System Approaches

| Year    | Winner(s)  | Key Methods                        | Notable Techniques              | Human Score/Rank                  |
|---------|------------|------------------------------------|---------------------------------|-----------------------------------|
| 2023    | NJUST-KMG  | Pegasus summarization + PaddleOCR  | BRIO contrastive ranking        | Norm.ROUGE-2 (4.489)              |
| 2023    | USTC       | Pegasus summarization + BLIP-2     | Caption enrichment              | Norm.ROUGE-2 (2.418)              |
| 2023    | GPT-4V     | Proprietary image+text LLM         | Zero-shot human evaluation      | #1 choice in ≥70% of editor trials |
| 2024    | Ours*      | Pegasus/LLaMA2+OCR+filtering       | Ensemble, gating                | Human: 4.33 (long), 4.66 (short)  |

*2407.01897, reporting first place in both human-rated short and long caption tracks.

### Competition Protocols

- Large-scale, open-access train/test splits (~400k train, ~48k test; 2023 and 2024).
- Two tracks: short captions (≤20 words), long captions (≤50 words), with human and automatic metric leaderboards [2407.01897].
- Evaluation metrics prioritize normalized recall for length fairness.
- “Quality Subset” of reference captions (scored by GPT-4, then manually filtered) as an oracle comparison [2501.19353].

Key findings:

- OCR augmentation improves bigram recall where object labels are mission-critical.
- Paragraph filtering (informativeness thresholding, λ≈1.05–1.2) reduces distractor context and sharpens content alignment.
- Incorporating paper-specific style profiles yields 40–48% BLEU and 25–27% ROUGE improvements [2510.07993].

## 6. Key Technical Insights, Limitations, and Open Challenges

### Lessons

- Scientific figure captioning is fundamentally a task of context-driven summarization, not vision alone [2512.21789][2306.03491].
- Automatic metrics weakly track expert preferences; human (or LLM-in-the-loop) evaluation is indispensable [2501.19353][2512.21789].
- Personalization and style transfer, while increasing perceived authenticity, introduce potential factuality trade-offs [2512.21789][2510.07993].
- Vision-only or even multimodal systems pretrained on natural images often misalign with the composition and semantics of scientific plots and diagrams.

### Outstanding Challenges

- Captioning with incomplete or absent textual context remains unresolved; figures lacking mentioning paragraphs require new pretraining or reasoning resources [2512.21789].
- Adapting output to different reader expertise or communication norms; this suggests future research on controllable caption generation.
- Unified benchmarks for both captioning and deep figure content understanding (e.g., parsing, question answering) are lacking [2512.21789].
- Improving image-text grounding to avoid hallucinated facts or missed core insights.
- Real-world, in-process writer studies to measure AI tool impact on authoring workflows.
- Robust evaluation schemes that penalize hallucinations and reward domain-precise recall, particularly with future multimodal, author-style, or audience-adapted models.

### Current Limitations

- Most high-performing models still depend heavily on external or filtered context; free-standing image captioning remains far from solved [2501.19353].
- Pure text generation neglects visual signals that are not described in text or OCR [2407.01897].
- OCR errors, visually dense diagrams, and equation-heavy objects introduce cascading noise and performance bottlenecks [2512.21789][2306.03491].

## 7. Resources, Community Impact, and Future Directions

The SciCap initiative has released all major datasets (SciCap, SciCap+, LaMP-Cap), benchmarking code, and interactive caption drafting platforms (SciCapenter), facilitating broad adoption and reproducibility [2306.03491][2512.21789]. The annual challenges have attracted global participation, raising the quality bar each year.

Proposed future avenues include:

- Pretraining multimodal transformers directly on figure–mention pairs with figure-centric vision encoders.
- Incorporating structured external knowledge (e.g., ontologies) for improved interpretability [2306.03491].
- Adaptive, graph-based context selection and dynamic prompt refinement for more coherent, factual, and style-aligned output [2510.07993].
- Semi-automatic “human-in-the-loop” captioning workflows utilizing LMM output as assistive drafts for expert refinement [2501.19353].
- Audience-adaptive and style-controllable models that tailor captions to reader profiles [2512.21789].
- Benchmarks and metrics sensitive to factuality, visual–text consistency, and real-world scientific informativeness.

In summary, SciCap has systematically transformed figure captioning from a vision-task analog into a deeply contextual, multimodal, and human-centered challenge, anchoring research at the junction of robust AI, scientific communication, and accessibility [2512.21789][2407.01897][2501.19353][2510.07993][2306.03491][2110.11624].

Source: https://www.emergentmind.com/topics/scicap-project