---
title: 'MLBCAP: Multi-LLM Figure Captioning'
url: https://www.emergentmind.com/topics/multi-llm-collaborative-figure-caption-generation-mlbcap
type: topic
---

# MLBCAP: Multi-LLM Figure Captioning

Multi-LLM Collaborative Figure Caption Generation (MLBCAP) is a modular paradigm for generating high-quality, context-aware, and domain-relevant captions for scientific figures by orchestrating multiple large language models (LLMs), each specializing in specific subtasks. This approach arises from the recognition that figure captioning in academic documents is a complex, multimodal challenge—requiring reasoning over heterogeneous data (image, OCR tokens, textual mentions), mitigating low-quality annotations prevalent in paper corpora, and aligning outputs with the distinct style and communicative goals of scientific writing.

## 1. Motivation and Foundational Challenges

The scientific figure captioning problem is defined by the need to synthesize image content, document context, and domain conventions into coherent and informative descriptions. Prior approaches—either as vision-to-language or text summarization tasks—often underperform due to (a) limited integration of available modalities, and (b) the intrinsically low and variable quality of author-written captions typically found in resources like arXiv [2302.12324, 2501.02552].

Typical captioning corpora are encumbered by incomplete figure–caption alignment, absence of standardized "helpfulness" criteria, and coverage issues (roughly half the author-written captions are unhelpful for human understanding) [2302.12324, 2501.19353]. These limitations hinder supervised training regimes and expose models to exposure bias and hallucination risks, especially for long or complex captions.

These challenges motivate the MLBCAP schema: assembling a pipeline where the responsibilities for data filtering, candidate generation, and final refinement are distributed among specialized agents or models, thus leveraging their combined competencies for optimal caption production [2501.02552, 2501.01834].

## 2. Framework Decomposition and Model Roles

MLBCAP instantiates a workflow composed of three coordinated modules, each powered by distinct LLMs or multimodal agents [2501.02552]:

### (i) Quality Assessment Module
A multimodal LLM (e.g., LLaVA, GPT-4o) is tasked with assessing and filtering candidate training samples. It does so based on informativeness, alignment, and correctness of captions for each figure, as evaluated against multiple sources—figure images, mention paragraphs, and OCR text. Fine-tuning on a synthetic quality-annotated subset (with scores from 1–6) ensures that only samples rated “5” or “6” are retained, thus improving reliability for subsequent model training.

### (ii) Diverse Caption Generation Module
Multiple LLMs (specialized for domain adaptation, summarization, or generative diversity) generate candidate captions. These include:
- Foundation models (e.g., GPT-4o) used in few-shot or prompt-based settings.
- Fine-tuned models on domain data (e.g., LLaMA-3-8B, Yi-1.5-9B), adapted to the filtered dataset using multi-source prompts ([Figure], [Paragraphs], [Mentions], [OCR], etc.).
- Specialized summarization models (e.g., Pegasus), optimized for extracting key content from textual mentions and OCR snippets.

This ensemble ensures coverage across different reasoning styles, presentation forms, and granularity levels.

### (iii) Judgment and Refinement Module
A prominent language model (typically GPT-4o or a comparable LLM) then aggregates, ranks, and edits the candidate captions. This LLM acts both as a critic—screening for factual, stylistic, and structural quality—and as a final editor, addressing inaccuracies in numerics, nomenclature, or linguistic expression. MLBCAP supports generation of both “long” and “short” captions through controlled word limits, enhancing flexibility for varied editorial use cases.

### Table: MLBCAP Modular Workflow

| Module                  | Typical Model    | Purpose                            |
|-------------------------|------------------|------------------------------------|
| Quality Assessment      | LLaVA / GPT-4o   | Filter low-quality training data   |
| Diverse Caption Gen.    | GPT-4o, LLaMA, Pegasus | Generate candidate captions       |
| Judgment/Refinement     | GPT-4o           | Select and edit final caption      |

## 3. Evaluation Protocols and Empirical Effectiveness

MLBCAP is validated via both automatic and human-centric metrics. While traditional metrics (ROUGE, BLEU) are reported, their alignment with human preferences is weak—especially in academic domains where detail and informativeness outweigh mere n-gram overlap [2302.12324, 2501.19353]. As such, the efficacy of MLBCAP is primarily supported by human evaluation experiments [2501.02552].

Computer vision and academic writing experts consistently rank MLBCAP’s long captions above single-model baselines and even above author-written captions, particularly in terms of informativeness, clarity, and completeness. Ablation studies demonstrate that collaborative fusion (i.e., combining multiple LLM outputs with post-selection editing) contributes the largest improvements (e.g., raising the SciCap-Eval score from 5.390 to 5.440).

## 4. Model Specialization, Collaborative Coordination, and Technical Aspects

MLBCAP’s design encourages modular specialization:
- Some models are tuned for detail retrieval from visual data, mimicking label and relation map attention mechanisms analogous to those proposed in earlier work [1906.02850].
- Others, such as summarization LMs fine-tuned on mention paragraphs and OCR tokens, are optimized for extracting document context [2302.12324, 2306.03491].
- Ensemble models resolve tasks via cross-candidate consensus, rank aggregation, or reward-based selection (mirroring self-critical RL objectives and ensemble voting observed in vision–language research) [2306.11593, 2412.15484].

Judgment modules can use scoring metrics (including learned reward models from human feedback [2307.10867]) or utilize human-crafted scales with high inter-rater reliability.

The technical specification for each step includes explicit prompt structures with placeholders, controlled vocabularies for scientific nomenclature, and JSON-formatted output structures for interoperability. Training and inference rely on transformer-based models, with learning rates, batch sizes, and token limits optimized for each submodule [2501.02552].

## 5. Extensibility, Personalization, and Multiagent Variants

MLBCAP is extensible to various scientific domains and supports personalized captioning by conditioning on multimodal figure profiles, as shown in the LaMP-Cap framework [2506.06561]. By incorporating historical context (images, captions, related paragraphs from the same author or paper), MLBCAP can learn stylistic and content conventions, which enhances the alignment of model generations with intended authorial voice.

Agent-based variants (e.g., MoColl [2501.01834]) can further decompose the collaborative process as a series of question–answering steps: a VQA agent extracts domain-specific facts, while a generalist LLM synthesizes these facts into coherent, context-rich captions.

MLBCAP’s pipeline also supports personalized or domain-specific configuration. Models can be assigned roles based on known competencies (e.g., visual extraction, linguistic polish, factual validation), and final outputs can be selectively assembled according to user-defined criteria.

## 6. Limitations and Future Research Directions

Several open challenges remain for MLBCAP:
- **Quality of Reference Data**: Even after filtering, author-written captions in training data may still lack coverage or precision, necessitating further advances in synthetic quality scoring, human feedback integration, and iterative refinement protocols [2501.06317, 2307.10867].
- **Evaluation Bias**: Human-preferred outputs do not always align with standard metrics; the adoption of dual-metric evaluation for factuality and coverage (as in [2412.15484]) provides a more robust assessment framework but requires domain-specific benchmarks.
- **Coordination Overhead**: The orchestration of multiple LLMs imposes computational and logistical burdens, particularly when integrating external models or human-in-the-loop feedback.
- **Personalization**: Representing and leveraging author, domain, or venue profiles in a scalable, data-efficient manner is a developing area [2506.06561].
- **Data Privacy and Generalizability**: Personalized or context-aware systems must respect the privacy of research data while ensuring generalization across diverse scientific fields.
- **Automation vs. Human-in-the-Loop**: Studies reveal that authors rarely accept AI-generated captions verbatim—instead, they iteratively adapt, correct, or merge multiple suggestions [2501.06317]. MLBCAP systems that interleave AI generation and human review are thus best positioned for robust scientific communication.

## 7. Significance for the Future of Figure Captioning

The MLBCAP paradigm marks a transition from monolithic captioning pipelines toward compositional, collaborative systems that are robust to noisy training data, flexible in their semantic coverage, and amenable to iterative human interaction. This architecture generalizes well to multilingual, personalized, and multimodal figure captioning tasks, facilitating advances in automation, human–AI collaboration, and reproducible evaluation [2501.02552, 2302.12324, 2506.06561].

By foregrounding specialized LLMs for assessment, content generation, and critical revision, and by leveraging multi-source information (figure, OCR, mention text, prior figures), MLBCAP systems are positioned to meet the rising standards and nuanced demands of scientific figure communication.

Source: https://www.emergentmind.com/topics/multi-llm-collaborative-figure-caption-generation-mlbcap