Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation
Abstract: Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using LLMs, commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set. To address this, we propose Rigel, an automatic evaluation metric for image and video captioning, based on self-distilled score adaptation. The metric employs an evaluation-specific scoring head distilled from a frozen LLM, which captures judgment signals in a task-aligned space without relying on large-vocabulary token sets. We then refine the LLM backbone with human judgment data. To train Rigel, we constructed the Vid-Lepus dataset, which contains 3,338 video clips, 33,380 reference captions, and 5,637 candidate captions. Experiments on multiple benchmarks show that Rigel outperforms state-of-the-art metrics, achieving over 10-point improvements on ActivityNet-Fact in the reference-free setting.
Summary
- The paper introduces a novel two-phase framework that first distills a task-specific scoring head from a frozen LLM and then adapts the backbone using human judgments.
- It leverages Earth Mover’s Distance to train on ordinal labels, effectively reducing noise from irrelevant vocabulary tokens.
- Experimental results on diverse datasets demonstrate Rigel’s superior alignment with human annotations compared to traditional and LLM-based metrics.
Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation
Introduction and Motivation
Despite the proliferation of multimodal foundation models, automatic evaluation of image and video captioning remains a fundamental challenge. Classical metrics—such as BLEU, METEOR, ROUGE, CIDEr, SPICE—are known to align poorly with human judgments, especially for captions that are semantically correct but lexically diverse. Recent advances leverage vision-LLMs (VLMs) and LLMs for more data-driven evaluation, but even LLM-as-a-Judge approaches face a critical limitation: the mismatch between the LLM's large token vocabulary and the small, ordinal label set required for caption scoring. As a result, evaluation via LLMs is noisy, since logit distributions over all vocabulary tokens allocate significant probability mass to task-irrelevant tokens, degrading performance.
The Rigel Framework
Rigel addresses this vocabulary-label mismatch by introducing a self-distilled score adaptation framework for automatic image and video captioning evaluation. Rigel implements a two-phase process:
Figure 1: Overview of Rigel’s two-phase framework: Phase 1 self-distillation of a task-specific scoring head from a frozen LLM, and Phase 2 human-guided backbone adaptation with the scoring head frozen.
- Phase 1—Denoised Head Self-Distillation:
A lightweight evaluation-specific scoring head is distilled from a frozen LLM. Instead of relying on the LLM's language modeling head, which is inherently suited for next-token prediction over a massive vocabulary, Rigel’s scoring head projects hidden representations onto a small ordinal label set (e.g., scores 1–5). The head is trained using Earth Mover’s Distance (EMD) to capture ordinal relationships, using soft pseudo-labels derived from the model’s original logits. All LLM parameters remain frozen in this phase.
Figure 3: Training flow of Rigel—left: Phase 1 with the scoring head trained on five labels (EMD), LLM frozen; right: Phase 2 where only LoRA-adapted backbone is trained with human judgments, scoring head fixed.
- Phase 2—Human-Guided Score Adaptation: The LLM backbone undergoes LoRA-based parameter-efficient adaptation using supervised human judgment data, while the scoring head parameters remain fixed. The backbone’s representations are thus aligned with human scoring preferences.
Inference is performed by feeding an input visual/language triplet through the LLM and scoring head to produce a scalar, linearly normalized over [0,1].
Analysis of Previous Approaches
Traditional and embedding-based metrics (BLEU, CIDEr, CLIPScore, PAC-S, Polos) are primarily optimized for surface-level similarity (lexical/semantic overlap), missing fine-grained human-aligned properties such as descriptiveness and relevance. LLM-based metrics (e.g., FLEUR, G-VEval, HarmonicEval) have improved interpretability but are constrained by the aforementioned vocabulary-label mismatch, resulting in noisy logits for scoring tokens (see Figure 3 and Figure 4).
Figure 2: The logit magnitudes assigned by the LLM head show non-score tokens receive similar mass as score tokens, indicating noisy predictions in evaluation.
Figure 5: Logit distribution pattern over the Composite dataset, confirming the prevalence of noisy non-score token activations.
Rigel’s targeted scoring head and self-distillation directly alleviate this problem, moving away from the full vocabulary to a space focused on ordinal label accuracy.
Dataset Construction: Vid-Lepus
A significant constraint in prior work has been the lack of supervised datasets for video captioning evaluation. Rigel’s implementation introduces Vid-Lepus, a large-scale resource for training video evaluation metrics, comprising 3,338 video clips, 33,380 reference captions, 5,637 candidate captions, and 14,802 human judgments. Annotations span descriptiveness, relevance, and fluency. Outlier exclusions and manual review policies ensure annotation fidelity.
Experimental Results
Image Captioning:
Rigel demonstrates strong performance improvements across evaluation benchmarks (Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, FOIL), both in reference-based and reference-free settings. On the Composite dataset and FOIL, Rigel achieves the highest accuracy. In the reference-free setting, it outperforms all baselines, including supervised and LLM-based methods.
Video Captioning:
On VATEX-EVAL, ActivityNet-Fact, YouCook2-Fact, and ActivityNet-FOIL, Rigel surpasses all previous baselines in both reference-based and reference-free contexts. The improvement on ActivityNet-Fact in the reference-free setting exceeds 10 points, testifying to superior alignment with human judgments. The empirical study demonstrates that each phase (self-distillation, human adaptation) independently boosts performance, with their combined application leading to the best overall outcomes.
Ablation and Objective Study
Through ablation, exclusion of Phase 1 (scoring head self-distillation) or Phase 2 (human-guided backbone adaptation) each result in nontrivial decreases in Kendall’s and Pearson’s . EMD, the chosen Phase 1 training objective, outperforms KL divergence by modeling the ordinal nature of human labeling, indicating that label proximity information is essential for effective supervision and downstream evaluation.
Qualitative and Failure Analysis
Rigel consistently aligns more closely with human annotation compared to prior baselines in both reference-based and reference-free settings, as observed on the Nebula and VATEX-EVAL datasets.
Figure 6: Qualitative results on Nebula demonstrate Rigel’s efficacy for both reference-based and reference-free evaluation, but with identified failure modes such as textual misreading.
Figure 4: Successful evaluations on the VATEX-EVAL dataset for both settings, with Rigel producing scores close to human ground truth.
Error categorization reveals that most failures result from underestimation of accurate captions or alternative lexical forms (i.e., when references do not sufficiently cover valid variants), or occasional overestimation of inaccurate or underspecified candidates. These findings underscore the need for richer visual-linguistic grounding and further visual-token compression strategies.
Implications and Future Directions
Rigel establishes a robust automatic evaluation framework, resolving the persistent LM-vocabulary/ordinal-label mismatch within LLM-based caption assessment. The leveraging of EMD, human judgment adaptation, and task-specific scoring heads may generalize to other generative language evaluation domains. The release of Vid-Lepus strengthens the ecosystem for supervised metric development in video captioning.
The main limitation is dependence on accessible LLM hidden states, precluding use with proprietary APIs, and the potential suboptimality of self-distillation from frozen LM heads. Additionally, reliance on LoRA for alignment might restrict scalability. Further advancements may emerge from joint visual-linguistic adaptation and new mechanisms to handle reference incompleteness, especially for lexically diverse descriptions.
Conclusion
Rigel’s self-distilled score adaptation framework redefines automatic caption evaluation for images and videos, demonstrating clear, consistent gains across multiple human-aligned benchmarks. Its approach to the LM head mismatch problem significantly reduces noise in score prediction. Theoretical and practical extensions to LLM-based generative task evaluation are likely, particularly as human-feedback-oriented adaptation and dataset curation approaches continue to expand the supervision landscape.
Reference:
"Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation" (2606.29997)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper introduces Rigel, a new way to automatically judge how good captions are for images and videos. A “caption” is a sentence describing what’s in a picture or video. Rigel tries to score captions in a way that matches what humans think, because many current automatic scores don’t agree well with human opinions.
What questions does the paper ask?
- How can we make automatic caption scoring match human judgments better?
- Why do some LLM judges still give noisy or confusing scores?
- Can we design a scoring method that focuses on simple labels (like 1 to 5 stars) instead of getting distracted by a giant word vocabulary?
- Can we build training data with real human ratings to teach the model what “good” looks like?
How did the researchers do it?
The authors found a mismatch in many LLM-based evaluators: the model’s output layer is built to pick the next word from a huge dictionary, but a caption judge only needs to choose a small set of scores (like 1, 2, 3, 4, or 5). That huge dictionary makes scoring noisy.
Rigel fixes this with a two-phase approach:
Phase 1: Build a clean “scoring head” (self-distillation)
- Imagine the LLM as a powerful engine with a big “word picker” at the end. Rigel adds a small, dedicated “score picker” (the scoring head) that outputs just five numbers (for scores 1–5).
- “Self-distillation” means the model teaches this new scoring head using its own internal signals, but without changing the main engine. Think of it like the model training a smaller, calmer judge to replace its noisy, word-based judge.
- To train the scoring head, they use a gentle matching trick called Earth Mover’s Distance (EMD). Picture five buckets for scores 1–5 with sand in them. EMD measures how much sand you’d need to move to match two score distributions. This helps the new head learn smooth and sensible score predictions.
Phase 2: Align the engine with human ratings
- Next, they freeze the new scoring head and slightly adjust the LLM’s internal “engine” using real human ratings. They do this with LoRA, a way to make small, safe tweaks without retraining everything.
- By tuning the engine to produce features that the scoring head understands, the whole system learns to judge captions more like humans.
Data: Vid-Lepus for video, Spica for images
- They created a new dataset called Vid-Lepus for video caption evaluation training. It includes thousands of short video clips, reference captions (good examples), candidate captions (the ones to judge), and human ratings on a 1–5 scale across three aspects: descriptiveness, relevance, and fluency.
- For images, they use an existing dataset called Spica with human judgments.
How it works at test time
- Rigel takes the visual input (image or video), optional reference captions, and a candidate caption.
- It makes one forward pass through the LLM engine and the scoring head to produce a single smooth score between 0 and 1, where higher means better.
Main findings and why they matter
- Rigel consistently matches human ratings better than many popular metrics across lots of benchmarks for both images and videos.
- It works in two modes:
- Reference-based: judging a candidate caption while seeing human-written reference captions.
- Reference-free: judging a candidate caption without any references—just using the image or video.
- Rigel improved results by large margins on several tests. For example, it shows over 10-point improvements on ActivityNet-Fact in the reference-free setting, and very high accuracy on FOIL-style tests that check if captions say something wrong.
- The two-phase design matters: both the clean scoring head (Phase 1) and the human-guided tuning (Phase 2) each add measurable benefits. Together, they give the best performance.
- Vid-Lepus helps the field by providing human-labeled training data for video caption evaluation, which was missing before.
Implications and potential impact
- Better automatic judges mean fairer, more reliable benchmarks. That helps researchers and companies compare captioning systems and make real progress.
- The scoring head idea can be reused: any task where an LLM has to choose from a small set of labels (not a giant vocabulary) could benefit from a similar “clean score picker.”
- With stronger agreement to human opinions, future captioning systems can be trained and tuned more effectively, potentially leading to captions that are more accurate, readable, and trustworthy.
- The new Vid-Lepus dataset gives the community a foundation to build even better video caption evaluators.
Overall, Rigel is like giving a powerful model a dedicated, calm judge trained to think in “scores” instead of “words,” and then teaching the whole system to respect human ratings. This leads to smarter, more human-aligned caption evaluation.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, framed to guide future research.
- Phase 1 teacher quality: The self-distillation head is trained from LM-head score-token distributions of the same frozen LLM; it is unclear whether better “teacher” signals (e.g., ensemble teachers, calibrated external judges, or human-annotated soft labels) would yield stronger heads and reduce teacher-induced bias.
- Distillation objective choices: Only 1D-EMD with temperature scaling is studied; alternative ordinal objectives (e.g., CORAL, ordinal logistic regression, pairwise ranking losses, label smoothing strategies) and their impact on alignment and calibration remain unexplored.
- Temperature and token-set sensitivity: The method depends on the choice of score tokens and the distillation temperature τ; there is no analysis of sensitivity to τ, score-token vocabulary, or non-uniform/continuous label mappings beyond 1–5 bins.
- Scoring head design space: The scoring head is a lightweight MLP; the paper does not test alternative architectures (e.g., deeper heads, MoE, attention-based heads, monotonic constraints) or regularizers that might better encode ordinal structure.
- Training dynamics across phases: The head is frozen in Phase 2; it is unknown whether joint or alternating optimization, bi-level objectives, or periodic re-distillation would prevent mismatch between evolving backbone representations and a fixed head.
- Multi-dimensional judgments underused: Vid-Lepus collects descriptiveness, relevance, and fluency, but training collapses these into a single label; a multi-head or multi-task model that predicts per-dimension scores (and aggregates them) is not explored.
- Score calibration to human scale: The final score is normalized to [0,1] and evaluated via correlations; there is no analysis of calibration error (e.g., ECE/MCE, reliability curves) or mapping directly to 1–5 human scores with interpretable thresholds.
- Uncertainty estimation: The method outputs a mean score but not uncertainty; providing confidence intervals, variance-aware aggregation across references, or abstention mechanisms is left open.
- Prompt sensitivity and instruction design: Performance may depend on prompt templates that elicit hidden representations; robustness to paraphrased prompts, prompt tuning, or automatic instruction optimization is not examined.
- Backbone generality: Experiments are centered on small Qwen-based MLLMs (2–3B); the transferability of the approach across model families (e.g., LLaVA, InternVL, CLIP-based MLLMs) and sizes (from tiny to large) is only partially probed.
- Black-box compatibility: Rigel requires hidden states and thus excludes many proprietary APIs; exploring variants that rely only on logprobs (if available) or a black-box distillation route (student evaluator without hidden-state access) is needed.
- Computational efficiency: Despite claiming a single forward pass, there are no measurements of latency, memory, or throughput versus LLM-as-a-Judge baselines; trade-offs between speed and accuracy remain unknown.
- Cross-lingual and multi-lingual evaluation: All datasets appear English-centric; whether the method transfers to other languages, mixed-language captions, or cross-lingual references (and needed adaptations) is untested.
- Domain and distribution shift: Generalization to specialized domains (e.g., medical, charts/diagrams, egocentric, ASL/sign videos) or stylistically distinct captions is not studied; the robustness under domain shift is an open question.
- Scene-text and OCR-heavy cases: The failure case with misread signage highlights a weakness; integrating OCR-aware features, text-grounding, or benchmarks like TextCaps for evaluation-aware training is unexplored.
- Video temporal modeling choices: The impact of frame sampling rate, clip length, temporal stride, and long-context modeling on evaluation quality is not analyzed; ablations over T or temporal encoders are missing.
- Multi-reference aggregation: The method supports both reference-based and reference-free settings but does not study how the number and diversity of references affects scores, nor how to aggregate contradictory references.
- Stability across seeds and significance: Results are based on a single run; variance across random seeds, confidence intervals, and statistical significance tests for metric comparisons are not reported.
- System-level vs. instance-level evaluation: Correlations are reported largely at the instance level; the ability to correctly rank systems (system-level correlation) and to be robust to small per-instance perturbations is not analyzed.
- Candidate-distribution robustness: The training candidates come from specific captioners; the method’s robustness to new or adversarial caption distributions (e.g., very long, repetitive, toxic, or nonsensical captions) is not evaluated.
- Preference learning alternatives: The approach uses cross-entropy on discretized labels; leveraging pairwise human preferences or direct preference optimization (DPO-like) tailored for evaluators is untested.
- Data scale and bias in Vid-Lepus: With 14,802 judgments over 3,338 clips, dataset size and diversity may limit generalization; detailed analyses of annotator agreement, demographic bias, and topic/visual diversity are not provided.
- Long-video evaluation: The benchmarks focus on short clips; the method’s behavior on long-form videos (minutes to hours), where summarization and temporal grounding challenges are different, remains unassessed.
- Explainability and rationale: Unlike some baselines that produce structured explanations, Rigel provides no human-interpretable rationales or error attributions; adding rationale generation and auditing tools is an open direction.
- Continuous vs. discrete supervision: Human judgments are discretized to one-hot labels; directly learning from continuous scores (or modeling label distributions) and their benefits for calibration/robustness are unexamined.
- Joint training across tasks: Although the framework could, in principle, generalize beyond captioning (e.g., VQA, summarization), no experiments demonstrate portability or multi-task evaluators trained with shared heads.
- Proprietary-model bridging: A practical pathway to use proprietary MLLMs (e.g., via adapter layers, API-based surrogate modeling, or head distillation into an open student) is not proposed.
- Comprehensive error typology: Beyond a few qualitative examples, there is no systematic error analysis by category (object presence, attributes, relations, counts, hallucination, temporal order, OCR, pronoun coreference), limiting targeted improvements.
Practical Applications
Immediate Applications
Below are applications that can be deployed with today’s open multimodal LLMs and the Rigel training recipe (self-distilled scoring head + LoRA backbone tuning) and/or by directly using the released model and Vid-Lepus data.
- Automated caption quality assurance (QA) for image/video platforms
- Sectors: software, media/entertainment, e-commerce, accessibility
- Tools/products/workflows: “Rigel Score” API to score auto-generated captions; dashboard to flag low-quality captions; batch auditing for existing catalogs; FOIL-like checks to catch object/attribute inconsistencies
- Assumptions/dependencies: access to a compatible open MLLM backbone exposing hidden states; GPU for inference at scale; domain shift may require light calibration; scoring focuses on 1–5 ordinal labels
- Reference-free model selection and A/B testing for captioning systems
- Sectors: software/ML, MLOps
- Tools/products/workflows: CI/CD gate using Rigel score thresholds; regression tests for hallucination and relevance; prompt optimization loops that maximize Rigel scores
- Assumptions/dependencies: Goodhart’s law risks (over-optimizing to a single metric); recommended to pair with periodic human audits and complementary metrics
- Dataset curation and filtering for captioning corpora
- Sectors: academia, data engineering, e-commerce
- Tools/products/workflows: filter noisy or off-topic captions from web-scraped sets; triage samples for manual review based on low Rigel scores; stratified sampling by score bands
- Assumptions/dependencies: domain alignment (Vid-Lepus + Spica are general-domain); specialized domains may need small domain-specific calibration or additional human labels
- Accessibility and compliance checks for alt-text
- Sectors: accessibility, legal/compliance, education
- Tools/products/workflows: CMS plugin that flags insufficient alt-text quality; policy rules (e.g., minimum normalized score) for ADA/WCAG readiness; editorial assistance to improve captions
- Assumptions/dependencies: text-reading errors (scene text) are a known limitation; might need an OCR-aware variant for text-centric images (TextCaps-like scenarios)
- Crowdsourcing QA and human-in-the-loop annotation triage
- Sectors: crowdsourcing platforms, data labeling vendors, academia
- Tools/products/workflows: prioritize items with low or highly variable scores for additional review; consensus checking across dimensions (descriptiveness, relevance, fluency) using Rigel outputs and task-specific prompts
- Assumptions/dependencies: consistent rubric/prompting for each dimension; human spot-checks for edge cases
- Content moderation and trust & safety triage for captions
- Sectors: social media, media platforms, advertising
- Tools/products/workflows: flag potentially misleading or hallucinated captions for review; combine Rigel with safety filters to detect harmful mismatches between media and description
- Assumptions/dependencies: not a replacement for safety classifiers; threshold tuning needed to balance recall/precision
- Academic benchmarking and reproducible evaluation
- Sectors: academia, open-source communities
- Tools/products/workflows: benchmarking suites that include Rigel alongside lexical and embedding metrics; use of Vid-Lepus for supervised metric training; per-dataset leaderboards using Kendall’s τ, Spearman’s ρ, Pearson’s r
- Assumptions/dependencies: transparent release of model, code, and splits; consistent evaluation protocols
- Retail catalog QC and SEO support
- Sectors: e-commerce, marketing
- Tools/products/workflows: batch evaluation of product image/video captions; auto-suggestions to raise quality score; rejection criteria for vendor-submitted captions
- Assumptions/dependencies: product-specific jargon and compliance claims may require domain tuning or keyword constraints
- Video editing and creator tools with real-time caption feedback
- Sectors: creator economy, media production
- Tools/products/workflows: plugin that scores captions as users edit; highlights low-scoring segments; “quality badge” for captions above threshold
- Assumptions/dependencies: latency constraints; may require quantized/lightweight backbones or batching
- Procurement and internal policy gating for AI-generated captions
- Sectors: enterprise IT, public sector procurement
- Tools/products/workflows: acceptance criteria (e.g., ≥0.8 normalized Rigel) in RFPs; supplier audits against ActivityNet/FOIL-like probes
- Assumptions/dependencies: standardized prompts and score normalization; documented variance across datasets
Long-Term Applications
These require further research, domain adaptation, scaling, or development (e.g., additional human-labeled data, optimized deployments, new task formulations).
- Generalizing self-distilled evaluation heads beyond captioning
- Sectors: software, education, documentation, code tools
- Tools/products/workflows: evaluation heads for small ordinal label spaces in tasks like summarization quality, VQA correctness, chart/table description, code-comment relevance, translation adequacy
- Assumptions/dependencies: access to hidden states; new human-judgment datasets per task; careful rubric design to avoid token-vocabulary mismatch
- Reward models for training captioners (RLHF/RLAIF) to reduce hallucinations
- Sectors: software/ML, media, robotics
- Tools/products/workflows: use Rigel-style heads as reward functions in RL; multi-objective optimization across descriptiveness, relevance, fluency; active learning loops seeded by Vid-Lepus-like data
- Assumptions/dependencies: reward hacking risks; regular human evaluation and metric ensembles recommended
- Domain-specialized evaluators (healthcare, scientific imaging, industrial inspection)
- Sectors: healthcare, manufacturing, geospatial, automotive
- Tools/products/workflows: collect domain-specific human judgments; adapt Phase 2 with LoRA for specialty data; deploy as QA in report generation or anomaly description pipelines
- Assumptions/dependencies: strict compliance and safety standards (e.g., medical device regulations); need for expert-labeled judgments; risk and bias assessments
- Multilingual and multicultural caption evaluation
- Sectors: global media, education, public sector
- Tools/products/workflows: multilingual scoring heads (per language or shared); cross-lingual calibration using parallel captions; region-specific rubrics for cultural relevance
- Assumptions/dependencies: multilingual Vid-Lepus-like datasets; base MLLMs with strong multilingual capabilities; locale-sensitive guidelines
- Standardization and policy for multimodal evaluation
- Sectors: policy/regulation, standards bodies, enterprise governance
- Tools/products/workflows: NIST-style suites for caption quality, hallucination robustness (FOIL-like), and domain robustness; conformance badges; third-party audits using reference-free and reference-based protocols
- Assumptions/dependencies: stakeholder consensus on rubrics and metrics; periodic re-benchmarking to deter gaming
- On-device/private evaluation for accessibility (AR glasses, mobile)
- Sectors: consumer electronics, assistive tech
- Tools/products/workflows: compressed backbones with integrated scoring heads; low-latency evaluation for real-time caption feedback; privacy-preserving local inference
- Assumptions/dependencies: model compression and hardware acceleration; battery/runtime trade-offs; local OCR for text-heavy scenes
- Instructional and language-learning tutors for visual description
- Sectors: education, edtech
- Tools/products/workflows: practice modules where learners describe images/videos and receive rubric-aligned feedback; analytics by dimension (fluency, relevance, descriptiveness)
- Assumptions/dependencies: pedagogically validated rubrics; bias/fairness considerations; user privacy
- Data generation oversight for synthetic training corpora
- Sectors: software/ML, media, e-commerce
- Tools/products/workflows: filter or reweight synthetic captions by Rigel score; curriculum schedules that increase difficulty only when quality is high
- Assumptions/dependencies: mitigation against metric overfitting; ensemble metrics and human spot-checks
- Safety filters for robotics and autonomous systems
- Sectors: robotics, automotive, drones
- Tools/products/workflows: gate perception-language outputs used in planning; suppress low-confidence or likely-hallucinated scene descriptions before action execution
- Assumptions/dependencies: strict latency budgets; safety certification; integration with perception stacks and uncertainty estimation
- Compliance and auditing in regulated industries
- Sectors: finance, healthcare, advertising
- Tools/products/workflows: audit trails for media-caption consistency in disclosures and ads; automated checks for misrepresentation or missing details; human review escalation
- Assumptions/dependencies: legal interpretability of metric scores; explainability tooling; human governance
- OCR/text-centric and fine-grained attribute evaluation
- Sectors: retail (packaging), document/media analytics, navigation
- Tools/products/workflows: extend Rigel with OCR-aware prompts or multi-heads focused on text fidelity and fine-grained attributes
- Assumptions/dependencies: additional datasets with scene-text labels; integrated OCR modules; known limitation acknowledged in the paper’s failure case
Notes on cross-cutting dependencies and assumptions:
- Hidden-state access: Rigel’s scoring head requires access to MLLM hidden representations; closed/proprietary models may be incompatible without vendor support.
- Domain shift: performance may degrade in specialized domains; Phase 2 adaptation with domain judgments is recommended.
- Metric gaming: combine Rigel with complementary metrics and periodic human audits; rotate evaluation datasets to reduce overfitting.
- Compute and latency: although single-pass inference is efficient, production deployment still benefits from model optimization (quantization, batching).
- Ethical use: ensure fairness/bias audits when used for high-stakes decisions (e.g., accessibility compliance, healthcare).
Glossary
- Denoised Head Self-Distillation: A training phase that distills evaluation signals from the LM head into a compact scoring head better suited for ordinal labels. "consists of two phases: (i) Denoised Head Self-Distillation and (ii) Human-Guided Score Adaptation."
- Earth Mover's Distance (EMD): A distance between probability distributions (over ordered labels) used here as a loss to align predicted and target score distributions. "where denotes the one-dimensional Earth Mover's Distance (EMD)"
- Evaluation-specific scoring head: A lightweight prediction head that maps hidden representations to a small set of ordinal score labels. "The metric employs an evaluation-specific scoring head distilled from a frozen LLM"
- Frozen LLM: An LLM whose parameters are kept fixed (not updated) during part of training. "distilled from a frozen LLM"
- Hidden representation: The final-layer vector from the model used as input to the scoring head. "to obtain a final-layer hidden representation "
- Kendall's τ_b and τ_c: Rank correlation measures used to assess agreement between metric scores and human judgments. "we used Kendall's and "
- Language modeling (LM) head: The LLM’s output projection that produces logits over the full vocabulary, not ideal for small label sets. "The language modeling (LM) head operates over a large vocabulary set "
- LLM backbone: The main body of the LLM (excluding task-specific heads) that is adapted to align with human judgments. "We then refine the LLM backbone with human judgment data."
- LLM-as-a-Judge: A paradigm where an LLM is used to evaluate system outputs (e.g., captions) as a judge. "often referred to as LLM-as-a-Judge approaches"
- Logit: The unnormalized score output before softmax; here, vocabulary and score-token logits are analyzed for noise. "Logit distributions over score tokens (
1''–5'') and non-score tokens" - Low-Rank Adaptation (LoRA): A parameter-efficient fine-tuning method that inserts low-rank adapters into the backbone. "fine-tune only the backbone parameters via Low-Rank Adaptation (LoRA)"
- Multimodal LLM (MLLM): An LLM that processes multiple modalities (e.g., text and images/videos). "LLMs and multimodal LLMs (MLLMs)"
- Ordinal label set: An ordered set of discrete labels (e.g., 1–5) used to represent judgment scores. "small ordinal label set "
- Pearson's correlation coefficient (r): A statistic measuring linear correlation between predicted and human scores. "Pearson's correlation coefficient "
- Pseudo-label: A soft target distribution derived from the model itself and used for distillation. "a soft pseudo-label distribution"
- Reference-based setting: Evaluation that uses one or more human-written reference captions. "In the reference-based setting,"
- Reference-free setting: Evaluation without any reference captions, judging candidates directly against the input media. "in the reference-free setting"
- Score smoothing: Computing the final scalar score as the expectation under the predicted score distribution. "we compute via score smoothing"
- Score tokens: Special tokens (e.g., “1”–“5”) used by LLMs to represent ordinal scores. "score tokens (
1''–5'')" - Self-distillation: Training a model component using soft targets produced by the model itself to transfer knowledge. "learns an evaluation-specific scoring head via self-distillation"
- Self-distilled score adaptation: The overall two-phase framework that distills a scoring head and then adapts the backbone to human judgments. "self-distilled score adaptation framework"
- Temperature-scaled softmax: A softmax with a temperature parameter to control sharpness during distillation. "We then apply a temperature-scaled softmax"
- Vocabulary–label mismatch: The misalignment between predicting over a large vocabulary and needing only a small set of ordinal labels. "addressing the vocabulary-label mismatch between the LM head vocabulary and ordinal label set."
- Vocabulary logits: The logits over the entire token vocabulary produced by the LM head. "From the vocabulary logits, we extract those corresponding to the ordinal score tokens"
Collections
Sign up for free to add this paper to one or more collections.