---
title: AgenticIQA-Eval Benchmark
url: https://www.emergentmind.com/topics/agenticiqa-eval
type: topic
---

# AgenticIQA-Eval Benchmark

AgenticIQA-Eval is the benchmark introduced alongside the AgenticIQA framework for evaluating **VLM-based IQA agents** at the level of internal agent behavior rather than only final scalar quality prediction. It is described as “the first benchmark for assessing the planning, execution, and summarization capabilities of VLM-based IQA agents” and as a benchmark “measuring planning accuracy, execution precision, and summarization reliability through multiple-choice questions (MCQs)” [2509.26006]. Its central purpose is to diagnose whether a model can act as a modular, query-aware image-quality assessor that interprets the query, reasons about distortions, selects appropriate IQA tools, and synthesizes intermediate perceptual evidence into a final decision.

## 1. Concept and motivation

AgenticIQA-Eval arises from a mismatch between conventional IQA evaluation and the requirements of an **agentic IQA system**. Traditional FR-IQA and NR-IQA evaluation emphasizes scalar alignment with MOS through measures such as SRCC and PLCC, while some VLM-oriented settings emphasize low-level visual QA or description generation. The benchmark’s motivating claim is that these settings do not evaluate whether a system can perform the sequence of decisions required by a modular IQA pipeline: understand the query, determine whether the task is IQA or something else, infer or extract distortions, choose appropriate tools, execute them when needed, and integrate heterogeneous evidence into an answer [2509.26006].

The benchmark therefore targets **adaptive, modular, query-aware IQA** rather than fixed black-box scoring. The paper explicitly argues that score-only metrics ignore reasoning structure, that interpretation and scoring are often disconnected, that static benchmarks do not test adaptivity, and that low-level VLM benchmarks such as Q-Bench and LLVisionQA still do not directly test planner/executor/summarizer decomposition or tool selection. This suggests a shift from evaluating IQA systems as scalar predictors to evaluating them as **agentic perception systems** [2509.26006].

## 2. Relation to the AgenticIQA architecture

AgenticIQA itself follows a **plan–execute–summarize** design with three VLM-driven modules—Planner, Executor, and Summarizer—and decomposes IQA into four subtasks: **distortion detection**, **distortion analysis**, **tool selection**, and **tool execution**. The benchmark mirrors this architecture directly, so its tracks are not auxiliary tasks detached from the method but probes of the framework’s core decision points [2509.26006].

Formally, the executor subtasks are written as
$$
M_i = \mathcal{E}_i(x, t, P_t(i), \mathcal{T}), \quad i \in \{dd, da, ts, te\},
$$
with distortion detection
$$
D = \mathcal{E}_{dd}(x, t_{dd}),
$$
distortion analysis
$$
A_i = \left\{ \left(d_i, l_i, r_i\right) \mid d_i \in D_k \right\} = \mathcal{E}_{da}(x, t_{da}, D_k, O_k),
$$
tool selection
$$
T_i = \mathcal{E}_{ts}(d_i, t_{ts}, \mathcal{T}),
$$
and tool execution
$$
\hat{q}_i = \mathcal{E}_{te}(x, T_i).
$$
Here \(l_i \in \{\text{none, slight, moderate, severe, extreme}\}\) is the severity level and \(r_i\) is a textual explanation [2509.26006].

The Planner determines fields such as **query type** (`IQA` vs `Other`), **reference mode** (`Full-Reference` or `No-Reference`), **query scope** (`Global` or object-specific), **distortion source** (`Explicit` or `Inferred`), and boolean activation flags for distortion detection, distortion analysis, tool selection, and tool execution. The surrounding framework specifies JSON outputs for planner, distortion detection, distortion analysis, tool selection, and summarizer modules, but AgenticIQA-Eval itself scores capability through MCQs rather than direct structured-output matching [2509.26006].

The Executor operates over a tool library \(\mathcal{T}\) drawn from IQA-PyTorch. Listed FR tools include **TOPIQ\_FR**, **AHIQ**, **FSIM**, **LPIPS**, **DISTS**, **WaDIQaM\_FR**, **PieAPP**, **MS-SSIM**, **GMSD**, **SSIM**, **CKDN**, **VIF**, **PSNR**, and **VSI**. Listed NR tools include **QAlign**, **CLIPIQA**, **HyperIQA**, **TReS**, **MUSIQ**, **NIMA**, **BRISQUE**, **NIQE**, **MANIQA**, and **LIQE**; the paper notes inconsistent spelling for **UNIQIE / UNIQUE**. The appendix further states that tools have different distortion strengths, for example **TOPIQ** across blur, color, compression, noise, brightness, spatial distortions, and sharpness/contrast, **QAlign** across blur, color distortions, noise, brightness, spatial distortions, and sharpness, **UNIQUE** on lens blur, JPEG/JPEG2000, denoise artifact, spatial distortions, and contrast, **LIQE** on color diffusion, and **DISTS** on Gaussian blur [2509.26006].

## 3. Benchmark construction and annotation

AgenticIQA-Eval contains **1000** curated MCQ samples. Its image pool combines **500 distorted images from MICBench**, which contains “content from diverse sources with authentic and generative degradations,” with **500 pristine images from the Waterloo Exploration Database** that are synthetically degraded with one or two random distortions following the protocol of Descriptive IQA in the Wild. The final composition is **750 NR-IQA** instances and **250 FR-IQA** instances [2509.26006].

The benchmark is organized into three tracks aligned with the planner–executor–summarizer decomposition.

| Track | Samples | Evaluated capability |
|---|---:|---|
| Planner | 250 | Subtask configurations and valid evaluation plans |
| Executor | 500 | 250 distortion identification/severity + 250 tool selection |
| Summarizer | 250 | Use of intermediate perceptual state \(M_t\) to produce accurate responses |

Each sample is an **MCQ with a single ground-truth answer**. The question formats are **What**, **How**, **Which**, and **Yes/No**, and the appendix states that the benchmark maintains a **balanced distribution** across these question types. The appendix also states coverage across **FR and NR scenarios**, distortion types such as **blur, noise, compression**, and **planning configurations** [2509.26006].

Curation is manual and expert-verified. The paper states that all MCQs are manually curated, every question is cross-verified by **at least two expert annotators**, and the appendix further specifies that each question is independently verified by a second annotator with disagreements resolved by a **third expert adjudicator**. The answer options are designed to be “plausible yet discriminative,” so the benchmark is intended to measure reasoning rather than shortcut pattern matching [2509.26006].

Representative examples illustrate the three tracks. Planner questions test query interpretation, including object extraction from prompts such as “What is the primary distortion affecting the boat clarity?” Executor questions test distortion severity or tool appropriateness, for example whether **NIMA** is suitable for sharpness assessment. Summarizer questions require inference from distortion analysis and tool scores, such as attributing object obscurity to blur or out-of-focus artifacts [2509.26006].

## 4. Evaluation protocol, interfaces, and measured abilities

The benchmark’s primary metric is **accuracy**, reported independently for each subtrack and then overall. The paper is explicit that AgenticIQA-Eval does **not** use LLM-as-a-judge, free-form rubric scoring, SRCC/PLCC/KRCC, or explanation-similarity metrics for the benchmark proper. Labels are human-authored and human-verified, while scoring is automatic exact-answer matching over MCQs [2509.26006].

This design makes the benchmark narrower and more controlled than fully open-ended agent execution. Planning-oriented tasks are present; tool-use and tool-selection reasoning are present; instruction-following is present insofar as the model must interpret the query and role-specific behavior. By contrast, open-ended explanation generation is not the primary scoring format, multi-turn tasks are not described, and trajectory-level evaluation is not explicitly described. The benchmark is therefore best characterized as a **single-turn, MCQ-based, component-level evaluation benchmark** for planning, execution-related reasoning, and summarization over evidence [2509.26006].

The benchmark is nevertheless tightly coupled to the framework’s structured interfaces. The Planner is expected to emit JSON fields such as `query_type`, `query_scope`, `distortion_source`, `distortions`, `reference_mode`, `required_tool`, and a `plan` subobject containing boolean switches for `distortion_detection`, `distortion_analysis`, `tool_selection`, and `tool_execute`. Distortion detection outputs a `distortion_set`; distortion analysis outputs typed distortions, severity labels, and explanations; tool selection outputs a `selected_tools` mapping; and the Summarizer outputs either a multiple-choice `final_answer` with `quality_reasoning` or a free-form reasoning-plus-answer pair for instruction data [2509.26006].

The surrounding AgenticIQA framework also specifies normalization and fusion machinery for tool outputs. Tool scores are mapped to a common \([1,5]\) scale through a five-parameter monotonic logistic transformation,
$$
\hat{q}_i = \beta_1\left(\frac{1}{2} - \frac{1}{\exp(\beta_2(\tilde{q}_i - \beta_3))}\right) + \beta_4 \tilde{q}_i + \beta_5,
$$
where \(\tilde{q}_i\) is the raw tool score. The paper further defines a summarizer-side fusion using
$$
\bar{q} = \frac{1}{n} \sum_{i=1}^n \hat{q}_i
$$
and perceptual weights
$$
\alpha_c = \frac{\exp(-\eta(\bar{q} - c)^2)}{\sum_{j=1}^5 \exp(-\eta(\bar{q} - j)^2)},
$$
with \(\eta = 1\) in experiments, but notes that the printed final score equation contains a typesetting error. These formulas belong to the surrounding framework rather than the benchmark metric itself [2509.26006].

## 5. Empirical results

The main benchmark results are reported as average accuracy of agent-level performance under the AgenticIQA framework. **GPT-4o** is the strongest model overall at **76.30%**, while **Qwen2.5-VL\*** is the strongest open-source model at **75.40%**. **Human** performance is **81.80%**, leaving a visible human–model gap. Other reported systems include **mPLUG-Owl3**, **InternVL2.5**, **LLaVA-OneVision**, **Qwen2.5-VL**, **Q-Instruct**, **Q-SiT**, **Claude-3.5-Sonnet**, and **Gemini-2.0-Flash** [2509.26006].

The subtrack pattern is more informative than the overall ranking. Planning and summarization are generally easier than distortion and tool subtasks for many VLMs. The hardest component is **distortion reasoning**: the best Executor-Distortion score is **64.40%** from GPT-4o. Tool selection can remain relatively strong even when distortion reasoning is weaker; for example, **LLaVA-OneVision** reaches **77.20%** on Executor-Tool despite a substantially lower distortion score. The paper treats this as evidence that low-level perceptual reasoning remains a bottleneck for agentic VLMs [2509.26006].

The effect of agentic instruction tuning is also localized rather than uniform. The paper compares base **Qwen2.5-VL** with **Qwen2.5-VL\*** and reports an overall improvement from **73.20%** to **75.40%**, with the largest gain in **Executor Distortion**: **55.60% \(\rightarrow\) 63.20%**, a **+7.6** increase. Planner improves from **74.40%** to **76.80%**, Summarizer from **84.80%** to **85.20%**, but Executor-Tool decreases from **78.00%** to **76.40%**. The paper therefore concludes that the major benefit of agentic tuning is in **distortion reasoning**, not tool selection [2509.26006].

One model, **Q-SiT**, degrades sharply on this benchmark, reaching **29.60%** overall, which the paper attributes to fine-tuning on rigid fixed-format instructions that harms flexible agentic reasoning. By contrast, general-purpose VLMs show moderate planning and summarization ability due to stronger language competence, even when their low-level perceptual decisions remain weaker [2509.26006].

## 6. Interpretive significance, neighboring evaluation paradigms, and limitations

AgenticIQA-Eval is significant because it redefines IQA benchmarking around **behavioral decomposition**. Rather than asking only whether a system predicts a MOS-aligned score, it asks whether a VLM can behave like a competent IQA agent that parses intent, reasons about degradations, chooses tools intelligently, and synthesizes intermediate evidence. In that sense it functions as a benchmark for **agentic perception behavior**, not merely end-to-end quality prediction [2509.26006].

A common misconception follows from the acronym. Despite its name, AgenticIQA-Eval is distinct from **“IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering”** [2408.13545]. The latter concerns **interactive question answering**, where an LLM-based Evaluation Agent simulates a human user and then judges the dialogue. AgenticIQA-Eval instead concerns **image quality assessment**, is organized around planner/executor/summarizer roles, and is officially scored through MCQs rather than multi-turn interaction [2509.26006][2408.13545]. The similarity in naming obscures a substantial difference in object of evaluation: interactive QA in one case, agentic image-quality reasoning in the other.

The benchmark also has explicit and implicit limitations. The paper notes **error propagation**, since the framework assumes reliable decomposition and tool selection; **backbone dependence**, because performance is bounded by the underlying VLM; and **efficiency/scalability constraints**, because sequential agentic reasoning is slower and more resource-intensive. In broader-impact discussion it also notes that VLM-based IQA may inherit biases from training data and amplify subjective quality judgments across demographic or cultural contexts [2509.26006].

Benchmark-specific limitations are equally important. AgenticIQA-Eval is **MCQ-based**, so it does not directly score free-form explanation faithfulness. It does **not** explicitly score full trajectories or raw tool-execution correctness as a standalone metric. Robustness under major distribution shift is unclear, and the benchmark may depend on the particular distortion and tool ontology encoded by the framework. The paper therefore positions it as a meaningful step toward benchmarking **agentic IQA ability**, but not as a complete open-ended evaluation of all perceptual-agent behaviors [2509.26006].

Source: https://www.emergentmind.com/topics/agenticiqa-eval