Section-Level Multimodal Evaluation Protocol

• Section-Level Multimodal Evaluation Protocol is a framework that segments multimodal artifacts into discrete, semantically rich sections for localized assessment.
• It employs automated decomposition, modality alignment, and specific metrics to evaluate text, image, layout, and audio quality in each section.
• The approach enables targeted error diagnosis and continuous benchmarking, providing actionable feedback for model improvement and alignment.

A section-level multimodal evaluation protocol defines methods for assessing discrete, semantically meaningful sections within multimodal artifacts—such as webpage regions, document passages, dialogue turns, or image/text pairs—across multiple modalities (text, vision, layout, audio, etc.). This approach enables high-granularity, localized diagnosis of models’ capabilities, the isolation of error sources, and actionable feedback for generation and alignment improvements. Recent research has proposed a diverse set of technical implementations, architectural innovations, and theoretical frameworks that collectively establish new state-of-the-art practices in section-level multimodal evaluation.

1. Formal Frameworks and Task Decomposition

Section-level evaluation protocols are predicated on explicitly segmenting multimodal artifacts into constituent sections, assigning modality-aligned representations, and conducting focused assessment on each section. For instance, WebGen-V’s protocol (Wang et al., 17 Oct 2025) formally defines a processor that maps a webpage $W$ to a structured tuple:

$Z(W) = \{ S, T, I, M, B \}$

where $S$ is the ordered section list, $T$ the structured text assets, $I$ the classified image assets, $M$ the section and page-level metadata (e.g., color and typography), and $B$ the bounding boxes for rendered components.

These representations underpin pipelines where, instead of evaluating a whole artifact monolithically, each section $\hat{s}_i$ is evaluated in terms of relevant fine-grained metrics—such as text correctness, media alignment, or spatial layout consistency. This modularization is central not only in web generation (Wang et al., 17 Oct 2025) but also in dialogue summarization (Liu et al., 2 Oct 2025), multimodal reasoning (Zhou et al., 18 Oct 2024), and beyond.

2. Section-wise Pipeline: Processing, Representation, and Metrics

Protocols like WebGen-V and MDSEval (Wang et al., 17 Oct 2025, Liu et al., 2 Oct 2025) use agentic or algorithmic section decomposition, with each section preserving localization information (region in screenshot or DOM tree, embeddings, and asset linkage). Each section is then evaluated independently by an automatic or model-in-the-loop evaluator, leading to a feedback tuple:

$$F = \{ (\hat{s}_i, m_k, \text{score}, \text{reason}, \text{feedback}) \}$$

where $\hat{s}_i$ is the $i$th section, $m_k$ is the evaluation metric, and the rest record quantitative and qualitative output.

In MDSEval, a set of eight dimensions (such as Multimodal Coherence, Modality-specific Critical Information Coverage, Conciseness, Faithfulness, etc.) is annotated per section, enabling a multi-axis appraisal of model output quality. The MEKI (Mutually Exclusive Key Information) filtering process selects sections in which exclusive content is contributed by only one modality, directly enhancing the granularity and informativeness of the protocol (Liu et al., 2 Oct 2025):

$$\mathrm{MEKI}(I, T; S) = \lambda \cdot \mathrm{EKI}(I|T; S) + (1-\lambda)\cdot \mathrm{EKI}(T|I; S)$$

where EKI measures the projection of exclusive content.

Similarly, WebGen-V aligns the text, layout, and visual modalities for each isolated section (Wang et al., 17 Oct 2025), and assessments cover metrics such as spacing consistency (SPC), media positional accuracy (MP), text-image association (TIA), and overall multimodal coherence.

3. Multi-Modal Alignment and Model-Assisted Assessment

A distinguishing element in recent protocols is the use of advanced multimodal LLMs (e.g., GPT-4V, LLaVA-Critic, GPT-5) as automated “judges” or evaluators (Wang et al., 17 Oct 2025, Zhou et al., 18 Oct 2024, Xiong et al., 3 Oct 2024). These models are tasked not only with assigning a scalar or categorical score to the output for a given section but also with generating fine-grained error localization (such as pointing out mismatched text rendering, misaligned images, or hallucinated content).

In LLaVA-Critic (Xiong et al., 3 Oct 2024), for example, the instruction-following critic is trained on detailed per-section evaluation criteria, producing both numeric/rank outputs and natural-language justifications. The alignment with human annotators is reported as highly reliable (e.g., Pearson’s $r \approx 0.75$, Kendall’s $\tau \approx 0.93$ in matching human and GPT-4 judgments), enabling its use not only in benchmarking but also in preference learning and reward modeling for model improvement.

Evaluation outputs can be aggregated via various mathematical schemes. For example, MiCEval (Zhou et al., 18 Oct 2024) aggregates the per-step (section) scores using a geometric mean:

$$\text{Correctness}_{\text{overall}} = \left(\prod_{i=1}^n \text{Correctness}^{(i)} \right)^{1/n}$$

thus strongly penalizing any stage with poor correctness in a reasoning chain.

4. Empirical Validation, Error Diagnosis, and Benchmarking Outcomes

Experiments across multiple works demonstrate that section-level protocols uncover error types (e.g., in alignment, factuality, modality integration, and design fidelity) that page-level or holistic evaluation fails to reveal (Wang et al., 17 Oct 2025, Liu et al., 2 Oct 2025, Zhou et al., 18 Oct 2024). For example, WebGen-V’s section-level approach yields significant increases in detection of human-injected degradations (F1 from 0.46 to 0.78 for text, layout, and media categories). In MDSEval, section-level annotations facilitate pinpointing when and where multimodal coherence or faithfulness breaks down, exposing score concentration bias and enabling more robust evaluations.

Benchmarks such as MMMG (Yao et al., 23 May 2025) show that these protocols scale to a wide matrix of modalities and tasks (images, text, audio, interleaved generation), reporting human-model agreement in automated assessment as high as 94.3%, and enabling direct cross-model capability attribution at the section level.

5. Technical Implementation, Open Data, and Modular Pipelines

Most recent section-level protocols provide or advocate for open-source implementations and structured data formats that facilitate reproducibility and extensibility (Wang et al., 17 Oct 2025, Xiong et al., 3 Oct 2024). WebGen-V’s modular processor, for example, standardizes the HTML-to-structured-representation pipeline for continual real-world agentic crawling. Kaput et al. (Liu et al., 2 Oct 2025) provide all evaluation code and dataset, allowing practitioners to adapt or extend MEKI section-wise computation.

Section-level pipelines typically include:

• Automatic section decomposition (by DOM, visual cues, or semantic heuristics)
• Multi-modal feature extraction and structured representation assembly
• Model-assisted per-section metric calculation, with interfaces for manual overrides or hybrid human-in-the-loop scoring
• Optional feedback loop for iterative refinement, as described in the Gen-Eval-Refine methodology (Wang et al., 17 Oct 2025)

6. Research Implications and Future Directions

Section-level multimodal evaluation protocols support:

• Fine-grained diagnosis of model errors and capabilities, informing both model development and dataset curation
• Targeted model alignment by virtue of localized reward signals (e.g., in DPO/RL training)
• Continuous and scalable benchmarking as real-world artifacts and interfaces evolve, thanks to agentic and extensible crawling frameworks

Key open research areas include automating the section definition in highly complex or novel domains, constructing “superhuman” section-level feedback loops (e.g., scalable, multi-criteria LMM-judges), and formalizing human value and style alignment at section resolution for safety-critical or high-stakes applications.

7. Summary Table: Key Features by Protocol

Protocol	Section Segmentation	Evaluation Modality	Metrics/Outputs
WebGen-V (Wang et al., 17 Oct 2025)	DOM, visual, asset heuristics	Text, layout, vision	SPC, MP, TIA, per-section feedback
MDSEval (Liu et al., 2 Oct 2025)	Dialogue turn/topic	Text, image	8-dim. quality, MEKI
LLaVA-Critic (Xiong et al., 3 Oct 2024)	Image/question/response pairs	Text, image	Numeric score, justification
MiCEval (Zhou et al., 18 Oct 2024)	Reasoning step	Visual, reasoning	Step correctness, chain geometric mean

This section-level paradigm establishes a rigorous foundation for benchmarking and refining multimodal models, driving both research and engineering advances in high-fidelity, context-aware, and actionable model assessment.