DeViBench: Benchmark for Degraded Video

• DeViBench is a benchmark that provides large-scale, degradation-sensitive QA pairs to assess MLLM performance on low-bitrate video streams.
• It employs an automatic QA generation pipeline with dual MLLM verification to capture codec-induced blur and blocking artifacts.
• The framework highlights trade-offs among bitrate, accuracy, and latency, offering insights for optimizing real-time AI video communication.

The Degraded Video Understanding Benchmark (DeViBench) is a dataset and evaluation framework developed to systematically quantify the impact of video quality degradation—primarily aggressive bitrate reduction—on the response accuracy of Multimodal LLMs (MLLMs) in real-time video communication (RTC) scenarios where the “viewer” is an AI rather than a human. DeViBench is the first benchmark to provide large-scale, detail-sensitive question–answer (QA) pairs that by construction are sensitive to codec-induced blur and blocking artifacts, effectively tracing the critical interplay between networked video streaming and MLLM-based semantic understanding (Wu et al., 14 Jul 2025, Wu et al., 13 Feb 2026).

1. Motivation and Differentiation

Conventional RTC benchmarks and streaming-video LLM datasets, such as StreamingBench, focus either on human perceptual quality (measured by metrics such as VMAF) or LLM intelligence under high-fidelity video (e.g., 4 Mbps). Empirical evaluations have shown that only approximately 8% of QA pairs in these benchmarks are degradation-sensitive, with the majority unaffected by bitrate-induced blur or artifacts. However, video-based MLLM assistants frequently encounter queries—such as text recognition, object or detail counting, and attribute perception—that critically depend on video fidelity. DeViBench addresses this gap by generating QA pairs that are explicitly and verifiably correct on the high-quality (HQ) version of a video, but incorrect when evaluated on a severely degraded, low-bitrate (LB) version at 200 Kbps.

By systematically constructing and filtering QA pairs tied to bitrate-sensitive content, the benchmark provides a more stringent and informative evaluation for RTC systems where the ultimate consumer is an MLLM, not a human—capturing the unique failure modes and needs of AI-based video understanding (Wu et al., 14 Jul 2025, Wu et al., 13 Feb 2026).

2. Benchmark Design and Construction

Video Source and Dataset Composition

DeViBench repurposes base videos from StreamingBench, covering six diverse real-world scene types under different lighting and weather conditions, with a total raw video duration of 88,680 seconds (≈24.6 hours) (Wu et al., 13 Feb 2026). Each video is encoded at standard RTC resolutions (e.g., 720p or 1080p @ 30 FPS), then transcoded to produce a 200 Kbps LB variant. The HQ and LB clips are concatenated horizontally to facilitate direct comparison and automated QA generation.

The benchmark provides 1,968 free-response QA samples distributed across semantic categories (Table 1):

Semantic Category	Percentage (%)
Text-rich understanding	81.86
Attribute perception	11.53
Object perception	3.56
Counting	2.54
Action perception	0.36
Spatial reasoning	0.15

QA pairs cover both intra-frame (91.72%) and inter-frame (8.28%) temporal dependencies. DeViBench includes a manually spot-checked validation split (100 samples; 96% answerable, 93% correct), alongside a held-out test set (1,868 samples), supporting both hyperparameter tuning and unbiased evaluation (Wu et al., 13 Feb 2026).

Automatic QA Generation Pipeline

The QA construction pipeline utilizes large, state-of-the-art MLLMs as both generators and verifiers. The process comprises three steps (Wu et al., 14 Jul 2025):

1. Prompting: An MLLM is prompted to generate QA pairs that probe differences between the HQ and LB regions.
2. QA Filtering: QAs are tested on both HQ and LB: retained only if the HQ answer is correct and the LB answer incorrect, yielding ≈25.2% of initial candidates.
3. Cross-Verification: A second MLLM verifies QA pairs, retaining only those for which the HQ answer is stable across models, resulting in ≈14.6% survival rate from the original pool.

3. Degradation Scenarios and Parameterizations

DeViBench targets degradations relevant to real-world mobile RTC, primarily bitrate reduction and packet loss due to network fluctuations.

• Bitrate Reduction: The primary setting is a severe drop from ≥2 Mbps/HQ to 200 Kbps/LB, with spatial resolution and frame rate (30 FPS) held constant.
• Packet Loss/Frame Drops: Experiments parameterize packet loss as an i.i.d. rate $p_t$, with packets per frame $\kappa_t \approx$ frame_size/MTU. Frame success probability is $P_n = (1-p_t)^{\kappa_t}$, and the probability of at least one successful frame per MLLM query interval is $P_s = 1 - (1 - P_n)^K$, where $K= \lceil R_t/R_m \rceil$ (Wu et al., 14 Jul 2025).
• Network Emulation: Real-world 5G uplink traces are replayed through emulation tools (Mahimahi) with cap and drop-tail queue, using either WebRTC-GCC or BBR congestion control (Wu et al., 13 Feb 2026).

Notably, DeViBench does not utilize hand-crafted packet-loss formulae in its primary experiments; degradation is induced by encoder quantization parameter (QP) adjustments under bandwidth constraints.

4. Evaluation Protocols and Metrics

The primary evaluation task is video question answering (Video-QA): given a video segment (HQ or LB) and a natural-language question, the MLLM produces a free-form answer. For each QA in the test set:

• Answer correctness is determined via string-similarity measures as judged by a frozen MLLM (e.g., GLM-4.6V-Flash).
• Accuracy is defined as the fraction of QA pairs answered correctly.
• Latency metrics include average end-to-end frame latency ($D_t = D_{encode} + D_{transmit} + D_{decode}$) and stalling latency (delay due to retransmissions or awaiting the next usable frame).

Generative captioning metrics, such as BLEU or METEOR, are not used in DeViBench, as the focus is on binary QA correctness (Wu et al., 14 Jul 2025, Wu et al., 13 Feb 2026).

5. Key Experimental Findings

DeViBench experimental results reveal several critical patterns and trade-offs:

• Accuracy vs. Bitrate: Standard encoding requires ≈3,171 Kbps to reach 90% QA accuracy on MLLMs. The ZeCoStream context-aware streaming module achieves 90% at only ≈908 Kbps, and at 290 Kbps raises accuracy from 39% (standard) to 60%.
• Bitrate Thresholds: For context-agnostic streaming, QA accuracy collapses (<30%) below ~300 Kbps. Context-aware encoding supports >80% accuracy at 200 Kbps (Wu et al., 14 Jul 2025).
• Latency vs. Throughput: Aggressive bitrate reduction (operating below the “traditional ABR” region) enables sub-10 ms frame latency under 30 ms one-way delay conditions, compared to persistent stalling at higher but unstable bitrates.
• Adaptive Bitrate and Frame Rate: Response Capability-aware Adaptive Bitrate (ReCapABR)—which caps bitrate based on MLLM confidence saturation—reduces uplink bitrate by 47–70%, lowers latency by ≈148 ms, and increases the proportion of frames delivered with <200 ms latency from 65.5% (WebRTC-GCC baseline) to 82.5%.
• QA Sensitivity: Approximately 25% of randomly generated QA pairs are quality-sensitive, reflecting the need for benchmarks tailored to AI-based understanding rather than human perceptual thresholds (Wu et al., 13 Feb 2026).

End-to-end, DeViBench shows that joint context-aware encoding and loss-resilient scheduling can achieve AI-interactive video chat with 300 ms total latency at 150–400 Kbps uplink rates while retaining high MLLM response fidelity (Wu et al., 14 Jul 2025).

6. Streaming System Design Recommendations

DeViBench offers empirically supported guidelines for designing RTC systems dedicated to AI video understanding:

• Context-Aware Bit Allocation: Use CLIP-style cross-modal scores $\rho_{mn}$ for each spatial patch, and map to QP values via $$QP_{mn} = 51 \cdot [1 - ((\rho_{mn} + 1)/2)^\gamma]$$, $\gamma=3$, concentrating bits on semantically relevant regions (Wu et al., 14 Jul 2025).
• Loss-Resilient Frame Rate: Maintain sender frame rate $R_t$ to ensure $\kappa_t \approx$0 probability of delivering at least one usable frame per MLLM inference: $\kappa_t \approx$1.
• Bitrate Capping at Confidence Saturation: When MLLM’s normalized confidence gap $\kappa_t \approx$2 falls below threshold $\kappa_t \approx$3, adapt bitrate: $\kappa_t \approx$4, $\kappa_t \approx$5.
• Latency Budgeting: Minimize video transmission time to ≤68 ms to accommodate AI response time within a 300 ms total interaction budget.

7. Impact, Limitations, and Future Directions

DeViBench establishes the paradigm of evaluating RTC systems on “AI understanding” rather than human viewing, supporting co-optimization of network and MLLM modules. With its open-source release [https://github.com/pku-netvideo/DeViBench], DeViBench provides a reproducible baseline for future research targeting robust, detail-sensitive AI video assistants (Wu et al., 13 Feb 2026).

Identified research directions include proactive context prediction (enabling encoding before explicit user queries), semantic layering for long-term AI memory, and region-based token pruning to accelerate autoregressive MLLM inference (Wu et al., 14 Jul 2025). A plausible implication is the shift of network adaptation logic toward MLLM-centric metrics, with integrated QA-driven feedback loops enabling sustained accuracy in non-ideal networking conditions.