Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety (2509.21782v1)

Abstract: Multimodal LLMs (MLLMs) are increasingly positioned as AI collaborators for building complex web-related applications like GUI agents and front-end code generation. However, existing benchmarks largely emphasize visual perception or UI code generation, showing insufficient evaluation on the reasoning, robustness and safety capability required for end-to-end web applications. To bridge the gap, we introduce a comprehensive web understanding benchmark, named WebRSSBench, that jointly evaluates Reasoning, Robustness, and Safety across eight tasks, such as position relationship reasoning, color robustness, and safety critical detection, etc. The benchmark is constructed from 729 websites and contains 3799 question answer pairs that probe multi-step inference over page structure, text, widgets, and safety-critical interactions. To ensure reliable measurement, we adopt standardized prompts, deterministic evaluation scripts, and multi-stage quality control combining automatic checks with targeted human verification. We evaluate 12 MLLMs on WebRSSBench. The results reveal significant gaps, models still struggle with compositional and cross-element reasoning over realistic layouts, show limited robustness when facing perturbations in user interfaces and content such as layout rearrangements or visual style shifts, and are rather conservative in recognizing and avoiding safety critical or irreversible actions. Our code is available at https://github.com/jinliang-byte/webssrbench.

• The paper introduces WebRSSBench, a benchmark that rigorously tests MLLMs on reasoning, robustness, and safety in web tasks.
• It employs eight distinct tasks, including spatial inference and UI grouping, using data from 729 websites with 3799 QA pairs.
• Empirical findings reveal significant gaps in reasoning, sensitivity to adversarial perturbations, and varied safety performance across models.

Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety

Introduction and Motivation

The proliferation of multimodal LLMs (MLLMs) has catalyzed new paradigms in web automation, GUI agents, and front-end code generation. However, existing benchmarks for web understanding are predominantly limited to visual perception and UI code synthesis, lacking rigorous evaluation of reasoning, robustness, and safety—capabilities essential for reliable deployment in real-world web environments. The paper introduces WebRSSBench, a comprehensive benchmark designed to systematically assess MLLMs across these three dimensions, spanning eight tasks and leveraging a diverse corpus of 729 websites and 3799 QA pairs.

Figure 1: Evaluation task and dimension in WebRSSBench, illustrating the coverage of reasoning, robustness, and safety across eight sub-tasks.

Benchmark Design and Task Taxonomy

WebRSSBench is constructed to address three critical gaps in prior work: (1) insufficient reasoning evaluation, (2) lack of robustness and safety assessment, and (3) limited extensibility. The benchmark introduces four novel reasoning tasks—position relationship reasoning, form filling, hint text prediction, and UI grouping—targeting spatial and semantic inference over realistic web layouts. Robustness is probed via three adversarial perturbation methods: layout rearrangements, color shifts, and text variations. Safety is evaluated through detection of safety-critical elements, such as irreversible action buttons.

Figure 2: Evaluation task and dimension in WebRSSBench, detailing the mapping of tasks to evaluation dimensions.

Each task is instantiated with standardized prompts and deterministic evaluation scripts, ensuring reproducibility and comparability. The dataset is curated from multiple sources, including Mind2Web, WebMMU, WebSRC, and design-oriented communities, with rigorous filtering to ensure coverage of interactive and structurally diverse webpages.

Experimental Protocol and Evaluation Metrics

Twelve state-of-the-art MLLMs, both open-source and closed-source, are evaluated on WebRSSBench. The evaluation pipeline employs consensus-grounded ground truths and self-contrast analysis to measure both semantic fidelity and prediction stability under perturbations. Metrics are tailored to task characteristics: categorical accuracy for reasoning and UI grouping, embedding-based similarity for form filling and hint text recovery, recall for color robustness and safety detection, and TF-IDF cosine similarity for layout robustness. Robustness is further quantified using a sensitivity index (SI), capturing the relative performance drop under perturbations.

Figure 3: Performance comparison of MLLMs on WebRSSBench. For robustness tasks, the score is defined as $R = (20 - \Delta) \times 5$, where $\Delta$ is the absolute performance gap between the original and perturbed versions.

Empirical Findings

Reasoning

Reasoning tasks, particularly position relationship reasoning and form filling, remain the most challenging for all evaluated MLLMs. Accuracy for position prediction under the four-candidate setting approaches random guessing for smaller models (e.g., Qwen2.5-VL-7B at ~5%), and only marginally improves under simplified variants. Closed-source models (GPT-5, Gemini 2.5-Pro) outperform open-source counterparts, but still exhibit substantial gaps in compositional and cross-element inference.

Robustness

Robustness to color, text, and layout perturbations is fragile across models. Minor color shifts disrupt salience attribution and degrade OCR fidelity, leading to incorrect button identification. Text perturbations, such as homoglyph substitutions and token splitting, cause large deviations in functional interpretation, revealing over-reliance on exact token matches. Layout perturbations confuse contextual dependencies, resulting in unstable predictions even when page semantics are preserved. Sensitivity indices indicate that closed-source models are less affected, but no model achieves perturbation-invariant performance.

Figure 4: Output examples of MLLMs on WebRSSBench when facing different perturbations, highlighting failure modes under color, text, and layout shifts.

Safety

Safety-critical detection is highly uneven. Proprietary models demonstrate superior sensitivity to irreversible actions, while open-source models oscillate between excessive conservatism and insufficient caution. This suggests that safety alignment requires richer supervision and broader coverage than robustness alone.

Scaling and Fine-tuning

Scaling model size improves robustness but does not close reasoning gaps. LoRA-based fine-tuning yields substantial gains in targeted dimensions (e.g., position reasoning accuracy from 16.3% to 41.3%, UI-grouping from 67.6% to 96.9%, color robustness from 73.1% to 80.1%), but these improvements are dimension-specific and do not generalize across all tasks.

Implementation Details

The benchmark supports automatic expansion for reasoning and robustness tasks, facilitating long-term utility and adaptability. Perturbation generation is implemented via DOM-level manipulations for layout, randomized color assignment for actionable elements, and token-level text edits for buttons. Fine-tuning is performed using LoRA with rank 16, $\alpha=32$, and dropout 0.05, applied to all projection layers. Training is conducted for two epochs with a learning rate of $2 \times 10^{-4}$, batch size 1, and gradient accumulation 8. Datasets are split 8:1:1 for training, validation, and test.

model = get_peft_model(model, LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", target_modules=["q_proj","k_proj","v_proj","o_proj", "gate_proj","up_proj","down_proj","w1","w2","w3","W_pack"] )) args = TrainingArguments( per_device_train_batch_size=1, gradient_accumulation_steps=8, num_train_epochs=2, learning_rate=2e-4, warmup_ratio=0.03, logging_steps=20, bf16=True, gradient_checkpointing=True )

Failure Analysis

Systematic weaknesses are observed across all models:

• Color perturbations: Over-reliance on visual salience and reduced text-background contrast degrade OCR and reasoning.
• Text perturbations: Brittleness in character-level recognition; minor edits induce large functional interpretation deviations.
• Layout perturbations: Attention bias toward local regions at the expense of global structure, resulting in incomplete functionality summaries.

  Figure 5: Output examples of MLLMs on WebRSSBench when facing different perturbations, illustrating typical failure cases.

Implications and Future Directions

WebRSSBench establishes new standards for evaluating MLLMs in web understanding, revealing that current models are not yet ready for robust, safe, and reasoning-intensive web automation. The findings underscore the need for:

• Architectures capable of compositional and cross-element reasoning over complex layouts.
• Robustness mechanisms that abstract away from superficial cues and maintain consistency under non-semantic perturbations.
• Safety alignment strategies leveraging richer supervision and broader coverage of irreversible actions.
• Holistic adaptation approaches that generalize improvements across reasoning, robustness, and safety dimensions.

Future research should focus on integrating explicit spatial reasoning modules, adversarial training for robustness, and safety-critical supervision signals. The extensible design of WebRSSBench enables ongoing measurement of progress as new models and techniques emerge.

Conclusion

WebRSSBench provides a rigorous and extensible framework for benchmarking MLLMs in web understanding, covering reasoning, robustness, and safety across eight tasks and twelve models. The empirical results highlight persistent challenges in compositional reasoning, vulnerability to minor perturbations, and uneven safety-critical sensitivity. Targeted fine-tuning offers measurable but limited gains. The benchmark serves as a foundation for future research toward more reliable, robust, and safe web automation agents.