Papers
Topics
Authors
Recent
Search
2000 character limit reached

Robust-kbench: Benchmarking Model Robustness

Updated 2 July 2026
  • robust-kbench is a benchmarking ecosystem designed to stress-test CUDA-generated kernels and audio-visual models under real-world perturbations.
  • It employs multi-seed, multi-task evaluations with error-corrected LLM code generation and extensive corruption simulations to ensure reliable performance metrics.
  • Empirical findings reveal notable speedups and expose failure modes, guiding the development of more adaptive and robust AI systems.

robust-kbench is a term applied to multiple high-impact benchmarking frameworks and datasets focused on evaluating model robustness under challenging, non-ideal conditions. It is notably associated with two state-of-the-art resources: (1) the rigorous CUDA kernel benchmarking harness introduced to address the flaws of previous LLM code-generation evaluations (Lange et al., 16 Sep 2025), and (2) AVRobustBench, a comprehensive suite for testing audio-visual (AV) recognition models against co-occurring, correlated multimodal corruptions (Maharana et al., 31 May 2025). These frameworks establish stringent, domain-specific protocols to identify and quantify failure modes that escape conventional benchmarks.

1. Definition and Objectives

robust-kbench, in its primary form, is a CUDA-centric benchmark and agentic evaluation ecosystem designed to stress-test LLM-generated CUDA kernels for both functional correctness and deployment-level performance across diverse, non-trivial workloads and runtime contexts (Lange et al., 16 Sep 2025). It systematically eliminates benchmark loopholes (overfitting, trivialization, single-shape bias) and mandates rigorous multi-seed, multi-shape, forward/backward, and hardware-sensitive evaluation.

An alternative usage, especially in multimodal learning, refers to AVRobustBench (frequently synonymized as “robust-kbench”)—a suite for analyzing the test-time robustness of AV recognition systems subjected to co-occurring, correlated audio-visual corruptions, replicating real-world degradations beyond unimodal or i.i.d. perturbations (Maharana et al., 31 May 2025).

Both implementations share a central motivation: to expose the limitations of current methods under realistic noise and stress conditions, thereby promoting research on algorithms that generalize and remain reliable under deployment-level perturbations.

2. Benchmark Construction and Task Design

CUDA Kernel Benchmarking

robust-kbench for CUDA rigorously evaluates kernels generated—often via LLM pipelines—for realistic deep learning operations. Key features include:

  • Task Set Diversity: 12 forward-pass and 8 backward-pass tasks, including fused conv2d–ReLU–MaxPool, linear layers, fused cross-entropy, ResNet-18 blocks, transformer feed-forward components, LayerNorm, and RMSNorm.
  • Input/Initialization Variation: Each task is evaluated across multiple input shapes (both seen and held-out), initialization states (uniform, Xavier, normal), and across different random seeds.
  • Profiling Depth: Benchmarks are profiled at both high (PyTorch profiler) and low level (NVidia Compute Profiler, Clang-tidy), and on different GPU architectures (H100, RTX 4090, A6000).

Audio-Visual Recognition Robustness

In the AV domain, robust-kbench embraces four standard datasets: AudioSet-2C, VGGSound-2C, Kinetics-2C, and EpicKitchens-2C. Each original test set is augmented with 75 bimodal corruptions (15 joint corruption types × 5 severity levels), simulating deployment scenarios with simultaneous and correlated perturbations in both audio and visual streams. Corruptions span:

  • Digital noise (Gaussian, impulse, shot, speckle, JPEG compression)
  • Environmental artifacts (snow, frost, spatter, motion blur, rain, underwater)
  • Human-related disruptions (concert, smoke, crowd, interference)

This enables systematic, large-scale measurement of model degradation under realistic, co-occurring noise.

3. Evaluation Protocols and Metrics

CUDA Kernel Setting

robust-kbench enforces:

  • Sequential Workflow:
    • Translation Phase: PyTorch code is converted to CUDA kernels by LLMs, with iterative error-feedback correction.
    • Evolutionary Meta-Generation: 10 generations of LLM proposals (with self-verification and filtering), optimizing both correctness and runtime.
    • Generalization Testing: Evaluation of candidate kernels on held-out inputs to ensure generalizability.
  • Metrics:
    • Latency (mean over 2000+ runs)
    • Relative speedup: S=Ttorch/TcudaS = T_{\text{torch}} / T_{\text{cuda}}
    • Throughput (TFLOPS for GEMM)
    • Pass/fail correctness across all seeds and shapes
  • LLM-Based Self-Verification:
    • Three specialized verifiers (compilation, memory, numerical), tuned via small labeled datasets and majority vote ensembles. Compilation accuracy reaches ≈0.82; memory ≈0.80; numerical ≈0.73.
  • Scoring/Ranking:
    • Among proposals passing all checks, kernels are ranked by measured latency; robust performance is defined only if generalization holds across all held-out configurations and seeds.

AV Recognition Setting

  • Metrics:
    • For each corruption and severity, absolute (αi,s\alpha_{i,s}) and relative (ρi,s\rho_{i,s}) robustness scores are reported:

    αi,s=1−δAi,s100 ,ρi,s=1−δAi,sAcl\alpha_{i,s} = 1 - \frac{\delta A_{i,s}}{100}\,,\qquad \rho_{i,s} = 1 - \frac{\delta A_{i,s}}{A_{\mathrm{cl}}}

    where δAi,s\delta A_{i,s} is the accuracy drop from the clean baseline AclA_{cl}. - Main metrics: top-1 accuracy (VGGSound-2C, Kinetics-2C, EpicKitchens-2C), mean AP (AudioSet-2C). - Test-time adaptation (TTA) gains are measured in terms of post-TTA accuracy improvement at highest corruption severity.

  • Adaptation Protocols:

    • Standard online TTA (e.g., TENT, RPL, EATA, SAR, READ, SuMi) as well as the AV2C method (cross-modal query/key/value adaptation with selective entropy minimization).

4. Empirical Findings and Failure Mode Analysis

CUDA Kernel Results

  • Performance Gains:
    • Forward pass speedups: 1.3× (ResNet block) to 3.4× (MNIST Conv–ReLU–Pool); backward passes: ≈1.1–1.2×.
    • Notable improvements using LayerNorm—12.52× over TorchCompile.
  • Verification Efficiency:
    • Invalid kernel rate reduced from 30–45% to 15–20% with LLM-based verifiers; final valid proposal rate ≈80–85%.
  • Ablation Insights:
    • Model ensembling improves speedup by ~10–15%.
    • Context construction (“least-to-most” ordering) yields additional ~5% speedup.
    • Profiling feedback increases performance by 5–10%.
  • Generalization Challenges:
    • Simpler kernels lose up to 20% speedup on unseen shapes, indicating mild overfitting; complex tasks such as ResNet block exhibit reliable generalization.

Audio-Visual Robustness Results

  • Baseline Gaps:
    • Severe accuracy drops under bimodal corruptions (s=5s=5): UAVM (VGGSound-2C): clean 65.50%, corrupted 27.41%, ρ=0.42\rho=0.42.
    • CAV-MAE and EquiAV outperform self-supervised methods by 15–20 pts under identical corruptions.
  • TTA Effectiveness:
    • For bimodal shifts, standard TTA provides marginal gains or worsens accuracy (EATA and READ are exceptions, yielding modest improvements).
    • AV2C yields strongest improvement: +5.32 pts over READ and +8.50 pts over SuMi at maximal severity on VGGSound-2C; +1.39 pts over SuMi on Kinetics-2C.
  • Failure Mode Taxonomy:
    • Cross-attention imbalance (READ) causes dominance of visual tokens in the fused model as audio signal is strongly corrupted.
    • Entropy-penalized adaptation that selectively updates cross-modal attention achieves the best robustness.

5. Impact, Limitations, and Future Directions

  • CUDA kernel ecosystem:
    • robust-kbench enforces strict, loophole-free validation, ensuring that reported speedups translate to deployable, hardware-level gains.
    • The agentic pipeline—error-corrected LLM translation, evolutionary meta-generation, LLM-based verification—represents a practical template for automated code discovery and optimization in high-performance computing.
  • Multimodal recognition field:
    • AVRobustBench (robust-kbench) highlights the inadequacy of unimodal or synthetic robustness benchmarks for AV models; only co-occurring, correlated stressors expose the full spectrum of vulnerability.
    • Current TTA methods, primarily designed for unimodal perturbations, do not consistently transfer to the bimodal regime. AV2C establishes selective attention adaptation as a promising direction.
  • Open Problems and Extensions:
    • CUDA: Incorporate hardware-specific code generation (target-SM, tensor-core WMMA), fine-tune reinforcement learning agents on archived kernels, increase backward-operator coverage, and extend task diversity to sparse/mixed-precision contexts.
    • AV: Broaden corruptions to include streaming, asynchronous, or real-world time-varying disruptions; design unified adaptation frameworks that explicitly couple audio and visual uncertainty during TTA.
  • Resource Availability:

6. Comparative Summary Table

Domain Key Properties Representative Metrics
CUDA kernel gen. Multi-shape, multi-seed, forward+backward, LLM Speedup, TFLOPS, correctness
Audio-visual recog 75 bimodal corruptions, 4 datasets, TTA eval mAP, top-1, rel./abs. drop

Both robust-kbench implementations converge on the necessity of domain-specific, multi-faceted stress testing to drive algorithmic advances in real-world robust AI system development.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to robust-kbench.