---
title: 'CLEVER: Diverse Methods & Benchmarks'
url: https://www.emergentmind.com/topics/clever-7cddab6a-158e-4acc-ba10-e8d2f56903a1
type: topic
---

# CLEVER: Diverse Methods & Benchmarks

Searching arXiv for recent and canonical uses of “CLEVER” across research areas.
arxiv_search(query="CLEVER benchmark Lean verified code generation", max_results=5)
CLEVER is used in arXiv literature as the name of several distinct methods, benchmarks, and analytical constructs rather than a single unified framework. The term denotes, among other things, a benchmark for end-to-end verified code generation in Lean, a visually grounded commonsense knowledge acquisition framework, a robustness score for adversarial evaluation, a retrieval-based system for cross-lingual multi-hop knowledge editing, and a stream-based active learning system for robot semantic perception [2505.13938], [2211.12054], [1810.08640], [2407.10275], [2507.15499]. In parallel, “clever” also appears in work on benchmark validity and “Clever Hans” behavior, where high performance is attributed to shortcut exploitation rather than the intended capability [1911.00225], [2410.11672].

## 1. CLEVER as a benchmark for formally verified code generation

In formal methods and program synthesis, CLEVER stands for **Curated Lean Verified Code Generation Benchmark** and is designed for **end-to-end verified code generation** in Lean [2505.13938]. The benchmark contains **161 problems**, adapted from **161 of the 164 HumanEval tasks** that could be formalized in Lean 4. Each problem is organized into two certification stages. In **specification certification**, a model must generate a Lean specification \(\psi\) from the natural-language statement \(\nu\) and prove an isomorphism to a held-out human-written specification \(\psi^*\). In **implementation certification**, the model must generate an implementation \(\pi\) and prove that it satisfies the ground-truth specification. A task counts as solved only if both stages are fully verified by Lean’s type checker [2505.13938].

A central design choice is that CLEVER avoids **test-case supervision**, **LLM-generated annotations**, and specifications that leak implementation logic or allow vacuous solutions. Its specifications are **human-authored** and written as **non-computable propositions** in Lean’s `Prop`, precisely to prevent the model from copying executable structure from the specification into the program. The paper contrasts this with benchmarks in which partial property checking can be satisfied by trivial programs, and it emphasizes that CLEVER is intended to capture the **full natural-language intent** of the task [2505.13938].

The evaluation metric is **pass@k-seconds**, with \(k=600\) seconds. The paper reports that current models “all struggle to achieve full verification,” and that the best end-to-end result is only **1/161 problems**, even though compilation rates are substantially higher than proof success rates [2505.13938]. This makes CLEVER a frontier benchmark for program synthesis and formal reasoning, and it also yields fine-grained failure analysis because specification generation, equivalence proving, implementation synthesis, and correctness proving are scored separately.

## 2. CLEVER as visually grounded commonsense knowledge acquisition

In knowledge extraction, CLEVER is a framework for **visually grounded commonsense knowledge acquisition** that formulates commonsense knowledge extraction as a **distantly supervised multi-instance learning** problem over bags of images [2211.12054]. The starting point is the claim that text suffers from **reporting bias**: obvious facts such as \((\text{person}, \text{can\_hold}, \text{bottle})\) are rarely stated explicitly, whereas images directly depict grounded entity interactions. The paper reports that on Visual Genome, **83% of the triplets** sampled by the authors are not found in ConceptNet, motivating image-based acquisition as a complement to text-based commonsense extraction [2211.12054].

For an entity pair \((s,o)\), CLEVER constructs a bag of images \(\mathcal{B}_{(s,o)}=\{v_i\}_{i=1}^N\) and uses a **vision-language pre-training** encoder, specifically **VinVL**, to obtain instance representations. It then forms relation-aware bag representations
\[
\boldsymbol{B}_{(s,r_i,o)} = \sum_{j=1}^N \alpha^{r_i}_j\boldsymbol{v}_j, \qquad \alpha^{r_i}_j = \text{Softmax}_j ( \boldsymbol{v}_j^\top\boldsymbol{r}_i),
\]
and optimizes them with a contrastive objective rather than standard attention-only multi-instance learning [2211.12054]. The purpose of this **contrastive attention** is to address what the paper calls the **golden-query-only problem**, where positive relations are trained directly but negative relations are insufficiently separated.

The benchmark is built from **Visual Genome**, using the top **100 entity types** and **100 relation types**, with splits of **13,780 facts / 6,443 entity pairs / 55,911 images** for training, **1,166 facts / 678 entity pairs / 5,224 images** for validation, and **3,496 facts / 1,964 entity pairs / 13,722 images** for testing [2211.12054]. On held-out evaluation, CLEVER achieves **AUC 41.92** and **mAUC 26.57**, improving over the best PLM-based baseline, **Prompt-FT**, by **+3.9 AUC** and **+6.4 mAUC**. In human evaluation, the predicted commonsense scores show a **0.78 Spearman coefficient** with human judgment [2211.12054]. The framework is thus positioned not merely as a classifier of visual relations, but as a system for extracting grounded commonsense triplets with interpretable image-level support.

## 3. CLEVER as a robustness score for adversarial evaluation

In adversarial robustness, CLEVER stands for **Cross-Lipschitz Extreme Value for nEtwork Robustness** and is an **Extreme Value Theory (EVT) based robustness score** for large-scale deep neural networks [1810.08640]. Its objective is to provide an **attack-agnostic robustness score** by estimating a lower-bound-style quantity on the minimum perturbation required to change a classifier’s prediction. The core first-order construction estimates a local Lipschitz constant from sampled gradient norms and then converts that estimate into a robustness radius of the form
\[
\epsilon \;=\; \min\!\left(\frac{g_t(x_0)}{L_q^t},\, R\right),
\]
where \(g_t(x)=f_{c(x_0)}(x)-f_t(x)\) is a cross-class score difference [1810.08640].

The 2018 extension introduces a **second-order CLEVER score** for twice-differentiable classifiers. Defining
\[
a=\max_{x\in B}\|H(x)\|_{p,q},\quad b=\|\nabla g_t(x_0)\|_p,\quad \gamma=g_t(x_0),
\]
the formal robustness guarantee becomes
\[
\epsilon
=
\min\!\left(
\frac{-b+\sqrt{b^2+2a\gamma}}{a},
R
\right).
\]
The same paper also proposes combining CLEVER with **Backward Pass Differentiable Approximation (BPDA)** so that models with **non-differentiable input transformations** can still be evaluated in a white-box manner [1810.08640].

This line of work is accompanied by a direct critique. A separate paper argues that **gradient masking causes CLEVER to overestimate the size of perturbation needed to fool the model** and states that “an estimate of a bound is not a bound” [1804.07870]. Its counterexample uses staircase-like quantization and Lipschitz-continuous approximations with narrow ramps, showing that finite sampling can miss the regions where gradients are concentrated. The controversy is therefore methodological: CLEVER was introduced as a scalable, attack-independent robustness estimator, but its validity depends on whether sampled gradients faithfully represent local sensitivity [1810.08640], [1804.07870].

## 4. CLEVER in retrieval-based editing and stream-based semantic perception

A later use of the name appears in **CLEVER-CKE**, a retrieval-based framework for **cross-lingual multi-hop knowledge editing** [2407.10275]. The setting is harder than standard monolingual editing because the model must both transfer edited facts across languages and propagate those edits through a reasoning chain. To evaluate this, the paper constructs **CROLIN-MQUAKE**, a parallel benchmark derived from **MQUAKE-CF** and **MQUAKE-T**, with **3k examples** for the counterfactual set, **1.8k examples** for the temporal set, and **eight languages total**: English, German, Spanish, Chinese, Russian, Hindi, Bengali, and Swahili [2407.10275].

CLEVER-CKE is a **parameter-preserving** method with a **retrieve–verify–generate** pipeline. A retriever searches an external multilingual memory of edited facts, a verification step checks whether the retrieved fact is sufficiently close to the sub-question, and the LLM either uses the retrieved fact or falls back to internal knowledge. The retriever is trained with three losses: a **semantic distinction loss** \(L_{SD}\), a **cross-lingual edit consistency loss** \(L_{CLEC}\), and a **binary cross-entropy loss** \(L_{BCE}\), combined as
\[
L_{total} = L_{SD} + L_{CLEC} + L_{BCE}.
\]
Across three LLMs, eight languages, and two datasets, the paper reports gains of **up to 30%** over prior methods [2407.10275].

Another distinct usage is the robotics system **CLEVER: Stream-based Active Learning for Robust Semantic Perception from Human Instructions** [2507.15499]. Here the problem is **distribution shift** in robot perception. CLEVER is a **stream-based active learning system for robust semantic perception with Deep Neural Networks (DNNs)** that asks a human for support when uncertainty is high, receives instructions and demonstrations, and updates a Bayesian classifier online [2507.15499]. The architecture uses **unknown object segmentation** such as **Segment Anything (SAM)**, feature extraction with **DINOv2**, and **one multilayer perceptron (MLP) head per object class**.

The Bayesian formulation places a posterior over class-specific parameters,
\[
\rho^{(c, t)}=p(\boldsymbol{\theta}^{(c, t)} | \mathcal{D}^{(c, t)}),
\]
and prediction is performed by Bayesian marginalization [2507.15499]. Querying is driven by uncertainty after temporal filtering, and sample selection uses **BatchBALD**. In a user study with **13 participants**, the system reports, for **CLEVERv1**, a **success rate of 0.887 \pm 0.049**, **ECE 0.060 \pm 0.017**, **precision 0.933 \pm 0.020**, and **1.539 \pm 0.544** queries to reach 85% precision; it also reports average training time of **17.79 \pm 4.717 s** [2507.15499]. The paper positions this as the first realization of stream-based active learning on a real robot.

## 5. CLEVER and the diagnosis of shortcut learning

A substantial body of work uses “clever” in the **Clever Hans** sense: a system appears to solve the task, but does so by exploiting unintended cues. In commonsense NLP, **Balanced COPA** was introduced after an analysis showed that original COPA contained **single-token distributional imbalances** and that **RoBERTa trained only on the alternatives** reached about **59.6% accuracy**, above the **50%** random baseline [1911.00225]. The mirrored construction of Balanced COPA ensures that every alternative appears once as correct and once as incorrect, and human evaluation reports **100%** accuracy on original COPA and **97%** on Balanced COPA, indicating that the task remains solvable after shortcut cues are removed [1911.00225].

The same concern has been extended to modern LLM benchmarks. One study evaluates **19 multiple-choice benchmarks** with **logistic regression** over simple **unigrams**, **bigrams**, and **readability and diversity metrics**, using **Cohen’s \(\kappa\)** and treating **\(\kappa > 0.2\)** as detectable agreement [2410.11672]. It reports that **9/19 benchmarks** show evidence of agreement between simple-feature predictions and ground truth, with **Corporate Lobbying** and **SpaceNLI** showing **consistent \(\kappa > 0.6\)**. The paper also reports adjusted one-sided paired t-test results suggesting that benchmark-predictable items are easier for some model families, including **Meta**, **OpenAI**, **Mistral AI**, and **Writer** [2410.11672]. This literature treats benchmark internal validity as a first-class research problem.

Related work shows that Clever Hans effects are not restricted to supervised classification. In anomaly detection, an explainable AI framework based on **LRP / Deep Taylor Decomposition** reveals that density-based, reconstruction-based, and boundary-based models often detect anomalies for the wrong reasons, and it introduces a **Clever Hans score** defined as
\[
\text{Clever Hans score} = \text{detection accuracy} - \text{explanation accuracy}
\]
on **MNIST-C** and **MVTec** [2006.10609]. A broader study of unsupervised learning then argues that Clever Hans effects are **widespread in unsupervised learning**, including representation learning and anomaly detection, and attributes them primarily to **inductive biases** in the learning objective and architecture rather than only to dataset artifacts [2408.08041].

This diagnostic program also led to mitigation methods. **Class Artifact Compensation (ClArC)** uses explanation maps and **Spectral Relevance Analysis** to quantify artifact-prone classes and then applies either **A-ClArC** or **P-ClArC** to reduce reliance on class-specific artifacts [1912.11425]. In translationese classification, topic-label alignment derived from **LDA** and **BERTopic** is reinterpreted as a **topic floor**, with average alignment values around **0.55 to 0.62** and a proposed floor of about **0.62** [2308.13170]. In medical imaging, a study on ADNI T1-weighted MRI reports **990 AD MRIs** and **990 NC MRIs**, and finds that binarized, skull-stripped inputs retain performance comparable to non-binarized references, revealing a Clever Hans effect in which the network depends primarily on **volumetric atrophy**, **brain shape / mask geometry**, and **skull-stripping artifacts** rather than gray-white matter texture [2501.15831]. More recently, **Counterfactual Knowledge Distillation (CFKD)** addresses **Clever Hans strategies in image classifiers** by generating and validating counterfactuals; on five datasets it reports strong gains in **average group accuracy**, including **94.5** on Square and **83.9** on Follicles [2510.17524].

## 6. Other technical uses and recurrent disambiguation

The name also appears in several domain-specific constructs outside machine learning evaluation. In computational biology, **CLEVER: Clique-Enumerating Variant Finder** is a structural-variant caller that organizes read alignments into a graph where **max-cliques represent maximal contradiction-free groups of alignments** [1203.0937]. The method targets indels from roughly **20 bp up to 50,000 bp**, uses all alignments including concordant reads, and is reported to be particularly strong in the **20–100 bp** range [1203.0937].

In geometric computer vision, the phrase “clever elimination strategy” denotes an algebraic method for generating smaller and faster minimal solvers by first eliminating unknowns that do not appear in the linear measurement equations and then extending solutions to the remaining unknowns [1703.05289]. The paper demonstrates this on several relative pose problems, including \(f+E+f\), \(E+f\), and \(E+f+k\), reducing elimination templates from **\(31\times 46\) to \(21\times 36\)**, from **\(21\times 30\) to \(6\times 15\)**, and from **\(200\times 231\) to \(51\times 70\)**, respectively [1703.05289].

There is also a probabilistic usage in **“A clever (self-repelling) burglar”**, where the “cleverness” refers to a structural property of **true self-repelling motion**. The paper proves that, conditionally on the occupation-time density \(\Lambda_1(\cdot)\), the law of \(X_1\) is **uniform on a certain admissible interval**, a result contrasted with the Brownian-motion setting of Warren and Yor [1201.4057].

Finally, CLEVER is often confused with **CLEVR**, the synthetic VQA benchmark whose name is pronounced “clever.” CLEVR contains **100,000 rendered images**, **about 1 million automatically generated questions**, and **853,554 unique questions**, with **70k train / 15k val / 15k test** splits [1612.06890]. It was designed as a diagnostic dataset for **compositional language and elementary visual reasoning**, with functional programs, controlled attributes, and reduced question-conditional bias [1612.06890]. This suggests that, across fields, “CLEVER” and closely related naming conventions are repeatedly used for systems whose purpose is diagnostic rigor, formal control, or shortcut-resistant evaluation, even though the underlying technical objects are heterogeneous.

Source: https://www.emergentmind.com/topics/clever-7cddab6a-158e-4acc-ba10-e8d2f56903a1