CLEVER: Diverse Methods & Benchmarks

• CLEVER is a multifaceted term referring to diverse methods and benchmarks that span end-to-end verified code generation in Lean, visually grounded commonsense acquisition, adversarial robustness, and more.
• The CLEVER benchmark for formal verification challenges models with two-stage certification, requiring both specification generation and implementation proof, revealing significant difficulties in full verification.
• Additional CLEVER frameworks include retrieval-based cross-lingual editing, stream-based active learning for robotics, and diagnostic tools to detect shortcut learning in AI systems.

Searching arXiv for recent and canonical uses of “CLEVER” across research areas. arxiv_search(query="CLEVER benchmark Lean verified code generation", max_results=5) CLEVER is used in arXiv literature as the name of several distinct methods, benchmarks, and analytical constructs rather than a single unified framework. The term denotes, among other things, a benchmark for end-to-end verified code generation in Lean, a visually grounded commonsense knowledge acquisition framework, a robustness score for adversarial evaluation, a retrieval-based system for cross-lingual multi-hop knowledge editing, and a stream-based active learning system for robot semantic perception (Thakur et al., 20 May 2025, Yao et al., 2022, Weng et al., 2018, Khandelwal et al., 2024, Lee et al., 21 Jul 2025). In parallel, “clever” also appears in work on benchmark validity and “Clever Hans” behavior, where high performance is attributed to shortcut exploitation rather than the intended capability (Kavumba et al., 2019, Pacchiardi et al., 2024).

1. CLEVER as a benchmark for formally verified code generation

In formal methods and program synthesis, CLEVER stands for Curated Lean Verified Code Generation Benchmark and is designed for end-to-end verified code generation in Lean (Thakur et al., 20 May 2025). The benchmark contains 161 problems, adapted from 161 of the 164 HumanEval tasks that could be formalized in Lean 4. Each problem is organized into two certification stages. In specification certification, a model must generate a Lean specification $\psi$ from the natural-language statement $\nu$ and prove an isomorphism to a held-out human-written specification $\psi^*$. In implementation certification, the model must generate an implementation $\pi$ and prove that it satisfies the ground-truth specification. A task counts as solved only if both stages are fully verified by Lean’s type checker (Thakur et al., 20 May 2025).

A central design choice is that CLEVER avoids test-case supervision, LLM-generated annotations, and specifications that leak implementation logic or allow vacuous solutions. Its specifications are human-authored and written as non-computable propositions in Lean’s Prop, precisely to prevent the model from copying executable structure from the specification into the program. The paper contrasts this with benchmarks in which partial property checking can be satisfied by trivial programs, and it emphasizes that CLEVER is intended to capture the full natural-language intent of the task (Thakur et al., 20 May 2025).

The evaluation metric is pass@k-seconds, with $k=600$ seconds. The paper reports that current models “all struggle to achieve full verification,” and that the best end-to-end result is only 1/161 problems, even though compilation rates are substantially higher than proof success rates (Thakur et al., 20 May 2025). This makes CLEVER a frontier benchmark for program synthesis and formal reasoning, and it also yields fine-grained failure analysis because specification generation, equivalence proving, implementation synthesis, and correctness proving are scored separately.

2. CLEVER as visually grounded commonsense knowledge acquisition

In knowledge extraction, CLEVER is a framework for visually grounded commonsense knowledge acquisition that formulates commonsense knowledge extraction as a distantly supervised multi-instance learning problem over bags of images (Yao et al., 2022). The starting point is the claim that text suffers from reporting bias: obvious facts such as $(\text{person}, \text{can\_hold}, \text{bottle})$ are rarely stated explicitly, whereas images directly depict grounded entity interactions. The paper reports that on Visual Genome, 83% of the triplets sampled by the authors are not found in ConceptNet, motivating image-based acquisition as a complement to text-based commonsense extraction (Yao et al., 2022).

For an entity pair $(s,o)$, CLEVER constructs a bag of images $\mathcal{B}_{(s,o)}=\{v_i\}_{i=1}^N$ and uses a vision-language pre-training encoder, specifically VinVL, to obtain instance representations. It then forms relation-aware bag representations

$$\boldsymbol{B}_{(s,r_i,o)} = \sum_{j=1}^N \alpha^{r_i}_j\boldsymbol{v}_j, \qquad \alpha^{r_i}_j = \text{Softmax}_j ( \boldsymbol{v}_j^\top\boldsymbol{r}_i),$$

and optimizes them with a contrastive objective rather than standard attention-only multi-instance learning (Yao et al., 2022). The purpose of this contrastive attention is to address what the paper calls the golden-query-only problem, where positive relations are trained directly but negative relations are insufficiently separated.

The benchmark is built from Visual Genome, using the top 100 entity types and 100 relation types, with splits of 13,780 facts / 6,443 entity pairs / 55,911 images for training, 1,166 facts / 678 entity pairs / 5,224 images for validation, and 3,496 facts / 1,964 entity pairs / 13,722 images for testing (Yao et al., 2022). On held-out evaluation, CLEVER achieves AUC 41.92 and mAUC 26.57, improving over the best PLM-based baseline, Prompt-FT, by +3.9 AUC and +6.4 mAUC. In human evaluation, the predicted commonsense scores show a 0.78 Spearman coefficient with human judgment (Yao et al., 2022). The framework is thus positioned not merely as a classifier of visual relations, but as a system for extracting grounded commonsense triplets with interpretable image-level support.

3. CLEVER as a robustness score for adversarial evaluation

In adversarial robustness, CLEVER stands for Cross-Lipschitz Extreme Value for nEtwork Robustness and is an Extreme Value Theory (EVT) based robustness score for large-scale deep neural networks (Weng et al., 2018). Its objective is to provide an attack-agnostic robustness score by estimating a lower-bound-style quantity on the minimum perturbation required to change a classifier’s prediction. The core first-order construction estimates a local Lipschitz constant from sampled gradient norms and then converts that estimate into a robustness radius of the form

$$\epsilon \;=\; \min\!\left(\frac{g_t(x_0)}{L_q^t},\, R\right),$$

where $\nu$0 is a cross-class score difference (Weng et al., 2018).

The 2018 extension introduces a second-order CLEVER score for twice-differentiable classifiers. Defining

$\nu$1

the formal robustness guarantee becomes

$\nu$2

The same paper also proposes combining CLEVER with Backward Pass Differentiable Approximation (BPDA) so that models with non-differentiable input transformations can still be evaluated in a white-box manner (Weng et al., 2018).

This line of work is accompanied by a direct critique. A separate paper argues that gradient masking causes CLEVER to overestimate the size of perturbation needed to fool the model and states that “an estimate of a bound is not a bound” (Goodfellow, 2018). Its counterexample uses staircase-like quantization and Lipschitz-continuous approximations with narrow ramps, showing that finite sampling can miss the regions where gradients are concentrated. The controversy is therefore methodological: CLEVER was introduced as a scalable, attack-independent robustness estimator, but its validity depends on whether sampled gradients faithfully represent local sensitivity (Weng et al., 2018, Goodfellow, 2018).

4. CLEVER in retrieval-based editing and stream-based semantic perception

A later use of the name appears in CLEVER-CKE, a retrieval-based framework for cross-lingual multi-hop knowledge editing (Khandelwal et al., 2024). The setting is harder than standard monolingual editing because the model must both transfer edited facts across languages and propagate those edits through a reasoning chain. To evaluate this, the paper constructs CROLIN-MQUAKE, a parallel benchmark derived from MQUAKE-CF and MQUAKE-T, with 3k examples for the counterfactual set, 1.8k examples for the temporal set, and eight languages total: English, German, Spanish, Chinese, Russian, Hindi, Bengali, and Swahili (Khandelwal et al., 2024).

CLEVER-CKE is a parameter-preserving method with a retrieve–verify–generate pipeline. A retriever searches an external multilingual memory of edited facts, a verification step checks whether the retrieved fact is sufficiently close to the sub-question, and the LLM either uses the retrieved fact or falls back to internal knowledge. The retriever is trained with three losses: a semantic distinction loss $\nu$3, a cross-lingual edit consistency loss $\nu$4, and a binary cross-entropy loss $\nu$5, combined as

$\nu$6

Across three LLMs, eight languages, and two datasets, the paper reports gains of up to 30% over prior methods (Khandelwal et al., 2024).

Another distinct usage is the robotics system CLEVER: Stream-based Active Learning for Robust Semantic Perception from Human Instructions (Lee et al., 21 Jul 2025). Here the problem is distribution shift in robot perception. CLEVER is a stream-based active learning system for robust semantic perception with Deep Neural Networks (DNNs) that asks a human for support when uncertainty is high, receives instructions and demonstrations, and updates a Bayesian classifier online (Lee et al., 21 Jul 2025). The architecture uses unknown object segmentation such as Segment Anything (SAM), feature extraction with DINOv2, and one multilayer perceptron (MLP) head per object class.

The Bayesian formulation places a posterior over class-specific parameters,

$\nu$7

and prediction is performed by Bayesian marginalization (Lee et al., 21 Jul 2025). Querying is driven by uncertainty after temporal filtering, and sample selection uses BatchBALD. In a user study with 13 participants, the system reports, for CLEVERv1, a success rate of 0.887 \pm 0.049, ECE 0.060 \pm 0.017, precision 0.933 \pm 0.020, and 1.539 \pm 0.544 queries to reach 85% precision; it also reports average training time of 17.79 \pm 4.717 s (Lee et al., 21 Jul 2025). The paper positions this as the first realization of stream-based active learning on a real robot.

5. CLEVER and the diagnosis of shortcut learning

A substantial body of work uses “clever” in the Clever Hans sense: a system appears to solve the task, but does so by exploiting unintended cues. In commonsense NLP, Balanced COPA was introduced after an analysis showed that original COPA contained single-token distributional imbalances and that RoBERTa trained only on the alternatives reached about 59.6% accuracy, above the 50% random baseline (Kavumba et al., 2019). The mirrored construction of Balanced COPA ensures that every alternative appears once as correct and once as incorrect, and human evaluation reports 100% accuracy on original COPA and 97% on Balanced COPA, indicating that the task remains solvable after shortcut cues are removed (Kavumba et al., 2019).

The same concern has been extended to modern LLM benchmarks. One study evaluates 19 multiple-choice benchmarks with logistic regression over simple unigrams, bigrams, and readability and diversity metrics, using Cohen’s $\nu$8 and treating $\nu$9 as detectable agreement (Pacchiardi et al., 2024). It reports that 9/19 benchmarks show evidence of agreement between simple-feature predictions and ground truth, with Corporate Lobbying and SpaceNLI showing consistent $\psi^*$0. The paper also reports adjusted one-sided paired t-test results suggesting that benchmark-predictable items are easier for some model families, including Meta, OpenAI, Mistral AI, and Writer (Pacchiardi et al., 2024). This literature treats benchmark internal validity as a first-class research problem.

Related work shows that Clever Hans effects are not restricted to supervised classification. In anomaly detection, an explainable AI framework based on LRP / Deep Taylor Decomposition reveals that density-based, reconstruction-based, and boundary-based models often detect anomalies for the wrong reasons, and it introduces a Clever Hans score defined as

$\psi^*$1

on MNIST-C and MVTec (Kauffmann et al., 2020). A broader study of unsupervised learning then argues that Clever Hans effects are widespread in unsupervised learning, including representation learning and anomaly detection, and attributes them primarily to inductive biases in the learning objective and architecture rather than only to dataset artifacts (Kauffmann et al., 2024).

This diagnostic program also led to mitigation methods. Class Artifact Compensation (ClArC) uses explanation maps and Spectral Relevance Analysis to quantify artifact-prone classes and then applies either A-ClArC or P-ClArC to reduce reliance on class-specific artifacts (Anders et al., 2019). In translationese classification, topic-label alignment derived from LDA and BERTopic is reinterpreted as a topic floor, with average alignment values around 0.55 to 0.62 and a proposed floor of about 0.62 (Borah et al., 2023). In medical imaging, a study on ADNI T1-weighted MRI reports 990 AD MRIs and 990 NC MRIs, and finds that binarized, skull-stripped inputs retain performance comparable to non-binarized references, revealing a Clever Hans effect in which the network depends primarily on volumetric atrophy, brain shape / mask geometry, and skull-stripping artifacts rather than gray-white matter texture (Tinauer et al., 27 Jan 2025). More recently, Counterfactual Knowledge Distillation (CFKD) addresses Clever Hans strategies in image classifiers by generating and validating counterfactuals; on five datasets it reports strong gains in average group accuracy, including 94.5 on Square and 83.9 on Follicles (Bender et al., 20 Oct 2025).

6. Other technical uses and recurrent disambiguation

The name also appears in several domain-specific constructs outside machine learning evaluation. In computational biology, CLEVER: Clique-Enumerating Variant Finder is a structural-variant caller that organizes read alignments into a graph where max-cliques represent maximal contradiction-free groups of alignments (Marschall et al., 2012). The method targets indels from roughly 20 bp up to 50,000 bp, uses all alignments including concordant reads, and is reported to be particularly strong in the 20–100 bp range (Marschall et al., 2012).

In geometric computer vision, the phrase “clever elimination strategy” denotes an algebraic method for generating smaller and faster minimal solvers by first eliminating unknowns that do not appear in the linear measurement equations and then extending solutions to the remaining unknowns (Kukelova et al., 2017). The paper demonstrates this on several relative pose problems, including $\psi^*$2, $\psi^*$3, and $\psi^*$4, reducing elimination templates from $\psi^*$5 to $\psi^*$6, from $\psi^*$7 to $\psi^*$8, and from $\psi^*$9 to $\pi$0, respectively (Kukelova et al., 2017).

There is also a probabilistic usage in “A clever (self-repelling) burglar”, where the “cleverness” refers to a structural property of true self-repelling motion. The paper proves that, conditionally on the occupation-time density $\pi$1, the law of $\pi$2 is uniform on a certain admissible interval, a result contrasted with the Brownian-motion setting of Warren and Yor (Dumaz, 2012).

Finally, CLEVER is often confused with CLEVR, the synthetic VQA benchmark whose name is pronounced “clever.” CLEVR contains 100,000 rendered images, about 1 million automatically generated questions, and 853,554 unique questions, with 70k train / 15k val / 15k test splits (Johnson et al., 2016). It was designed as a diagnostic dataset for compositional language and elementary visual reasoning, with functional programs, controlled attributes, and reduced question-conditional bias (Johnson et al., 2016). This suggests that, across fields, “CLEVER” and closely related naming conventions are repeatedly used for systems whose purpose is diagnostic rigor, formal control, or shortcut-resistant evaluation, even though the underlying technical objects are heterogeneous.