Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection

Abstract: Existing benchmarks for LLM-based vulnerability detection compress model performance into a single metric, which fails to reflect the distinct priorities of different stakeholders. For example, a CISO may emphasize high recall of critical vulnerabilities, an engineering leader may prioritize minimizing false positives, and an AI officer may balance capability against cost. To address this limitation, we introduce SecLens-R, a multi-stakeholder evaluation framework structured around 35 shared dimensions grouped into 7 measurement categories. The framework defines five role-specific weighting profiles: CISO, Chief AI Officer, Security Researcher, Head of Engineering, and AI-as-Actor. Each profile selects 12 to 16 dimensions with weights summing to 80, yielding a composite Decision Score between 0 and 100. We apply SecLens-R to evaluate 12 frontier models on a dataset of 406 tasks derived from 93 open-source projects, covering 10 programming languages and 8 OWASP-aligned vulnerability categories. Evaluations are conducted across two settings: Code-in-Prompt (CIP) and Tool-Use (TU). Results show substantial variation across stakeholder perspectives, with Decision Scores differing by as much as 31 points for the same model. For instance, Qwen3-Coder achieves an A (76.3) under the Head of Engineering profile but a D (45.2) under the CISO profile, while GPT-5.4 shows a similar disparity. These findings demonstrate that vulnerability detection is inherently a multi-objective problem and that stakeholder-aware evaluation provides insights that single aggregated metrics obscure.

• The paper introduces a comprehensive, role-specific evaluation framework that assesses LLMs on 35 dimensions tailored to stakeholder requirements in vulnerability detection.
• It benchmarks 12 LLMs across 406 tasks using both Code-in-Prompt and Tool-Use paradigms, revealing significant score divergences among different roles.
• The study highlights trade-offs between precision, recall, cost, and throughput, advocating for customized model selection in security-critical environments.

SecLens: Role-Specific Evaluation of LLMs for Security Vulnerability Detection

Motivation and Problem Statement

Current benchmarks for LLM-based vulnerability detection typically reduce model performance to a single aggregate score. This unidimensional approach is insufficient for real-world organizational deployment where divergent stakeholder requirements exist—for instance, a CISO valuing critical recall, an engineering lead prioritizing low false positives, and an AI officer balancing cost and capability. The "SecLens" framework (2604.01637) addresses this critical evaluation gap by introducing a comprehensive, role-specific assessment architecture that surfaces differential model suitability across operational contexts.

Framework Design

SecLens-R operationalizes stakeholder-aware evaluation by defining 35 shared dimensions spanning 7 measurement categories: Detection, Coverage and Consistency, Reasoning and Evidence, Operational Efficiency, Tool-Use and Navigation, Risk and Severity, and Robustness. Each of five stakeholder roles—CISO, Chief AI Officer, Security Researcher, Head of Engineering, and AI-as-Actor—selects between 12 and 16 dimensions, assigning an 80-point weight budget among them. The consequence is a highly granular, role-weighted composite "Decision Score" enabling direct score divergence quantification across lenses.

Figure 1: Category-level weight distribution for the five stakeholder roles, highlighting their distinct evaluative emphases.

SecLens evaluates models via two distinct access paradigms: Code-in-Prompt (CIP), which isolates reasoning from a single code snippet, and Tool-Use (TU), which requires repository navigation, tool invocation, and stateful task completion. The dimensional scoring automatically adjusts for the available evidence per layer, dynamically excluding inapplicable components.

Experimental Setup

A panel of 12 contemporary LLMs (Anthropic Claude, Google Gemini, OpenAI GPT, and several high-performing open-weight and routed models) was benchmarked on 406 tasks curated from 93 projects across 10 programming languages and 8 OWASP-aligned vulnerability categories. Tasks were organized as paired TP (vulnerable) and post-patch samples, ensuring class balance and robust metric computation. The evaluation layers and prompt protocol were tightly standardized.

Results: Divergence and Stakeholder-Dependent Assessment

A central, empirically validated finding is that single-score leaderboards obscure substantial cross-role score divergence. For instance, a model such as Qwen3-Coder or GPT-5.4 may receive an A-grade (score $\geq$ 75) from the Head of Engineering due to high precision and actionable finding rate, but only a D (score $<$ 50) from the CISO, mainly due to poor critical vulnerability recall.

Figure 2: Per-role Decision Scores for 12 models (CIP layer), highlighting stark discrepancies across stakeholder priorities.

Role Divergence Index (RDI) quantification demonstrates up to 31-point variation for a single model—the maximum observed gap between any two stakeholder role scores per model.

Figure 3: Role Divergence Index per model (CIP layer); red indicates models whose evaluation is highly contingent on the evaluator’s role.

Notably, the AI-as-Actor lens is most lenient, with all models achieving A grades, reflecting the widespread robustness and format compliance of contemporary LLMs. By contrast, the CISO lens is the most stringent, significantly penalizing any model with category or severity recall gaps. Models with conservative prediction strategies, such as Qwen3-Coder, enjoy high engineering and AI-Actor scores due to low false positives, but are systematically penalized by security-focused lenses.

Granular Performance Analysis

Evaluation per vulnerability category reveals further that model strengths are highly non-uniform; no model dominates all categories. For example, top F1 scores in categories like Memory Safety, Cryptographic Failures, and SSRF are distributed across Claude, Gemini, and Kimi models, while Qwen3-Coder consistently underperforms in recall-heavy dimensions.

Figure 4: F1 score heatmap by model and vulnerability category (CIP layer), illustrating that top models are not universally optimal across categories.

The framework also reveals efficiency-performance trade-offs. Cost-per-task and throughput vary by orders of magnitude across models and layers, and leaderboard rank does not correlate with efficiency.

Figure 5: Cost per task versus leaderboard score (CIP layer, log-scale), illustrating substantial inefficiency in some high-performing models.

Practical and Theoretical Implications

The findings critically demonstrate that model selection for vulnerability detection cannot be reduced to optimizing for a single metric or leaderboard position. For deployment, organizations must align model selection to their operational priorities and risk appetite; the same LLM can be optimal for engineering velocity (due to high precision and throughput) but unsuitable for risk mitigation (if it demonstrates poor critical recall).

Role-specific weighting of operational dimensions enables more principled, explainable, and transparent tradeoff navigation. Further, the YAML-based weight profile implementation supports organizational customization, facilitating rapid adjustment of evaluative criteria as requirements evolve.

Theoretically, this work strengthens the case for multi-criteria decision analysis as essential in the ML security evaluation landscape. Aggregate scores conflate crucial detection and cost/efficiency/robustness dimensions, risking suboptimal or non-defensible deployment decisions.

Limitations and Future Directions

Weight assignment, while anchored in practitioner literature and frameworks (ISO/IEC 25010, GQM, NIST CSF), retains subjective elements and may warrant further empirical refinement or stakeholder survey validation. The evaluation cohort size, particularly for rare vulnerability categories, bounds statistical confidence in specific dimensions. Current cost analysis is limited by provider tracking capabilities, and TU evaluation remains cost-prohibitive for continual deployment.

Future work should focus on broader SAST false-positive corpora for improved specificity assessment, integration with additional benchmarks (e.g., SecVulEval, SEC-bench, TOSSS), and development of tools for interactive, stakeholder-driven profile customization. Longitudinal model tracking and multi-run statistical analysis will enhance confidence in role-scored evaluation trends.

Conclusion

SecLens-R provides a comprehensive, extensible architecture for role-specific LLM vulnerability detection evaluation. Empirical analysis confirms that aggregate benchmarks obscure decisive differences in model suitability across stakeholder contexts. The framework enables precise, explainable tradeoff analysis—an essential step toward robust, defensible deployment of LLMs in security-critical environments.