SecLens-R: Stakeholder Scoring Framework

• SecLens-R is a multi-dimensional scoring framework that transforms granular LLM outputs into role-aware Decision Scores on a 0–100 scale.
• Its methodology leverages 35 rigorously normalized metrics across seven categories and five stakeholder profiles, ensuring evaluations reflect diverse organizational priorities.
• Empirical results reveal key trade-offs in LLM vulnerability detection, highlighting how multi-objective optimization aids informed model selection.

SecLens-R is a multi-stakeholder, multi-dimensional scoring framework that evaluates the security vulnerability detection capabilities of LLMs by transforming granular per-task outputs into role-aware “Decision Scores” on a 0–100 scale. It addresses the limitations of unitary metrics in existing LLM-based vulnerability detection benchmarks by exposing the divergent priorities of different stakeholders—including CISOs, engineering leads, and chief AI officers—each of whom requires distinct evidentiary criteria and trade-off considerations. SecLens-R leverages 35 rigorously normalized scalar metrics grouped in seven categories, and defines five role-specific profiles with tailored dimension weights. The framework thus operationalizes multi-objective optimization, enabling informed model selection aligned with organizational priorities rather than a single aggregated score (Halder et al., 2 Apr 2026).

1. Framework Architecture and Dimension Taxonomy

SecLens-R structures model evaluation around 35 normalized dimensions (D1–D35), each computed from per-task outputs such as verdicts, CWE tags, vulnerability locations, cost, tool use, and generated reasoning. These metrics are systematically grouped into seven measurement categories:

Category	Indices	Selected Example Dimensions
Detection	D1–D8	MCC, Recall, Precision, F1 Score, CWE-ID Accuracy, Actionability
Coverage & Consistency	D9–D13	CWE Coverage Breadth, Worst-Category Floor, Cross-Language Cons.
Reasoning & Evidence	D14–D17	Evidence Completeness, Reasoning Presence, FP Reasoning Quality
Operational Efficiency	D18–D23	Cost per Task, Wall-Time, Throughput, MCC per Dollar
Tool-Use & Navigation	D24–D27	Tool Calls per Task, Navigation Efficiency, Tool Effectiveness
Risk & Severity	D28–D30	Severity-Weighted Recall, Critical Miss Rate, Severity Coverage
Robustness	D31–D35	Parse Success Rate, Format Compliance, Graceful Degradation

Each metric is scalar, normalized to $[0,1]$, and derived from granular scoring objects (e.g., intersection-over-union in code locations, source-sink-data-flow evidence, cost efficiency) constructed from evaluation outputs. This taxonomy underpins the multi-objective, role-driven aggregation process.

2. Role-Specific Weighting Profiles

SecLens-R codifies five distinct stakeholder profiles, reflecting priorities observed in security operations, AI governance, and software engineering. Each role selects a subset $S^{(r)}$ of 12–16 dimensions and assigns integer weights $w_i^{(r)}$ summing to 80:

• CISO (16 dims): Trustworthy, severity-aware detection. Emphasizes recall on high/critical vulnerabilities, penalizes misses, balances detection, coverage, and robustness.
• Chief AI Officer (CAIO, 14 dims): Capability versus cost. Values autonomous completion, MCC per dollar, throughput, and tool-use efficiency.
• Security Researcher (13 dims): Depth of reasoning and evidence. Focuses on CWE accuracy, location granularity, and evidence completeness.
• Head of Engineering (13 dims): Developer-oriented. Favors high precision and actionable findings with low costs and minimal false positives.
• AI as Actor (13 dims): Autonomy and robustness. Centers on agent reliability, parse success, and resilience to errors and rare cases.

The weighted composition is strictly normalized, with $\sum_{i\in S^{(r)}} w_i^{(r)} = 80$ for each role. No dummy or unutilized weights are permitted.

3. Aggregation Mechanism: The Decision Score

For each evaluated model $m$ and stakeholder role $r$, the composite Decision Score captures a role-aligned assessment on a $[0,100]$ scale. Let $s_i^{(m)}$ denote the normalized model score for dimension $i$, and $A^{(r)} \subseteq S^{(r)}$ the subset of active dimensions in the current setting (e.g., excluding location or tool-use metrics when unavailable):

$S^{(r)}$0

This weighted average ensures comparability even when certain metrics cannot be computed in the current evaluation (e.g., lack of severity labels or unavailable tool-use dimensions). The composite nature operationalizes stakeholder priorities as explicit trade-offs, contrasting with single-metric reporting.

4. Evaluation Protocols and Dataset Characteristics

SecLens-R sits atop a benchmark assembled from 406 tasks: 203 True-Positive pre-patch and 203 Post-Patch negatives, sourced from 93 open-source projects. The dataset covers ten programming languages (PHP, Go, Python, C#, Ruby, Java, C, Rust, JavaScript, C++) and eight OWASP-aligned vulnerability categories, notably Broken Access Control, Cryptographic Failures, Injection, Improper Input Validation, SSRF, Authentication Failures, Data Integrity Failures, and Memory Safety.

Two evaluation settings are defined:

• Code-in-Prompt (CIP): The vulnerable function is directly inlined in the LLM prompt, enabling pure, single-turn scoring (verdict, CWE classification, reasoning). Location (D7, D8) and tool-use (D24–D27) metrics are omitted.
• Tool-Use (TU): The model conducts multi-turn code navigation in a sandboxed environment, invoking tools (read_file, search, list_dir) to perform real-world audit tasks. This enables evaluation on navigation and tool efficacy metrics, raising both discrimination and cost.

5. Empirical Results, Divergence, and Trade-Offs

Empirical application of SecLens-R to 12 frontier LLMs reveals pronounced divergence in role-specific evaluations. The Role Divergence Index (RDI) for a model $S^{(r)}$1 is defined as:

$S^{(r)}$2

Observed RDIs reach ≈31 points (e.g., Qwen3-Coder 31.1, GPT-5.4 30.8), with the same model scoring A (76.3) for Head of Engineering but D (45.2) for CISO in CIP mode. No model universally dominates across all lenses; Gemini 3 Flash leads CIP leaderboard (49.6%) and performs well for both CISO (73.3, B) and CAIO (68.1, B), while Claude Haiku 4.5, ranked 8th overall (43.8%), is second for CISO (71.2, B). The AI-as-Actor lens yields the most lenient grading (all models A, 77.9–87.5), with robustness largely solved among leading LLMs. CISO is the strictest, heavily penalizing missed critical vulnerabilities.

The framework surfaces non-trivial multi-objective trade-offs. Conservative detection (high precision, low recall) models excel for engineering stakeholders but perform poorly for CISO priorities that weight severity-adjusted recall. Conversely, broad recall enhances CISO scores but reduces engineering-grade due to increased false positives and operational overhead.

Notably, TU mode, though introducing higher computational cost (10–100× increase), adds discriminative power along navigation and interaction metrics, though CIP alone captures 65–70% of the overall discriminative variance, providing an efficiency-reliability trade-off.

6. Significance and Organizational Implications

SecLens-R establishes a first-of-kind, stakeholder-driven multi-dimensional evaluation protocol for LLM-based vulnerability detection, illuminating that reliable, robust security QA is a multi-objective problem. By mapping granular model outputs to role-specific composite scores through tailored weightings, the framework offers organizations an explicit, rigorous method for aligning model selection with risk tolerance, cost sensitivity, and operational needs. Role-aware grading elucidates inherent trade-offs and uncovers which LLMs are best suited for distinct organizational mandates, exposing the incompleteness of aggregated single-number benchmarks.

This suggests that adoption of SecLens-R and similar multi-objective protocols may foster a new standard of evidence in AI security evaluation, grounding critical procurement and deployment decisions in empirically transparent, role-aware analytics (Halder et al., 2 Apr 2026).