PVAF: Continuous LLM Vulnerability Assessment

• PrompTrend Vulnerability Assessment Framework is a continuous, community-driven system that identifies and scores LLM vulnerabilities through real-time, multi-platform data collection.
• It employs a scalable architecture with ingestion agents, a detailed vulnerability database, and a multi-dimensional scoring engine to integrate socio-technical factors.
• Empirical findings reveal that community-driven psychological attacks often outperform technical exploits, prompting a shift toward dynamic monitoring and policy adaptation.

Searching arXiv for the PrompTrend paper and closely related vulnerability-assessment work to ground the article in current preprints. PrompTrend Vulnerability Assessment Framework (PVAF) is the scoring and assessment component of PrompTrend, a system for continuous community-driven vulnerability discovery and assessment for LLMs. It is motivated by the observation that static benchmarks fail to capture LLM vulnerabilities emerging through community experimentation in online forums. PrompTrend collects vulnerability data across platforms, evaluates them using multidimensional scoring, and uses an architecture designed for scalable monitoring. In a cross-sectional analysis of 198 vulnerabilities collected from online communities over a five-month period (January–May 2025) and tested on nine commercial models, the framework reports that advanced capabilities correlate with increased vulnerability in some architectures, psychological attacks significantly outperform technical exploits, and platform dynamics shape attack effectiveness with measurable model-specific patterns. The framework achieves 78% classification accuracy and reports limited cross-model transferability, positioning continuous socio-technical monitoring as an alternative to traditional periodic assessment (Gasmi et al., 25 Jul 2025).

1. Conceptual scope

PrompTrend is designed around a specific problem formulation: vulnerability discovery for LLMs is treated as a continuous process driven by community experimentation rather than as a closed benchmark-construction task. The framework therefore targets vulnerabilities that appear, mutate, and propagate through public online venues, including Reddit, Discord, GitHub, Twitter/X, and specialized security forums. In this setting, the unit of analysis is a deduplicated vulnerability “prompt” enriched with temporal, social, and technical metadata, rather than a static benchmark item (Gasmi et al., 25 Jul 2025).

This orientation matters because the framework treats attack discovery as socio-technical. Its reported findings do not reduce vulnerability to model-internal technical weakness alone. Instead, they connect attack success to community adoption, cross-platform spread, temporal persistence, and model-specific response patterns. The paper’s summary claim that community-driven psychological manipulation is the dominant threat vector for current LLMs directly frames PVAF as a method for assessing both prompt content and the ecosystem in which that content circulates (Gasmi et al., 25 Jul 2025).

A recurrent misconception addressed by the reported results is that capability advancement necessarily improves safety. The reported model trends challenge that assumption. Another misconception is that technical obfuscation is the primary operational threat surface; the analysis instead reports stronger effects for psychological and social-engineering patterns. These claims are empirical within the reported dataset rather than universal, but they define the framework’s scope.

2. System architecture and data flow

PrompTrend is built as a scalable, fault-tolerant pipeline that continuously ingests, enriches, scores, stores, and visualizes community-discovered LLM vulnerabilities. Its architecture comprises Multi-Platform Data Ingestion Agents, a Vulnerability Database, a Scoring Engine (PVAF), and a Monitoring Dashboard. The ingestion layer includes a Reddit Agent for r/ChatGPT, r/PromptEngineering, and r/LocalLLaMA; a Discord Agent for public AI servers; a GitHub Agent for repos, issues, and code snippets; a Twitter/X Agent for security researchers and threads; and a Specialized Security Forums Agent (Gasmi et al., 25 Jul 2025).

Component	Reported function
Multi-Platform Data Ingestion Agents	Collect raw posts, repos, tweets, and forum content
Vulnerability Database	Store deduplicated prompts with 47-field metadata and cross-platform provenance linking
Scoring Engine (PVAF)	Compute multi-dimensional risk scoring with dynamic modifiers and scheduled retesting
Monitoring Dashboard	Provide real-time alerts, vulnerability cards, and aggregate trend views

The data flow is specified in staged form. Agents poll all platforms hourly. Stage 1 Filtering applies a keyword lexicon of 127 terms and a relevance threshold of at least 0.7, yielding approximately 43,000 candidates per day. Stage 2 Enrichment runs in parallel and adds temporal context, social signals, technical indicators, and content preservation via semantic fingerprint. Deduplication and cross-platform coordination then use semantic hashing with cosine similarity at least 0.85 together with propagation network analysis. Stage 3 Scoring uses the PVAF engine to compute composite risk and assign Low, Moderate, or High tiers. The processed items are stored in a document-oriented database with hierarchical schema and exposed through real-time visualization and batch analytics (Gasmi et al., 25 Jul 2025).

The database schema is notable because each item carries 47 fields of metadata, including social signals, temporal context, and technical indicators. The dashboard adds operational affordances: real-time alerts for high-risk items, vulnerability cards with score history, platform journey, and risk tier, and aggregate trend views such as model comparisons and transformation heatmaps. Scheduled retesting at 7, 30, 90, and 180 days embeds longitudinal reassessment into the architecture rather than treating evaluation as one-shot (Gasmi et al., 25 Jul 2025).

3. Multi-dimensional scoring methodology

PVAF scores each vulnerability $v$ along six core dimensions and applies dynamic modifiers. The six dimensions and their weights are defined explicitly:

Core dimension	Weight
Harm Potential (HP)	$w_1=0.20$
Exploit Sophistication (ES)	$w_2=0.20$
Community Adoption (CA)	$w_3=0.15$
Cross-Platform Efficacy (CPE)	$w_4=0.15$
Temporal Resilience (TR)	$w_5=0.15$
Propagation Velocity (PV)	$w_6=0.15$

The composite score is

$$\text{PVAF}(v)=\sum_{i=1}^{6} w_i\,S_i(v)+\sum_j M_j(v),$$

where $S_i(v)\in[0,100]$ is the raw score for dimension $i$, and $w_1=0.20$0 are dynamic modifiers. Reported examples include Mutation Factor at +5–+15 and Corporate Response at –5––20; Academic Citation is also named as a modifier class. The framework therefore combines intrinsic severity components with context-sensitive adjustments (Gasmi et al., 25 Jul 2025).

The initial triage regime uses a simplified formula because only three metadata dimensions are available in Phase 1:

$w_1=0.20$1

Risk tiers are then defined as Low Risk for 0–33, Moderate Risk for 34–66, and High Risk for 67–100. This two-stage structure distinguishes between early screening and later, metadata-rich reassessment. A plausible implication is that the framework is intended to support both rapid alerting and more calibrated downstream prioritization, but the reported formulas remain fixed to the six-dimensional weighted scheme and its dynamic modifiers (Gasmi et al., 25 Jul 2025).

4. Experimental setup and metrics

The reported evaluation uses 198 unique community-discovered vulnerabilities collected during January–May 2025. The source distribution is Discord 43%, Reddit 31%, GitHub 18%, and Forums 8%. The models under test are nine commercial models: Azure OpenAI GPT-4, O1, O3-Mini, and GPT-4.5; and Anthropic Claude Sonnet 3.5, Haiku, Sonnet 3.7, Sonnet 4, and Opus 4. Transformation Strategies number 71 and are grouped into eight categories: Encoding, Linguistic, Psychological, Structural, Technical, Advanced, Social Engineering, and Academic/Theoretical (Gasmi et al., 25 Jul 2025).

The evaluation protocol is defined over vulnerabilities $w_1=0.20$2, transformation strategies $w_1=0.20$3, and models $w_1=0.20$4. For each triple, the transformation is applied to the vulnerability, executed on the model, and the response is classified into one of four outcomes: BLOCKED, FAIL, NEUTRAL, or ERROR. The total theoretical run count is

$w_1=0.20$5

while actual executions with retries are reported as 199 368 (Gasmi et al., 25 Jul 2025).

The framework’s primary predictive metric is classification accuracy,

$w_1=0.20$6

The paper also reports Precision $w_1=0.20$7, Recall $w_1=0.20$8, and $w_1=0.20$9. For attack-centric analysis it introduces Attack Success Rate,

$w_2=0.20$0

Transformation Effectiveness,

$w_2=0.20$1

and the Model Vulnerability Index,

$w_2=0.20$2

These metrics allow the framework to move between classification quality, transformation-level efficacy, and model-level relative vulnerability within one experimental design (Gasmi et al., 25 Jul 2025).

5. Reported empirical findings

The model-level analysis reports significant stratification with Friedman $w_2=0.20$3. The reported failure rates are Claude 4 Sonnet at 4.1% with 95% CI [3.8–4.4%], Claude 4 Opus at 3.7% [3.4–4.0%], Claude 3.7 Sonnet at 1.3% [1.1–1.5%], Haiku at 0.9% [0.7–1.1%], GPT-4 at 1.9% [1.6–2.1%], O1 at 1.6% [1.4–1.8%], O3-Mini at 1.1% [0.9–1.3%], and GPT-4.5 at 0.6% [0.5–0.7%]. The reported OpenAI trend is GPT-4 to GPT-4.5, from 1.9% to 0.6%, whereas the reported Claude trend is Haiku to Claude 4, from 0.9% to 4.1% (Gasmi et al., 25 Jul 2025).

The comparison between attack classes is one of the framework’s central findings. In the top-category table, Psychological attacks led by Emotional Manipulation achieve 4.9% success with effect size Cramér’s V 0.187 and $w_2=0.20$4, while Technical attacks led by Base64 Obfuscation achieve 2.7% with V 0.098 and $w_2=0.20$5. The reported aggregate comparison states that psychological attacks nearly double technical obfuscations, 4.9% versus 2.7%, with Cochran’s $w_2=0.20$6. This directly supports the paper’s claim that psychological manipulation is a dominant operational threat vector (Gasmi et al., 25 Jul 2025).

Platform dynamics are similarly differentiated. Discord contributes 85 vulnerabilities, with 2.8% success rate, average PVAF 31.2, and Psychological as dominant type at 52%. Reddit contributes 62 vulnerabilities, with 2.1% success rate, average PVAF 24.7, and Roleplay as dominant type at 48%. GitHub contributes 36 vulnerabilities, with 1.3% success rate, average PVAF 19.3, and Technical as dominant type at 67%. Forums contribute 15 vulnerabilities, with 1.7% success rate, average PVAF 28.9, and Mixed as dominant type at 41%. The interaction table further reports Discord at 3.9% for Claude versus 1.6% for OpenAI with $w_2=0.20$7, and GitHub at 0.9% for Claude versus 1.8% for OpenAI with $w_2=0.20$8. Cross-platform propagation analysis is reported to confirm Discord → Claude vulnerability alignment (Gasmi et al., 25 Jul 2025).

The framework also reports limited cross-model transferability: only 16.9% of transformations with greater than 2% success transfer across model families. Calibration and predictive association are quantified by a correlation between PVAF score and actual success of $w_2=0.20$9, an estimated increase of 2.8 percentage points in jailbreak risk for each +10-point PVAF increment with $w_3=0.15$0 and 95% CI [0.024–0.032], ROC AUC of 0.72 for binary jailbreak classification, and minimal calibration error up to PVAF 50. Together these results indicate that the framework is moderately predictive, better calibrated in the lower-to-middle score range, and sensitive to model-family-specific exploit structure (Gasmi et al., 25 Jul 2025).

6. Operational and policy implications

The framework’s socio-technical interpretation is explicit. Emotional appeals are reported to exploit the model’s “helpfulness” objective, mirroring human social-engineering vulnerabilities, while current alignment pipelines based on refusal tuning are reported not to adequately guard against nuanced narrative framing. In parallel, the reported capability–security inversion claim states that Claude’s latest versions regress in safety as context windows and “helpfulness” increase surface area for psychological attacks, whereas OpenAI’s iterative refusal-tuning and latent-thought monitors demonstrate that safety can improve independently of raw capability (Gasmi et al., 25 Jul 2025).

Operational recommendations follow from these findings. The paper recommends adopting a model-specific vulnerability monitoring strategy, with the example that Claude 4 implies Discord psychological defenses and GPT-4 implies GitHub code obfuscation checks. It further recommends integrating PVAF or similar multi-dimensional scoring into CI/CD pipelines for prompt filtering and model-release gating, collaborating with community platforms to instrument early-warning signals such as bridge-node detection in propagation networks, and allocating security resources based on PVAF tiers so that emerging moderate/high-risk items receive rapid response even if technically “low” complexity. The argument for dynamic, continuous assessment is supported by the claim that static benchmarks and periodic red teaming fail to capture community-driven emergence and refinement cycles, whereas scheduled retesting at 7/30/90/180 days and dynamic modifiers provide longitudinal visibility (Gasmi et al., 25 Jul 2025).

The policy implications are stated in standards language. OWASP/ATLAS guidelines are recommended to be extended to include psychological-attack vectors and community-adoption metrics, and regulators are said to need continuous threat-intelligence integration rather than snapshot compliance reports. A plausible implication is that PVAF is intended not merely as an internal scoring scheme but as a candidate template for ongoing governance-oriented monitoring.

In adjacent vulnerability-assessment research, VulStamp addresses a different but related problem: description-free vulnerability assessment for software vulnerabilities. It combines static analysis, LLM-based intention extraction, a prompt-tuned model, and RL-based prompt tuning, and outputs a severity class from 0=low to 3=critical for C/C++ vulnerabilities. PrompTrend, by contrast, scores community-discovered LLM vulnerabilities using Harm Potential, Exploit Sophistication, Community Adoption, Cross-Platform Efficacy, Temporal Resilience, and Propagation Velocity, together with dynamic modifiers and scheduled retesting (Shen et al., 13 Jun 2025). This contrast suggests a distinction between code-centric severity assessment and socio-technical vulnerability monitoring: both are “vulnerability assessment” frameworks, but they operate on different objects, evidence sources, and temporal assumptions.