Harm Emergence Prediction in AI

• Harm emergence prediction is a composite research area that forecasts how, where, and when harms arise in AI systems by integrating retrospective analysis, anticipatory scenario elicitation, and real-time monitoring.
• It emphasizes stakeholder-specific, temporally dynamic, and multimodal indicators rather than relying on simple binary safe/unsafe classifications.
• Methods include incident signal analysis, ethical matrix forecasting, and hidden-state monitoring to guide adaptive governance and mitigate evolving AI risks.

Harm emergence prediction denotes a family of methods for identifying how harms appear, concentrate, and evolve across the AI lifecycle. In the current literature, the object of prediction ranges from realized damage in incident corpora to anticipated bias-to-harm pathways, long-horizon indirect harms of model outputs, same-pass forecasts of harmful drift in hidden states, and step-wise harmful behaviors inside reasoning traces (Vei et al., 12 Sep 2025, Tantalaki et al., 27 Nov 2025, Sun et al., 26 Jun 2025, Park et al., 13 May 2026, Kakkar et al., 21 Apr 2026). The field is therefore not a single technique but a composite research area spanning retrospective incident analysis, pre-deployment scenario elicitation, online monitoring, and process-level safety diagnostics, with recurrent emphasis on stakeholder specificity, temporal dynamics, and multimodal or sociotechnical context rather than binary safe/unsafe judgments alone (Mengesha et al., 23 Apr 2026, Paz, 14 Dec 2025, Li et al., 13 Jan 2026).

1. Conceptual foundations

A central distinction in the literature is between harms, risks, and severity. "AI Harmonics" defines AI harms as “physical, psychological, or otherwise consequential damage experienced by individuals, groups, organizations, society, or the environment” from AI use or misuse, while AI risks are probability × impact of adverse outcomes and are therefore forward-looking rather than realized damage (Vei et al., 12 Sep 2025). The same work treats severity as ordinal rather than cardinal: it assumes reliable judgments of relative severity without assuming that one harm is numerically some fixed multiple of another. This is consequential for prediction, because many deployed systems generate sparse, noisy, and heterogeneous evidence for which ordinal ranking is more defensible than calibrated loss estimation.

The concept of emergence is also broader than simple occurrence. In "AI Harmonics", emergence is operationalized as uneven distributions in which harms concentrate in some categories and on the most severely affected stakeholders; repeated measurement of such concentration is presented as a way to monitor emergent harm patterns over time (Vei et al., 12 Sep 2025). A different but complementary view appears in "From Linear Risk to Emergent Harm: Complexity as the Missing Core of AI Governance", where harm is characterized as “emergent, delayed, redistributed, and amplified through feedback loops and strategic adaptation by system actors” inside complex adaptive socio-technical systems (Paz, 14 Dec 2025). Under that framing, prediction cannot be reduced to static ex ante classification, because regulation, deployment, and organizational adaptation alter the system that generates harm.

A further refinement concerns the status of disagreement. "PluriHarms" does not model objective harm directly; it models human judgments of harmfulness and treats disagreement as signal rather than noise (Li et al., 13 Jan 2026). Its benchmark is organized along two dimensions, the harm axis from benign to harmful and the agreement axis from agreement to disagreement, making explicit that borderline prompts are often exactly those where human judgment is most plural and unstable. This suggests that harm emergence prediction is not only about forecasting adverse outcomes, but also about forecasting where evaluative consensus itself breaks down.

2. Incident-based quantification and trajectory analysis

One major paradigm starts from realized incidents and converts them into structured predictive signals. "AI Harmonics" uses incidents $n_k$, harm categories $c_i$, and stakeholder groups $h_j$ to compute category-specific stakeholder frequency profiles,

$f_{ij} = \frac{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i}, h_{j})}{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i})},$

and then defines the severity-adaptive concentration metric $AIH_i$ as the area under an ordinal “derivative” Lorenz curve,

$AIH_i = \int_0^1 \ell_i(x)\,dx.$

The purpose is to measure whether harms cluster at high-severity stakeholder ranks without requiring cardinal severity estimates. On annotated AIAAIC incidents, Political & economic harms receive the highest $AIH$ at 0.85, followed by Physical and Psychological at 0.73, while Financial & business and Autonomy are lowest at 0.51 and 0.53, respectively (Vei et al., 12 Sep 2025). The paper explicitly interprets such concentration as a monitoring signal for emergent harms.

A second incident-based approach separates exposure from hazard trends. "A pragmatic classification of AI incident trajectories" argues that raw incident counts conflate deployment growth, harm frequency per unit exposure, and reporting propensity, and therefore proposes monitoring questions in the SORT format together with separate estimation of exposure $E(t)$ and harm-per-exposure trend $\hat{H}$ (Mengesha et al., 23 Apr 2026). It then classifies trajectories as Escalating, Mitigating, Concentrating, or Receding depending on whether exposure and harm-per-exposure are increasing or decreasing. In the paper’s worked examples, conversational AI harms involving responses that encourage, or fail to discourage, suicidal ideation or self-harm are classified as Escalating, whereas autonomous vehicle incidents involving injury or property damage per mile are classified as Mitigating (Mengesha et al., 23 Apr 2026). This moves prediction away from raw counts and toward directional claims about whether underlying hazard is intensifying.

A third observational resource is "RealHarm", a deployer-centric dataset of 136 annotated real-world language-model failures, split into 68 unsafe and 68 safe interactions (Jeune et al., 14 Apr 2025). It reports reputational damage as the predominant organizational harm and misinformation as the most common hazard category, with the hazard taxonomy covering Interaction Disconnect, Unsettling Interaction, Operational Disruption, Brand Damaging Conduct, Criminal Conduct, Violence and Toxicity, Bias & Discrimination, Privacy Violation, Misinformation and Fabrication, and Vulnerable Individual Misguidance (Jeune et al., 14 Apr 2025). The empirical guardrail evaluation in the same work exposes a protection gap: many real incidents would not have been prevented by deployed moderation or guardrail systems. Taken together, these incident-based approaches treat emergence as measurable structure in observed failures rather than as purely speculative risk.

3. Pre-deployment anticipation through scenarios, vignettes, and ethical matrices

A second major paradigm predicts harm before deployment by eliciting structured expectations about future failures. "AHA!: Facilitating AI Impact Assessment by Generating Examples of Harms" begins with an AI deployment scenario, generates stakeholders and problematic AI behaviors, places them into an ethical matrix, and then uses vignettes to elicit harms from crowd workers and GPT-3 (Buçinca et al., 2023). Across five scenarios, the framework surfaces 4113 harms and derives an eight-category taxonomy that includes Quality-of-service harms, Representational harms, Harms affecting people’s well-being, Legal and reputational harms, Allocational harms, Loss of rights or agency, and Other social and societal harms (Buçinca et al., 2023). The paper reports that prompting both crowds and a LLM yields more diverse examples of harms than either source alone, which suggests that anticipatory coverage benefits from heterogeneous generators even without probabilistic forecasting.

"ECHO" formalizes this anticipatory logic more explicitly as a mapping from stakeholders and bias types to harm sets within a domain: $(s_k, b_m, d) \mapsto EM^d_{k,m}.$ Its workflow comprises stakeholder generation, vignette construction, attachment of a harm multiple-choice question, annotation by humans and GPT-4, aggregation with majority threshold $c_i$0, and production of both a descriptive ethical matrix $c_i$1 and an inferential ethical matrix $c_i$2 (Tantalaki et al., 27 Nov 2025). Bias types include Representation bias, Measurement bias, Algorithmic bias, Evaluation bias, and Deployment bias, while harm types are adapted from Shelby et al. and include allocative, quality-of-service, interpersonal, and representational harms (Tantalaki et al., 27 Nov 2025). In disease diagnosis and hiring, omnibus $c_i$3 tests identify several statistically meaningful bias–harm associations, such as Diagnosis–Patient group at $c_i$4, $c_i$5, $c_i$6 (Tantalaki et al., 27 Nov 2025). The framework is explicitly intended for early phases, before development or deployment, when no model or dataset yet exists.

What distinguishes this class of methods is that they operationalize harm emergence as a latent pathway rather than an observed incident count. They do not estimate realized frequency; they estimate what harms are likely to arise for which stakeholders when a particular bias, failure mode, or design choice is instantiated in context. This suggests a complementary relationship between pre-deployment anticipation and incident-based monitoring: the former enumerates plausible pathways, while the latter measures which pathways are materializing.

4. Online forecasting, latent-state monitoring, and process-level safety

A third paradigm predicts harms during or immediately before generation. "Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation" formulates harm emergence as a planning problem over event trajectories. It adapts a POMDP/world-model perspective in which a prompt–response pair seeds a directed event graph $c_i$7, and each generated event is represented as

$c_i$8

where $c_i$9 is an event description, $h_j$0 is likelihood, $h_j$1 is temporal horizon, and $h_j$2 is impact severity (Sun et al., 26 Jun 2025). A breadth-first search over event scripts generates downstream societal events, population strata likely to be affected, and textual feedback $h_j$3, which is then used to refine the original response and also to create preference pairs for Direct Preference Optimization (Sun et al., 26 Jun 2025). The paper reports over 20% improvement on a new indirect-harm dataset and an average win rate exceeding 70% against strong baselines on AdvBench, SafeRLHF, and WildGuardMix (Sun et al., 26 Jun 2025). Here, emergent harm is neither a content tag nor a policy violation, but a downstream trajectory of plausible societal consequences.

"AERIC: Anticipatory Hidden-State Monitoring for Implicit Harmful Dialogue" moves the predictive locus inside the decoder. It studies same-pass anticipatory monitoring under the constraint that a monitor can read hidden states $h_j$4 and a cached prompt vector $h_j$5, but may not invoke an additional forward pass through the base model (Park et al., 13 May 2026). Its risk score is

$h_j$6

where $h_j$7 is a future hazard head, $h_j$8 a support head, and $h_j$9 a prompt-conditioned residual head, and the online decision signal is an exponential moving average

$f_{ij} = \frac{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i}, h_{j})}{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i})},$0

With only 387 trainable head parameters, AERIC improves AUROC from 0.6830 to 0.7143 on DiaSafety and from 0.8219 to 0.8582 on Harmful Advice relative to Qwen3GuardStream-4B, while adding only 2.34% mean latency on the fixed-generation benchmark where Qwen3Guard-Stream-4B adds 79.40% (Park et al., 13 May 2026). The benchmark results on HarmBench DirectRequest and SocialHarmBench further show trigger@64 values up to 0.6849 and 0.7363, respectively, under the source-side safe-budget rule (Park et al., 13 May 2026). This operationalizes emergence as short-horizon harmful drift detectable before explicit toxic text appears.

Two related strands make the process-level view more fine-grained. "RoTRAG" performs turn-level harm detection in multi-turn dialogue by retrieving human-written Rules of Thumb, deciding through a routing classifier whether a new turn requires fresh normative grounding, and then classifying the current turn using accumulated RoT history and dialogue history (Lee et al., 19 Apr 2026). "HarmThoughts" annotates 56,931 sentences from 1,018 reasoning traces with a 16-way taxonomy of harmful reasoning behaviors, distinguishing early behaviors such as Refusal Suppression and Intent Rationalization from later behaviors such as Task Decomposition, Domain Knowledge Elicitation, and Obfuscation (Kakkar et al., 21 Apr 2026). The paper shows that existing detectors are much better at coarse harmful-versus-safe discrimination than at fine-grained early-stage behaviors, which are precisely the behaviors needed for reliable harm emergence prediction (Kakkar et al., 21 Apr 2026). This suggests that online prediction increasingly depends on process signals rather than final outputs alone.

5. Domain-specific applications and modalities

The field is strongly domain-dependent, and several application areas illustrate distinct prediction targets. In online self-harm support communities, "Helping or Hurting? Predicting Changes in Users' Risk of Self-Harm Through Online Community Interactions" predicts whether distressed users are helped or harmed by the replies they receive in a thread and reports a macro-F1 score of up to 0.69 (Soldaini et al., 2018). The object of prediction there is not a general harmful utterance but a change in user risk state induced by social interaction.

In clinical NLP, "Why Do Self-Harm Prediction Models Struggle to Generalise? Lexical and Semantic Variations in Emergency Department Triage Notes" examines cross-site degradation in self-harm detection from emergency department triage notes (Chen et al., 1 Jun 2026). The paper reports within-site AUPRC of approximately 0.85 at Royal Melbourne Hospital and cross-site AUPRC of approximately 0.78 when transferring to Latrobe Regional Health, and attributes the drop to lexical and documentation differences despite consistent core themes such as self-poisoning and self-injury (Chen et al., 1 Jun 2026). The predictive problem here is early clinical signal detection, but the main scientific point is that institutional variation changes how harms are textually manifested.

In multimodal moderation, "MuPHI" targets harmful semantics that emerge from the interaction of image and text rather than from either modality in isolation (Saha et al., 28 May 2026). The dataset contains 1594 image–text pairs, with 623 harmful and 971 benign examples, and MuPHIRM trains a vision-LLM with structured reasoning and rule-based rewards for outcome correctness, format, evidence alignment, and consistency (Saha et al., 28 May 2026). On MuPHI, MuPHIRM reaches macro-F1 of 90.2 relative to 78.4 for the Label+Rationale baseline, and it also improves cross-dataset and leave-one-class-out generalization (Saha et al., 28 May 2026). This instantiates harm emergence prediction as compositional semantic inference.

In agentic systems, "SkillHarm" shifts attention from outputs to reusable skill packages as a persistent attack surface (Ning et al., 1 Jun 2026). The benchmark contains 879 attack samples across 71 skills and 57 user tasks, and distinguishes Fixed-Payload Poisoning from Self-Mutating Poisoning, where a first benign-looking session silently mutates persistent skill content and harm emerges only on later reuse (Ning et al., 1 Jun 2026). Current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP, and the paper shows that many apparent failures are actually cases where the agent never engaged with the poisoned file, motivating conditional ASR as a better intrinsic vulnerability signal (Ning et al., 1 Jun 2026). Harm emergence is therefore explicitly lifecycle-dependent.

A more physical notion of emergence appears in "Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View with a Reachability Prior", where emergence prediction refers to where new objects not currently visible will appear in future ego-view frames (Makansi et al., 2020). The Emergence Prediction Network improves FDE from 21.48 to 15.89 and NLL from 22.99 to 21.03 when augmented with a reachability prior (Makansi et al., 2020). Although this work is not framed as AI governance, it provides a perception-level template for predicting physical hazard emergence through multimodal, probabilistic future occupancy.

6. Evaluation regimes, governance uses, and unresolved problems

Evaluation in harm emergence prediction is heterogeneous because the target variable itself changes across paradigms. PluriHarms evaluates continuous human harmfulness judgments and disagreement using 150 prompts, 15,000 ratings, and 100 annotators, and shows that personalization materially improves prediction of human ratings relative to non-personalized baselines (Li et al., 13 Jan 2026). AERIC uses AUROC, AUPRC, trigger@$f_{ij} = \frac{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i}, h_{j})}{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i})},$1, and mean withheld tokens for streaming dialogue risk (Park et al., 13 May 2026). HarmThoughts evaluates Macro-F1, Macro-AUPRC, and Top-$f_{ij} = \frac{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i}, h_{j})}{\sum_{k=1}^{K} \mathds{1}(n_{k}, c_{i})},$2 accuracy for behavior labels in reasoning traces (Kakkar et al., 21 Apr 2026). In physical prediction, FLN and EPN rely on FDE, IOU, and NLL (Makansi et al., 2020). This diversity indicates that harm emergence prediction is better understood as a family of structurally related problems than as a single benchmark task.

The governance literature uses these signals operationally. "AI Harmonics" proposes continuous incident collection, stakeholder annotation, severity ordering, concentration measurement, and harm prioritization so that policymakers and organizations can target high-concentration categories (Vei et al., 12 Sep 2025). "A pragmatic classification of AI incident trajectories" offers a way to decide what can and cannot be claimed from public incident data by separating exposure from harm-rate trends and labeling questions as Escalating, Mitigating, Concentrating, Receding, or Unclassifiable (Mengesha et al., 23 Apr 2026). "From Linear Risk to Emergent Harm" argues that regulation should be treated as intervention rather than control and should integrate dynamic system mapping, causal reasoning, simulation, monitoring, and iterative revision (Paz, 14 Dec 2025). Across these works, prediction is tied less to one-off compliance than to ongoing system stewardship.

Several limitations recur. Incident datasets are incomplete and biased toward visible, reported, and often English-language events; "AI Harmonics" and the trajectory-classification paper both stress incompleteness and reporting effects (Vei et al., 12 Sep 2025, Mengesha et al., 23 Apr 2026). Severity ordering is value-laden and context-dependent, even when ordinal formulations reduce cardinal assumptions (Vei et al., 12 Sep 2025). ECHO is careful not to claim full causal identification; its bias–harm mappings remain associational despite directional questionnaire design (Tantalaki et al., 27 Nov 2025). Cross-site generalization remains difficult in clinical settings because lexical variation shifts predictive features across institutions (Chen et al., 1 Jun 2026). MuPHI is primarily English and relies partly on automated rationale evaluation, while AERIC’s same-pass monitoring remains sensitive to threshold calibration and does not specify an intervention policy beyond triggering (Saha et al., 28 May 2026, Park et al., 13 May 2026). SkillHarm shows that static scanners and defensive system prompts still fail to reliably mitigate lifecycle-aware attacks (Ning et al., 1 Jun 2026).

The cumulative implication is that harm emergence prediction is moving from static harmfulness classification toward temporally structured, stakeholder-aware, and mechanism-aware forecasting. Incident concentration metrics, ethical matrices, long-horizon simulation, hidden-state forecasters, reasoning-trace taxonomies, pluralistic human judgments, and lifecycle-aware attack benchmarks each isolate a different aspect of emergence. Together they suggest that robust prediction requires combining empirical incident monitoring, prospective scenario analysis, online process signals, and adaptive governance rather than treating harm as a fixed attribute of isolated model outputs.