A Red-Team Study of Anthropic Fable 5 & Opus 4.8 Models

Abstract: We evaluate the adversarial robustness of two frontier LLMs developed by Anthropic, Fable 5 and Opus 4.8, against four families of automated jailbreak attack across 7 826 harmful intents spanning a ten-category harm taxonomy. Using the HackAgent red-teaming framework, hundreds of thousands of adversarial attempts were generated and every apparent success was independently re-adjudicated by a panel of three judge models (majority vote). Both models resist the majority of attacks, but the residual surface is larger than aggregate framing suggests: it is dominated by adaptive iterative attacks, while static obfuscation is near-fully neutralised. The strongest adaptive search (tree-of-attacks) breaks Opus 4.8 on 11.5% of intents overall, whereas Fable 5 stays in the single digits (6.1% worst-case). Aggregate rates therefore should not be read as reassurance. Even in these hardened configurations, the two models produced 1 620 (Opus 4.8) and 702 (Fable 5) panel-confirmed harmful completions spanning every harm category, located automatically, cheaply, and within the first one or two refinement steps by an attacker model with no human expert in the loop. The reasonable conclusion is that even the best, most-tested frontier models remain reliably breakable under sustained automated pressure.

• The paper demonstrates that adaptive adversarial attacks expose significant residual jailbreak vulnerabilities in Fable 5 and Opus 4.8.
• It employs a rigorous black-box evaluation across 7,826 harmful intents and multiple attack families to identify concentration of failure modes.
• The results reveal that safety-trained models yield high harmful completions with minimal iterations, emphasizing the need for dynamic, context-sensitive defenses.

Adversarial Robustness of Anthropic Fable 5 and Opus 4.8: A Systematic Red-Teaming Study

Introduction

This study presents a rigorous evaluation of adversarial robustness in Anthropic's Fable 5 and Opus 4.8 LLMs, leveraging automated red-teaming across a broad taxonomy of harmful intents. The paper systematically measures the residual jailbreak surface by orchestrating hundreds of thousands of adaptive and static attack attempts, aiming to delineate not only aggregate robustness statistics but also the underlying structure of failure modes. All adversarial attempts are strictly adjudicated via a multi-model judge panel, ensuring robust filtering of spurious or marginal cases.

Experimental Methodology

The threat model is a pure black-box: the attacker interacts with model APIs without access to model internals. Target systems are Anthropic's Fable 5 and Opus 4.8, evaluated on 7,826 curated harmful intents spanning 10 harm categories and 55 subcategories, reflecting a comprehensive cross-section of documented LLM risks.

Four attack families were evaluated:

• Tree of Attacks with Pruning (TAP): Adaptive, multi-step prompt search; depth-3 tree with pruning.
• Prompt Automatic Iterative Refinement (PAIR): Iterative attack/rewrite loop with up to 12 iterations.
• Persuasive Adversarial Prompts (PAP): Persuasion-based reframing using role-play, authority, and human-centric tactics.
• H4RM3L: A suite of static obfuscation techniques including encoding, payload-splitting, and template manipulation.

Attacks are classified as succeeded (i.e., “confirmed jailbreaks”) only when a majority of three different open LLM judges independently rate the response as genuinely harmful.

Results and Analysis

Comparative Robustness and Attack Efficacy

Key Numerical Results:

• Under the strongest adaptive attack (TAP), Opus 4.8 yielded 11.5% confirmed jailbreaks across harmful intents, while Fable 5 remained under 6.1% even in the worst-case.
• Static obfuscation is rendered largely ineffective: despite 50,000+ attempts, ASR for H4RM3L decorators was $\leq 0.2\%$ for both models.
• Opus 4.8 produced 1,620 confirmed harmful completions; Fable 5, 702—each spanning all harm categories.

Category Granularity:

• Weaknesses are not uniformly distributed. For Opus 4.8, TAP reached 27.6% ASR on child-safety, with significant failures in criminal/economic (14.7%), content (13.2%), and cybersecurity (16.6% for PAIR). Fable 5's peak exposures are ethical/social and child-safety but avoids large cybersecurity failures.
• Subcategory analysis emphasizes concentrated vulnerability. For instance, Opus 4.8 peaks at 11.5% on phishing/ransomware and 8.6% on exploit development; Fable 5 peaks at 5.8% on misinformation/disinformation and harassing speech.

Iteration Yield and Adversarial Cost:

• Successful jailbreaks in adaptive attacks are highly front-loaded: most are achieved in one to two iterations, implying that neither extended prompt search nor high computation cost is necessary for reliable exploitation.

Structural Insights

Adaptive Attacks Dominate Residual Surface:

• The vast majority of successful jailbreaks are attributable to adaptive and contextual attacks. Static attack strategies are neutralized in these modern safety-trained models.

Contextual, Not Lexical, Vulnerabilities:

• Surviving attacks typically manipulate the context or narrative of the request rather than its lexical content. This evades surface-level filters and underscores the limits of input sanitization in production safety pipelines.

Model-Specific Idiosyncrasies and Shared Modes:

• Both models show shared vulnerabilities in child-safety reframings, indicative of a contextual failure not addressable by surface-level mitigation. The variability in cybersecurity exposure between models highlights the impact of fine-tuning and policy configuration, with Opus 4.8 particularly lagging in these domains.

Theoretical and Practical Implications

This study reveals that aggregate robustness rates are misleadingly reassuring. Even hardened, heavily safety-aligned LLMs can be reliably jailbroken by automated, adaptive attackers operating purely as black boxes and with no expert-in-the-loop intervention. The sheer number and diversity of confirmed harmful completions at moderate ASR underscores the persistent gap between safety under casual and adversarial interaction.

For practical deployment, these findings underscore that the residual risk, even when numerically a minority, translates into a continuous stream of adversarially-induced harmful outputs at production scale. The study also illustrates the diminishing returns of static mitigation strategies and points decisively toward the necessity for dynamic, context-sensitive, multi-turn semantic monitoring and defense strategies.

Limitations

The results are contingent on controlled benchmarks and current model versions. Production safety layers (such as pre- and post-output filtering, logging, or human moderation) are not simulated. Judge panel adjudication is more robust but still imperfect, and PAIR attack comparisons for Fable 5 are conservative lower bounds due to incomplete coverage.

Conclusion

The paper provides a quantitatively rigorous and methodologically conservative adversarial robustness assessment of leading Anthropic LLMs, elucidating that the residual jailbreak surface is substantial, concentrated in adaptive attacks, and nontrivial at deployment scale. Even the most heavily safety-trained models remain reliably breakable under sustained, fully-automated, black-box adversarial pressure. Future alignment and defense research must prioritize context- and narrative-aware robustness and adopt evaluation regimes capable of scaling with the sophistication of adversarial methods.