GoldenTransformer Fault Injection

• GoldenTransformer is a modular and extensible fault injection framework designed for evaluating transformer-based models' resilience against hardware faults.
• It supports diverse fault types—including weight corruption, activation injections, and attention-level disruptions—with granular control over injection points.
• Integrated with PyTorch and HuggingFace Transformers, it offers reproducible experiments, detailed metric logging, and interactive visualizations for robust analysis.

GoldenTransformer is a modular and extensible fault injection framework specifically designed for evaluating the resilience of large transformer-based models, particularly LLMs, to induced hardware faults. Built upon PyTorch and HuggingFace Transformers, GoldenTransformer provides a unified Python-based interface for injecting a diverse array of faults—including weight corruption, activation injections, and attention-level disruptions—across logical and structural points within transformer architectures. By addressing challenges such as complex computation graphs, nonuniform layer definitions, and latent parameter dependencies, GoldenTransformer supports reproducible experimentation, metric logging, and interactive visualization, thereby serving as a robust platform for model robustness analysis and fault-aware design in real-world LLM deployments (Howard, 13 Sep 2025).

1. Framework Architecture and Core Modules

GoldenTransformer adopts a modular architecture composed of five primary modules:

• FaultInjector: Serves as the central API wrapper for HuggingFace transformer models. It leverages Python introspection to discover injectable submodules, such as model.transformer.h.* for decoder blocks and model.encoder.layer for encoder-only architectures. FaultInjector interposes directly on PyTorch’s nn.Module hooks, wrapping forward passes to enable fault injection without altering model code.
• Fault Modules (BaseFault and Subclasses): Fault types (e.g., WeightCorruption, ActivationFault, AttentionMaskFault) are implemented as subclasses of a BaseFault class. Each implements standardized inject() and revert() methods, with customizable mathematical operators to define new fault classes as needed.
• ExperimentRunner: Manages experiment orchestration, including model and dataset loading, configuration of faults and metrics, batching, metric collection, forward computations, and rollback. Outputs are organized into timestamped directories containing JSON logs, raw metrics, and generated plots.
• Metrics: Implements a hierarchy of metric classes such as Accuracy, LatencyMetric, Perplexity, and DivergenceMetric. Each metric provides compute_baseline() and compute_faulty() interfaces, supporting delta-based reporting and aggregation.
• Visualization: Includes Bokeh and Matplotlib-based wrappers for generating fault severity curves, layer-wise sensitivity heatmaps, and error-bar charts. Results can be exported in PNG, SVG, or interactive HTML formats.

This modular structure enables plug-and-play extension and fine-grained control over both fault models and experimentation pipelines.

2. Fault Injection Models and Operators

GoldenTransformer supports three principal classes of injected faults—weight corruption, activation injection, and attention-level disruptions—each with multiple configurable operators.

2.1 Weight Corruption

• Gaussian Perturbation: Applies stochastic perturbations to weight tensors, defined as $W' = W + \delta W$ where $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$. The severity parameter $\sigma$ modulates the noise variance.
• Mantissa Bit-Flip: Randomly flips mantissa bits in IEEE-754 float32 weights; for each $w \in W$, with probability $p$, one of the 23 mantissa bits is flipped, i.e., $w' = \text{FlipBit}(w, b), \ b \in \{0, ..., 22\}$, with bit-flip rate $p$ controlling severity.

2.2 Activation Injection

• Additive Noise: Adds Gaussian noise element-wise to activation tensors, $A' = A + \eta$, where $\eta \sim \mathcal{N}(0, \sigma_a^2)$.
• Random Zeroing: Implements dropout-like corruptions, $A' = A \odot B$ with $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$0, where $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$1 controls the probability of zeroing out elements.

2.3 Attention-Level Disruptions

• Mask Corruption: Modifies binary attention masks $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$2 by randomly disabling a $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$3-fraction of attention links, $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$4, with $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$5.
• Head Dropout: Entire attention head outputs $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$6 are zeroed with probability $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$7:

$\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$8

This granular taxonomy of corruptions allows for the systematic probing of transformer robustness in the context of diverse hardware fault models.

3. Injection Points and Experimental Workflow

3.1 Logical and Structural Injection Points

GoldenTransformer supports injection at multiple logical and structural levels within transformer models:

• Encoder layers (e.g., in BERT or DistilBERT)
• Decoder blocks (e.g., in GPT architectures)
• Multi-headed attention modules (query, key, value projections)
• Feed-forward MLP sublayers
• LayerNorm parameters
• Attention masks and intermediate caches

3.2 Experiment Execution

A canonical experimental workflow consists of:

1. Configuration: Specification of fault types, severities, target layers (e.g., WeightCorruption(σ=0.05, layers=[0..9])) and metrics (e.g., Accuracy(), LatencyMetric(num_runs=3)).
2. Initialization: Instantiation of ExperimentRunner with model, tokenizer, dataset, faults, and metrics.
3. Baseline Pass: Computation and storage of reference outputs and metrics under fault-free conditions.
4. Faulty Pass: For each fault, injection into the model, inference and metric collection, followed by restoration via revert().
5. Aggregation and Visualization: Compilation of trial-level results into JSON, plotting of sensitivity or severity curves, and export of graphical outputs.

The workflow ensures that parameter versions are tracked and restored for isolating single-fault effects, addressing cascading dependencies typical in transformer architectures.

4. Metrics, Logging, and Visualization

GoldenTransformer provides comprehensive robustness metrics, logging, and visualization support:

• Accuracy Degradation: $\delta W_{ij} \sim \mathcal{N}(0, \sigma^2)$9.
• Failure Rate: $\sigma$0.
• Output Divergence: For generation tasks, cumulative KL divergence across time steps: $\sigma$1.
• Perplexity: For language modeling, $\sigma$2.

Results are logged as timestamped JSON files at per-trial and per-layer granularity, with CSV summaries for tabular ingestion. Visualization modules generate:

• Layer-wise bar charts with $\sigma$3 confidence intervals
• Fault-severity response curves (e.g., accuracy vs. $\sigma$4)
• Head-by-layer sensitivity heatmaps

Export options include static (PNG, SVG) as well as interactive (HTML) formats for post-hoc exploration.

5. Experimental Case Studies

GoldenTransformer’s utility is demonstrated via controlled experiments:

5.1 Weight Faults in Text Classification (DistilBERT/IMDB)

• Setup: textattack/distilbert-base-uncased-imdb; 50 IMDB samples; Gaussian noise in layers 0–9; severities $\sigma$5 and $\sigma$6; 30 random seeds.
• Metric: Accuracy with $\sigma$7 confidence intervals.

Layer	Baseline Acc	Acc @ p=0.05	Acc @ p=0.10
0	0.88 ± 0.02	0.75 ± 0.03	0.60 ± 0.02
1	0.88 ± 0.02	0.68 ± 0.05	0.48 ± 0.03

Findings: Early layers (e.g., 1) experience the largest variance and most pronounced accuracy degradation, with some mid-layers exhibiting higher robustness. Increased fault severity ($\sigma$8) produces narrower error bars, indicating more consistent degradation.

5.2 Perplexity under Bit Flips (GPT-2/WikiText2)

• Setup: GPT-2 (117M); first 100 WikiText2 lines (32 tokens); single-bit mantissa flips in layers 0–9; 30 seeds.
• Metric: Log-scale perplexity.

Findings: Statistically significant perplexity increase observed only in layer 7 injections. GPT-2 displays high resilience to sparse mantissa bit-flips at this scale, emphasizing the relevance of targeted hardening for specific layers.

6. Usage Guidelines and Extensibility

Practical usage of GoldenTransformer involves systematic parameter selection and protocol adherence:

• Fault severity schedules should be pre-defined (e.g., $\sigma$9; flip rates $w \in W$0).
• At least 30 randomized trials per configuration are recommended for estimating confidence intervals.
• Experiments should focus on a single fault class to avoid interaction confounding; prototyping is expedited using small validation subsets.
• Configuration files (INI/JSON) are version-controlled, and ExperimentRunner logs include Git commit and random seed snapshots for reproducibility.
• Extensibility is supported through subclassing of BaseFault and Metric, exposing APIs via goldentransformer.* namespaces.

An example custom fault operator:

$w \in W$1

GoldenTransformer’s open-source repository includes comprehensive documentation, Jupyter tutorials, and CI tests, ensuring reproducibility and community adoption. This framework enables detailed, layer-level, and operator-specific robustness analyses for transformer models, supporting both academic study and robust fault-tolerant system design (Howard, 13 Sep 2025).