SynthCTI: LLM Cyber Threat Data Augmentation

• SynthCTI is an LLM-driven framework that generates synthetic CTI sentences to balance underrepresented MITRE ATT&CK techniques.
• It partitions CTI data by technique, employs HDBSCAN clustering for semantic extraction, and prompts LLMs to create lexically diverse and semantically faithful sentences.
• Fine-tuning transformer classifiers on the augmented corpus yields significant macro-F1 improvements and accelerates optimization for minority classes.

SynthCTI is an LLM-driven data augmentation framework for cyber threat intelligence (CTI) mining that generates synthetic CTI sentences for underrepresented MITRE ATT&CK techniques in order to improve technique mapping under label scarcity and class imbalance. The framework partitions CTI sentences by technique, derives cluster-level semantic context from each class, prompts an LLM to produce additional sentences that are intended to be lexically diverse and semantically faithful, and then fine-tunes downstream transformer classifiers on the augmented corpus. On CTI-to-MITRE and TRAM, this procedure yields consistent macro-$F_1$ improvements across ALBERT, DistilBERT, BERT, and SecureBERT; for example, SecureBERT on CTI-to-MITRE increases from 0.4412 to 0.6558 (Ruiz-Ródenas et al., 21 Jul 2025).

1. Problem setting and scope

CTI mining involves extracting structured insights from unstructured threat data, and one of its central tasks is mapping threat descriptions to MITRE ATT&CK techniques. The underlying classification problem is difficult for two reasons emphasized in the SynthCTI formulation: the scarcity of high-quality labeled CTI data and severe class imbalance, with many techniques represented by very few examples. The framework is therefore positioned as a data-generation response to the long tail of underrepresented techniques rather than as a purely architectural modification to the classifier (Ruiz-Ródenas et al., 21 Jul 2025).

The operational objective is straightforward. A corpus of CTI sentences, each labeled with a MITRE ATT&CK technique, is expanded with synthetic sentences targeted at underrepresented classes. The augmented set is then used to fine-tune transformer classifiers so that the deployed model can map new CTI sentences to MITRE techniques, thereby speeding up analyst workflows and improving coverage of rare classes. A notable empirical consequence is that smaller models augmented with SynthCTI can outperform larger models trained without augmentation, which shifts emphasis from model scale alone to the interaction between representation quality and class-balanced supervision (Ruiz-Ródenas et al., 21 Jul 2025).

2. Framework architecture

SynthCTI is organized into two phases: System Training and System Deployment. During training, the input corpus is partitioned by technique class. For each class, sentences are embedded with all-MiniLM-L6-v2, clustered with HDBSCAN, summarized into cluster-specific prompt features, and passed to Gemma-3 4B to generate $G_i$ synthetic sentences for class $i$. The final augmented training set is the union of original and synthetic samples. In deployment, the fine-tuned model performs technique mapping on previously unseen CTI sentences (Ruiz-Ródenas et al., 21 Jul 2025).

The clustering stage is central because it supplies the semantic scaffolding for generation. The embedding model produces 384-dimensional vectors, and HDBSCAN is run with default Euclidean distance on normalized embeddings. HDBSCAN does not require pre-specifying $k$; it identifies clusters at multiple density levels and labels low-density points as noise. Each point $i$ receives both a cluster label $c(i)$ and a membership probability $p(i)$, which is subsequently used for exemplar selection in prompt construction (Ruiz-Ródenas et al., 21 Jul 2025).

Cluster quality is analyzed with the silhouette coefficient and the Davies–Bouldin index:

$s(i)=\frac{b(i)-a(i)}{\max\{a(i),\,b(i)\}},$

where $a(i)$ is the mean intra-cluster distance and $b(i)$ the smallest mean distance to points in another cluster, and

$G_i$0

The analysis labels classes as “strong” when

$G_i$1

and as “weak” when

$G_i$2

These thresholds are reported as empirical analysis criteria rather than as universal decision rules (Ruiz-Ródenas et al., 21 Jul 2025).

3. Prompt construction and synthetic-sample generation

For each HDBSCAN cluster within a technique, SynthCTI extracts a structured set of prompt features. The prompt includes top-2 few-shot examples selected by membership probability $G_i$3, LDA topics derived from TF-IDF vectors, 10–12 keyphrases extracted via KeyBERT, synonym-expanded keyphrases, tone labels, and a text-type signal based on average sentence count. The resulting template contains explicit sections such as [Examples], [Key Topics], [Keyphrases], and [Synonym Keyphrases], followed by an instruction to generate $G_i$4 sentences using the specified tone mixture (Ruiz-Ródenas et al., 21 Jul 2025).

Synonym expansion is scored with

$G_i$5

where $G_i$6 and $G_i$7 are normalized embeddings, $G_i$8 is the Zipf frequency of candidate synonym $G_i$9, and $i$0 is a tunable weight, with $i$1 given as an example. The top 3 synonyms are retained. Tone is estimated from Flesch Reading Ease and Gunning Fog scores and mapped to {formal, neutral, informal}; cluster tone is then assigned by majority rule when the top label exceeds the second by at least 20%, otherwise a dual-tone instruction is permitted (Ruiz-Ródenas et al., 21 Jul 2025).

The number of synthetic samples per class is computed from the class-frequency mean:

$i$2

where $i$3 is the original count for class $i$4 and $i$5 is the total number of classes. This mechanism directs generation pressure toward minority classes and leaves classes above the mean unaugmented. A plausible implication is that SynthCTI operationalizes class balancing through controlled text synthesis rather than by resampling alone (Ruiz-Ródenas et al., 21 Jul 2025).

Semantic fidelity and lexical diversity are explicitly quantified. Fidelity is measured as average cosine distance between original and synthetic embeddings:

$i$6

while lexical diversity is measured with Self-BLEU by treating one synthetic sentence as candidate and the rest as references. The reported Diversity–Similarity plot places each class in the plane of Self-BLEU and cosine distance to expose redundancy and semantic drift simultaneously (Ruiz-Ródenas et al., 21 Jul 2025).

4. Experimental configuration and reported performance

The evaluation uses two public CTI datasets: CTI-to-MITRE with 12,945 sentences and 188 techniques, and TRAM with approximately 26,000 sentence-level annotations. Each dataset is split 80% train and 20% test with stratification to keep all classes represented, and synthetic data is added only to the training portion. Fine-tuned classifiers are trained with the HuggingFace Trainer API on an NVIDIA A100 40 GB using learning rate $i$7, batch size 32, 10 epochs, AdamW with a linear scheduler, and dropout 0.1 for BERT, ALBERT, and SecureBERT, or 0.2 for DistilBERT (Ruiz-Ródenas et al., 21 Jul 2025).

The model set includes albert-base-v2 (11.8 M), distilbert-base (67 M), bert-base-uncased (110 M), and SecureBERT (125 M). Reported gains are measured primarily in macro-$i$8, reflecting the importance of performance on minority techniques rather than only aggregate accuracy (Ruiz-Ródenas et al., 21 Jul 2025).

Model	CTI-to-MITRE macro-$i$9	TRAM macro-$k$0
ALBERT	0.3496 $k$1 0.5256	0.6200 $k$2 0.7013
DistilBERT	0.4053 $k$3 0.6013	0.6375 $k$4 0.7420
BERT	0.5181 $k$5 0.6302	0.6645 $k$6 0.7412
SecureBERT	0.4412 $k$7 0.6558	0.6468 $k$8 0.7419

On CTI-to-MITRE, the relative gains are +50.4% for ALBERT, +48.4% for DistilBERT, +21.6% for BERT, and +48.6% for SecureBERT. On TRAM, the gains are +13.2%, +16.4%, +11.5%, and +14.7%, respectively. Accuracy also increases; one reported example is SecureBERT on CTI-to-MITRE, which rises from 0.7220 to 0.7811. These results support the claim that synthetic CTI generation can materially improve downstream technique mapping even when the base classifier is relatively compact (Ruiz-Ródenas et al., 21 Jul 2025).

5. Ablations, failure modes, and methodological significance

The reported ablations indicate that data augmentation changes not only final performance but also training dynamics. Models trained with synthetic data reach peak $k$9 in fewer epochs; for example, SecureBERT reaches 0.60 in 6 epochs, whereas the non-augmented version does not reach that level within 10 epochs. This suggests that the synthetic corpus can improve optimization efficiency by densifying class manifolds for sparse techniques (Ruiz-Ródenas et al., 21 Jul 2025).

Class-level analysis via UMAP separates “strong” and “weak” settings in a way that aligns with sample count. Classes with at least 20 original samples yield tight, well-overlapped synthetic clusters, while classes with fewer than 10 samples produce scattered, noisy outputs. The Diversity–Similarity analysis reports that techniques such as T1564 achieve a balance of cosine approximately 0.14 and Self-BLEU approximately 0.36, whereas very rare classes exhibit either semantic drift, with cosine approximately 0.01, or near-duplication, with Self-BLEU greater than or approximately equal to 0.5. A plausible implication is that augmentation quality depends strongly on whether the source class already contains enough internal semantic structure for clustering to extract stable prompt features (Ruiz-Ródenas et al., 21 Jul 2025).

The framework also exposes several qualitative failure modes. Reported issues include entity overfitting, exemplified by repetition of “DarkVishnya,” hallucinations under low context, and prompt outputs that surface inherent MITRE boundary ambiguities such as T1092 versus T1200. A subtechnique ablation further shows that techniques with fewer subtechniques, specifically 1–2, receive larger accuracy gains than techniques with 4–5 subtechniques. This pattern is consistent with the paper’s broader conclusion that class structure, not merely class frequency, affects the utility of synthetic augmentation (Ruiz-Ródenas et al., 21 Jul 2025).

Future directions identified for the framework include integrating NER-based entity normalization, federated LLM prompting, and refined tone/topic controls to mitigate rare-class noise. These directions preserve the core premise of SynthCTI—generation conditioned by class-internal semantics—while targeting the main sources of degradation observed in the rarest techniques (Ruiz-Ródenas et al., 21 Jul 2025).

6. Terminological ambiguity and adjacent meanings

The term “SynthCTI” is not unique to cybersecurity. In medical-imaging literature, synthetic CT imaging is sometimes abbreviated as sCT or SynthCTI, where it refers to generating CT-like images from non-CT modalities, most commonly MRI or CBCT, in order to recover the electron density information required for radiotherapy dose calculation (Thummerer et al., 24 Feb 2025). This is a distinct research area with separate datasets, objectives, and evaluation criteria.

Within that usage, the SynthRAD2025 Grand Challenge dataset provides 2,362 patient studies collected at five European academic radiation oncology centers, subdivided into 890 MRI–CT pairs and 1,472 CBCT–CT pairs across head-and-neck, thorax, and abdomen. The dataset is intended for training, validation, and benchmarking of synthetic CT generation algorithms and includes standardized pre-processing, deformable registration for validation/testing, MetaImage storage, and metrics such as mean absolute error in Hounsfield units, Dice similarity coefficient for organ-at-risk contours, and gamma analysis pass rates for dose comparison (Thummerer et al., 24 Feb 2025).

A further example appears in MRI-based transcranial focused ultrasound planning, where a synthetic CT pipeline combines a 3D U-Net plus transformer MRI-to-sCT network with accelerated acoustic simulation methods. In that setting, the reported goal is precise tFUS targeting without a conventional CT, and the method is evaluated by head MAE, skull MAE, voxel-wise $i$0, focal-width measures, targeting deviation, normalized pressure error, and simulation time (Gao et al., 11 Jul 2025). This suggests that the same label, “SynthCTI,” can denote either a cybersecurity data-augmentation framework for MITRE technique mapping or a shorthand for synthetic CT imaging, depending on disciplinary context.