AZERG: Zero-Shot Grading & CTI Extraction

• AZERG is defined by its dual usage: a zero-shot essay grading approach leveraging gaze-derived cognitive supervision and a STIX extraction framework for cyber threat intelligence.
• In essay grading, AZERG integrates token embeddings with binned gaze features in a multi-task neural architecture to improve cross-prompt generalization and scoring accuracy.
• In cyber threat intelligence, AZERG automates the extraction of entities and relationships from unstructured reports into standardized STIX-compliant JSON bundles for efficient sharing.

AZERG denotes two distinct research usages in the literature. In one usage, introduced only as a synonym in a later exposition rather than by the original authors, AZERG refers to the strict zero-shot automatic essay grading setting in which a model is trained on essays from seven prompts and evaluated on an entirely unseen eighth prompt, with no target-prompt essays used for training or validation; the defining study augments scoring with gaze-derived cognitive supervision to improve cross-prompt transfer (Mathias et al., 2021). In a separate and unrelated usage, AZERG is the name of a semi-automated large-language-model-driven framework for converting unstructured cyber threat intelligence text into STIX-compliant structured entities, relationships, and JSON bundles suitable for TAXII-enabled sharing (Lekssays et al., 22 Jul 2025).

1. Terminological scope

The label “AZERG” is therefore not semantically uniform across domains. In educational NLP, it is an expositional shorthand for zero-shot AEG. In cyber threat intelligence, it is the proper name of a tool/framework. The two usages share neither task formulation nor implementation, and the overlap is purely nominal.

Usage of “AZERG”	Domain	Definition
AZERG as zero-shot AEG	Automatic essay grading	Train on seven prompts and test on an unseen eighth prompt with no target-prompt supervision
AZERG as STIX extraction framework	Cyber threat intelligence	Semi-automated LLM framework for extracting STIX entities and relationships from CTI reports

A common source of confusion is that the 2021 essay-grading paper does not use the acronym “AZERG”; it is synonymous only in the supplied exposition. By contrast, the 2025 cybersecurity paper introduces AZERG explicitly as the tool/framework name, but does not expand the acronym (Mathias et al., 2021, Lekssays et al., 22 Jul 2025).

2. AZERG in zero-shot automatic essay grading

In the essay-grading literature, AZERG denotes a prompt-agnostic evaluation regime rather than a specific architecture. Standard prompt-specific AEG trains and validates on essays written to the same prompt as the test essays. Cross-domain AEG trains on source prompts and tests on a target prompt, but prior work often uses the target prompt for development or hyperparameter adaptation. Zero-shot AEG, by contrast, trains on essays from seven prompts and tests on the eighth prompt with no target-prompt essays used for training or validation (Mathias et al., 2021).

The underlying dataset is the ASAP Automated Student Assessment Prize AES corpus, containing 12,978 essays across eight prompts. The prompts span persuasive, source-dependent response, and narrative writing, with markedly different score ranges.

Prompt	Score range	Essay type
1	2–12	persuasive
2	1–6	persuasive
3	0–3	source-dependent response
4	0–3	source-dependent response
5	0–4	source-dependent response
6	0–4	source-dependent response
7	0–30	narrative
8	0–60	narrative

Cross-prompt generalization is difficult because rubrics and score ranges differ by prompt, genre and length distributions vary widely, lexical and discourse regularities are prompt-specific, and models can learn spurious prompt-specific correlations. The zero-shot protocol therefore tests whether a scoring model can generalize without access to target-prompt cues during either training or validation. In the formulation associated with AZERG, the central idea is to reduce prompt-specific overfitting by integrating cognitive signals from human gaze behavior as an auxiliary task during training (Mathias et al., 2021).

3. Cognitive supervision and model architecture in essay grading

The cognitively aided formulation uses a small gaze dataset originally reused from Mathias et al. (2020): 48 essays drawn from the source-dependent prompts, annotated by 8 annotators whose average agreement with ground truth is approximately 0.646 QWK across those essays. Gaze is mapped onto text through Interest Areas, with each on-screen word treated as a unique IA. Only fixations within IAs are retained; background fixations are ignored. From these events, the model uses five token-level gaze attributes: Dwell Time (DT), First Fixation Duration (FFD), IsRegression (IR), Run Count (RC), and Skip (Mathias et al., 2021).

Continuous gaze features are binned to reduce inter-subject variance arising from reader idiosyncrasies, and both essay scores and binned gaze values are scaled to $[0,1]$ during training. At the representation level, token embeddings $x_t \in \mathbb{R}^d$ and gaze features $g_t \in \mathbb{R}^p$ are concatenated into early-fused inputs $z_t = [x_t; g_t]$. The architecture then applies a word-level CNN, sentence pooling, a sentence-level LSTM, and attention pooling over sentence representations. The reported implementation uses 50-dimensional GloVe embeddings, a CNN with kernel size 5 and 100 filters, an LSTM with 100 hidden units, a dense layer with sigmoid output, dropout 0.5, RMSProp with initial learning rate 0.001 and momentum 0.9, batch size 200, and 50 epochs (Mathias et al., 2021).

Training is multi-task. The primary scoring objective is mean squared error on normalized essay scores,

$$L_{\text{score}} = \frac{1}{N}\sum_i (\hat{y}_i - y_i)^2,$$

while the auxiliary gaze loss is a weighted sum of token-level MSE terms,

$L_{\text{gaze}} = \sum_a w_a L_a,$

with weights $w_{DT}=0.05$, $w_{FFD}=0.05$, $w_{IR}=0.01$, $w_{RC}=0.01$, and $x_t \in \mathbb{R}^d$0. The joint objective is

$x_t \in \mathbb{R}^d$1

No prompt features are used, and no adversarial or domain-adaptation component is introduced. The intended inductive bias is that auxiliary gaze prediction pushes the shared encoder toward cognitively salient, and therefore more transferable, representations (Mathias et al., 2021).

4. Evaluation, performance, and limitations of essay-grading AZERG

Evaluation is reported under a prompt-holdout protocol described as five-fold cross-validation over prompts, where one target prompt is selected for testing and the remaining prompts are used for training and validation; the test score is taken from the epoch with the best development QWK on non-target prompts. The comparison is between a text-only single-task model (“No Gaze”) and the cognitively aided multi-task model (“Gaze”). Performance is measured with Quadratic Weighted Kappa, using weights

$x_t \in \mathbb{R}^d$2

and

$x_t \in \mathbb{R}^d$3

Mean QWK across prompts improves from 0.449 to 0.498, with the paper characterizing this as an average gain of almost 5 percentage points; improvements marked with an asterisk are significant under a two-tailed paired t-test with $x_t \in \mathbb{R}^d$4 (Mathias et al., 2021).

The gains are broad across most prompts: 0.319 to 0.423, 0.391 to 0.439, 0.508 to 0.545, 0.548 to 0.626, 0.548 to 0.628, 0.599 to 0.600, and 0.362 to 0.420. The main negative result is Prompt 8, where QWK declines from 0.316 to 0.286. The reported interpretation is distributional shift: Prompt 8 contains much longer narrative essays, and the gaze-derived auxiliary signal does not overcome the mismatch in length and discourse structure when such long essays are absent from training (Mathias et al., 2021).

Several constraints delimit the scope of these results. Gaze is not required at inference time; it functions only as auxiliary supervision during training, consistent with the “grade without seeing” idea. However, gaze collection is expensive, the gaze dataset is limited to 48 essays and 8 annotators, coverage is confined to source-dependent prompts, and the reader population is specific. The paper does not report per-feature or per-fusion ablations, so the relative importance of DT, FFD, IR, RC, and Skip remains unresolved. Hardware details and sampling rate for eye tracking are not specified in the paper itself, and no adversarial or explicitly domain-invariant training is used (Mathias et al., 2021).

5. AZERG as a STIX extraction framework for cyber threat intelligence

In cyber threat intelligence, AZERG is a semi-automated framework for converting unstructured CTI reports into STIX-compliant structured outputs. Its motivation is operational: CTI analysts spend large amounts of time on manual and repetitive curation, with a survey of 468 analysts reporting that 66% spend more than 50% of their time on tedious work and 64% believe automation would significantly streamline efforts. The paper also cites a nine-year analysis of more than 6 million STIX objects showing severe underuse of STIX relationship objects, a heavy skew toward indicator objects exceeding 90%, delays of 43–109 days for non-URL objects, and multiple quality problems including spelling errors, incorrect designations, duplicates, and nonstandard vocabulary (Lekssays et al., 22 Jul 2025).

AZERG targets the main STIX object families relevant to report structuring. For STIX Domain Objects, it detects names and descriptions of Attack Pattern, Course of Action, Identity, Indicator, Infrastructure, Location, Malware, Threat Actor, Tool, Campaign, and Vulnerability. For Cyber-observable Objects, it identifies indicator subtypes such as Directory, Domain Name, Email Address, File, IPv4/IPv6 Address, MAC Address, URL, and Windows Registry Key. For STIX Relationship Objects, it enforces the type constraints encoded in the STIX relationship matrix, which includes 38 relationship types such as indicates, targets, uses, exfiltrates-to, authored-by, and communicates-with (Lekssays et al., 22 Jul 2025).

The framework is designed to address timeliness, coverage, and data quality simultaneously. It produces a validated STIX JSON bundle that can be consumed by TAXII-enabled platforms. At the same time, it does not attempt to populate all STIX attributes, because attribute completion is described as complex and often dependent on external sources. The focus is on names, types, and linkages (Lekssays et al., 22 Jul 2025).

6. Pipeline, data resources, and modeling choices in CTI AZERG

AZERG decomposes extraction into four sequential subtasks. T1 performs entity detection, first using regular expressions for structured indicators of compromise and then an LLM extractor for remaining STIX entities. T2 performs entity type identification. T3 performs related pair detection under the STIX relationship matrix, with the model given all valid labels for a type pair together with “is not related to” and “not sure.” T4 performs relationship type identification for pairs marked as related. These stages are arranged in a pipeline: pre-processing, T1, verification, T2, verification, T3 with STIX constraints, verification, T4, verification, and STIX JSON generation. Human validation is inserted after each stage to correct, add, or delete outputs before the next step, thereby mitigating error propagation (Lekssays et al., 22 Jul 2025).

Pre-processing converts HTML and PDF documents into plain text, retaining headings for section structure. HTML is handled with BeautifulSoup, PDFs with PyPDF, and section splitting uses headings when available or otherwise length-based segmentation with overlap. IoC extraction precedes LLM execution through IoCFinder and IoCParser. Mixed formats such as code snippets and commands are retained, whereas images and charts are not processed. Prompt templates enumerate STIX definitions and allowed output spaces, and DSPy Signatures are used to encode task I/O specifications (Lekssays et al., 22 Jul 2025).

The training corpus contains 141 real-world CTI reports compiled into AZERG Data and AnnoCTRPlus, yielding 4,011 entities and 2,075 relationships across 914 passages with an average of 59.69 words per passage. Train/test separation is performed at the vendor level, with campaigns in training excluded from test. The training split contains 806 passages, 2,664 entities, and 1,510 relations; the test split contains 108 passages, 1,347 entities, and 565 relations. Annotation is aligned strictly with the STIX standard and cross-verified by additional experts, although no numeric inter-annotator agreement is reported (Lekssays et al., 22 Jul 2025).

Model selection begins from several open- and closed-parameter instruction models, including GPT4o, Mistral-7B-Instruct-v0.3, Gemma-2-9b-it, Qwen2-7B-Instruct, Llama-3.1-8B-Instruct, InternLM2_5-7b-chat, and Phi-3-mini-instruct. After continual fine-tuning, Mistral-7B-Instruct-v0.3 is selected as the base model. The authors build both specialized task models—AZERG-S-T1 through AZERG-S-T4—and a single mixed-task model, AZERG-MixTask. Fine-tuning uses LoRA within LLaMA-Factory, with learning rate $x_t \in \mathbb{R}^d$5; inference uses vLLM with temperature 0.7, top-$x_t \in \mathbb{R}^d$6 0.1, and max tokens 1024 for T1 and 10 for T2–T4. Training runs on an NVIDIA A100-80GB system under Ubuntu 22.04, and inference runs on a single NVIDIA V100 32GB (Lekssays et al., 22 Jul 2025).

7. Empirical results, error modes, and deployment implications in CTI AZERG

Evaluation uses precision, recall, and F1 on the held-out test split, with

$x_t \in \mathbb{R}^d$7

On real-world test data, AZERG-MixTask reaches F1 scores of 84.43% for T1, 88.49% for T2, 95.47% for T3, and 84.60% for T4. The abstract summarizes these as improvements of 2–25% across tasks relative to baselines and state-of-the-art methods. Exact comparisons show especially large margins on T1, T2, and T4, with more modest gains on T3 (Lekssays et al., 22 Jul 2025).

The test-set detail is informative. For T1, AZERG-MixTask attains precision 0.9092, recall 0.7880, and F1 0.8443, outperforming GPT4o at 0.6277 F1, base Mistral at 0.5871, AttaKG at 0.3941, EXTRACTOR at 0.3023, and GliNER at 0.2159. For T2, the best specialized model reaches precision/recall/F1 of 0.8923, slightly above AZERG-MixTask at 0.8849. For T3, AZERG-MixTask records precision 0.9224, recall 0.9893, and F1 0.9547, exceeding GPT4o at 0.9315 and Mistral at 0.9035. For T4, AZERG-MixTask reaches 0.8460, above GPT4o at 0.7946 and Mistral at 0.7097. Reported per-query latencies are 2.57 s for T1, 1.54 s for T2, 0.58 s for T3, and 0.36 s for T4 (Lekssays et al., 22 Jul 2025).

The error analysis shows that T1 and T4 are the hardest stages. In T1, semantic ambiguity and contextual interpretation produce false positives such as algorithm names or descriptive concepts, while recall losses are concentrated in non-IoC categories including Tools, Malware variants, Threat Actor aliases, and Identities; more than 25% of Identity mentions are missed because of contextual ambiguity. T2 exhibits confusions among Tools, Infrastructure, Malware, Threat Actors, and Identities. T3 is strong but can over-interpret verbs not mapped to STIX relations and omit valid links in multi-entity paragraphs. T4 frequently confuses semantically adjacent labels such as uses, communicates-with, exfiltrates-to, and targets. Error amplification is a recurring risk: code and other non-linguistic text can trigger cascades of entity and relation errors, and unresolved aliases can cause multiple misses (Lekssays et al., 22 Jul 2025).

Operationally, AZERG is designed for SOC and intelligence-sharing workflows. It enforces STIX compliance at several points: entity typing is constrained to STIX taxonomies, invalid type pairs are excluded from relation proposals, and relationship labels are selected only from those allowed for the source–target types. The final product is a shareable STIX bundle suitable for TAXII ingestion. The framework remains explicitly human-in-the-loop, and the paper treats that design as a precision safeguard rather than an implementation inconvenience. Limitations include incomplete STIX attribute coverage, lack of image or chart processing, unspecified licensing terms for released resources, absence of reported numeric inter-annotator agreement, and the possibility that generalization degrades on new vendors or especially long and complex reports (Lekssays et al., 22 Jul 2025).