AQUA-LLM: Cybersecurity and Audio QA Framework

• AQUA-LLM is a dual-purpose framework that refers to both a cybersecurity QA evaluation setup and an audio QA dataset generation pipeline, each with distinct technical approaches.
• In cybersecurity, the system benchmarks small LLMs across configurations—base, quantized, fine-tuned, and fine-tuned quantized—balancing accuracy, robustness, and efficiency for edge deployments.
• For audio, the AQUALLM pipeline automates large-scale QA dataset creation by extracting candidate answers, generating questions, filtering, and paraphrasing, enhancing multimodal QA tasks.

Searching arXiv for the exact AQUA-LLM/AQUALLM entries and closely related names to ground the article. AQUA-LLM designates more than one line of research in the recent arXiv literature. In its most explicit hyphenated usage, it refers to an evaluation framework for studying accuracy, quantization, and adversarial robustness trade-offs in small LLMs for cybersecurity question answering under four deployment configurations: base, quantized-only, fine-tuned, and fine-tuned combined with quantization (Gungor et al., 16 Sep 2025). A closely related near-homonymous usage, “AQUALLM,” denotes a fully automated, end-to-end pipeline that transforms existing audio-caption corpora into large-scale Audio Question Answering datasets through candidate answer extraction, question generation, filtering, and paraphrasing (Behera et al., 2023). The name therefore occupies an ambiguous position: it can denote either a cybersecurity QA benchmarking framework or, in the earlier spelling variant, an audio QA data-generation system.

1. Terminological Scope and Disambiguation

The 2025 cybersecurity work defines AQUA-LLM as an evaluation framework for benchmarking several state-of-the-art small LLMs under four distinct configurations—base, quantized-only, fine-tuned, and fine-tuned combined with quantization—specifically for cybersecurity QA (Gungor et al., 16 Sep 2025). Its stated motivation is that quantization can improve deployability on resource-constrained edge devices, but may degrade model accuracy and increase susceptibility to adversarial attacks.

By contrast, the 2023 audio work introduces “AQUALLM: Audio Question Answering Data Generation Using LLMs,” a scalable AQA data generation pipeline that relies on existing audio-caption annotations and state-of-the-art LLMs to generate expansive, high-quality AQA datasets (Behera et al., 2023). The difference in spelling is operationally significant. The former is an evaluation framework centered on compression, robustness, and efficiency in cybersecurity QA; the latter is a data-generation framework for audio question answering.

This suggests that “AQUA-LLM” is not a single, stable technical term across arXiv, but a label shared by distinct LLM-centered systems with different task definitions, datasets, and optimization criteria.

2. Cybersecurity QA Formulation

In the cybersecurity setting, AQUA-LLM measures three core properties of each model-configuration: accuracy, robustness, and efficiency (Gungor et al., 16 Sep 2025). Accuracy is defined on a multiple-choice QA evaluation set of $N$ questions as

$$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$

Robustness is measured through Attack Success Rate (ASR) on adversarial prompts crafted via direct prompt-injection:

$$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$

A lower ASR denotes a more robust model. Efficiency is measured as average per-question latency $t$ in seconds, with speedup defined relative to the full-precision base model as

$Speedup_C = \frac{t_{\text{base}}}{t_C}.$

The framework evaluates six open-source models, all approximately $7$–$8$B parameters: Meta LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct, Phi-3.5-Mini-Instruct, Foundation-Sec-8B, Qwen 2.5-7B-Instruct, and DeepSeek-R1-Distill. The datasets are CyberBench—converted to multiple-choice QA, with SecMMLU for train and CyQuiz for evaluation—and CyberMetric, described as retrieval-augmented, expert-validated multiple-choice QA. Hardware consists of an NVIDIA A100 GPU, a 16-core CPU, and 32 GB RAM.

The framework’s central analytical object is the interaction among four deployment modes: $B$ for the pre-trained base model, $Q$ for 4-bit NF4 quantized-only inference, $FT$ for LoRA fine-tuning on the 16-bit base model, and $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$0 for LoRA fine-tuning on the pre-quantized 4-bit model. This organization makes the trade-off landscape explicit rather than treating compression or fine-tuning as isolated interventions.

3. Quantization and Fine-Tuning Regimes

AQUA-LLM applies post-training, 4-bit “NormalFloat” (NF4) quantization using the BitsAndBytes library (Gungor et al., 16 Sep 2025). For a transformer weight matrix $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$1, each element $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$2 is replaced by its 4-bit NF4 approximation,

$$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$3

where the exponent $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$4 and low-bit mantissa $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$5 are chosen to best match the distribution of $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$6. BitsAndBytes additionally applies “double quantization,” splitting metadata and further compressing scale factors, yielding roughly a 75% reduction in GPU memory footprint from 16-bit to 4-bit.

The fine-tuning mechanism is parameter-efficient LoRA fine-tuning on both full-precision and quantized models. The reported hyperparameters are adapter rank $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$7, scaling $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$8, no dropout, AdamW with peak learning rate $$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.$$9, and 60 gradient-update steps. The loss is standard cross-entropy over the correct multiple-choice label. For CyberMetric, the split is 850 train and 150 test, further divided into 50-question and 100-question subsets; for CyberBench (SecMMLU), the split is 116 train and 128 test, again with overlapping 50-question and 100-question subsets.

In the $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$0 configuration, LoRA modules are injected into the attention projections of the pre-quantized 4-bit model, the quantized backbone remains frozen except for the low-rank adapters, and the adapters are merged post-training into a single 4-bit checkpoint. A common misconception is that quantization and fine-tuning operate independently. The reported setup instead treats them as coupled interventions whose interaction materially affects both predictive quality and attack susceptibility.

4. Reported Trade-offs in Accuracy, Robustness, and Efficiency

The main reported result is that no single deployment mode simultaneously maximizes accuracy, robustness, and efficiency (Gungor et al., 16 Sep 2025). Quantization alone improves efficiency but yields the lowest accuracy and robustness. Fine-tuning alone restores or exceeds the highest accuracy, but often increases ASR relative to the base model. The combined fine-tuned quantized setting preserves most of the efficiency gains of quantization while recovering most of the accuracy and reducing ASR below both $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$1 and $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$2 in many cases.

Configuration	Average QA accuracy	ASR / efficiency summary
Base (B)	CyberBench $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$3; CyberMetric $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$4	Avg ASR $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$5
Quantized only (Q)	CyberBench $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$6; CyberMetric $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$7	Avg ASR $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$8; best efficiency
Fine-tuned (FT)	CyberBench $$ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.$$9; CyberMetric $t$0	CyberBench ASR $t$1; CyberMetric ASR $t$2
Fine-tuned Quantized (FTQ)	CyberBench $t$3; CyberMetric $t$4	CyberBench ASR $t$5; CyberMetric ASR $t$6

For latency, the reported speedups for quantized inference are 1.25× for Meta-Llama-3.1-8B, 1.26× for Mistral-7B, 1.10× for Phi-3.5, 1.17× for Foundation-Sec-8B, 1.13× for Qwen 2.5-7B, and 1.28× for DeepSeek-R1. On CyberMetric, the corresponding base and quantized latencies are 0.140 s and 0.112 s for Meta-Llama-3.1-8B, 0.118 s and 0.093 s for Mistral-7B, 0.124 s and 0.112 s for Phi-3.5, 0.136 s and 0.116 s for Foundation-Sec-8B, 0.138 s and 0.122 s for Qwen 2.5-7B, and 0.154 s and 0.120 s for DeepSeek-R1.

The paper’s interpretive claim is that quantization-aware, adversarially informed fine-tuning is the key to deploying robust, efficient LLMs on edge devices for security-critical question answering. A plausible implication is that robustness cannot be inferred from task accuracy alone: the $t$7 condition reaches the strongest QA scores, but the $t$8 condition offers a more balanced operating point once ASR and inference latency are included.

5. AQUALLM in Audio Question Answering

The earlier “AQUALLM” framework addresses a different problem: the generation of large-scale Audio Question Answering datasets from audio-caption corpora (Behera et al., 2023). It is described as a fully automated, end-to-end pipeline with four sequential modules—Candidate Answer Extraction Module (CAM), Question Generation Module (QGM), Question-Answer Filtering Module (QAFM), and Question Paraphrasing Module (QPM).

CAM uses spaCy to produce a dependency parse and part-of-speech tags from a caption $t$9, identifying six “in-caption” answer-candidate categories—noun phrases including named entities, verbal phrases, adjectival phrases, adverbial phrases, cardinal numbers, and the token “yes”—and two “out-of-caption” categories, the Boolean “no” and the numeric “zero.” This yields a candidate set

$Speedup_C = \frac{t_{\text{base}}}{t_C}.$0

For each $Speedup_C = \frac{t_{\text{base}}}{t_C}.$1 pair with $Speedup_C = \frac{t_{\text{base}}}{t_C}.$2, QGM prompts a T5 model fine-tuned on SQuAD1.1 to generate a question whose answer is exactly $Speedup_C = \frac{t_{\text{base}}}{t_C}.$3. QAFM then re-answers the generated question with a separate T5 model fine-tuned on SQuAD2.0 and computes token-level $Speedup_C = \frac{t_{\text{base}}}{t_C}.$4 between the original candidate and the re-answered output, retaining only pairs with $Speedup_C = \frac{t_{\text{base}}}{t_C}.$5. QPM uses a T5 model fine-tuned on the Quora Question Pairs dataset to generate the top $Speedup_C = \frac{t_{\text{base}}}{t_C}.$6 paraphrases, yielding up to $Speedup_C = \frac{t_{\text{base}}}{t_C}.$7 questions per answer candidate.

The reported total number of question instances for a caption set $Speedup_C = \frac{t_{\text{base}}}{t_C}.$8 is

$Speedup_C = \frac{t_{\text{base}}}{t_C}.$9

where $7$0 and $7$1. Applying the framework to three public corpora produces three benchmark datasets.

Dataset	Audio clips / questions	Unique answers
AQUALLM-Clotho	5 929 / 438 600	25 235
AQUALLM-AudioCaps	51 308 / 728 310	35 008
AQUALLM-MACS	3 930 / 268 875	11 874

The paper compares these to DAQA, described as the largest prior human- or algorithmically-generated AQA dataset, with 100 000 audios, 599 294 questions, and 36 answer labels. For downstream evaluation, the state-of-the-art MWAFM audio-question classifier is trained on each AQUALLM dataset by treating answers as classification labels. Reported binary classification accuracies are 68.75% for ClothoAQA as baseline, 95.58% for AQUALLM-Clotho, 95.86% for AQUALLM-AudioCaps, and 96.84% for AQUALLM-MACS. The same source reports that models trained on AQUALLM datasets exhibit markedly better transfer to out-of-domain audio questions, with informal cross-evaluation accuracy remaining above 90%, whereas a model trained on human-annotated ClothoAQA falls below 65%.

This near-homonymous usage matters because it represents a different understanding of what an “AQUA-LLM” system can be: not a deployment benchmark for compact cybersecurity QA models, but an automated data-construction apparatus for multimodal QA.

6. Related AQUA-Named Systems and Conceptual Boundaries

The broader literature contains several additional systems whose names are close enough to create confusion, but whose technical aims are distinct. “AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs” is an approximation strategy for attention that computes a universal projection matrix via SVD in an offline phase and then dynamically selects sparse dimensions based on projected query magnitudes during online inference; on Llama-3.1-8B it reports a 25% reduction in attention dot-product computation with a statistically insignificant impact on performance (S et al., 14 Sep 2025). “AQUA: A LLM for Aquaculture & Fisheries” is a domain-adapted model built on Mistral-7B-Instruct-v0.3 with LoRA rank-8 adapters and an agentic data pipeline called AQUADAPT, trained on approximately 3 million high-fidelity QA pairs after curation and LLM-judge filtering (Narisetty et al., 28 Jul 2025).

“AquiLLM” is a lightweight, modular RAG system for research groups, built on Django, PostgreSQL with pgvector, and a React front end, with ingestion, embedding, retrieval, generation, and collection-level privacy control for private scholarly materials (Campbell et al., 25 Jul 2025). “AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains” is a GPU memory-management framework for responsive LLM inference under bursty load, reporting 20× responsiveness improvement compared to the state of the art and 4× throughput improvement over a single long prompt in the abstract (Kumar et al., 2024). “AquaChat++” is a multi-ROV inspection framework for aquaculture net pens in which a high-level LLM planning layer converts natural-language commands into symbolic multi-agent inspection plans and a low-level control layer handles trajectory tracking, battery management, and thruster fault compensation (Saad et al., 6 Aug 2025).

These neighboring usages do not redefine AQUA-LLM itself, but they do establish a naming environment in which “AQUA,” “AQUA-LLM,” and “AQUALLM” refer to materially different objects: an evaluation framework for cybersecurity QA, a data-generation pipeline for audio QA, an attention approximation method, a domain-specialized aquaculture LLM, a privacy-aware RAG stack, a GPU offloading system, and an LLM-assisted robotic inspection architecture. For technical writing, explicit citation and spelling are therefore essential.