---
title: 'AQUA-LLM: Cybersecurity and Audio QA Framework'
url: https://www.emergentmind.com/topics/aqua-llm
type: topic
---

# AQUA-LLM: Cybersecurity and Audio QA Framework

Searching arXiv for the exact AQUA-LLM/AQUALLM entries and closely related names to ground the article.
AQUA-LLM designates more than one line of research in the recent arXiv literature. In its most explicit hyphenated usage, it refers to an evaluation framework for studying accuracy, quantization, and adversarial robustness trade-offs in small large language models for cybersecurity question answering under four deployment configurations: base, quantized-only, fine-tuned, and fine-tuned combined with quantization [2509.13514]. A closely related near-homonymous usage, “AQUALLM,” denotes a fully automated, end-to-end pipeline that transforms existing audio-caption corpora into large-scale Audio Question Answering datasets through candidate answer extraction, question generation, filtering, and paraphrasing [2312.17343]. The name therefore occupies an ambiguous position: it can denote either a cybersecurity QA benchmarking framework or, in the earlier spelling variant, an audio QA data-generation system.

## 1. Terminological Scope and Disambiguation

The 2025 cybersecurity work defines AQUA-LLM as an evaluation framework for benchmarking several state-of-the-art small LLMs under four distinct configurations—base, quantized-only, fine-tuned, and fine-tuned combined with quantization—specifically for cybersecurity QA [2509.13514]. Its stated motivation is that quantization can improve deployability on resource-constrained edge devices, but may degrade model accuracy and increase susceptibility to adversarial attacks.

By contrast, the 2023 audio work introduces “AQUALLM: Audio Question Answering Data Generation Using Large Language Models,” a scalable AQA data generation pipeline that relies on existing audio-caption annotations and state-of-the-art LLMs to generate expansive, high-quality AQA datasets [2312.17343]. The difference in spelling is operationally significant. The former is an evaluation framework centered on compression, robustness, and efficiency in cybersecurity QA; the latter is a data-generation framework for audio question answering.

This suggests that “AQUA-LLM” is not a single, stable technical term across arXiv, but a label shared by distinct LLM-centered systems with different task definitions, datasets, and optimization criteria.

## 2. Cybersecurity QA Formulation

In the cybersecurity setting, AQUA-LLM measures three core properties of each model-configuration: accuracy, robustness, and efficiency [2509.13514]. Accuracy is defined on a multiple-choice QA evaluation set of $N$ questions as
$$
A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{u}_i=y_i\}\times 100\%.
$$
Robustness is measured through Attack Success Rate (ASR) on adversarial prompts crafted via direct prompt-injection:
$$
ASR = \frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{\text{response}_j \text{ is harmful}\}\times 100\%.
$$
A lower ASR denotes a more robust model. Efficiency is measured as average per-question latency $t$ in seconds, with speedup defined relative to the full-precision base model as
$$
Speedup_C = \frac{t_{\text{base}}}{t_C}.
$$

The framework evaluates six open-source models, all approximately $7$–$8$B parameters: Meta LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct, Phi-3.5-Mini-Instruct, Foundation-Sec-8B, Qwen 2.5-7B-Instruct, and DeepSeek-R1-Distill. The datasets are CyberBench—converted to multiple-choice QA, with SecMMLU for train and CyQuiz for evaluation—and CyberMetric, described as retrieval-augmented, expert-validated multiple-choice QA. Hardware consists of an NVIDIA A100 GPU, a 16-core CPU, and 32 GB RAM.

The framework’s central analytical object is the interaction among four deployment modes: $B$ for the pre-trained base model, $Q$ for 4-bit NF4 quantized-only inference, $FT$ for LoRA fine-tuning on the 16-bit base model, and $FTQ$ for LoRA fine-tuning on the pre-quantized 4-bit model. This organization makes the trade-off landscape explicit rather than treating compression or fine-tuning as isolated interventions.

## 3. Quantization and Fine-Tuning Regimes

AQUA-LLM applies post-training, 4-bit “NormalFloat” (NF4) quantization using the BitsAndBytes library [2509.13514]. For a transformer weight matrix $W$, each element $w$ is replaced by its 4-bit NF4 approximation,
$$
Q_{\mathrm{NF4}}(w)\approx \mathrm{sign}(w)\cdot 2^{e(w)}\cdot m(w),
$$
where the exponent $e(w)$ and low-bit mantissa $m(w)$ are chosen to best match the distribution of $W$. BitsAndBytes additionally applies “double quantization,” splitting metadata and further compressing scale factors, yielding roughly a 75% reduction in GPU memory footprint from 16-bit to 4-bit.

The fine-tuning mechanism is parameter-efficient LoRA fine-tuning on both full-precision and quantized models. The reported hyperparameters are adapter rank $r=16$, scaling $\alpha=16$, no dropout, AdamW with peak learning rate $\eta_{\max}=2\times 10^{-4}$, and 60 gradient-update steps. The loss is standard cross-entropy over the correct multiple-choice label. For CyberMetric, the split is 850 train and 150 test, further divided into 50-question and 100-question subsets; for CyberBench (SecMMLU), the split is 116 train and 128 test, again with overlapping 50-question and 100-question subsets.

In the $FTQ$ configuration, LoRA modules are injected into the attention projections of the pre-quantized 4-bit model, the quantized backbone remains frozen except for the low-rank adapters, and the adapters are merged post-training into a single 4-bit checkpoint. A common misconception is that quantization and fine-tuning operate independently. The reported setup instead treats them as coupled interventions whose interaction materially affects both predictive quality and attack susceptibility.

## 4. Reported Trade-offs in Accuracy, Robustness, and Efficiency

The main reported result is that no single deployment mode simultaneously maximizes accuracy, robustness, and efficiency [2509.13514]. Quantization alone improves efficiency but yields the lowest accuracy and robustness. Fine-tuning alone restores or exceeds the highest accuracy, but often increases ASR relative to the base model. The combined fine-tuned quantized setting preserves most of the efficiency gains of quantization while recovering most of the accuracy and reducing ASR below both $FT$ and $Q$ in many cases.

| Configuration | Average QA accuracy | ASR / efficiency summary |
|---|---:|---|
| Base (B) | CyberBench $\approx 69.8\%$; CyberMetric $\approx 70.4\%$ | Avg ASR $\approx 52.3\%$ |
| Quantized only (Q) | CyberBench $\approx 67\%$; CyberMetric $\approx 67\%$ | Avg ASR $\approx 77.9\%$; best efficiency |
| Fine-tuned (FT) | CyberBench $\approx 97.2\%$; CyberMetric $\approx 98.8\%$ | CyberBench ASR $\approx 84.1\%$; CyberMetric ASR $\approx 76.3\%$ |
| Fine-tuned Quantized (FTQ) | CyberBench $\approx 94.9\%$; CyberMetric $\approx 98.3\%$ | CyberBench ASR $\approx 64.6\%$; CyberMetric ASR $\approx 71.6\%$ |

For latency, the reported speedups for quantized inference are 1.25× for Meta-Llama-3.1-8B, 1.26× for Mistral-7B, 1.10× for Phi-3.5, 1.17× for Foundation-Sec-8B, 1.13× for Qwen 2.5-7B, and 1.28× for DeepSeek-R1. On CyberMetric, the corresponding base and quantized latencies are 0.140 s and 0.112 s for Meta-Llama-3.1-8B, 0.118 s and 0.093 s for Mistral-7B, 0.124 s and 0.112 s for Phi-3.5, 0.136 s and 0.116 s for Foundation-Sec-8B, 0.138 s and 0.122 s for Qwen 2.5-7B, and 0.154 s and 0.120 s for DeepSeek-R1.

The paper’s interpretive claim is that quantization-aware, adversarially informed fine-tuning is the key to deploying robust, efficient LLMs on edge devices for security-critical question answering. A plausible implication is that robustness cannot be inferred from task accuracy alone: the $FT$ condition reaches the strongest QA scores, but the $FTQ$ condition offers a more balanced operating point once ASR and inference latency are included.

## 5. AQUALLM in Audio Question Answering

The earlier “AQUALLM” framework addresses a different problem: the generation of large-scale Audio Question Answering datasets from audio-caption corpora [2312.17343]. It is described as a fully automated, end-to-end pipeline with four sequential modules—Candidate Answer Extraction Module (CAM), Question Generation Module (QGM), Question-Answer Filtering Module (QAFM), and Question Paraphrasing Module (QPM).

CAM uses spaCy to produce a dependency parse and part-of-speech tags from a caption $c$, identifying six “in-caption” answer-candidate categories—noun phrases including named entities, verbal phrases, adjectival phrases, adverbial phrases, cardinal numbers, and the token “yes”—and two “out-of-caption” categories, the Boolean “no” and the numeric “zero.” This yields a candidate set
$$
S(c)=\{a_1,a_2,\dots,a_N\}.
$$
For each $(c,a)$ pair with $a\in ICAC$, QGM prompts a T5 model fine-tuned on SQuAD1.1 to generate a question whose answer is exactly $a$. QAFM then re-answers the generated question with a separate T5 model fine-tuned on SQuAD2.0 and computes token-level $F_1$ between the original candidate and the re-answered output, retaining only pairs with $F_1(a,\bar{a})\ge 0.55$. QPM uses a T5 model fine-tuned on the Quora Question Pairs dataset to generate the top $k=5$ paraphrases, yielding up to $(1+k)$ questions per answer candidate.

The reported total number of question instances for a caption set $C$ is
$$
\mathrm{Total\ Q}=\sum_{c\in C}N_c\cdot(1+k),
$$
where $N_c=|S(c)|$ and $k=5$. Applying the framework to three public corpora produces three benchmark datasets.

| Dataset | Audio clips / questions | Unique answers |
|---|---|---:|
| AQUALLM-Clotho | 5 929 / 438 600 | 25 235 |
| AQUALLM-AudioCaps | 51 308 / 728 310 | 35 008 |
| AQUALLM-MACS | 3 930 / 268 875 | 11 874 |

The paper compares these to DAQA, described as the largest prior human- or algorithmically-generated AQA dataset, with 100 000 audios, 599 294 questions, and 36 answer labels. For downstream evaluation, the state-of-the-art MWAFM audio-question classifier is trained on each AQUALLM dataset by treating answers as classification labels. Reported binary classification accuracies are 68.75% for ClothoAQA as baseline, 95.58% for AQUALLM-Clotho, 95.86% for AQUALLM-AudioCaps, and 96.84% for AQUALLM-MACS. The same source reports that models trained on AQUALLM datasets exhibit markedly better transfer to out-of-domain audio questions, with informal cross-evaluation accuracy remaining above 90%, whereas a model trained on human-annotated ClothoAQA falls below 65%.

This near-homonymous usage matters because it represents a different understanding of what an “AQUA-LLM” system can be: not a deployment benchmark for compact cybersecurity QA models, but an automated data-construction apparatus for multimodal QA.

## 6. Related AQUA-Named Systems and Conceptual Boundaries

The broader literature contains several additional systems whose names are close enough to create confusion, but whose technical aims are distinct. “AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs” is an approximation strategy for attention that computes a universal projection matrix via SVD in an offline phase and then dynamically selects sparse dimensions based on projected query magnitudes during online inference; on Llama-3.1-8B it reports a 25% reduction in attention dot-product computation with a statistically insignificant impact on performance [2509.11155]. “AQUA: A Large Language Model for Aquaculture & Fisheries” is a domain-adapted model built on Mistral-7B-Instruct-v0.3 with LoRA rank-8 adapters and an agentic data pipeline called AQUADAPT, trained on approximately 3 million high-fidelity QA pairs after curation and LLM-judge filtering [2507.20520].

“AquiLLM” is a lightweight, modular RAG system for research groups, built on Django, PostgreSQL with pgvector, and a React front end, with ingestion, embedding, retrieval, generation, and collection-level privacy control for private scholarly materials [2508.05648]. “AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains” is a GPU memory-management framework for responsive LLM inference under bursty load, reporting 20× responsiveness improvement compared to the state of the art and 4× throughput improvement over a single long prompt in the abstract [2407.21255]. “AquaChat++” is a multi-ROV inspection framework for aquaculture net pens in which a high-level LLM planning layer converts natural-language commands into symbolic multi-agent inspection plans and a low-level control layer handles trajectory tracking, battery management, and thruster fault compensation [2508.06554].

These neighboring usages do not redefine AQUA-LLM itself, but they do establish a naming environment in which “AQUA,” “AQUA-LLM,” and “AQUALLM” refer to materially different objects: an evaluation framework for cybersecurity QA, a data-generation pipeline for audio QA, an attention approximation method, a domain-specialized aquaculture LLM, a privacy-aware RAG stack, a GPU offloading system, and an LLM-assisted robotic inspection architecture. For technical writing, explicit citation and spelling are therefore essential.

Source: https://www.emergentmind.com/topics/aqua-llm