SRAG-MAV: Structured Extraction for Chinese Hate Speech

• SRAG-MAV is a structured framework that reformulates quadruplet extraction into triplet extraction with deterministic reconstruction of hatefulness labels.
• It leverages self-retrieval augmented generation by using the annotated training set as retrieval memory to guide structured output and reduce ambiguity.
• The multi-round accumulative voting mechanism stabilizes inference by aggregating outputs across rounds, which enhances exact-match performance.

SRAG-MAV is a structured large-language-model framework introduced for CCL25-Eval Task 10 on the STATE ToxiCN benchmark for Fine-Grained Chinese Hate Speech Recognition (FGCHSR). The framework combines Task Reformulation (TR), Self-Retrieval-Augmented Generation (SRAG), and Multi-Round Accumulative Voting (MAV) in a single inference and training pipeline built on Qwen2.5-7B. Its central design is to simplify the target structure, use the annotated training set itself as retrieval memory, and stabilize structured predictions through repeated inference with frequency-thresholded aggregation (Wang et al., 24 Jul 2025).

1. Task setting and formal output structure

The framework is designed for a benchmark in which a system must extract a structured quadruplet from Chinese social media text. The required fields are Target, Argument, Targeted Group, and Hateful. The paper emphasizes that this problem is harder than ordinary hate-speech classification because the model must recover structured, context-dependent information rather than output a single label (Wang et al., 24 Jul 2025).

The reported dataset scale is 4,000 training samples and 1,602 test samples, with training conducted on 4× NVIDIA L40S 40GB GPUs. Evaluation is based on three scores. Hard Score is exact F1 on exact quadruplet matches. Soft Score is a partial-match F1 requiring the same Targeted Group, the same Hateful label, and Target and Argument similarity above 50%. Average Score is the mean of Hard and Soft. These definitions matter because SRAG-MAV is optimized for exact structured extraction rather than only coarse classification, and the framework’s design directly targets error modes that degrade exact-match performance.

The paper’s problem framing implies a strong dependence between structural consistency and evaluation sensitivity. A plausible implication is that even small output inconsistencies can sharply reduce Hard Score, which helps explain why the framework allocates separate mechanisms to structure simplification, contextual grounding, and inference stabilization.

2. End-to-end pipeline and backbone configuration

The SRAG-MAV pipeline is specified as a nine-stage process. It first reformulates the task from quadruplet extraction to triplet extraction. It then embeds all training texts with a Chinese retrieval encoder, retrieves similar training examples for each input, constructs prompts by concatenating retrieved examples with the input, and fine-tunes Qwen2.5-7B on these retrieval-enriched training prompts. At inference time, it retrieves top-$k$ similar training samples for each test input, runs multi-round inference over multiple prompts, votes accumulatively until one triplet candidate reaches threshold $\tau$, and finally converts the selected triplet back to a quadruplet by inferring the hatefulness label (Wang et al., 24 Jul 2025).

The implementation stack is explicit. The backbone is Qwen2.5-7B. Fine-tuning uses LLaMA-Factory. Inference uses vLLM. The retrieval model is bge-large-zh-v1.5. Reported hyperparameters are: fine-tuning temperature 0.7, MAV inference temperature 0.1, retrieval top-$k$ 10, and voting threshold $\tau$ 200. The paper notes that larger thresholds generally improve results, but increase computational cost.

Architecturally, the framework is not a generic external-corpus RAG system. Its retrieval source is the benchmark’s own annotated training set, and its output is a structured extraction object rather than free-form answer text. This places SRAG-MAV at the intersection of prompt-based structured generation, example retrieval, and test-time ensembling.

3. Task Reformulation (TR)

The Task Reformulation component is motivated by a dataset regularity: when the Targeted Group is "no-hate", the Hateful label is also "no-hate"; otherwise, the hatefulness is "hate" (Wang et al., 24 Jul 2025). On that basis, the framework replaces direct quadruplet prediction with triplet prediction.

The original target is

$(Target, Argument, Targeted\ Group, Hateful)$

and the reformulated target is

$(Target, Argument, Targeted\ Group).$

The final quadruplet is reconstructed deterministically:

• if Targeted Group = no-hate, then Hateful = no-hate
• else Hateful = hate

The paper characterizes this as a structure simplification step. It reduces the number of structured elements that must be generated directly, lowers the chance of inconsistent outputs, and simplifies sequence generation. In the reported ablation, adding TR to the base model moves performance from 23.70 / 47.03 / 35.365 to 24.33 / 47.35 / 35.840 in Hard / Soft / Average terms, indicating a modest but measurable gain.

Conceptually, TR converts a partially redundant output schema into a more compact target space. This suggests that part of SRAG-MAV’s effectiveness comes not from stronger parametric knowledge alone, but from aligning the prediction interface with a deterministic property of the dataset.

4. Self-Retrieval-Augmented Generation (SRAG)

The SRAG component uses the training set itself as the retrieval corpus rather than relying on an external knowledge base. The paper argues that this is especially suitable when external high-quality domain corpora may be unavailable and when the training data already contains annotated structured examples that can function as few-shot demonstrations (Wang et al., 24 Jul 2025).

All training texts are encoded with bge-large-zh-v1.5 and stored in a retrieval database. Similarity is computed by cosine similarity:

$$\text{sim}(x, x_i) = \cos(\mathbf{e}_x, \mathbf{e}_{x_i})$$

where $\mathbf{e}_x$ and $\mathbf{e}_{x_i}$ are embeddings from the retrieval model.

During training, the system retrieves the most similar sample, excludes the input itself, concatenates the retrieved sample with the original input, and trains the model to output the corresponding triplet. During inference, it retrieves the top-$k$ most similar training samples for each test input; each retrieved sample is paired with the test input, yielding $\tau$0 distinct prompts.

The paper interprets SRAG as a form of dynamic in-context learning. Retrieved examples are semantically close to the test input, provide task format guidance, bias the model toward valid structured outputs, and reduce ambiguity in difficult cases. In the ablation, adding SRAG on top of TR improves performance from 24.33 / 47.35 / 35.840 to 25.30 / 47.85 / 36.575, which is larger than the gain from TR alone.

A notable aspect of SRAG is that retrieval is not auxiliary metadata retrieval in the usual RAG sense. It is retrieval over labeled instances whose value lies in prompt conditioning and output regularization. That design makes the retrieval memory both domain-specific and annotation-aware.

5. Multi-Round Accumulative Voting (MAV) and empirical performance

The MAV component is introduced to address output variability across prompts and decoding rounds. The paper states that MAV draws inspiration from the idea behind Parallel Scaling Law (PARSCALE): use multiple diverse input variants and aggregate outputs instead of relying on a single pass (Wang et al., 24 Jul 2025).

Its procedure is explicit. The system retrieves top-$\tau$1 similar training examples, builds $\tau$2 prompts, runs inference on each prompt, collects the predicted triplets, counts how often each triplet appears, continues rounds of inference until the most frequent triplet’s count exceeds threshold $\tau$3, selects that triplet as the final answer, and converts the triplet into a quadruplet by restoring the hatefulness label. The aggregation rule is described as:

$\tau$4

subject to

$\tau$5

The paper describes this as a frequency-thresholded accumulation scheme rather than a single majority vote over a fixed number of samples. Its intended effects are reduced dependence on any one prompt, smoothing of decoding randomness, favoring repeated consistent outputs, and trading extra inference cost for better stability.

Empirically, the full SRAG-MAV system achieves Hard Score: 26.66, Soft Score: 48.35, and Average Score: 37.505 on the STATE ToxiCN test set. The reported baselines include Qwen2.5-7B: 23.70 / 47.03 / 35.365, GPT-4o: Average Score 15.63, mT5-base: 27.605 average, Mistral-7B: 34.670 average, LLaMA3-8B: 35.175 average, ShieldLM-14B-Qwen: 34.585 average, and ShieldGemma-9B: 35.315 average. Relative to the vanilla Qwen2.5-7B baseline, the full system improves by +2.96 Hard, +1.32 Soft, and +2.14 Average.

The ablation sequence is cumulative and consistent with the architectural rationale:

• Base Model: 23.70 / 47.03 / 35.365
• + TR: 24.33 / 47.35 / 35.840
• + TR + SRAG: 25.30 / 47.85 / 36.575
• + TR + SRAG + MAV: 26.66 / 48.35 / 37.505

The sensitivity analysis for MAV tests thresholds

$\tau$6

With $\tau$7, the paper reports increases from 25.30 to 26.66 in Hard Score, from 47.85 to 48.35 in Soft Score, and from 36.575 to 37.505 in Average Score, with the most notable improvement occurring at higher thresholds.

6. Nomenclature, scope, and related arXiv usages

In current arXiv usage, SRAG-MAV most directly denotes the framework for Fine-Grained Chinese Hate Speech Recognition introduced in "System Report for CCL25-Eval Task 10: SRAG-MAV for Fine-Grained Chinese Hate Speech Recognition" (Wang et al., 24 Jul 2025). The name should be distinguished from several unrelated systems that may invite confusion because they share either the MAV acronym or adjacent terminology.

One source of confusion is SCARF, the framework introduced in "A System for Comprehensive Assessment of RAG Frameworks" (Rengo et al., 10 Apr 2025). That paper explicitly states that SCARF is the framework name and that it is not explicitly called the “SRAG-MAV Framework”; the wording is described there as a naming variant or a misremembered label. SCARF is a modular, black-box evaluation framework for deployed Retrieval-Augmented Generation (RAG) systems rather than a hate-speech extraction system.

A second source of ambiguity comes from MAV robotics literature. "Global-Local MAV Detection under Challenging Conditions based on Appearance and Motion" introduces GLAD, a global-local detector for micro aerial vehicles, with components such as a global detector, a local detector, an adaptive local search region, and a detector switcher (Guo et al., 2023). Likewise, "Efficient Submap-based Autonomous MAV Exploration using Visual-Inertial SLAM Configurable for LiDARs or Depth Cameras" presents an autonomous exploration framework for MAVs based on visual-inertial SLAM, submaps, frontier computation, and next-best-view planning (Papatheodorou et al., 2024). These systems are unrelated in task, model class, and evaluation setting.

Within its own scope, SRAG-MAV is best understood as a retrieval-augmented structured extraction architecture for a narrow benchmarked task. The paper’s stated practical takeaway is that it is especially useful when high-quality external knowledge is unavailable. Its computational tradeoff is also explicit: larger voting thresholds improve results but increase cost. A plausible implication is that the framework is most attractive when exact structured output quality is more critical than single-pass inference latency.