LionGuard 2: Singapore Localised Moderation

• LionGuard 2 is a lightweight, multilingual classifier designed for Singaporean code-mixed texts such as Singlish, focusing on local linguistic nuances.
• It combines frozen pre-trained OpenAI text embeddings with a compact multi-head ordinal classification head to efficiently moderate content.
• The system emphasizes data efficiency and rapid retraining, outperforming larger LLM-based guardrails on localized moderation benchmarks.

LionGuard 2 is a lightweight, multilingual, Singapore-localised content moderation classifier designed for code-mixed and low-resource settings, particularly everyday Singlish spanning English, Chinese, Malay, and partial Tamil. It combines frozen pre-trained OpenAI text embeddings with a compact multi-head ordinal classification head, and was proposed as an alternative to large LLM-based guardrails that require substantial training data and expensive fine-tuning or inference. The system is reported to support localisation through a Singapore-specific safety taxonomy, local slang coverage, and code-switching preservation, while operating with a 0.85M-parameter classifier occupying 3.2 MB on disk and retraining in under two minutes on CPUs (Tan et al., 21 Jul 2025).

1. Problem setting and localisation target

LionGuard 2 was developed to address what the paper characterises as localisation gaps in multilingual moderation. Many multilingual moderation systems are described as underperforming on localised, code-mixed inputs, slang, and dialectal variation, especially in Singaporean discourse where English, Chinese, Malay, and Tamil are routinely blended. The target setting is not generic multilingual moderation, but moderation under Singapore-specific linguistic and policy constraints, including Singlish, Hokkien expressions, Malay borrowings, and protected-group references defined in Singapore’s legal context (Tan et al., 21 Jul 2025).

The model explicitly targets English, Chinese, Malay, and partial Tamil. Its intended scope includes local slang and acronyms, protected traits aligned to Singapore’s Constitution and tripartite guidelines, and toxicity embedded in local forums. This positioning distinguishes LionGuard 2 from systems optimized primarily for standard English or for broad multilingual benchmarks without strong locale adaptation. A plausible implication is that the work treats moderation as a domain adaptation problem in which local semantics, policy definitions, and code-switching behavior are primary determinants of downstream performance.

The paper frames the design choice around small, “data-efficient” and “lightweight” classifiers as a response to the cost structure of larger guardrails. Large LLM guardrails in the 8–12B range are described as demanding substantial training data, often exceeding one million examples, as well as costly fine-tuning and inference. LionGuard 2 instead freezes the embedding model and trains only a classifier head, thereby avoiding large decoder inference while preserving multilingual transfer (Tan et al., 21 Jul 2025).

2. Data curation, annotation, and safety taxonomy

The training corpus consists of local comments, synthetic prompts, and selected English safety datasets. The local component contains 20,333 texts extracted from Singaporean forums and subreddits. Synthetic data adds 2,098 chatbot-style prompts generated by gpt-4o-mini through a transformation prompt using self-reflection and chain-of-thought to preserve safety attributes from local comments. Public English data that was retained includes WildGuardTrain with 2,558 prompts and responses, Reddit Suicide Detection with 924 posts, and PH_titles with 294 adult-site titles. The final training set contains 26,207 texts in total, while the test material includes 6,249 raw comments and 7,058 synthetic comments, using a time-series split (Tan et al., 21 Jul 2025).

The paper reports several filtering and composition decisions as critical. The final training mixture maintains approximately 87% safe and 13% unsafe examples through controlled down-sampling. An 85:15 ratio of local-to-English data is reported to maximise binary F1 across benchmarks. Deduplication is performed using k-nearest neighbours over text-embedding-3-large vectors, with removal of pairs above the 95th percentile cosine similarity. The paper also reports that several public English datasets were dropped because ablations showed reductions of up to 30% in binary F1, including Aegis 2.0, Aya Red-teaming, and HateXplain variants filtered for Singapore-relevant targets (Tan et al., 21 Jul 2025).

The label taxonomy has six categories: Hateful, Insults, Sexual, Physical Violence, Self-Harm, and Misconduct. Four of these are ordinal. Hateful is divided into Level 1 (Discriminatory) and Level 2 (Hate Speech); Sexual into Level 1 (Not for minors) and Level 2 (Not for all ages); Self-Harm into Level 1 (Ideation) and Level 2 (Action/Suicide); and Misconduct into Level 1 (Not socially acceptable) and Level 2 (Illegal). Insults and Violence are single-level categories. The taxonomy is explicitly ordered, so that a Level 2 violation implies a Level 1 breach within the same category (Tan et al., 21 Jul 2025).

Annotation is semi-supervised. A panel of six human annotators and six candidate LLMs was used, and an Alt-Test methodology selected Gemini 2.0 Flash, o3-mini-low, and Claude 3.5 Haiku as the three LLMs best aligned with human judgments. These models then labeled all examples using a Singapore-specific taxonomy prompt. The paper does not report inter-annotator agreement metrics such as Cohen’s $\kappa$; instead, alignment is assessed via Alt-Test statistics. It also reports that machine translation of local comments into Malay or Tamil using gpt-4o-mini degraded downstream F1, so no machine-translated Chinese, Malay, or Tamil data was used; non-English supervision came only through natural code-mixing (Tan et al., 21 Jul 2025).

3. Architecture and ordinal formulation

LionGuard 2 uses pre-trained multilingual embeddings as a frozen front end and a compact multi-head ordinal classifier as the trainable component. The embedding model selected was OpenAI’s text-embedding-3-large, chosen after comparison against six multilingual encoders under an identical classifier-head setup. According to the paper, text-embedding-3-large achieved the highest binary F1 on hold-outs, by as much as approximately 20% over the next-best model. The paper does not specify embedding dimensionality, tokenization, pooling, or long-text segmentation strategy; the embeddings are used as returned by the API (Tan et al., 21 Jul 2025).

The classifier architecture comprises one binary head for safe versus unsafe prediction and six category heads. A compact shared MLP with two shared dense layers and dropout 0.2 feeds all heads. Hidden sizes are not specified. The full classifier head contains 0.85M parameters and occupies 3.2 MB on disk (Tan et al., 21 Jul 2025).

For ordinal categories, the model predicts cumulative threshold probabilities. Let $c$ index a category and let $y_c \in \{0,1,2\}$ denote the gold level, where 0 means absence, 1 means Level 1, and 2 means Level 2. The head outputs

$(p_1, p_2) = \sigma(\mathrm{Dense2}(h)),$

with

$p_1 = P(y_c > 0), \qquad p_2 = P(y_c > 1),$

subject to

$0 \le p_2 \le p_1 \le 1.$

At test time, reconstruction with threshold $\tau$ is

$\hat{y}_c = 0 \ \text{if}\ p_1 < \tau,$

$$\hat{y}_c = 1 \ \text{if}\ p_1 \ge \tau \ \text{and}\ p_2 < \tau,$$

$\hat{y}_c = 2 \ \text{if}\ p_2 \ge \tau.$

The loss for an ordinal head is binary cross-entropy over the two cumulative thresholds:

$c$0

The binary head uses standard BCE against the safe/unsafe label. The paper states the monotonicity condition $c$1 but does not specify how that constraint is enforced in implementation (Tan et al., 21 Jul 2025).

The work also reports architectural baselines involving LoRA-tuned LlamaGuard-3-8B and a fine-tuned snowflake-arctic-embed-l-v2.0. On a Singlish subset of roughly 31k sentences, fine-tuning these larger models did not yield better test performance; LionGuard 2 is reported to match LlamaGuard-3-8B and Arctic-Embed-2.0 while being substantially cheaper to retrain (Tan et al., 21 Jul 2025).

4. Training configuration and data efficiency

Training uses Adam with learning rate $c$2, batch size 64, and 10 epochs with early stopping of patience 3. Dropout is 0.2 in the shared layers. The classifier is trained on CPUs only, whereas large-model experiments were run on AWS g4dn.xlarge with a T4 16GB GPU or on A100 40GB hardware. Reported retraining time is under two minutes, and the classifier size remains 3.2 MB (Tan et al., 21 Jul 2025).

The paper attributes performance primarily to data quality and locality rather than to classifier complexity. High-quality local data and code-mixed Singlish samples are described as decisive. Semi-supervised labeling with the three selected LLM annotators is used to emulate human supervision at scale, and an active filtering pipeline implements a five-stage funnel with consensus filtering, category rebalancing, safe down-sampling, and near-duplicate removal. The rejection of machine-translated Malay and Tamil data is presented as a deliberate data-quality decision because translation-based augmentation degraded performance and could distort toxicity or context (Tan et al., 21 Jul 2025).

Ablations quantify the contribution of data locality. With identical architecture, local-only training yields 80.3% average F1, English-only training yields 45.3%, and combined local plus English training yields 81.5%. The paper therefore identifies localised data as the dominant factor. It also identifies embedding choice as decisive: text-embedding-3-large produces the best F1, even though the appendix notes that raw cross-lingual cosine alignment may be weaker than for some alternatives. This suggests that embedding suitability for moderation is better measured by task-level transfer than by intrinsic similarity metrics alone (Tan et al., 21 Jul 2025).

5. Evaluation across localised and general benchmarks

LionGuard 2 is evaluated on 17 benchmarks: one internal test set and 16 public benchmarks. The localised public suite contains RabakBench, SGHateCheck, and SGToxicGuard, each evaluated in Singlish (SS), Chinese (ZH), Malay (MS), and Tamil (TA). The general English suite contains BeaverTails, SORRY-Bench, OpenAI Moderation evaluation, and SimpleSafetyTests. The primary metric is binary F1 at threshold 0.5. For LionGuard 2, this score is taken from the dedicated binary head; for baselines, an example is treated as unsafe if any harm category exceeds 0.5. Calibration metrics such as ECE or Brier score are not reported (Tan et al., 21 Jul 2025).

The central empirical pattern is that LionGuard 2 leads on most Singapore-localised benchmarks in Singlish, Malay, and Chinese, while Tamil remains weaker.

Benchmark	LionGuard 2 results	Comparative note
RabakBench	SS 88.1, MS 87.8, ZH 78.4, TA 66.6	Highest on SS, MS, ZH; below top Tamil baseline
SGHateCheck	SS 98.8, MS 92.1, ZH 97.4, TA 64.5	Highest on SS, MS, ZH; Tamil moderate
SGToxicGuard	SS 99.7, MS 98.2, ZH 99.2, TA 71.5	Leads on SS, MS, ZH; Tamil comparable to GCP
BeaverTails	73.7	Below AWS Bedrock 76.4; above LG3-8B 68.2 and LG4-12B 67.0
SORRY-Bench	73.7	Above LG3-8B 62.5 and LG4-12B 58.7
OpenAI Moderation eval	70.5	Below LG3-8B 82.2, LG4-12B 77.5, OpenAI Mod 77.1, GCP 74.8
SimpleSafetyTests	100.0	Above LG4-12B 98.0, LG3-8B 87.6, AWS 84.4, OpenAI Mod 81.0

On RabakBench, the best cited baselines include GCP Model Armor at 73.4 for Tamil, OpenAI Moderation at 69.7 for Malay, Azure at 66.0 for Singlish, and LlamaGuard 4 12B at 65.2 for Chinese; LionGuard 2 exceeds these baselines in SS, MS, and ZH but not in TA. On SGHateCheck, it surpasses OpenAI Moderation in SS, GCP in ZH, and remains behind Azure in Tamil. On SGToxicGuard, it leads in SS, MS, and ZH, while its Tamil result of 71.5 is close to GCP’s 71.9 (Tan et al., 21 Jul 2025).

The general English picture is mixed rather than uniformly dominant. LionGuard 2 outperforms LlamaGuard 3 and 4 on BeaverTails and SORRY-Bench, but underperforms stronger baselines on OpenAI Moderation evaluation. On SimpleSafetyTests, where the benchmark is unsafe-only and F1 is approximately recall, the system achieves 100.0. This suggests that the model’s strengths are most pronounced in the localised moderation regime for which it was designed, rather than on every English benchmark equally (Tan et al., 21 Jul 2025).

The paper also reports robustness to orthographic noise. On the RabakBench Singlish noisy variant involving random casing, punctuation, and misspellings, LionGuard 2 drops from 87.1 to 85.6, a reduction of 1.5%. OpenAI Moderation drops from 64.0 to 52.2, whereas LionGuard 1 changes from 58.4 to 64.2. The small degradation for LionGuard 2 suggests tolerance to noisy user-generated text, though the paper does not isolate which architectural or data choices contribute most strongly to that robustness (Tan et al., 21 Jul 2025).

Native-speaker red-teaming adds an additional perspective. Across 391 hand-crafted multilingual cases, LionGuard 2 obtains F1 85.0 in Chinese versus Azure’s 82.6, and 81.4 in Malay versus Azure’s 73.1. In Tamil it reaches 41.1, placing it in the middle of the pack, and its overall weighted F1 is 72.7 compared with Azure’s 74.2. Detailed per-category F1 values, reported in the appendix for datasets such as RabakBench, BeaverTails, and SimpleSafetyTests, lie roughly in the 30–70% range for fine-grained labels, indicating that category attribution remains materially harder than binary unsafe detection (Tan et al., 21 Jul 2025).

6. Deployment, operational characteristics, and release

LionGuard 2 is deployed as a safety module on the Singapore Government’s AI Guardian platform, specifically within Sentinel Guardrails and GovText Guardrails. It supports filtering of both user prompts and model responses. The deployment mode uses CPU-only inference, with end-to-end throughput of approximately 300 tokens per second. The embedding step accounts for most latency at roughly 250 tokens per second, while the classifier itself runs at approximately $c$3 tokens per second. The paper states that batching and caching embeddings can significantly increase throughput (Tan et al., 21 Jul 2025).

The system’s hardware footprint is minimal for the classifier component: 3.2 MB with frozen embeddings supplied through an external API. The paper characterises this as suitable for synchronous pipelines and horizontally scalable when batching is available. Monitoring is provided through internal API access and systems for tracking usage and abuse, and the authors recommend human-in-the-loop review for high-stakes deployments. Specific escalation workflows and cost estimates are not provided (Tan et al., 21 Jul 2025).

The reported inference procedure is straightforward. Text is first embedded with text-embedding-3-large. The embedding is passed to the classifier, which returns a binary safe/unsafe probability and per-category ordinal probabilities $c$4 and $c$5. In evaluation, the binary decision uses threshold 0.5, and ordinal reconstruction also uses 0.5. The paper notes that operational thresholds may be raised or lowered depending on risk tolerance, with a conservative blocking rule based on the maximum over categories or a nuanced policy based on the binary head, while noting minor binary/category misalignment rates (Tan et al., 21 Jul 2025).

The model weights are released at https://huggingface.co/govtech/lionguard-2, and a synthetic training-data subset is released at https://huggingface.co/datasets/govtech/lionguard-2-synthetic-instruct. Full training data is not released because of privacy and copyright constraints. Reproducibility is partial: the training configuration is specified, but seeds, exact environment, and licenses are not reported (Tan et al., 21 Jul 2025).

7. Limitations, fairness, and implications for multilingual moderation

The principal fairness and localisation issue identified in the paper is Tamil underperformance. Tamil support is explicitly described as partial, with weaker results attributed to limited high-quality data and weaker embedding support. Translation-based augmentation worsened Tamil performance substantially: for RabakBench TA, the baseline LionGuard 2 score is 66.5, but a variant using SS+MS+TA translated data falls to 23.1, and a TA-only translation variant reaches 21.2. The authors identify future directions as sourcing better Tamil data, exploring tokenisation methods, and possibly Tamil-centric encoders (Tan et al., 21 Jul 2025).

The system also exhibits a mismatch between the binary head and category heads. Averaged over 43,075 samples across five benchmarks, the paper reports 4.19% over-prediction, where the binary head flags unsafe while all categories remain below threshold, and 0.70% under-prediction. The paper characterises this mismatch as conservative because harmful text is rarely allowed through, and suggests future work on joint calibration or explicit constraints to reduce inconsistencies. Since calibration metrics are not reported, the practical operating characteristics of these probabilities beyond fixed-threshold classification remain under-specified (Tan et al., 21 Jul 2025).

A further limitation is dependence on a closed-source embedding model, text-embedding-3-large. Future embedding changes may require retraining and renewed benchmarking, and the paper notes that open-source alternatives may be preferable for strict reproducibility. Several implementation details remain unspecified, including embedding dimensionality, tokenization and pooling strategy, monotonicity enforcement in ordinal heads, hidden-layer widths, long-text handling, and random seeds (Tan et al., 21 Jul 2025).

More broadly, the work argues that localised moderation benefits more from authentic local data, taxonomy design, and careful semi-supervised annotation than from scaling classifier size alone. The paper’s summary claim is that a small embedding-based, ordinal multi-head classifier trained on high-quality local data can outperform or match much larger commercial and open-source guardrails across 17 benchmarks, especially for Singapore-localised Singlish, Chinese, and Malay. A plausible implication is that multilingual safety systems may need to be treated as locale-specific measurement and data-engineering problems rather than as purely model-scaling problems (Tan et al., 21 Jul 2025).