Code-Mixed Singlish Safety Corpus
- Code-Mixed Singlish Safety Corpus is a collection of localized safety datasets that capture Singapore-specific code-mixing, informal spellings, and culturally nuanced unsafe content.
- The datasets are derived from large-scale online discourse and refined through human curation across modalities such as moderation, translation, alignment, and red-teaming.
- The corpus lineage spans resources like LionGuard, RabakBench, and SGToxicGuard, providing quantitative benchmarks and qualitative insights to enhance safety in multilingual contexts.
Code-Mixed Singlish Safety Corpus denotes a line of Singapore-contextualized safety data resources centered on Singlish, an English-based creole or vernacular characterized by heavy code-mixing, colloquial spellings, slang, expressive tone, and culturally embedded harmful speech. In the recent literature, the term does not denote a single canonical public benchmark. Instead, it refers to several closely related artifacts with different scopes: the large-scale Singlish moderation corpus underlying LionGuard, a small human-curated safety-oriented corpus derived from LionGuard for toxicity-preserving translation, the Singlish source split of RabakBench, the proprietary SGToxicityPrompts pipeline for alignment, and the Singlish component of the multilingual red-teaming dataset SGToxicGuard (Foo et al., 2024, Ge et al., 16 Jul 2025, Chua et al., 8 Jul 2025, Lim et al., 18 Feb 2025, Hu et al., 18 Sep 2025).
1. Scope and corpus lineage
The most stable lineage begins with LionGuard, which introduced a Singapore-contextualized moderation classifier trained on a large Singlish corpus constructed from Singaporean online discourse (Foo et al., 2024). Later work reused or extended this localized safety setting in distinct ways. The toxicity-aware translation paper did not present a large standalone benchmark with a full data card; instead, it described a human-curated safety-oriented corpus derived from LionGuard and used to build few-shot translation demonstrations for Singlish to Chinese, Malay, and Tamil (Ge et al., 16 Jul 2025). RabakBench then treated Singlish as the core localized source corpus for a multilingual safety benchmark, while SGToxicityPrompts and SGToxicGuard pursued safety alignment and red-teaming respectively (Chua et al., 8 Jul 2025, Lim et al., 18 Feb 2025, Hu et al., 18 Sep 2025).
| Resource | Reported scale or role | Primary use |
|---|---|---|
| LionGuard corpus | 138,000 labeled texts | Moderation training and benchmarking |
| Translation few-shot pool | 20 Singlish sentences, balanced between benign and harmful content | Toxicity-preserving translation demonstrations |
| RabakBench Singlish subset | 1,341 Singlish examples | Source language for multilingual benchmark |
| SGToxicityPrompts pipeline | 8,000 sampled texts, split equally into safe and unsafe statements | Safety alignment data construction |
| SGToxicGuard Singlish tasks | 2,314 conversation, 182 QA, 2,314 tweet composition | Multilingual red-teaming |
A common misconception is that there is already one standardized, fully specified public dataset named exactly “Code-Mixed Singlish Safety Corpus.” The record is more heterogeneous. The translation work explicitly states that its corpus is a compact, human-curated demonstration pool rather than a large standalone benchmark, while the larger reusable Singlish safety dataset in the same research lineage is the LionGuard corpus (Ge et al., 16 Jul 2025, Foo et al., 2024).
2. Linguistic and safety rationale
The corpus family is motivated by the fact that Singlish is not treated as merely standard English with slang. The papers describe it as an English-based variety or creole with lexical items from Chinese, Malay, Tamil, Hokkien, and other local varieties, alongside orthographic variation, local particles, and informal register (Foo et al., 2024, Chua et al., 8 Jul 2025). This makes safety evaluation difficult because harmfulness may be carried not only by overt profanity but also by tone, implied insult, local idioms, and culturally specific references.
The LionGuard work argues that existing moderation APIs such as OpenAI Moderation API, Perspective API, and LlamaGuard are not sufficiently adapted to Singlish because they often miss Singapore-specific slang, local dysphemisms and slurs, code-mixed expressions, and culturally specific references that are harmless in one context but unsafe in Singapore (Foo et al., 2024). The translation work sharpens the same point in the multilingual setting: ordinary translation systems often sanitize profanity or flatten the social meaning of toxic utterances, which is undesirable when the application is moderation, benchmarking, or faithful downstream analysis (Ge et al., 16 Jul 2025).
Concrete examples of localized unsafe content reported in the literature include terms such as “ceca,” “sinkie,” “amdk,” “tiong,” “bbfa,” “kkj,” “knn,” “kns,” and “ccb,” as well as sexual or vulgar code-mixed forms such as “piak,” “fap,” and “syt” (Foo et al., 2024). RabakBench’s generation prompts similarly encouraged strong Singlish tone, vulgar and localized lexical choices, and no proper grammar, with examples such as “cb,” “kimak,” “nabei,” “CECA,” “sinkie,” “humji,” and “ahtiong” (Chua et al., 8 Jul 2025). This suggests that the safety problem is simultaneously lexical, pragmatic, and sociocultural.
3. LionGuard and the first large-scale Singlish safety corpus
LionGuard provides the clearest large-scale realization of a code-mixed Singlish safety corpus. The data were collected from HardwareZone forum comments, especially the Eat-Drink-Man-Woman section, and from Singapore-related Reddit communities including r/Singapore, r/SingaporeHappenings, and r/SingaporeRaw, covering 2020 to 2023 and initially yielding about 8.9 million comments (Foo et al., 2024). Because unsafe content was rare, the authors selected threads about controversial topics in Singapore, selected threads containing offensive words, and then randomly sampled 69,000 potentially unsafe texts together with 69,000 additional texts from the remaining pool for topic and language diversity, producing a final 138,000-text Singlish dataset (Foo et al., 2024).
The annotation schema supported both binary unsafe detection and multi-label category detection. LionGuard used seven safety categories: hateful, harassment, encouraging public harm, encouraging self-harm, sexual, toxic, and violent (Foo et al., 2024). Final labels supported a binary unsafe label plus the seven category labels, and the split was 70% train, 15% validation, and 15% test with thread-level preservation (Foo et al., 2024). Positive label counts were reported as 537 hateful, 101 harassment, 147 public harm, 82 self-harm, 695 sexual, 7,295 toxic, 153 violent, and 8,375 unsafe under the binary label (Foo et al., 2024).
Labeling was performed with three safety-tuned LLMs—GPT-3.5-turbo (0613), Claude 2.0, and PaLM 2 (text-bison-002)—using context prompting, few-shot prompting, and chain-of-thought prompting in JSON format (Foo et al., 2024). The authors compared majority vote and consensus, and chose consensus for training because they prioritized label accuracy over coverage (Foo et al., 2024). Human validation involved 95 Singapore-resident workers, 11,997 unique texts, and three workers per text, with workers allowed to choose “I Don’t Know” (Foo et al., 2024). Human full agreement was reported as 52.9% for binary unsafe, 67.3% for toxic, 70.6% for hateful, 95.5% for self-harm, and 94.3% for violent, underscoring the subjectivity of localized moderation judgments (Foo et al., 2024).
LionGuard’s evaluation used PR-AUC because of severe class imbalance. On binary unsafe detection, LionGuard achieved 0.819 PR-AUC, compared with 0.675 for Moderation API, 0.588 for Perspective API, and 0.459 for LlamaGuard (Foo et al., 2024). Category scores included 0.480 for hateful, 0.413 for harassment, 0.491 for public harm, 0.507 for self-harm, 0.485 for sexual, 0.827 for toxic, and 0.514 for violent (Foo et al., 2024). The best model combination was BGE Large plus a Ridge classifier, while masked language modeling fine-tuning on a separate 500k-text sample for 30 epochs had negligible effect (Foo et al., 2024).
4. The human-curated translation corpus derived from LionGuard
The translation paper narrowed the corpus from large-scale moderation training to a compact, high-precision demonstration pool for toxicity-preserving translation (Ge et al., 16 Jul 2025). It framed the data as a Code-Mixed Singlish Safety Corpus drawn from LionGuard and used as a testbed for inclusive NLP and multicultural LLM safety, especially for translating Singlish into Chinese, Malay, and Tamil (Ge et al., 16 Jul 2025). The authors selected 20 Singlish sentences, balanced between benign and harmful content, to create a compact prompt demonstration pool (Ge et al., 16 Jul 2025).
Construction proceeded through a three-round iterative annotation process. In Round 1, for each sentence, the authors generated three zero-shot translations using GPT-4o mini, DeepSeek-R1, and Gemini 2.0 Flash; annotators then selected any number of acceptable translations or wrote a custom translation if none captured the intended meaning and tone (Ge et al., 16 Jul 2025). In Round 2, annotators reviewed the top two LLM outputs from Round 1 plus any human-provided alternatives and selected up to two preferred candidates (Ge et al., 16 Jul 2025). In Round 3, remaining candidates were ranked and annotators chose the single best translation, with the version receiving the most votes becoming part of the final few-shot pool (Ge et al., 16 Jul 2025).
The purpose of this iterative design was to reduce manual translation burden, preserve informal tone and toxicity, converge toward culturally faithful examples, and identify which model outputs were trustworthy enough to reuse as demonstrations (Ge et al., 16 Jul 2025). The appendix included Label Studio-style screenshots for the initial selection, re-evaluation of top outputs, and final choice, making the workflow explicitly human-in-the-loop rather than an automatic mining procedure (Ge et al., 16 Jul 2025).
Several reported properties indicate how much curation was necessary. Chinese and Tamil each retained 9 LLM-generated translations in the final pool, whereas Malay retained only 2 (Ge et al., 16 Jul 2025). The average number of custom human translations per sentence was 8.8 for Malay, 6.4 for Chinese, and 5.6 for Tamil (Ge et al., 16 Jul 2025). The paper linked Malay’s lower retention of LLM outputs to orthographic variability and the need for colloquial spellings that better matched Singlish tone (Ge et al., 16 Jul 2025). To assess similarity between final curated examples and original LLM candidates, the authors computed character-level substring overlap and reported a median of 0.47 and an average of 0.54, indicating moderate proximity but meaningful human revision (Ge et al., 16 Jul 2025).
This corpus was operationalized as a retrieval pool for few-shot prompting. For a source sentence, the system computed semantic similarity to each curated example and included the top- most similar examples in the prompt, with tested values and best values of 15 for Chinese, 10 for Malay, and 20 for Tamil (Ge et al., 16 Jul 2025). The prompt instructed the model not to soften the tone, to preserve impoliteness in a culturally appropriate way, and to analyze slang and emotional intensity before translating; the final prompt asked for an Explanation followed by a Translation (Ge et al., 16 Jul 2025).
5. Subsequent extensions: alignment, multilingual benchmarking, and red-teaming
A separate trajectory used localized Singlish safety data for alignment rather than moderation or translation. “Safe at the Margins” introduced and used a proprietary Singlish safety and toxicity corpus pipeline built around SGToxicityPrompts, described as curated Singlish texts from HardwareZone’s Eat-Drink-Man-Woman forum and selected Singapore-related subreddits (Lim et al., 18 Feb 2025). The authors sampled 8,000 texts, split equally into 4,000 benign and 4,000 toxic statements, then turned them into conversational prompts using 21 manually designed prompt templates (Lim et al., 18 Feb 2025). Unsafe prompts were paired with GPT-4o-generated refusals, safe prompts retained the original SEA-Lion response, and LionGuard was used to filter prompt templates; templates were dropped from the safe subset because they did not have at least 80% of safe prompts below the LionGuard high recall threshold (Lim et al., 18 Feb 2025).
RabakBench further broadened the notion of a Singlish safety corpus into a multilingual benchmark localized to Singapore’s linguistic context (Chua et al., 8 Jul 2025). Its Singlish component is the core localized source corpus, built from real Singlish web comments and LLM-based red-teaming against LionGuard and other guardrails, then transformed into evaluation-style prompts while preserving Singlish syntax, informal tone, slang, code-mixing, and culturally grounded meaning (Chua et al., 8 Jul 2025). The final dataset comprises 1,341 examples per language across four languages, for 5,364 examples overall, with the Singlish examples labeled under a six-category multi-label taxonomy: hateful, insults, sexual, physical violence, self-harm, and all other misconduct, with severity levels for all except insults and physical violence (Chua et al., 8 Jul 2025).
SGToxicGuard introduced a different benchmarking paradigm based on toxicity red-teaming in Singapore’s low-resource languages (Hu et al., 18 Sep 2025). It is multilingual rather than Singlish-only, but Singlish is central to its relevance as a code-mixed local variety. The dataset extends HateCheck and SGHateCheck into three real-world scenarios—conversation, question answering, and tweet composition—and reports 2,314 Singlish instances for conversation, 182 for QA, and 2,314 for tweet composition (Hu et al., 18 Sep 2025). Its Singlish examples include local particles such as “lah,” “lor,” and “one,” showing that code-mixed pragmatic markers are integral to the red-teaming design (Hu et al., 18 Sep 2025).
6. Evaluation paradigms, related methodologies, and limitations
The evaluation paradigms used across these corpora differ substantially. LionGuard is a moderation dataset with binary and multi-label classification under PR-AUC, F1, agreement, and precision/recall analysis (Foo et al., 2024). The translation corpus emphasizes semantic fidelity and round-trip consistency through embedding-based direct translation and back-translation similarity computed with text-embedding-3-large, followed by human ratings of meaning and tone on a 1–5 scale (Ge et al., 16 Jul 2025). RabakBench uses binary safety detection after mapping external taxonomies to its own schema, and SGToxicGuard uses hateful response rate and bias rate under conversation, QA, and tweet composition red-teaming (Chua et al., 8 Jul 2025, Hu et al., 18 Sep 2025).
The translation evaluation reported that GPT-4o mini performed best overall among Gemini 2.0 Flash, Grok 3 Beta Mini, DeepSeek-R1, and GPT-4o mini (Ge et al., 16 Jul 2025). In human evaluation of 200 GPT-4o mini translations, average ratings were 3.83 for machine and 4.07 for gold in Chinese, 4.09 and 4.08 in Malay, and 2.49 and 3.30 in Tamil (Ge et al., 16 Jul 2025). The authors interpreted Chinese and Malay outputs as close to gold references, while Tamil was substantially weaker, likely due to fewer annotators, greater structural distance, and difficulty preserving Singlish’s local slang and profanity in Tamil (Ge et al., 16 Jul 2025).
A closely related, though non-Singlish-specific, methodology comes from work on code-mixed perturbations in multilingual safety evaluation (Banerjee et al., 20 May 2025). That paper reused a culturally grounded safety dataset and transformed it into code-mixed prompts with English as the embedded language under the Matrix Language Frame model, maintaining a 60:40 ratio between matrix-language and embedded-language content (Banerjee et al., 20 May 2025). Its main finding was that code-mixed prompts substantially worsened safety alignment and increased attack success rate across ten matrix languages and twelve sociopolitical domains (Banerjee et al., 20 May 2025). Because Singlish is itself an English-based code-mixed variety with strong cultural specificity, this methodology is directly relevant as a design template, even though the paper did not introduce a Singlish corpus (Banerjee et al., 20 May 2025). A plausible implication is that future Singlish corpus design can combine the ecological validity of LionGuard-style local data with the controlled perturbation analysis used in multilingual code-mixing research.
The principal limitation across the literature is fragmentation. LionGuard is large-scale and reusable for moderation, but not designed as a multilingual translation benchmark (Foo et al., 2024). The translation corpus is precise and human-verified, but intentionally small, with only 20 demonstration sentences (Ge et al., 16 Jul 2025). SGToxicityPrompts is proprietary and lacks a public standalone release protocol (Lim et al., 18 Feb 2025). RabakBench offers public multilingual coverage and human-verified translations, but its taxonomy differs from LionGuard’s and its benchmark framing is binary safety detection after taxonomy reconciliation (Chua et al., 8 Jul 2025). SGToxicGuard is optimized for adversarial behavioral evaluation rather than moderation training (Hu et al., 18 Sep 2025). Taken together, these resources indicate that “Code-Mixed Singlish Safety Corpus” is best understood as an evolving research area: a family of localized, code-mixed safety datasets that jointly support moderation, translation, alignment, and red-teaming in Singapore’s multilingual environment.
