DEBERTA-S2M: Enhanced DeBERTa Model

• DEBERTA-S2M is an enhanced DeBERTa-based language model that integrates a Single-turn to Multi-turn (S2M) data augmentation pipeline for improved conversational QA.
• It leverages architectural innovations like Squeeze-and-Excitation blocks and sentiment augmentation to boost cyberbullying detection accuracy.
• Empirical studies demonstrate that DEBERTA-S2M achieves state-of-the-art results on both conversational QA tasks and cyberbullying classification benchmarks.

DEBERTA-S2M is a designation for an enhanced DeBERTa-based LLM architecture integrated with specialized modifications for improved conversational question answering (CQA) and cyberbullying detection. The term refers to distinct but related innovations: (1) the application of DeBERTa within the Single-turn to Multi-turn (S2M) conversational QA data augmentation pipeline (Li et al., 2023), and (2) the architectural modifications for synergistic deep-feature fusion in cyberbullying classification, including Squeeze-and-Excitation blocks and sentiment augmentation (Kumar, 19 Jun 2025). Both advances demonstrate substantial empirical gains over prior state-of-the-art approaches in their target domains.

1. Foundation: DeBERTa Architecture and the S2M Paradigm

The backbone of DEBERTA-S2M is the DeBERTa model ("Decoding-enhanced BERT with Disentangled Attention") (He et al., 2020), which achieves superior performance through disentangled attention—encoding token content and relative position separately. Attention scores are computed via multiple cross-terms: $$A_{i,j} = h_i h_j^T + h_i p_{j|i}^T + p_{i|j} h_j^T + p_{i|j} p_{j|i}^T$$ where $h_i$ and $h_j$ are content vectors, and $p_{i|j}, p_{j|i}$ are position vectors.

"DEBERTA-S2M" as an Editor's term (when anchored by (Li et al., 2023)) signifies the deployment of DeBERTa models fine-tuned or pre-trained on multi-turn synthetic conversational QA corpora, constructed by converting single-turn datasets via the S2M pipeline. This combination yielded top-ranking QuAC leaderboard performance and improved multi-turn CQA modeling.

Table: DeBERTa Core Innovations

Component	Contribution
Disentangled Attention	Flexible modeling of content/position in context
Enhanced Mask Decoder	Absolute position embeddings for MLM decoding
Virtual Adversarial Training (SiFT)	Scale-invariant fine-tuning robustness

2. S2M Data Transformation and Augmentation Pipeline

The S2M framework, as introduced in (Li et al., 2023), is a three-stage pipeline enabling the transformation of standalone single-turn QA datasets into multi-turn conversational resources suitable for CQA:

1. QA Pair Generator: Uses self-training models (e.g., RGX) to produce and curate diverse candidate QA pairs from each document, filtering out redundancy via union search and credit scoring.
2. QA Pair Reassembler and Knowledge Graph Construction: Constructs a passage-level knowledge graph using OpenIE triple extraction and customized triple join algorithms, then aligns QA pairs to graph nodes to sequence candidate pairs into coherent multi-turn dialogues.
3. Question Rewriter: Trains a seq2seq model (using the R-CANARD reverse rewriting dataset) to recast standalone questions into conversational, history-dependent follow-up forms.

This methodology ensures augmented datasets maintain dialogic coherence, topical flow, and linguistic diversity, closing the distributional gap between single-turn and multi-turn QA.

3. Enhanced DeBERTa-Based Architectures for Classification (Cyberbullying Detection)

In (Kumar, 19 Jun 2025), DEBERTA-S2M refers to a specific hybrid model for cyberbullying detection, incorporating the following enhancements over standard DeBERTa:

• Squeeze-and-Excitation (SE) Block: Global pooling and excitation recalibrate feature/channel importance after contextual encoding.
• Dimensional Reduction and Batch Normalization: Two-layer projection (768→384→192) retains salient features and reduces model complexity.
• Sentiment Integration: External sentiment analysis (VADER) generates feature vectors appended to DeBERTa outputs for richer affective modeling.
• Feature Selection: Employs Mutual Information or L1 regularization to retain top-K discriminative features.
• Gated Broad Learning System (GBLS) Classifier: Multi-head attention, adaptive gating (inspired by LSTM/GRU), shortcut connections, and normalization drive robust, adaptive classification.

Key formula for SE block recalibration: $$\mathbf{s} = \sigma( \mathbf{W}_2 \cdot \delta(\mathbf{W}_1\mathbf{z}))$$ where $\mathbf{z}$ is the pooled feature vector, $\delta$ is ReLU, and $\sigma$ is sigmoid activation.

4. Empirical Performance and Evaluation

Conversational QA (QuAC leaderboard, S2M):

• DeBERTa+S2M: F1 = 76.3, HEQ-Q = 73.6, HEQ-D = 17.9 (No. 1 at submission).
• S2M surpasses SIMSEEK and RGX (synthetic) data variants, despite smaller training corpus size.

Cyberbullying Detection (Kumar, 19 Jun 2025):

• ModifiedDeBERTa+GBLS achieves:
  • HateXplain: 79.3% accuracy, F1 = 0.781, ROC-AUC = 0.863
  • SOSNet: 95.41% accuracy, F1 = 0.9526
  • Mendeley-I: 91.37% accuracy, F1 = 0.9138
  • Mendeley-II: 94.67% accuracy, F1 = 0.9473, ROC-AUC = 0.9823
• Consistently outperforms deep LSTM/CNN, transformer, and compact hybrid baselines.
• Ablation studies confirm the incremental value of SE, sentiment features, and feature selection.

Table: Key DEBERTA-S2M Results

Task/Dataset	Model Variant	Accuracy / F1
QuAC (CQA)	DeBERTa+S2M	F1 76.3 / HEQ-Q 73.6
HateXplain	ModifiedDeBERTa+GBLS	Acc. 79.3, F1 0.781
SOSNet	ModifiedDeBERTa+GBLS	Acc. 95.41, F1 0.9526
Mendeley-I	ModifiedDeBERTa+GBLS	Acc. 91.37, F1 0.9138
Mendeley-II	ModifiedDeBERTa+GBLS	Acc. 94.67, F1 0.9473

5. Explainability, Transparency, and Robustness

DEBERTA-S2M models feature comprehensive interpretability mechanisms:

• Token-Level Attribution (Integrated Gradients): Identifies which tokens contribute most to toxicity flags or QA relevance.
• LIME-based Local Explanations: Surrogate models provide per-instance rationales for predictions.
• Confidence Calibration: Systematically aligns prediction confidence with true success rates, supporting human-in-the-loop moderation.
• Error Analysis: Illuminates failure modes, especially in implicit bias, sarcasm/irony, and nuanced criticism—informing future improvements.

6. Practical Implications and Future Directions

DEBERTA-S2M, as instantiated in both S2M-augmented conversational QA and hybrid cyberbullying detection pipelines, demonstrates the efficacy of integrating transformer-based contextual encoders with advanced data augmentation and post-encoding feature engineering. The approach is scalable, empirically validated, and robust to distributional shifts, making it suitable for large-scale deployment in moderation, QA, and dialog-centric NLP systems.

This suggests that further research into joint training objectives, deeper feature fusion, and task-specific augmentation—particularly with integrated explainability—has strong prospects for advancing both conversational modeling and content moderation systems.