Duluth at SemEval-2026 Task 6: DeBERTa with LLM-Augmented Data for Unmasking Political Question Evasions

Abstract: This paper presents the Duluth approach to SemEval-2026 Task 6 on CLARITY: Unmasking Political Question Evasions. We address Task 1 (clarity-level classification) and Task 2 (evasion-level classification), both of which involve classifying question--answer pairs from U.S.\ presidential interviews using a two-level taxonomy of response clarity. Our system is based on DeBERTa-V3-base, extended with focal loss, layer-wise learning rate decay, and boolean discourse features. To address class imbalance in the training data, we augment minority classes using synthetic examples generated by Gemini 3 and Claude Sonnet 4.5. Our best configuration achieved a Macro F1 of 0.76 on the Task 1 evaluation set, placing 8th out of 40 teams. The top-ranked system (TeleAI) achieved 0.89, while the mean score across participants was 0.70. Error analysis reveals that the dominant source of misclassification is confusion between Ambivalent and Clear Reply responses, a pattern that mirrors disagreements among human annotators. Our findings demonstrate that LLM-based data augmentation can meaningfully improve minority-class recall on nuanced political discourse tasks.

• The paper demonstrates that synthetic minority-class augmentation and focal loss significantly improve the detection of political evasions in presidential interviews.
• The system leverages DeBERTa-V3-base enhanced by LLM-generated data and discourse feature integration to mitigate class imbalance and subtle linguistic cues.
• The approach outperforms baselines and highlights the need for hedging-aware models to better handle ambiguous response boundaries.

DeBERTa with LLM-Augmented Data for Political Evasion Detection at SemEval-2026

Problem Definition and Task Framing

The paper addresses SemEval-2026 Task 6 (CLARITY), focused on the automated classification of political evasive discourse. Specifically, given question–answer (QA) pairs from U.S. presidential interviews, Task 1 aims to sort responses into three clarity classes: (1) Clear Reply, (2) Ambivalent, and (3) Clear Non-Reply. Task 2 targets a nine-way classification over specific evasion types, nested within this taxonomy. Both tasks present challenges typical of political dialogue modeling: subtle linguistic cues, weak or conflicting signals in the input, and pronounced class imbalance toward ambiguous responses.

System Overview and Methodological Contributions

The Duluth system is built on DeBERTa-V3-base, integrating several modifications critical for the target task:

• Synthetic Minority-Class Augmentation: To mitigate severe class imbalance (e.g., only 10% Clear Non-Reply labels), the authors generate additional training samples for minority classes using context-aware generation from Gemini 3 and EDA-inspired paraphrase via Claude Sonnet 4.5.
• Focal Loss Objective: Instead of standard cross-entropy, focal loss re-weights updates to emphasize hard and underrepresented examples, thereby boosting minority-class recall.
• Layerwise Learning Rate Decay (LLRD): Fine-tuning occurs with progressively smaller learning rates at lower transformer layers, preserving general pretrained knowledge while adapting higher layers for domain specificity.
• Discourse Feature Integration: Two Boolean features (e.g., affirmative_questions, multiple_questions) from the raw data are incorporated by concatenation before final classification.
• Hyperparameter Considerations: Gradient accumulation and a cosine annealing schedule with warmup are used in tandem with early stopping, providing regularization and efficient training.

These architectural choices reflect a union of LLM-augmented data preparation with careful neural adaptation, optimized for the nuances of political QA analysis.

Experimental Results

Data Augmentation Efficacy

LLM-generated data yields robust improvements over both classical and standard transformer-based baselines, with the Gemini-augmented DeBERTa-V3-base attaining a Macro F1 of 0.76 on Task 1 (evaluation phase), compared to 0.69 without augmentation. Paraphrase augmentation alone (Claude Sonnet 4.5) achieves an intermediate F1 of 0.74.

Comparison to Baselines

The best baseline (BERT-base) achieves 0.56 F1. Classical ML approaches (TF-IDF+LR, SVM, Random Forest) all remain below 0.45, while trivial majority prediction gives only 0.27 F1—showing the necessity of both strong representation learning and minority-class-focused methods for this task.

Error Analysis

Despite improved minority-class recall, the confusion matrices consistently show that the main failure mode is between Ambivalent and Clear Reply categories. This boundary is inherently fuzzy: even human annotators display substantial disagreement (inter-annotator $\kappa = 0.65$), and the dominant misclassification pattern mirrors this ambiguity. Notably, the classifier reliably distinguishes between Clear Reply and Clear Non-Reply, suggesting the modeling and augmentation strategies most strongly benefit explicitly evasive cases.

Error breakdown on the test set reveals 68% of errors are confusion between Ambivalent and Clear Reply. These result from cases where responses include both concrete statements and hedging, or where terse direct answers are mistaken for ambiguity. The system's decision boundary corresponds closely to human interpretive uncertainty, limiting further gains in the absence of improved discourse modeling or annotation protocols.

Subtask 2 Results

For nine-way evasion classification, the system’s Macro F1 drops from 0.45 (development) to 0.28 (evaluation), attributed to distribution shift and extreme class imbalance. This sharp reduction demonstrates the limits of augmentation and focal loss in more fine-grained, under-resourced multi-class settings.

Theoretical and Practical Implications

This work shows that targeted LLM-based data generation, combined with loss function re-weighting and domain-specific feature integration, is effective for complex discourse classification under severe class skew. However, improvements are mostly realized for explicit non-replies; the persistent confusion between Ambivalent and Clear Reply categories reflects theoretical limits imposed by both label noise and the properties of political language itself.

Practical applications include improved automated analysis of politician interviews, media monitoring, and support for political science studies on evasive language. However, the blurring of direct/ambiguous boundaries suggests that fully automating nuanced discourse understanding will likely require multi-task learning or additional sources of external world knowledge.

Future Prospects

The authors highlight three promising research directions:

1. Multi-Task Learning: Jointly predicting clarity and fine-grained evasion labels could leverage the hierarchical nature of the task and exploit complementary supervision.
2. Hedging-Aware Modeling: Explicitly modeling lexical, pragmatic, or structural markers of hedging could circumscribe the Ambivalent/Clear Reply confusion region.
3. Robust Augmentation and Validation: For fine-grained tasks, more aggressive augmentation and validation protocols targeting minority classes are needed to address distribution shift.

Broader theoretical advances may require rethinking label taxonomies or integrating multi-speaker or interactional context to resolve ambiguity inherent in political dialogue.

Conclusion

The Duluth system demonstrates that enriching transformer-based classifiers with LLM-generated minority-label data, focal loss, and discourse-aware features leads to state-of-the-art results on political evasion classification, particularly for explicit non-replies. The technique is robust within practical resource constraints, although further improvement in fine-grained evasion detection is limited by annotation ambiguity and class imbalance. Advancing the field will require architectural innovations, richer modeling of hedging, and principled handling of inter-annotator disagreement as a core aspect of political discourse classification.