TeSent: Telugu Sentiment Benchmark

• TeSent is a benchmark dataset for sentence-level sentiment classification in Telugu, offering three sentiment classes with explainable and fairness-aware evaluations.
• It provides human-annotated rationales and an associated fairness corpus (TeEEC) to audit demographic bias in diverse social media and news domains.
• The resource supports multi-domain NLP tasks and demonstrates that rationale-supervised training can enhance both model accuracy and interpretability.

TeSent is a benchmark dataset for sentence-level 3-class sentiment classification in Telugu that was designed to couple predictive modeling with explainability and fairness evaluation. Introduced in “TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu,” it provides sentiment labels, human-annotated rationales, and an associated fairness corpus called TeEEC for auditing demographic bias in Telugu sentiment and emotion tasks (Kumar et al., 2 Aug 2025). The benchmark was motivated by the underrepresentation of Telugu in NLP despite a reported speaker base of about 96 million, and by the absence of a Telugu resource that jointly supports large-scale sentiment modeling, rationale-based explanation analysis, and systematic fairness evaluation.

1. Scope, task definition, and motivation

TeSent is built for sentence-level sentiment classification with three labels: Positive, Negative, and Neutral (Kumar et al., 2 Aug 2025). The paper positions it as the first comprehensive Telugu sentiment benchmark that jointly includes gold sentiment labels, human rationales, and a fairness benchmark through TeEEC. Its target is therefore broader than conventional classification accuracy: it is intended to support research on whether Telugu sentiment models are accurate, whether their explanations align with human reasoning, and whether their predictions exhibit demographic bias.

The motivating gap is threefold. First, prior Telugu sentiment resources are described as limited by small scale, weak or insufficient annotation quality, unclear protocol descriptions, limited domain diversity, and in some cases translation-based data creation rather than naturally occurring Telugu text. Second, Telugu benchmarks had not been designed to evaluate plausibility of explanations against human evidence spans. Third, fairness auditing resources for Telugu sentiment models were largely absent. TeSent addresses all three by pairing naturally collected Telugu data with multi-annotator labels, rationale annotations, and a separate fairness evaluation suite.

The corpus is explicitly multi-domain. The collection process covers text from YouTube comments, Telugu blog comments, news websites, and Facebook and Instagram comments. The paper describes the benchmark as spanning approximately 20 domains, while the appendix lists a broader topical inventory including Religion, Education, Entertainment, Sports, Technology, Government, Health, Politics, Social Media, News, Environment, Economics, and others (Kumar et al., 2 Aug 2025).

2. Corpus construction and annotation workflow

The raw corpus initially contained 154,959 sentences. Preprocessing first removed non-Telugu and code-mixed content using Unicode for Telugu, discarding sentences containing non-Telugu fonts while explicitly retaining punctuation and emojis at that stage. This reduced the corpus to roughly 120,000 sentences. Removing emoji-only and punctuation-only items yielded roughly 90,000. A first deduplication pass used Jaccard similarity between sentence bigrams with a 75% threshold, producing 86,519 unique sentences. A second semantic deduplication pass used mBERT embeddings and pairwise cosine similarity with a 90% threshold, reducing the dataset to 26,150 sentences (Kumar et al., 2 Aug 2025).

The collection pipeline was source-specific. For YouTube, the authors used YouTube API library v3, retrieved up to 50 top videos per search phrase, and filtered comments with Langdetect at a 90% confidence threshold. Blog data were collected with a custom BeautifulSoup v4 script starting from the telugublogworld aggregator. News headlines were scraped from Eenadu, Andhra Jyothi, Way2News (Telugu), and Sakshi. Facebook and Instagram content was collected with Instant Data Scraper using generic hashtags such as #telugumemes and #teluguvideos. In the appendix, YouTube comments account for 51.2% of the data; exact percentages for the other sources are not stated in the body text (Kumar et al., 2 Aug 2025).

An anonymization stage replaced personally identifiable information with placeholders including <NUMBER>, <EMAIL>, <LINK>, <PHONE>, <PERCENT>, <DATE>, <TIME>, and <MONEY>. Then 650 sentences were randomly selected for internal testing and annotator recruitment, leaving a working set of 25,500 sentences.

Annotation was performed on a custom in-house platform with secure login, one-sentence-at-a-time display, primary sentiment selection, optional secondary sentiment selection, rationale highlighting by double-clicking, and a “Mark as Bad Sentences” option for grammatically incorrect, incomplete, off-topic, or uninterpretable inputs. Annotations and rationales were stored in MongoDB. The workload was organized into 17 pools, each with 10 batches, each batch containing 150 sentences. Annotator engagement features included a progress bar, a gamified leaderboard, and periodic manual reminders (Kumar et al., 2 Aug 2025).

Recruitment proceeded through academic mailing lists, social media, and internal university networks at SRM University AP. Participants had to be native Telugu speakers proficient in reading and understanding Telugu. Out of 112 responses, 95 annotators were selected, 74 completed at least one batch, and 21 defaulted. Compensation was ₹100 per completed batch of 150 sentences, and the process was overseen by 6 Telugu-speaking authors (Kumar et al., 2 Aug 2025).

3. Final dataset, rationales, and annotation agreement

From the 25,500 annotated working sentences, 23,144 were judged valid. From the 650 internal/test sentences, 517 were valid, giving 23,661 valid annotated sentences overall. Label finalization proceeded by majority voting over primary labels, after discarding annotations where the primary and secondary labels formed a direct contradiction, denoted in the paper as <+,-> pair contradictions. In the subset with secondary labels, 61 instances were removed due to a technical glitch; after contradiction filtering and majority voting, 2,272 of the remaining examples were retained. This yielded 22,585 sentences, after which Telugu-speaking authors manually removed 80 malformed or duplicate sentences, producing the final benchmark size of 22,505 sentences (Kumar et al., 2 Aug 2025).

The final class distribution is 44.99% Neutral, 26.64% Positive, and 28.37% Negative. The dataset is therefore not perfectly balanced, with Neutral as the largest class. The annotation protocol was two-layered. Annotators always assigned a primary sentiment label and, for Positive or Negative cases, were required to select a human rationale; for Neutral, rationale selection was optional. Annotators could additionally assign an optional secondary sentiment label, but no rationale was collected for that secondary label (Kumar et al., 2 Aug 2025).

The rationale annotations are the benchmark’s key explainability substrate. They are defined as the specific words or phrases highlighted by annotators as the basis for the primary sentiment label. The paper uses them both for rationale-supervised training and as ground truth explanations for evaluating post-hoc explanation methods. At the same time, the paper does not report detailed descriptive statistics such as average rationale length, average number of rationale spans per sentence, or token coverage percentages.

Inter-rater agreement is reported with Krippendorff’s Alpha and Fleiss’ Kappa. For the full 3-class task, the paper gives

$\alpha = 0.4797, \qquad \kappa = 0.4796,$

which it describes as moderate agreement. After excluding Neutral-labeled sentences, agreement rises to

$\alpha = 0.8992,$

indicating very strong agreement for positive-versus-negative discrimination. The paper interprets the lower 3-class agreement as reflecting the ambiguity of the Neutral category, and notes that about 80% of annotators have agreement levels above 82% with the other two annotators (Kumar et al., 2 Aug 2025).

4. Modeling setup and classification performance

The paper fine-tunes five pretrained transformer models with Telugu support: mBERT (bert-base-multilingual-cased), XLM-R, MuRIL, IndicBERT, and Te-BERT. Each model is trained in two settings: without rationales (wo/r) and with rationales (w/r) (Kumar et al., 2 Aug 2025).

Rationale-supervised training adds a KL regularizer that aligns model attention with human rationales. The paper computes a rationale-derived distribution $\hat{A}$ by taking the rationale vector over tokens, taking its mean, replacing zeros with $1\mathrm{e}{-8}$, and applying softmax. The model-side attention distribution $A$ is taken from [CLS]-to-token attention in the last layer, averaged across heads. The modified objective is

$$\text{Modified Loss} = \text{CE Loss} + \lambda \cdot \text{KL}(A \parallel \hat{A}).$$

For neutral sentences with no rationale, only the cross-entropy term is used. Hyperparameters were selected by grid search over batch sizes 16, 32, 64, learning rates from $1\mathrm{e}{-5}$ to $5\mathrm{e}{-5}$, and, for rationale-based training, $\lambda$ from 0.1 to 0.7. The final configuration used batch size 64, learning rate $2\mathrm{e}{-5}$, Adam, and 4 epochs, with $\alpha = 0.8992,$0 performing best overall for rationale supervision (Kumar et al., 2 Aug 2025).

In the main results, IndicBERT wo/r attains the highest reported accuracy at 0.737, while IndicBERT w/r attains the highest reported AUROC at 0.893. Te-BERT wo/r reaches 0.736 accuracy and 0.877 AUROC; Te-BERT w/r reaches 0.734 accuracy and 0.890 AUROC. MuRIL improves from 0.710/0.708/0.870 to 0.722/0.722/0.885 in Accuracy/F1/AUROC when rationales are added. mBERT shows the largest absolute gain from rationale supervision, moving from 0.647/0.647/0.822 to 0.673/0.668/0.839. The paper’s summary claim is cautious: training with rationales may improve model accuracy, and it improves AUROC for all five models (Kumar et al., 2 Aug 2025).

5. Explainability benchmark and evaluation suite

TeSent’s explainability component evaluates six post-hoc local explainers: LIME, SHAP using default partition SHAP, Gradient, Integrated Gradients, Gradient × Input, and Integrated Gradients × Input. The experiments use the ferret-xai library with default hyperparameters (Kumar et al., 2 Aug 2025).

The benchmark distinguishes between plausibility and faithfulness. Plausibility measures alignment with human rationales, using token-level Intersection over Union $\alpha = 0.8992,$1, token-level F1 $\alpha = 0.8992,$2, and AUPRC $\alpha = 0.8992,$3. Faithfulness measures consistency with the model’s own decision process, using Comprehensiveness $\alpha = 0.8992,$4, Sufficiency $\alpha = 0.8992,$5, and correlation with Leave-One-Out scores $\alpha = 0.8992,$6. The paper names these metrics but does not provide explicit equations for them.

The clearest finding is that rationale supervision substantially improved plausibility across all considered metrics. For example, with mBERT + SHAP, plausibility improves from $\alpha = 0.8992,$7 without rationales to $\alpha = 0.8992,$8 with rationales. With mBERT + LIME, the corresponding values move from $\alpha = 0.8992,$9 to $\hat{A}$0. For IndicBERT + SHAP, plausibility changes from $\hat{A}$1 to $\hat{A}$2. The paper states that SHAP and LIME are generally the strongest explainers in terms of alignment with human rationales, while Integrated Gradients × Input is described as a robust gradient-based method (Kumar et al., 2 Aug 2025).

By contrast, the faithfulness results are reported as mixed and inconclusive. For IndicBERT + LIME, the paper gives $\hat{A}$3 without rationales and $\hat{A}$4 with rationales: one metric improves while others worsen. For Te-BERT + SHAP, $\hat{A}$5 and $\hat{A}$6 increase slightly, while $\hat{A}$7 worsens. The paper therefore concludes that rationale supervision reliably improves human alignment of explanations, but does not establish a consistent effect on faithfulness (Kumar et al., 2 Aug 2025).

6. Fairness benchmark: TeEEC

TeEEC, the Equity Evaluation Corpus in Telugu, is the fairness component associated with TeSent. It is adapted from the English EEC and is intended for sentiment- and emotion-related Telugu NLP tasks. The benchmark targets two sensitive attribute families:

$\hat{A}$8

for gender, and

$\hat{A}$9

for religion. The paper treats male as the privileged gender group and Hindu as the privileged religion group, based on demographics of native Telugu speakers (Kumar et al., 2 Aug 2025).

TeEEC is built from 11 translated templates from EEC. Seven are of the type that include emotion words, such as <Person> feels <emotion word>., and four omit emotion words, such as I saw <person> in the market. The authors select 10 commonly used emotion words from each of the themes Anger, Fear, Joy, and Sadness, and adapt the morphology of these words to Telugu syntax. The paper specifies that Templates 3, 4, and 7 use the noun form, while the remaining templates use the adjective form. For person placeholders, the benchmark uses 20 common Telugu names for each of the six gender-religion combinations and 10 noun phrases for each group; noun phrases are not used for religion comparisons. The gender bias dataset contains 39,760 sentences, divided equally across male and female groups; the exact total size of the religion dataset is not explicitly stated (Kumar et al., 2 Aug 2025).

Fairness is evaluated with counterfactual polarity comparison and Equalized Odds. The paper samples 1320 counterfactual pairs for gender and 1200 counterfactual pairs for religion, balanced across the three sentiment classes and averaged over 10 runs. The Polarity Difference Score (PDS) is defined as

$1\mathrm{e}{-8}$0

with

$1\mathrm{e}{-8}$1

A positive PDS indicates that the privileged group is assigned more positive polarity on average. Equalized Odds is assessed through absolute differences in true positive rate and false positive rate, denoted $1\mathrm{e}{-8}$2 and $1\mathrm{e}{-8}$3, with lower values indicating better fairness (Kumar et al., 2 Aug 2025).

The paper’s overall conclusion is that rationale supervision does not always improve every fairness metric, but overall it tends to reduce gender and religion bias, especially religion bias. The strongest gender example is IndicBERT, where rationale training changes $1\mathrm{e}{-8}$4 from 0.013 to 0.001, $1\mathrm{e}{-8}$5 from 0.014 to 0.001, and $1\mathrm{e}{-8}$6 from 0.007 to 0.000. For religion, mBERT on the Hindu-Muslim comparison changes from $1\mathrm{e}{-8}$7 to $1\mathrm{e}{-8}$8, $1\mathrm{e}{-8}$9 to $A$0, and $A$1 to $A$2 after rationale training. Some models show mixed or negligible changes, but the paper reports that increases in measured bias are usually small (Kumar et al., 2 Aug 2025).

7. Position in the literature and name disambiguation

Within the literature represented here, TeSent refers specifically to the Telugu sentiment benchmark described above (Kumar et al., 2 Aug 2025). It should be distinguished from several similarly named but substantively different works. “Entropy Sentinel: Continuous LLM Accuracy Monitoring from Decoding Entropy Traces in STEM” studies slice-level accuracy estimation for LLMs from decoding entropy profiles and notes that “TeSent” is a plausible shorthand for Entropy Sentinel, but that work concerns monitoring under domain shift rather than Telugu sentiment classification (Buffa et al., 13 Jan 2026). “TransSent: Towards Generation of Structured Sentences with Discourse Marker” is a discourse-aware generation model based on head-discourse, relation, and tail-discourse structure, not a sentiment benchmark (Wu et al., 2019). “TNCSE: Tensor’s Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings” concerns unsupervised sentence embedding geometry and norm-aware contrastive objectives rather than explainable or fairness-aware sentiment classification (Zong et al., 17 Mar 2025).

In that landscape, TeSent occupies a specific niche: it is a benchmarking infrastructure for Telugu NLP that combines supervised sentiment labels, rationale annotations for explainability analysis, and a fairness corpus for demographic auditing. A plausible implication is that its main contribution is not a new classifier architecture but a reproducible experimental substrate for studying how rationale supervision changes classification quality, explanation plausibility, and bias behavior in a low-resource language setting (Kumar et al., 2 Aug 2025).