---
title: 'TeSent: Telugu Sentiment Benchmark'
url: https://www.emergentmind.com/topics/tesent
type: topic
---

# TeSent: Telugu Sentiment Benchmark

TeSent is a benchmark dataset for **sentence-level 3-class sentiment classification in Telugu** that was designed to couple predictive modeling with **explainability** and **fairness** evaluation. Introduced in “TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu,” it provides sentiment labels, **human-annotated rationales**, and an associated fairness corpus called **TeEEC** for auditing demographic bias in Telugu sentiment and emotion tasks [2508.01486]. The benchmark was motivated by the underrepresentation of Telugu in NLP despite a reported speaker base of about **96 million**, and by the absence of a Telugu resource that jointly supports large-scale sentiment modeling, rationale-based explanation analysis, and systematic fairness evaluation.

## 1. Scope, task definition, and motivation

TeSent is built for **sentence-level sentiment classification** with three labels: **Positive**, **Negative**, and **Neutral** [2508.01486]. The paper positions it as the **first comprehensive Telugu sentiment benchmark** that jointly includes **gold sentiment labels**, **human rationales**, and a **fairness benchmark** through **TeEEC**. Its target is therefore broader than conventional classification accuracy: it is intended to support research on whether Telugu sentiment models are accurate, whether their explanations align with human reasoning, and whether their predictions exhibit demographic bias.

The motivating gap is threefold. First, prior Telugu sentiment resources are described as limited by small scale, weak or insufficient annotation quality, unclear protocol descriptions, limited domain diversity, and in some cases translation-based data creation rather than naturally occurring Telugu text. Second, Telugu benchmarks had not been designed to evaluate **plausibility** of explanations against human evidence spans. Third, fairness auditing resources for Telugu sentiment models were largely absent. TeSent addresses all three by pairing naturally collected Telugu data with multi-annotator labels, rationale annotations, and a separate fairness evaluation suite.

The corpus is explicitly **multi-domain**. The collection process covers text from **YouTube comments**, **Telugu blog comments**, **news websites**, and **Facebook and Instagram comments**. The paper describes the benchmark as spanning approximately **20 domains**, while the appendix lists a broader topical inventory including Religion, Education, Entertainment, Sports, Technology, Government, Health, Politics, Social Media, News, Environment, Economics, and others [2508.01486].

## 2. Corpus construction and annotation workflow

The raw corpus initially contained **154,959 sentences**. Preprocessing first removed non-Telugu and code-mixed content using **Unicode for Telugu**, discarding sentences containing non-Telugu fonts while explicitly retaining punctuation and emojis at that stage. This reduced the corpus to roughly **120,000** sentences. Removing emoji-only and punctuation-only items yielded roughly **90,000**. A first deduplication pass used **Jaccard similarity between sentence bigrams** with a **75% threshold**, producing **86,519 unique sentences**. A second semantic deduplication pass used **mBERT embeddings** and pairwise **cosine similarity** with a **90% threshold**, reducing the dataset to **26,150 sentences** [2508.01486].

The collection pipeline was source-specific. For YouTube, the authors used **YouTube API library v3**, retrieved up to **50 top videos per search phrase**, and filtered comments with **Langdetect** at a **90% confidence threshold**. Blog data were collected with a custom **BeautifulSoup v4** script starting from the `telugublogworld` aggregator. News headlines were scraped from **Eenadu**, **Andhra Jyothi**, **Way2News (Telugu)**, and **Sakshi**. Facebook and Instagram content was collected with **Instant Data Scraper** using generic hashtags such as `#telugumemes` and `#teluguvideos`. In the appendix, **YouTube comments** account for **51.2%** of the data; exact percentages for the other sources are not stated in the body text [2508.01486].

An anonymization stage replaced personally identifiable information with placeholders including `<NUMBER>`, `<EMAIL>`, `<LINK>`, `<PHONE>`, `<PERCENT>`, `<DATE>`, `<TIME>`, and `<MONEY>`. Then **650 sentences** were randomly selected for internal testing and annotator recruitment, leaving a **working set of 25,500 sentences**.

Annotation was performed on a **custom in-house platform** with secure login, one-sentence-at-a-time display, primary sentiment selection, optional secondary sentiment selection, rationale highlighting by **double-clicking**, and a “**Mark as Bad Sentences**” option for grammatically incorrect, incomplete, off-topic, or uninterpretable inputs. Annotations and rationales were stored in **MongoDB**. The workload was organized into **17 pools**, each with **10 batches**, each batch containing **150 sentences**. Annotator engagement features included a progress bar, a gamified leaderboard, and periodic manual reminders [2508.01486].

Recruitment proceeded through academic mailing lists, social media, and internal university networks at SRM University AP. Participants had to be **native Telugu speakers** proficient in reading and understanding Telugu. Out of **112** responses, **95** annotators were selected, **74** completed at least one batch, and **21** defaulted. Compensation was **₹100 per completed batch of 150 sentences**, and the process was overseen by **6 Telugu-speaking authors** [2508.01486].

## 3. Final dataset, rationales, and annotation agreement

From the **25,500** annotated working sentences, **23,144** were judged valid. From the **650** internal/test sentences, **517** were valid, giving **23,661 valid annotated sentences** overall. Label finalization proceeded by majority voting over primary labels, after discarding annotations where the primary and secondary labels formed a direct contradiction, denoted in the paper as **`<+,-> pair`** contradictions. In the subset with secondary labels, **61** instances were removed due to a technical glitch; after contradiction filtering and majority voting, **2,272** of the remaining examples were retained. This yielded **22,585** sentences, after which Telugu-speaking authors manually removed **80 malformed or duplicate sentences**, producing the final benchmark size of **22,505 sentences** [2508.01486].

The final class distribution is **44.99% Neutral**, **26.64% Positive**, and **28.37% Negative**. The dataset is therefore not perfectly balanced, with Neutral as the largest class. The annotation protocol was two-layered. Annotators always assigned a **primary sentiment label** and, for **Positive** or **Negative** cases, were required to select a **human rationale**; for **Neutral**, rationale selection was optional. Annotators could additionally assign an **optional secondary sentiment label**, but no rationale was collected for that secondary label [2508.01486].

The rationale annotations are the benchmark’s key explainability substrate. They are defined as the specific words or phrases highlighted by annotators as the basis for the primary sentiment label. The paper uses them both for **rationale-supervised training** and as **ground truth explanations** for evaluating post-hoc explanation methods. At the same time, the paper does **not** report detailed descriptive statistics such as average rationale length, average number of rationale spans per sentence, or token coverage percentages.

Inter-rater agreement is reported with **Krippendorff’s Alpha** and **Fleiss’ Kappa**. For the full 3-class task, the paper gives
$$
\alpha = 0.4797, \qquad \kappa = 0.4796,
$$
which it describes as **moderate agreement**. After excluding Neutral-labeled sentences, agreement rises to
$$
\alpha = 0.8992,
$$
indicating very strong agreement for positive-versus-negative discrimination. The paper interprets the lower 3-class agreement as reflecting the ambiguity of the **Neutral** category, and notes that about **80% of annotators** have agreement levels above **82%** with the other two annotators [2508.01486].

## 4. Modeling setup and classification performance

The paper fine-tunes five pretrained transformer models with Telugu support: **mBERT** (`bert-base-multilingual-cased`), **XLM-R**, **MuRIL**, **IndicBERT**, and **Te-BERT**. Each model is trained in two settings: **without rationales** (**wo/r**) and **with rationales** (**w/r**) [2508.01486].

Rationale-supervised training adds a KL regularizer that aligns model attention with human rationales. The paper computes a rationale-derived distribution $\hat{A}$ by taking the rationale vector over tokens, taking its mean, replacing zeros with \(1\mathrm{e}{-8}\), and applying softmax. The model-side attention distribution \(A\) is taken from **[CLS]-to-token attention in the last layer**, averaged across heads. The modified objective is
$$
\text{Modified Loss} = \text{CE Loss} + \lambda \cdot \text{KL}(A \parallel \hat{A}).
$$
For neutral sentences with no rationale, only the cross-entropy term is used. Hyperparameters were selected by grid search over batch sizes **16, 32, 64**, learning rates from **\(1\mathrm{e}{-5}\)** to **\(5\mathrm{e}{-5}\)**, and, for rationale-based training, \(\lambda\) from **0.1 to 0.7**. The final configuration used **batch size 64**, **learning rate \(2\mathrm{e}{-5}\)**, **Adam**, and **4 epochs**, with **\(\lambda = 0.6\)** performing best overall for rationale supervision [2508.01486].

In the main results, **IndicBERT wo/r** attains the highest reported **accuracy** at **0.737**, while **IndicBERT w/r** attains the highest reported **AUROC** at **0.893**. **Te-BERT wo/r** reaches **0.736** accuracy and **0.877** AUROC; **Te-BERT w/r** reaches **0.734** accuracy and **0.890** AUROC. **MuRIL** improves from **0.710/0.708/0.870** to **0.722/0.722/0.885** in **Accuracy/F1/AUROC** when rationales are added. **mBERT** shows the largest absolute gain from rationale supervision, moving from **0.647/0.647/0.822** to **0.673/0.668/0.839**. The paper’s summary claim is cautious: training with rationales **may improve model accuracy**, and it improves AUROC for all five models [2508.01486].

## 5. Explainability benchmark and evaluation suite

TeSent’s explainability component evaluates six post-hoc local explainers: **LIME**, **SHAP** using default **partition SHAP**, **Gradient**, **Integrated Gradients**, **Gradient × Input**, and **Integrated Gradients × Input**. The experiments use the **ferret-xai** library with default hyperparameters [2508.01486].

The benchmark distinguishes between **plausibility** and **faithfulness**. Plausibility measures alignment with human rationales, using **token-level Intersection over Union** \(I(\uparrow)\), **token-level F1** \(F(\uparrow)\), and **AUPRC** \(A(\uparrow)\). Faithfulness measures consistency with the model’s own decision process, using **Comprehensiveness** \(C(\uparrow)\), **Sufficiency** \(S(\downarrow)\), and **correlation with Leave-One-Out scores** \(L(\uparrow)\). The paper names these metrics but does **not** provide explicit equations for them.

The clearest finding is that rationale supervision **substantially improved plausibility across all considered metrics**. For example, with **mBERT + SHAP**, plausibility improves from \(A=0.583,\ F=0.410,\ I=0.290\) without rationales to \(A=0.628,\ F=0.453,\ I=0.325\) with rationales. With **mBERT + LIME**, the corresponding values move from \(A=0.572,\ F=0.406,\ I=0.282\) to \(A=0.630,\ F=0.455,\ I=0.323\). For **IndicBERT + SHAP**, plausibility changes from \(A=0.626,\ F=0.464,\ I=0.333\) to \(A=0.635,\ F=0.479,\ I=0.338\). The paper states that **SHAP** and **LIME** are generally the strongest explainers in terms of alignment with human rationales, while **Integrated Gradients × Input** is described as a robust gradient-based method [2508.01486].

By contrast, the **faithfulness** results are reported as mixed and inconclusive. For **IndicBERT + LIME**, the paper gives \(C=0.529,\ S=0.029,\ L=0.419\) without rationales and \(C=0.483,\ S=0.049,\ L=0.432\) with rationales: one metric improves while others worsen. For **Te-BERT + SHAP**, \(C\) and \(L\) increase slightly, while \(S\) worsens. The paper therefore concludes that rationale supervision reliably improves **human alignment** of explanations, but does **not** establish a consistent effect on faithfulness [2508.01486].

## 6. Fairness benchmark: TeEEC

**TeEEC**, the **Equity Evaluation Corpus in Telugu**, is the fairness component associated with TeSent. It is adapted from the English **EEC** and is intended for **sentiment- and emotion-related Telugu NLP tasks**. The benchmark targets two sensitive attribute families:
$$
\{\text{male}, \text{female}\}
$$
for gender, and
$$
\{\text{Hindu}, \text{Muslim}, \text{Christian}\}
$$
for religion. The paper treats **male** as the privileged gender group and **Hindu** as the privileged religion group, based on demographics of native Telugu speakers [2508.01486].

TeEEC is built from **11 translated templates** from EEC. Seven are of the type that include emotion words, such as `<Person> feels <emotion word>.`, and four omit emotion words, such as `I saw <person> in the market.` The authors select **10 commonly used emotion words** from each of the themes **Anger**, **Fear**, **Joy**, and **Sadness**, and adapt the morphology of these words to Telugu syntax. The paper specifies that **Templates 3, 4, and 7** use the **noun form**, while the remaining templates use the **adjective form**. For person placeholders, the benchmark uses **20 common Telugu names** for each of the six gender-religion combinations and **10 noun phrases** for each group; noun phrases are not used for religion comparisons. The **gender bias dataset** contains **39,760 sentences**, divided equally across male and female groups; the exact total size of the religion dataset is not explicitly stated [2508.01486].

Fairness is evaluated with **counterfactual polarity comparison** and **Equalized Odds**. The paper samples **1320 counterfactual pairs for gender** and **1200 counterfactual pairs for religion**, balanced across the three sentiment classes and averaged over **10 runs**. The **Polarity Difference Score (PDS)** is defined as
$$
PDS = \frac{1}{n} \sum_{i=1}^n f(C_i^a) - f(C_i^b),
$$
with
$$
f(\text{negative})=1,\quad f(\text{neutral})=2,\quad f(\text{positive})=3.
$$
A positive PDS indicates that the privileged group is assigned more positive polarity on average. Equalized Odds is assessed through absolute differences in **true positive rate** and **false positive rate**, denoted \(TPR_{diff}\) and \(FPR_{diff}\), with lower values indicating better fairness [2508.01486].

The paper’s overall conclusion is that rationale supervision does **not always** improve every fairness metric, but **overall it tends to reduce gender and religion bias**, especially religion bias. The strongest gender example is **IndicBERT**, where rationale training changes \(PDS\) from **0.013** to **0.001**, \(TPR_{diff}\) from **0.014** to **0.001**, and \(FPR_{diff}\) from **0.007** to **0.000**. For religion, **mBERT** on the Hindu-Muslim comparison changes from \(PDS=0.092\) to \(0.053\), \(TPR_{diff}=0.048\) to \(0.016\), and \(FPR_{diff}=0.232\) to \(0.008\) after rationale training. Some models show mixed or negligible changes, but the paper reports that increases in measured bias are usually small [2508.01486].

## 7. Position in the literature and name disambiguation

Within the literature represented here, **TeSent** refers specifically to the Telugu sentiment benchmark described above [2508.01486]. It should be distinguished from several similarly named but substantively different works. **“Entropy Sentinel: Continuous LLM Accuracy Monitoring from Decoding Entropy Traces in STEM”** studies slice-level accuracy estimation for LLMs from decoding entropy profiles and notes that “TeSent” is a plausible shorthand for **Entropy Sentinel**, but that work concerns monitoring under domain shift rather than Telugu sentiment classification [2601.09001]. **“TransSent: Towards Generation of Structured Sentences with Discourse Marker”** is a discourse-aware generation model based on head-discourse, relation, and tail-discourse structure, not a sentiment benchmark [1909.05364]. **“TNCSE: Tensor’s Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings”** concerns unsupervised sentence embedding geometry and norm-aware contrastive objectives rather than explainable or fairness-aware sentiment classification [2503.12739].

In that landscape, TeSent occupies a specific niche: it is a **benchmarking infrastructure** for Telugu NLP that combines supervised sentiment labels, rationale annotations for explainability analysis, and a fairness corpus for demographic auditing. A plausible implication is that its main contribution is not a new classifier architecture but a reproducible experimental substrate for studying how **rationale supervision** changes classification quality, explanation plausibility, and bias behavior in a low-resource language setting [2508.01486].

Source: https://www.emergentmind.com/topics/tesent