---
title: 'Sentiment140 Lexicon: Clarifying Twitter Sentiment'
url: https://www.emergentmind.com/topics/sentiment140-lexicon
type: topic
---

# Sentiment140 Lexicon: Clarifying Twitter Sentiment

Searching arXiv for papers directly related to Sentiment140 and Twitter sentiment lexicons.
The Sentiment140 Lexicon is best understood, in the literature summarized here, as an ambiguous label at the intersection of Twitter sentiment corpora and lexicon-based polarity resources. The most important clarification is negative: "Lexicon-Based Sentiment Analysis on Text Polarities with Evaluation of Classification Models" does **not** define or derive a new lexicon called a “Sentiment140 lexicon,” nor does it construct any lexicon specifically tailored to the Sentiment140 dataset. Instead, it uses the Sentiment140 Twitter dataset containing **1.6 million unprocessed tweets** and applies existing generic lexicon-based sentiment tools—specifically **TextBlob** and **VADER**—to recalculate tweet sentiment, introduce a **neutral** class, and then support downstream TF-IDF-based supervised classification [2409.12840]. In that sense, Sentiment140 is the corpus substrate, while the lexicons are external resources embedded in off-the-shelf analyzers rather than lexicons induced from the corpus itself.

## 1. Terminological scope and recurrent confusion

Within this research area, “Sentiment140 Lexicon” is often treated as if it named a corpus-derived lexical resource. The evidence presented here supports a more careful reading: the Sentiment140 connection is primarily a matter of **application to Twitter data**, not lexicon construction. The paper centered on Sentiment140 explicitly states that the lexicons are **external resources embedded in TextBlob and VADER** and are **not induced from the 1.6 million tweets** [2409.12840].

This distinction matters because lexicon-based sentiment analysis and Sentiment140-style Twitter sentiment analysis are not identical methodological objects. A Twitter classifier paper can be highly relevant to Sentiment140 without using a Sentiment140-specific lexicon. For example, "Twitter Sentiment Analysis" discusses Twitter sentiment analysis in detail, cites the distant-supervision work commonly associated with the Sentiment140 dataset/project, and uses lexicon-derived features from **MPQA**, but it does **not** introduce or analyze a Sentiment140 lexical resource [1509.04219].

| Resource or term | Role in the literature | What it is not |
|---|---|---|
| Sentiment140 dataset | Twitter corpus with **1.6 million unprocessed tweets** | Not itself a lexicon |
| TextBlob and VADER in the Sentiment140 pipeline | Off-the-shelf lexicon-based analyzers applied to tweets | Not induced from Sentiment140 |
| SlangSD | Slang sentiment dictionary for short informal text | Not a replacement for broad Twitter lexicons |

A plausible implication is that the expression “Sentiment140 Lexicon” should be used cautiously unless the intended referent is made explicit: dataset, distant-supervision paradigm, or external sentiment lexicon applied to Twitter text.

## 2. Sentiment140 as corpus substrate rather than lexical inventory

In the 2024 pipeline, the dataset side is unambiguous. The paper uses the **Twitter Sentiment140 corpus** with **1.6 million unprocessed tweets**, originally containing approximately equal numbers of **positive** and **negative** examples. It emphasizes that these are “unprocessed no-emoticon tweets” for training and testing, even though the source remains Sentiment140 [2409.12840].

The preprocessing workflow is designed to standardize noisy Twitter text before lexicon lookup and subsequent TF-IDF feature construction. The stated operations include cleaning tweets with **regular expressions**, removing **numbers**, **URLs**, **Twitter usernames**, **special characters**, and other “non-essential tokens,” removing **English stop words**, maintaining a **slang list** for abbreviations, applying **lemmatization** or **stemming** where needed, and replacing slang forms with their complete forms [2409.12840].

These steps are significant because they reveal the operational meaning of lexicon use in a Sentiment140 setting. The lexicon is not learned from the corpus; rather, the corpus is normalized so that existing lexical analyzers can operate more effectively on tokenized and standardized microblog text. This suggests a pipeline in which lexical resources serve as annotation and interpretation instruments over a pre-existing Twitter corpus, rather than as corpus-extracted lexical artifacts.

## 3. Lexicon scoring, neutrality induction, and sentiment interpretation

The lexicon-based stage uses **TextBlob** and **VADER** as off-the-shelf sentiment analyzers. TextBlob provides two scores: **polarity**, ranging from \(-1\) to \(1\), and **subjectivity**, ranging from \(0\) to \(1\). Negative polarity values indicate negative sentiment, positive values indicate positive sentiment, and values near zero are treated as neutral. The paper does **not** provide a numeric TextBlob threshold for neutrality, so the decision rule can only be described qualitatively [2409.12840].

For **VADER**, the thresholding rule is explicit:

$$
\text{label}(t)=
\begin{cases}
\text{positive}, & s_{\text{VADER}}(t) > 0.05\\
\text{negative}, & s_{\text{VADER}}(t) < -0.05\\
\text{neutral}, & -0.05 \le s_{\text{VADER}}(t) \le 0.05
\end{cases}
$$

Here, \(t\) is a tweet, \(s_{\text{VADER}}(t)\) is the VADER sentiment score for tweet \(t\), and \(\text{label}(t)\) is the assigned sentiment class [2409.12840].

This thresholding rule is central because it introduces **neutrality** into a dataset that originally had only positive and negative labels. The paper argues that neutral sentiment matters because it can filter irrelevant reviews or moderate personality judgments. Neutrality therefore emerges not from a new lexicon, but from **thresholding the outputs of generic lexicon analyzers**. After preprocessing and rescoring, some tweets originally labeled positive or negative become neutral under TextBlob or VADER [2409.12840].

At the interpretive level, the paper presents lexicons as identifying “intensity of emotion and subjectivity at word levels.” Positive tweets are illustrated with words such as **“thanks,” “good,” and “best,”** while negative tweets include **“sorry,” “hurts,” “sucks,” “dangerous,” and “grief.”** Neutral tweets contain more informational or less affective words such as **“work,” “morning,” “update,” and “twitter.”** However, the paper does **not** specify a formal additive word-level scoring equation and treats the internal scoring mechanics of TextBlob and VADER as black-box outputs [2409.12840]. It also indicates that VADER is “more sophisticated for social media analysis” than TextBlob and better captures “subtle idiosyncrasies” of social-media content, implying a better fit to Twitter language without isolating that effect experimentally [2409.12840].

## 4. Hybrid pipeline and benchmarked classifiers

The Sentiment140-based workflow is hybrid rather than purely lexical. After lexicon-based rescoring and class reinterpretation, the paper uses **TF-IDF** for feature extraction and states that features are ranked to remove words with lower polarity and subjectivity. It further states that the difference between original and truncated text is controlled using the **Jaccard similarity measure**, with a randomly selected threshold between **60\% and 80\% similarity**. The text does **not** provide a TF-IDF formula, a Jaccard formula, or an exact rule linking polarity and subjectivity to feature elimination [2409.12840].

The downstream comparison involves five classifiers evaluated on **60–40**, **70–30**, and **80–20** train-test splits, with **5-fold cross-validation** used to corroborate performance. The preferred final setting is the **70–30 split** [2409.12840].

| Classifier | 60–40 | 70–30 | 80–20 |
|---|---:|---:|---:|
| Random Forest | 79% | **81%** | 79% |
| Multinomial Logistic Regression | 76% | 78% | 77% |
| SVM | 69% | 79% | 78% |
| Naive Bayes | 61% | 58% | 57% |
| XGBoost | 73% | 78% | 78% |

The best overall model is **Random Forest at 81% accuracy**, and the conclusion is that Random Forest slightly outperforms the others across splits [2409.12840].

The three-class analysis on the 70–30 split reinforces the same pattern. For **positive tweets**, Random Forest achieves the best precision (**85%**) and F1 (**87%**), while Multinomial Logistic Regression attains very high recall (**93%**). For **neutral tweets**, performance is substantially weaker and more variable: Random Forest has the best neutral recall (**79%**) and F1 (**71%**), whereas Multinomial Logistic Regression has high neutral precision (**83%**) but extremely poor neutral recall (**7%**). For **negative tweets**, Random Forest again performs best overall, with precision around **84%** and F1 around **72%**, while SVM has strong negative recall [2409.12840].

The resulting interpretation is explicit in the paper: lexicon-based sentiment scoring is **not itself the final benchmarked model**. Rather, lexicon outputs create or refine class labels—especially neutral labels—and help interpret salient words, while the final predictive comparison is among standard supervised classifiers.

## 5. Coverage gaps, slang, and complementary lexical resources

A major limitation of generic lexicons in Twitter sentiment analysis is coverage of **slang expressions** and **multiword informal phrases**. "SlangSD: Building and Using a Sentiment Dictionary of Slang Words for Short-Text Sentiment Classification" introduces **SlangSD**, presented as the **first sentiment dictionary specifically for slang words and phrases**, precisely because slang sentiment words are often absent from existing dictionaries or sentiment lexicons [1608.05129].

SlangSD is built from **Urban Dictionary**, existing sentiment lexicons, and **Twitter** usage, and assigns sentiment on a real-valued discrete scale from \(-2\) to \(2\). Its three-stage labeling procedure consists of direct transfer from existing sentiment lexicons, Twitter-based sentiment inference via co-occurrence, and iterative sentiment propagation over Urban Dictionary **related words**. The final resource contains **96,462 slang words and phrases** [1608.05129].

Relative to Sentiment140-style lexical resources, the paper is explicit that SlangSD should be viewed as **complementary** rather than substitutive. It does **not** mention Sentiment140 directly, but it targets a specific coverage gap in short, informal, social-media-style text: expressions such as **“out of the park”** and **“shit hot”**, whose surface form can be misleading or highly community-specific [1608.05129]. This suggests that a broad Twitter lexicon and a slang-first lexicon occupy different but compatible roles.

The same broader point appears in Twitter classifier work that uses external lexicons without centering on Sentiment140. In "Twitter Sentiment Analysis," the system uses **MPQA** for prior polarity and objectivity/subjectivity features, but the final retained polarity features are **learned unigram word models**, **number of positive emoticons**, and **number of negative emoticons**, with **MPQA score** treated as redundant [1509.04219]. That result does not invalidate lexicons; rather, it indicates that Twitter-specific lexical statistics and surface cues can dominate generic lexicon summary scores in microblog classification.

## 6. Theoretical and multilingual perspectives

The most systematic theoretical treatment in this group of papers comes from "Unsupervised Learning for Lexicon-Based Classification," which is not about Sentiment140 specifically but directly applies to any binary positive/negative lexicon. The paper shows that the standard lexicon heuristic—counting words from two opposed lexicons and choosing the larger count—is Bayes-optimal only under strong assumptions: **completeness**, **equal predictiveness**, **equal coverage**, and **equal priors** [1611.06933].

Its central critique is that **equal predictiveness** is rarely true in practice. In real sentiment lexicons, words differ in ambiguity, intensity, domain specificity, and frequency effects. The paper therefore replaces a single shared sentiment strength with per-word strengths \(\gamma_i\), learned **without labeled data** from cross-lexicon co-occurrence statistics, and derives a weighted scoring rule based on \(\log \frac{1+\gamma_i}{1-\gamma_i}\) [1611.06933]. For Sentiment140-style use, this provides a principled account of why flat lexicon counting may underperform: not all positive or negative tokens are equally informative.

The same literature also places Sentiment140-style lexicon questions in a multilingual setting. "Generating Sentiment Lexicons for German Twitter" compares semi-automatic translation of English polarity lists with automatic lexicon generation methods applied directly to German resources and German Twitter data. On **7,992 manually annotated tweets**, the best semi-automatic lexicon combination reaches a macro-averaged \(F_1\)-score of **0.589**, outperforming the best dictionary-based method (**0.479**) and the best corpus-based method (**0.419**) [1610.09995]. The paper does **not** mention Sentiment140 directly, but it is highly relevant to the problem of building a Twitter sentiment lexicon in another language.

Taken together, these results indicate that “Sentiment140 Lexicon” is less a single stable object than a methodological nexus: a Twitter corpus used as input data, off-the-shelf or complementary lexicons applied to that data, and broader questions about weighting, slang coverage, and cross-lingual transfer. The clearest synthesis from the evidence is that the 2024 Sentiment140 paper applies **existing lexicon resources to Sentiment140**; it does **not** build a Sentiment140-specific lexicon [2409.12840].

Source: https://www.emergentmind.com/topics/sentiment140-lexicon