NRC Hashtag Sentiment Lexicon

• The paper introduced a large-scale, tweet-specific lexicon that enhances sentiment classification by leveraging sentiment-word hashtags and yielding significant F-score improvements.
• NRC Hashtag Sentiment Lexicon is a resource automatically induced via distant supervision from tweets, providing graded polarity scores using pointwise mutual information.
• The lexicon effectively supports robust feature engineering by integrating unigram, bigram, and non-contiguous entries with specialized techniques such as negation handling and normalization.

Searching arXiv for the specified paper and closely related work to ground the article. arxiv_search(query="(Mohammad et al., 2013) OR \"NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets\" OR NRC Hashtag Sentiment Lexicon", max_results=5) arxiv_search(query="tweet sentiment lexicon PMI hashtags emoticons SemEval 2013 NRC", max_results=10) The NRC Hashtag Sentiment Lexicon is a tweet-specific sentiment resource introduced in the NRC-Canada system for SemEval-2013 sentiment analysis. It was created to provide large-scale associations between words and polarity, specifically positive versus negative, and to serve as a feature source for supervised classifiers operating on tweets and SMS. Each entry is assigned a real-valued sentiment association score derived from co-occurrence statistics between terms and sentiment-word hashtags in a large tweet corpus. In the SemEval-2013 systems described in "NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets," the lexicon was one of the most influential feature sources, contributing substantial gains in both message-level and term-level sentiment classification (Mohammad et al., 2013).

1. Purpose and conceptual role

The lexicon was designed as a large-scale, automatically induced sentiment lexicon specialized for Twitter. Its central function is to encode associations between terms and polarity using weak supervision from sentiment-indicative hashtags. In contrast to manually curated lexicons, it targets platform-specific language, including lexical items and constructions characteristic of tweets.

Within the NRC-Canada systems, the lexicon formed part of a broader feature stack that also included surface-form, semantic, and sentiment features. The paper reports that, in the message-level task, lexicon-based features provided a gain of 5 F-score points over all others, and the system ranked first on tweets with an F-score of 69.02 and first on SMS with an F-score of 68.46. For term-level sentiment, the system ranked first on tweets with 88.93 and second on SMS with 88.00; after a post-competition bug fix, the tweet score became 89.10 and the SMS score 88.34 (Mohammad et al., 2013).

A common misconception is to treat the resource as a general-purpose sentiment lexicon. The construction procedure and the reported usage indicate instead that it is tailored to Twitter circa 2012. This suggests that its principal value lies in domain-matched sentiment modeling rather than in universal lexical polarity annotation.

2. Corpus construction and distant supervision

The lexicon was induced from tweets containing sentiment-word hashtags used as distant labels. A tweet was labeled positive if it contained one of the positive seed hashtags and negative if it contained one of the negative seed hashtags. The description states that the authors used a list of 78 English seed words closely related to “positive” and “negative,” selected from Roget’s Thesaurus, and further specifies 32 positive seeds, such as #good and #excellent, and 36 negative seeds, such as #bad and #terrible.

The corpus comprised 775,000 tweets collected by polling the Twitter API every four hours from April to December 2012. The seed words and downstream tooling are English, but the paper does not explicitly declare a language restriction for the hashtag-collection step. For lexicon induction, the paper specifies that “pairs with certain punctuations, ‘@’ symbols, and some function words were removed” when constructing pair entries. It does not detail additional normalization or tokenization for lexicon creation itself. By contrast, the downstream classifiers used the CMU Twitter tokenizer and normalized URLs to http://someurl and user IDs to @someuser (Mohammad et al., 2013).

This construction procedure is a canonical instance of distant supervision. Its advantage is scale: weak labels obtained from hashtags make it possible to estimate polarity associations for a large number of terms without manual annotation. Its cost is label noise, since sarcasm, topic hashtags, or mixed-sentiment tweets can distort the relationship between hashtag and overall polarity.

3. Representation, entry types, and scoring

The lexicon contains three classes of entries: unigrams, bigrams, and non-contiguous pairs. Non-contiguous pairs include unigram–unigram, unigram–bigram, and bigram–bigram combinations that are not necessarily adjacent in the tweet.

Entry type	Number of entries
Unigrams	54,129
Bigrams	316,531
Non-contiguous pairs	308,808

This breadth provides strong coverage of Twitter-specific language, including multiword expressions and non-adjacent associations. The paper does not report explicit minimum frequency thresholds or smoothing for entry selection beyond the stated filtering of some punctuation, @-symbols, and function words.

Sentiment association is quantified with pointwise mutual information (PMI). Let $x$ and $y$ be events; then

$$\operatorname{PMI}(x, y) = \log \frac{P(x, y)}{P(x)\,P(y)}.$$

For a term $w$, the sentiment score is defined as

$$\text{score}(w) = \operatorname{PMI}(w, \text{positive}) - \operatorname{PMI}(w, \text{negative}).$$

Positive values indicate association with positive sentiment, negative values indicate association with negative sentiment, and the magnitude reflects the strength of association. The paper does not specify the logarithm base, whether any smoothing was applied, whether counts were token-level or tweet-level co-occurrences, or any minimum-count thresholds. It also does not provide sample lexicon entries with numeric scores or any distributional statistics such as histograms (Mohammad et al., 2013).

Because the score is a difference of two PMI terms, it is centered around zero by construction. This suggests a signed association measure rather than a categorical label inventory: the lexicon supports graded feature extraction rather than only binary lookup.

4. Integration into message-level and term-level classifiers

The NRC-Canada systems used linear SVM classifiers for both message-level and term-level sentiment analysis. In the message-level classifier, lexicon-derived features were computed for each lexicon, both manual and automatic, and for each token group: unigrams, bigrams, and non-contiguous pairs. The feature set included the count of tokens with $\text{score}(w,p) > 0$ for polarity or emotion $p$, the sum $\sum_{w \in \text{tweet}} \text{score}(w,p)$, the maximum $\max_{w \in \text{tweet}} \text{score}(w,p)$, and the score of the last token with $\text{score}(w,p) > 0$. These features were computed not only over all tokens, but also per POS tag, separately for hashtags, and separately for all-caps tokens.

Negation handling was explicit. In message-level classification, negated contexts, for example starting at “no” or “shouldn’t” and ending at punctuation, trigger a _NEG suffix on following tokens and on lexicon-polarity features such as POLARITY_positive_NEG. The description states that this effectively flips polarity contributions downstream.

The term-level classifier used similar lexicon-derived features within the target span: count with score $y$0, sum of scores, max score, and the non-zero score of the last token. It also incorporated context features from a 4-word window on each side of the target, with lexicon features computed separately for the target and for the context. Negation handling differed slightly: if a negation word occurs immediately before or within the target, polarities of all tokens after the negation are flipped (Mohammad et al., 2013).

These design choices show that the lexicon was not used merely for dictionary matching. Rather, it was embedded in a structured feature engineering scheme that exploited aggregation operators, positional effects, POS partitioning, capitalization, hashtags, and negation scope.

5. Empirical contribution and ablation results

The empirical impact of the lexicon is reported through system-level results and ablation studies. On the SemEval-2013 message-level tweet task, the full system obtained an overall F-score, macro-averaged over positive and negative, of 69.02. Removing all lexicon features reduced performance to 60.42, a drop of 8.60 points. Removing automatic lexicons alone reduced performance to 63.78, a drop of 5.24 points. Removing the NRC Hashtag lexicon alone reduced performance to 65.22, a drop of 3.80, while removing the Sentiment140 lexicon alone reduced performance to 65.25, a drop of 3.77.

On the message-level SMS task, where no SMS-specific training was used, the full system achieved 68.46. Removing all lexicons reduced this to 59.73, a drop of 8.73. Removing automatic lexicons reduced it to 67.12, a drop of 1.34. The description notes that SMS benefited more from manual lexicons than automatic ones.

On the term-level tweet task, the overall F-score was 88.93, increasing to 89.10 after a post-competition bug fix. Removing lexicons reduced the score to 85.15, a drop of 3.95. On the term-level SMS task, the score was 88.00, increasing to 88.34 after the bug fix; removing lexicons reduced it to 83.70, a drop of 4.64 (Mohammad et al., 2013).

Evaluation setting	Full system	Without lexicons
Message-level tweets	69.02	60.42
Message-level SMS	68.46	59.73
Term-level tweets	88.93	85.15
Term-level SMS	88.00	83.70

These ablations establish the lexicon as a major source of predictive power rather than a marginal auxiliary feature. For tweet message-level classification in particular, the automatic lexicons alone account for a large fraction of the total system advantage.

6. Relation to the NRC Sentiment140 lexicon and methodological significance

The paper contrasts the NRC Hashtag Sentiment Lexicon with the NRC Sentiment140 lexicon. The crucial difference is the supervision signal: the hashtag lexicon uses sentiment-word hashtags as distant labels, whereas the Sentiment140 lexicon uses emoticons as distant labels from the Sentiment140 corpus of 1.6 million tweets. The two lexicons are otherwise built identically in terms of PMI scoring and entry types.

The Sentiment140 lexicon is larger, with 62,468 unigrams, 677,698 bigrams, and 480,010 non-contiguous pairs. Despite that size difference, both lexicons contributed roughly similar improvements on the tweet message-level task: removing either one led to an approximately 3.8-point F-score drop. This indicates that the hashtag-based and emoticon-based labeling schemes provide complementary supervision signals rather than a simple size-driven hierarchy of usefulness (Mohammad et al., 2013).

Methodologically, the comparison is significant because it isolates the effect of weak-label source while holding the scoring framework and entry types constant. A plausible implication is that distant supervision quality depends not only on corpus scale but also on the semantic relation between labeling cue and target sentiment phenomenon.

7. Availability, application guidance, and limitations

The lexicons are stated to be freely available at www.purl.com/net/sentimentoftweets, although specific licensing terms are not detailed. For application, the paper recommends a robust Twitter tokenizer such as the CMU tool to align tokens and bigrams with lexicon entries. It also recommends consistent normalization of URLs and user mentions if uniform treatment is desired, though such normalization is described explicitly for the classifiers rather than for lexicon induction.

The reported feature design guidance is specific: summation, counts, max, and last-token features of lexicon scores are effective; these should be computed over whole messages, per POS, and over special subsets such as hashtags and all-caps tokens. Negation handling should be implemented through scope modeling and suffixing with _NEG, or through polarity flipping, to avoid false positives from local negation. The lexicon is most effective when combined with strong ngram baselines rather than used in isolation.

Several limitations are explicit. The resource is domain-specific, being tailored to Twitter circa 2012, and performance may degrade on other domains or on more recent usage without adaptation. Hashtag-based labels are weak signals, so sarcasm, topic hashtags, and mixed sentiments can introduce noise into PMI estimates. The seed list is English and anchored to words closely related to “positive” and “negative,” which may bias coverage toward particular registers or topics. The paper does not report class distributions in the hashtag corpus, so potential imbalance between positive and negative tweets is not quantified. Coverage gaps may arise for out-of-vocabulary tokens, evolving slang, code-switching, and non-English content, and the lexicon does not include explicit handling of intensifiers or diminishers apart from elongated-word features in the classifier (Mohammad et al., 2013).

The recommended operational response to these limitations is conservative rather than architectural: periodically refresh or augment the lexicon with newer tweets using the same pipeline, use both hashtag- and emoticon-derived lexicons, implement careful negation handling, combine lexicon features with strong ngram features, and evaluate on in-domain data. If the target medium is SMS or another non-Twitter domain, the paper further indicates that manually curated lexicons may be especially useful, since SMS benefited more from manual lexicons than from automatic ones.