Document-Level Sentiment Classification

• Document-level sentiment classification is the process of assigning a single overall sentiment label to an entire document, capturing varied evidence from distributed text cues.
• The methodology incorporates lexicon-based aggregation, sparse and distributed representations, and advanced neural architectures to manage long-range dependencies.
• Empirical studies show that integrating discourse structure, sentence importance, and tailored supervision regimes enhances both accuracy and efficiency.

Searching arXiv for recent and foundational papers on document-level sentiment classification to ground the article in published work. I’m looking up relevant arXiv papers on document-level sentiment classification, including hierarchical, discourse-aware, lexicon-based, and transfer-learning approaches. Document-level sentiment classification is the task of assigning a single sentiment label or rating to an entire document rather than to individual words, sentences, or aspects. In the classical setting, the document is assumed to discuss a single entity or topic, and the objective is to infer its overall polarity, typically positive versus negative, from evidence that may be distributed unevenly across the text and altered by negation, discourse organization, or domain-specific vocabulary (Stalidis et al., 2015, Choi et al., 2021, Barnes et al., 2020). The research literature spans lexicon-based aggregation, sparse and distributed document representations, hierarchical recurrent and convolutional architectures, discourse-aware composition, pretrained LLMs, and target-specific extensions such as author-to-entity sentiment and document-level aspect sentiment classification (Paramesha et al., 2013, Bhatia et al., 2015, Xu et al., 2016, Bastan et al., 2020, Wang et al., 2019).

1. Task definitions and label spaces

A standard formalization treats a document as a sequence of sentences, $D=\{S_1,S_2,\dots,S_n\}$, and maps it to one sentiment label from a predefined class set. The label space varies substantially by corpus and application. Binary polarity remains common in review analysis, as in movie-review datasets and several multilingual settings (Stalidis et al., 2015, Barnes, 2019, Nassar et al., 2022). Short-text sentiment work also uses three classes—positive, negative, and neutral—while large review corpora may use discrete rating labels such as 1–5 or 1–6 stars (Hamdan et al., 2016, Barnes et al., 2020). In entity-centered news sentiment, the target is not the document “in general” but the author’s sentiment toward the main person entity; the underlying annotation scheme uses five fine-grained ratings and collapses them into three coarse classes (Bastan et al., 2020).

The apparent simplicity of the output label obscures several distinct problem formulations. Cross-domain document classification asks a model trained on one product domain to generalize to another, even though vocabularies and polarities shift; examples in the literature include words that are positive in electronics but irrelevant or differently valenced in video games (Paramesha et al., 2013). Multi-aspect variants predict a sentiment value for each aspect mentioned in a document rather than a single document-wide polarity (Zeng et al., 2019, Wang et al., 2019). Zero-shot aspect-level classification goes further by attempting to derive aspect sentiment classifiers from document-level review ratings alone, without aspect-level annotations (Deng et al., 2022).

These formulations imply that document-level sentiment classification is not a single homogeneous task. A plausible implication is that architectural and representational choices are strongly conditioned by whether the target is overall polarity, ordinal rating, aspect-specific sentiment, or author stance toward an entity.

2. Representations and feature engineering

Early and hybrid approaches emphasize document representation as the central design choice. A representative example is Hybrid Weighted Word2Vec (HWW2V), defined as

$$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$

This representation concatenates Bag-of-Words features, TF-IDF weighted Word2Vec embeddings, and lexicon-based sentiment values derived from SentiWordNet (Stalidis et al., 2015). In its weighted embedding component, each word vector $v_i$ is scaled by

$w_{ij} = f_{ij}\, \log \frac{D}{D_i},$

and the document vector is

$s_j = \sum_i w_{ij}\, v_i.$

The implementation described in that work uses binary term presence for the TF component while retaining document-frequency variation through the IDF term (Stalidis et al., 2015).

Lexicon-based document representations constitute another major line. SentiWordNet-based systems use polarity weights rather than simple positive/negative term counts, often after POS tagging. One document-level formulation constructs sentence-level features from positive and negative scores, term counts, and score ratios for adjectives, adverbs, verbs, and nouns, then obtains document features by “Summation of all features over sentences” (Paramesha et al., 2013). In HWW2V, the lexicon subvector is four-dimensional—positive, objective, negative, and unknown-word ratio—and is obtained by averaging sentence-level sentiment vectors over the document (Stalidis et al., 2015). Annotation-centered work uses an even more specialized representation: the sentiment of a research paper is inferred from comments, highlights, and underlines, with comment polarity computed from SentiWordNet scores and aggregated through a meta-annotation-aware weighted average (Shukla, 2011).

Supervised term weighting generalizes the sparse-vector tradition by replacing uniform lexical counts with class-discriminative weights. In that framework,

$$w_{ij} = local\_weight \times global\_weight \times normalization,$$

and the class-specific score of a term is aggregated as

$score(f)=\max_{c \in C}(gmetric(f,c)).$

Across Twitter, restaurant, and laptop review datasets, fifteen global weighting metrics and four local schemes were compared, with recurrently strong metrics including $rf$, $kl$, and $wllr$, and recurrently weak metrics including $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$0 and $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$1 (Hamdan et al., 2016). The analysis associates poor weighting schemes with narrow term-score distributions whose means are close to zero (Hamdan et al., 2016).

Sentence-aggregated feature engineering remains important in morphologically rich and lexicon-driven settings. In Arabic document classification, sentence polarity is first computed from term scores using formulas such as $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$2, after which each document is represented by seven sentence-based features: counts of positive and negative sentences, maximum positive and negative sentence scores, and the first, middle, and last sentence scores (Nassar et al., 2022). This suggests that even in neural-dominant eras, hand-designed document summaries remain competitive when combined with language-specific preprocessing and polarity rules.

3. Neural architectures for long and structured documents

A central problem for document-level sentiment models is long-range dependency. Standard recurrent networks and even ordinary LSTMs may fail to preserve sentiment evidence over long texts, which motivated Cached Long Short-Term Memory neural networks (CLSTM). CLSTM divides memory into $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$3 groups with distinct forgetting-rate intervals,

$$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$4

so that each group operates at a different timescale. The grouped cell update is

$$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$5

The slowest group is used as the document representation, and the bidirectional variant B-CLSTM achieved the best reported results in that study: 46.2% accuracy and 2.112 MSE on IMDB, 59.8% and 0.549 on Yelp 2013, and 61.9% and 0.496 on Yelp 2014 (Xu et al., 2016).

Hierarchical models encode document structure more explicitly. A systematic comparison across English, French, German, Japanese, and Norwegian evaluated flat baselines, hierarchical CNNs, Hierarchical Attention Networks (HAN), ULMFiT, and multilingual BERT on document-level star-rating classification (Barnes et al., 2020). The strongest overall model was HAN with average accuracy 66.4, followed by a flat recurrent attention model at 66.1, while mBERT reached 64.9 on average (Barnes et al., 2020). The best model depended on the language: mBERT was best for English, French, and German, HAN for Norwegian, and ULMFiT for Japanese (Barnes et al., 2020). The same study also found that mBERT performs better on short documents, whereas HAN performs better on longer documents, especially beyond 10 sentences, with a Pearson ranked correlation of 0.41 and $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$6 (Barnes et al., 2020).

Other architectures pursue document-specific interaction patterns. Hierarchical Interaction Networks (HIN) model bidirectional summary-document interactions at character, segment, and document levels, then refine the resulting representation with a Sentiment-based Rethinking mechanism. On Toys Games, Sports Outdoors, and News datasets, HIN reached 77.5, 76.7, and 89.0 accuracy, and HIN-SR improved these to 78.1, 77.2, and 89.3 (Wei et al., 2020). In Urdu, a hybrid BiLSTM-SLMFCNN architecture using pretrained 300-dimensional Urdu embeddings, a BiLSTM with 150 hidden units, and a single-layer multi-filter CNN with filter sizes 3, 4, and 5 achieved 83%, 79%, 83%, and 94% accuracy on IMDB Small, IMDB Medium, IMDB Large, and a customer-support dataset respectively (Irum et al., 23 Jan 2025).

Taken together, these results support two recurrent themes: document length materially changes the behavior of sentiment models, and explicit hierarchical or multi-timescale structure can be advantageous when sentiment is dispersed across long texts.

4. Sentence importance, discourse, negation, and other compositional structure

A major critique of flat document encoders is that they treat a document as a bag of sentences. One response is to estimate sentence importance directly. A model based on ALBERT sentence embeddings, class-similarity features, a gate layer, and a GRU-based document encoder defines sentence importance as

$$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$7

This model achieved 0.548 accuracy on IMDB, 0.833 on Yelp-hotel, 0.882 on Yelp-rest, and 0.876 on Amazon, outperforming listed baselines that did not model importance differences (Choi et al., 2021). Its analysis reported that about 81.6% of test documents had normalized importance-score standard deviation greater than 0.2, indicating substantial non-uniformity in sentence contributions (Choi et al., 2021).

Discourse-aware methods address a related issue at a coarser rhetorical level. Using Rhetorical Structure Theory, discourse units can be reweighted according to depth in a dependency representation of the rhetorical tree: $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$8 Document sentiment is then computed by weighting EDU-level bag-of-words sentiment scores before aggregation (Bhatia et al., 2015). On movie reviews, depth-based reweighting improved lexicon-based accuracy from 68.3 to 72.6 on Pang and Lee, and from 74.9 to 78.9 on the Socher et al. corpus (Bhatia et al., 2015). A recursive neural model over RST trees further improved supervised classification, reaching 84.1 and 85.6 accuracy on the two datasets (Bhatia et al., 2015).

Negation is another persistent compositional difficulty. LTG-Oslo’s NEGES 2019 system for Spanish reviews used a hierarchical multi-task BiLSTM in which lower layers were shared between negation detection and document-level sentiment classification, with a CRF-Viterbi negation head and a hierarchical sentiment head (Barnes, 2019). The multi-task model achieved 72.5 development accuracy versus 71.4 for both a bag-of-words baseline and the single-task neural variant, and 66.2 on the test set (Barnes, 2019). Arabic document-level work encoded negation and intensification through explicit rules: polarity is flipped when a negation word precedes a term, and intensified positive or negative terms are pushed to 1 or $$\text{HWW2V}(d) = [\text{BoW}(d) \; ; \; \text{WeightedWord2Vec}(d) \; ; \; \text{SentimentLexicon}(d)].$$9 respectively (Nassar et al., 2022). The best document-level setting in that study—rules plus $v_i$0, seven features, and ANN—reached F-scores of 0.94 for the positive class and 0.93 for the negative class (Nassar et al., 2022).

These results converge on a common point: document-level sentiment depends not only on which sentiment-bearing words occur, but also on where they occur, how rhetorically central they are, and whether local composition reverses or amplifies polarity.

5. Supervision regimes and target-specific extensions

The literature includes several departures from standard fully supervised whole-document polarity. Weakly supervised document-level multi-aspect sentiment classification uses target-opinion word pairs as supervision rather than gold aspect labels. In the variational formulation,

$v_i$1

and learning proceeds by maximizing a variational lower bound in which latent aspect polarity explains observed opinion words (Zeng et al., 2019). On TripAdvisor and BeerAdvocate, VWS-DMSC obtained 0.7561 and 0.7538 test accuracy, outperforming multiple weakly supervised baselines and approaching fully supervised systems trained with hundreds of labels per aspect (Zeng et al., 2019).

A related but conceptually distinct line makes document supervision explicitly compositional. AF-DSC begins with potential aspects acquired heuristically by POS tagging, encodes the review with BERT, restricts attention to aspect candidates, and composes document sentiment from aspect-focused representations: $v_i$2 The model is trained with document rating prediction plus Word Sentiment Prediction and Mask Word Prediction, with total loss

$v_i$3

After training, aspect-level sentiment is obtained as a by-product of document-level supervision (Deng et al., 2022). On SemEval-2014, AF-DSC reached 69.61 accuracy and 61.69 macro-F1 on restaurant reviews, and 65.41 accuracy and 57.81 macro-F1 on laptop reviews, using only 30K document-level training examples (Deng et al., 2022).

Target-specific document sentiment in news exposes further limitations of generic document encoders. PerSenT contains 5,339 documents, 38,352 paragraphs, and 3,223 unique entities, with both paragraph-level and document-level annotations of the author’s sentiment toward the main person entity (Bastan et al., 2020). Direct document-level BERT was the strongest baseline, at 0.48 macro-F1 on the standard test set and 0.42 on the frequent-entity test set, but paragraph-level BERT followed by majority or average aggregation performed worse (Bastan et al., 2020). The paper explicitly concludes that simply fine-tuning document-level BERT representations is not adequate and that making paragraph-level decisions and aggregating them over the entire document is also ineffective (Bastan et al., 2020).

Document-level aspect sentiment classification has also been framed as an interpretable selection problem. A hierarchical reinforcement learning model first selects aspect-relevant clauses, then sentiment-relevant words within those clauses, and finally predicts the aspect rating from the selected evidence (Wang et al., 2019). On TripUser, TripAdvisor, and BeerAdvocate, the full HRL system achieved 62.84, 52.27, and 41.39 accuracy, together with 0.351, 0.662, and 1.503 MSE, outperforming listed baselines and providing explicit clause- and word-level rationales (Wang et al., 2019).

6. Evaluation practice, empirical patterns, and model-selection considerations

Evaluation protocols vary, but several patterns recur. Accuracy is the dominant metric in classical document-level polarity work and many multilingual review settings (Stalidis et al., 2015, Barnes et al., 2020). Other studies report macro-F1, precision, recall, MSE, and AUC, especially when label spaces are ordinal, multiclass, or imbalanced (Bastan et al., 2020, Xu et al., 2016, Irum et al., 23 Jan 2025). Cross-validation remains common in lexicon-driven and classical machine-learning studies, including 10-fold cross-validation on the RT movie-review dataset and in SentiWordNet-based cross-domain experiments (Stalidis et al., 2015, Paramesha et al., 2013). Large neural studies more often use fixed train/dev/test splits, as in Amazon multilingual reviews, PerSenT, and Urdu benchmarks (Barnes et al., 2020, Bastan et al., 2020, Irum et al., 23 Jan 2025).

Empirical comparisons are highly dataset-specific. On Pang and Lee’s RT dataset of 10,662 processed snippets, HWW2V improved every evaluated classifier relative to its single-feature baselines, with the best result of 0.796 accuracy from SVM with RBF kernel (Stalidis et al., 2015). In short-text three-class sentiment, supervised weighting metrics improved over binary baselines, but no single metric dominated across Twitter, restaurant, and laptop reviews (Hamdan et al., 2016). Cross-domain SentiWordNet-based classification yielded prior baseline accuracies ranging from 56.77% for a term-count approach to 68.63% for an averaging approach at threshold 0, while the optimized weighted dataset produced “considerably lower” 10-fold cross-validation performance than single testing on the whole weighted dataset because of overfitting and noise (Paramesha et al., 2013). This suggests that score comparisons across studies are methodologically unsafe unless domain, label space, and validation protocol are aligned.

A final issue is model selection under multiple operational criteria. CPC-CMS formalizes this by weighting accuracy, precision, recall, F1-score, specificity, MCC, Cohen’s Kappa, and optionally efficiency through Cognitive Pairwise Comparison and a weighted decision matrix

$v_i$4

On three social-media datasets, ALBERT was best when efficiency was excluded; when runtime was included, XGBoost became best in one case, Random Forest in another, and ALBERT remained best in the third (Li et al., 18 Jul 2025). The same work reports dramatic runtime differences, such as 0.324 seconds for Bernoulli Naive Bayes versus 12041.739 seconds for ALBERT in one dataset (Li et al., 18 Jul 2025). The practical implication is explicit in that study: no single classifier always performs better once time consumption is treated as a criterion (Li et al., 18 Jul 2025).

Across the literature, several limitations recur: domain mismatch, dependence on parser or tagger quality, lexicon coverage gaps, overfitting under small-data cross-validation, translation noise, difficulty with long documents and many entities, and the absence of gold supervision for sentence importance or target-focused evidence selection (Paramesha et al., 2013, Barnes, 2019, Choi et al., 2021, Bastan et al., 2020, Irum et al., 23 Jan 2025). At the same time, the accumulated evidence supports a stable conclusion: document-level sentiment classification improves when representations encode more than raw lexical presence—whether through weighted semantics, discourse structure, sentence importance, target awareness, or explicit multi-criteria model selection (Stalidis et al., 2015, Bhatia et al., 2015, Choi et al., 2021, Li et al., 18 Jul 2025).