NEWSCOPE: Diversified News Retrieval

• NEWSCOPE is a two-stage retrieval framework that uses sentence-level clustering to balance topical relevance with diversity.
• It employs dense candidate retrieval followed by OPTICS-based sentence clustering to uncover complementary facets of news events.
• Empirical evaluations show significant gains in diversity metrics with minimal trade-offs in relevance across varied benchmarks.

NEWSCOPE is a two-stage framework for diverse news retrieval that enhances event coverage by explicitly modeling semantic variation at the sentence level. It addresses a central weakness of conventional news retrieval: results are optimized for topical relevance but often highly redundant, limiting users’ exposure to complementary viewpoints. Dense and lexical retrievers, including BM25 and DPR-like bi-encoders, tend to return content that reiterates the same narrative, entities, and framing; NEWSCOPE mitigates redundancy by modeling diversity at the sentence level and explicitly optimizing viewpoint coverage during re-ranking, yielding ranked lists that remain relevant while capturing distinct facets of the event (Tang et al., 27 Aug 2025).

1. Problem setting and conceptual basis

The system is formulated around an event headline $h$ and a large corpus of news paragraphs. The task is to retrieve a top-$k$ list $R = \{r_1, \ldots, r_k\}$ that satisfies two simultaneous criteria: relevance, meaning that each $r_i$ is topically related to the event, and diversity, meaning that the set $R$ collectively covers multiple complementary aspects or perspectives with minimal redundancy (Tang et al., 27 Aug 2025).

The central design choice is to treat sentences, rather than paragraphs or articles, as the atomic meaning units for diversification. Paragraphs often mix multiple aspects, so coarse article-level or paragraph-level embeddings can mask fine-grained differences. Sentence-level clustering isolates subtopics, viewpoints, and factual facets more explicitly. This makes the diversification objective more interpretable and allows re-ranking to operate on uncovered informational aspects rather than on undifferentiated item-level dissimilarity alone (Tang et al., 27 Aug 2025).

This design also clarifies what NEWSCOPE means by “comprehensive event understanding.” The system does not merely spread retrieved items apart in embedding space. It attempts to surface complementary information by identifying semantically coherent sentence clusters across candidate paragraphs, then selecting paragraphs that cover as many uncovered clusters as possible while preserving headline relevance. A plausible implication is that the framework is designed for coverage of event facets rather than for stylistic heterogeneity or source balancing in isolation.

2. Two-stage retrieval pipeline

Stage 1 is a dense, relevance-first candidate retrieval step. Headlines and paragraphs are embedded with a bilingual dense sentence/paragraph encoder referred to as bilingual-embedding-large; in experiments, a multilingual dense retriever, DenseRetr, fine-tuned on semantic textual similarity and natural language inference tasks in the Augmented SBERT family, is used. A vector index supports efficient nearest-neighbor search, and FAISS is cited as the library for large-scale dense similarity search. Similarity is cosine similarity in embedding space, and for each headline the system retrieves the top-$N = 100$ most similar paragraphs. These candidates are then forwarded to Stage 2 for diversity-aware re-ranking (Tang et al., 27 Aug 2025).

Stage 2 makes diversity explicit by operating at the sentence level. Each candidate paragraph $p \in P$ is segmented into sentences $s \in p$, and each sentence is encoded as a dense vector using the same family of sentence encoders as in Stage 1. Sentence embeddings from the candidate pool are clustered with OPTICS, a density-based clustering method that does not require specifying the number of clusters $k$. OPTICS is used to identify clusters by exploring the intrinsic density structure of the sentence embedding space, thereby separating dense regions of semantically similar sentences. The resulting cluster set is written as $C = \{c_1, \ldots, c_M\}$ and is intended to capture distinct informational aspects or perspectives of the event (Tang et al., 27 Aug 2025).

The re-ranking stage is greedy. Let $k$0 be the selected paragraph set, initialized as empty; let $k$1 be the set of uncovered clusters; and let $k$2 denote the sentence clusters present in paragraph $k$3. NEWSCOPE implements two re-ranking variants. GreedySCS is cluster-coverage-first and scores each paragraph by the number of uncovered clusters it contains:

$k$4

At each step it selects

$k$5

adds $k$6 to $k$7, and updates $k$8 (Tang et al., 27 Aug 2025).

GreedyPlus refines this by weighting clusters by relevance and adding a paragraph-level relevance term. For each cluster $k$9,

$R = \{r_1, \ldots, r_k\}$0

where $R = \{r_1, \ldots, r_k\}$1 is computed with bge-reranker-large. Paragraphs are then scored by

$R = \{r_1, \ldots, r_k\}$2

Selection proceeds greedily until $R = \{r_1, \ldots, r_k\}$3. The reported default is $R = \{r_1, \ldots, r_k\}$4, which provides a strong relevance–diversity balance; increasing $R = \{r_1, \ldots, r_k\}$5 favors relevance, with slightly higher F1 and lower diversity, while decreasing it favors diversity, with higher $R = \{r_1, \ldots, r_k\}$6 and $R = \{r_1, \ldots, r_k\}$7 and lower F1 (Tang et al., 27 Aug 2025).

3. Diversity modeling and evaluation metrics

A distinctive feature of NEWSCOPE is its use of three interpretable diversity metrics. These metrics are defined over the retrieved set $R = \{r_1, \ldots, r_k\}$8, where $R = \{r_1, \ldots, r_k\}$9 is the paragraph embedding from the Stage 1 encoder and cosine distance is $r_i$0 (Tang et al., 27 Aug 2025).

Average Pairwise Distance (APD, $r_i$1) measures internal semantic spread of the top-$r_i$2 results:

$r_i$3

Positive Cluster Coverage (PCC, $r_i$4) measures how many distinct sentence-level clusters are covered by the retrieved set, with positivity defined by paragraph-level relevance labels:

$r_i$5

Information Density Ratio (IDR, $r_i$6) measures the density of unique information units per sentence:

$r_i$7

Higher $r_i$8 indicates more unique cluster coverage per sentence, that is, less redundancy and greater information richness (Tang et al., 27 Aug 2025).

The paper argues that these metrics are diagnostically useful because they separate semantic spread, aspect coverage, and information efficiency. APD quantifies spread, PCC quantifies aspect or perspective coverage, and IDR quantifies information richness per sentence. This suggests that hyperparameter tuning can be made more transparent than in diversification methods that rely only on opaque pairwise penalties.

4. Benchmarks and annotation protocol

NEWSCOPE is evaluated on two paragraph-level benchmarks, LocalNews and DSGlobal (Tang et al., 27 Aug 2025).

Benchmark	Scale	Notes
LocalNews	103 events, 5,296 paragraphs	Google News “Full Coverage”; average 7.1 sentences/paragraph, 124.9 words/paragraph
DSGlobal	147 global news events, 7,532 paragraphs	Adapted from DiverseSumm; average 7.5 sentences/paragraph, 123 words/paragraph

LocalNews is built from Google News “Full Coverage,” aggregating multiple sources for the same event to mitigate source bias, with accesses from varied IPs, devices, and login states for stability checks. Paragraph segmentation is capped at 512 tokens with sentence boundary preservation. For each event, a one-sentence abstractive summary generated with GPT-4o-mini serves as the query headline $r_i$9. Paragraph relevance labels are obtained via three rounds with majority voting, with 98.7% agreement, and “outside full coverage” paragraphs are auto-labeled as irrelevant. A human review of 100 samples yielded 99% acceptance. The collection is primarily English reporting and spans regional, national, international, mainstream, independent, domain-specific, and regional media. Splits and licensing are not specified (Tang et al., 27 Aug 2025).

DSGlobal contains 147 global news events and 7,532 paragraphs and is intended for global event coverage at paragraph granularity. Its segmentation and annotation protocols are analogous to those used for LocalNews. A plausible implication is that the two benchmarks jointly test both local, source-diverse reporting and broader international event coverage.

5. Empirical performance, ablations, and efficiency

The baselines are BM25, DenseRetr, MMR, and DkMIPS. Relevance is evaluated with Precision, Recall, and F1-score; no nDCG@k is reported. Diversity is evaluated with APD ($R$0), IDR ($R$1), and PCC ($R$2) (Tang et al., 27 Aug 2025).

On LocalNews at top-10, DenseRetr obtains $R$3, $R$4, $R$5, and $R$6. NEWSCOPE with GreedySCS obtains $R$7, $R$8, $R$9, and $N = 100$0. NEWSCOPE with GreedyPlus obtains $N = 100$1, $N = 100$2, $N = 100$3, and $N = 100$4. The reported interpretation is large diversity gains, including gains of $N = 100$5 to $N = 100$6 points in $N = 100$7 and $N = 100$8 to $N = 100$9 points in $p \in P$0 over DenseRetr, with small F1 changes. On LocalNews at top-50, DenseRetr yields $p \in P$1, $p \in P$2, $p \in P$3, $p \in P$4, while GreedyPlus yields $p \in P$5, $p \in P$6, $p \in P$7, $p \in P$8, combining a substantial APD increase with comparable or slightly higher F1 (Tang et al., 27 Aug 2025).

On DSGlobal at top-10, DenseRetr yields $p \in P$9, $s \in p$0, $s \in p$1, and $s \in p$2, whereas GreedyPlus yields $s \in p$3, $s \in p$4, $s \in p$5, and $s \in p$6. At top-50, DenseRetr yields $s \in p$7, $s \in p$8, $s \in p$9, and $k$0, while GreedyPlus yields $k$1, $k$2, $k$3, and $k$4. Across both datasets, the pattern is consistent: diversity increases substantially, while relevance is either preserved or traded off only slightly (Tang et al., 27 Aug 2025).

The ablation study clarifies the role of each term in GreedyPlus. Removing the diversity term makes the method behave like relevance-only ranking: F1 rises slightly, but diversity drops sharply, for example on LocalNews top-10 where $k$5 becomes $k$6 rather than $k$7. Removing the relevance term maximizes diversity but harms F1, for example on LocalNews top-20 where F1 drops to $k$8 rather than $k$9. The reported conclusion is that both terms are necessary to maintain relevance and diversity. Formal significance testing is not reported, though improvements are said to be consistent across both datasets and retrieval depths (Tang et al., 27 Aug 2025).

The full pipeline averages about $C = \{c_1, \ldots, c_M\}$0 seconds per event: approximately $C = \{c_1, \ldots, c_M\}$1 s for Stage 1 dense retrieval, $C = \{c_1, \ldots, c_M\}$2 s for sentence encoding and clustering, and $C = \{c_1, \ldots, c_M\}$3 s for greedy re-ranking. APD requires $C = \{c_1, \ldots, c_M\}$4 pairwise distance computations, and OPTICS scales with the number of sentences in the 100-candidate pool; because Stage 2 operates on a small per-query pool, the reported overhead is modest. This supports the claim that the approach is practically scalable (Tang et al., 27 Aug 2025).

6. Interpretation, limitations, and relation to adjacent systems

A qualitative case study illustrates the system’s intended behavior. For the query “OpenAI CEO Sam Altman warns of AI risks, calling for careful regulation amid global competition,” DenseRetr returns near-duplicates focused on the same interview and quotes, whereas NEWSCOPE with GreedyPlus surfaces complementary perspectives including Russia’s AI posture, hallucinations and reliability limits, adoption and benchmarks, education policy debates, and critical views on generative models’ creativity and reliability. The example is used to demonstrate that sentence-level clustering can capture distinct aspects and that the re-ranking strategy can balance relevance and novelty (Tang et al., 27 Aug 2025).

The paper also states several limitations. Viewpoint representation in low-resource locales can be constrained by source availability, and OPTICS clustering reflects the available sentence distribution. The system may surface conflicting narratives or misinformation if the source pool contains dubious content, because the current method optimizes diversity without explicit credibility controls. Cluster granularity remains sensitive to OPTICS density parameters, even though dependence on a fixed cluster count is removed. Inter-cluster relationships are not modeled explicitly; clusters are treated independently (Tang et al., 27 Aug 2025).

Relative to prior diversification methods, MMR operates at item-level similarity and does not incorporate sentence-level aspect signals, while DkMIPS penalizes pairwise similarity but lacks semantic transparency and fine-grained control over what is being diversified. In this sense, NEWSCOPE’s novelty lies in explicitly connecting sentence-level clusters to paragraph selection. Related systems address neighboring but distinct problems. NOVA, for example, focuses on assessment of polarizing media coverage through target-dependent sentiment, named-entity topics, and a staged visualization workflow rather than event-level diversity-aware retrieval (Dasu et al., 2024). NCEXPLORER, by contrast, centers on OLAP-like roll-up and drill-down operations over KG-integrated news cubes for due diligence and risk management rather than sentence-level viewpoint coverage (Wang et al., 2024).

The future directions named for NEWSCOPE include multilingual and cross-lingual retrieval via multilingual sentence encoders, multimodal news retrieval with image, video, and audio descriptors, dynamic clustering over time to track event evolution, adaptive density thresholds and adaptive $C = \{c_1, \ldots, c_M\}$5, and human-in-the-loop curation with cluster summaries and credibility or source-reputation signals. Taken together, these extensions indicate that NEWSCOPE is best understood as a retrieval framework for coverage-oriented diversification whose current contribution is fine-grained, interpretable sentence-level modeling, rather than as a complete solution to source reliability, stance adjudication, or temporal narrative synthesis (Tang et al., 27 Aug 2025).