Multi-sentence Scoring

• Multi-sentence scoring is a framework that assigns numerical scores to sentence pairs to evaluate semantic relationships across various language tasks.
• Models using IR baselines, embedding averages, RNNs, CNNs, and attention mechanisms demonstrate improved transferability and performance in ranking and retrieval.
• Robust evaluation with multiple random seeds and confidence intervals ensures reliable benchmarking across diverse natural language understanding datasets.

Multi-sentence scoring refers to the set of models, algorithms, and evaluation protocols that assign real-valued or categorical scores to pairs or groups of sentences, capturing their semantic relationships and supporting the solution of diverse natural language understanding tasks. Rather than focusing on single-sentence classification, multi-sentence scoring tasks consider functions of the form $f_2(s_0, s_1) \in \mathbb{R}$, evaluating relevance, semantic similarity, entailment, or other relationships between sentence pairs (or, in general, multiple sentences). This paradigm underlies a unified framework encompassing answer sentence selection, semantic textual similarity, next utterance ranking, paraphrase detection, and recognizing textual entailment, and serves as a critical building block for broader text comprehension and information retrieval systems.

1. Unified Formulation of Multi-Sentence Scoring Tasks

A central insight is that a wide array of NLU problems—including answer sentence selection, semantic textual similarity, next utterance ranking, RTE, and paraphrasing—can be reformulated as multi-sentence scoring tasks by learning a general function $f_2(s_0, s_1) \in \mathbb{R}$ that conveys the degree and type of semantic relationship between two sentences (Baudiš et al., 2016).

The unification is formalized as follows:

• Single-sentence scoring: $f_1(s) \in [0,1]$
• Pairwise sentence scoring: $f_2(s_0, s_1) \in \mathbb{R}$

This abstraction enables a common modeling framework applicable to retrieval, matching, ranking, and inference tasks. The empirical implication is that models—and even trained model weights—can often be transferred or reused across tasks traditionally viewed as distinct.

2. Model Architectures and Baseline Methods

Multiple neural architectures and baselines have been implemented and evaluated for multi-sentence scoring:

a. IR-Based Baselines

Information retrieval metrics such as TF-IDF and BM25 offer strong baselines. Here, one sentence is treated as the query and the other as the document. These models leverage weighted word overlap, with BM25 systematically adjusting for term frequency saturation and document length.

b. Embedding-Based Neural Baselines

Two principal techniques:

• “Avg” method: Computes the mean of pre-trained word embeddings (e.g., 300d GloVe), projects each sentence to a common vector space via a learned matrix ($E_i = \tanh(U \cdot \mathrm{avg}(e_i))$), and compares using either an MLP or a dot product.
• Deep Averaging Networks (DANs): Extend the avg approach by adding multiple dense layers (typically with ReLU nonlinearity) between the averaging and projection stages, capturing non-linear interactions.

c. Recurrent Neural Networks

Bidirectional GRU networks are employed to process each sentence, with 2N memory units (half forward, half backward) and aggregate the final hidden states into a sentence embedding. Dropout rates up to $p=0.8$ are applied at both input and output to regularize models, especially given the moderate size of many benchmarks.

d. Convolutional Neural Networks

Multi-channel CNNs are used with filter widths 2, 3, 4, 5. For longer filter widths, the number of filters per channel is typically halved. Sentence-wide max pooling summarizes activations, after which the representation is projected, mirroring the RNN setup.

e. RNN–CNN Hybrid Models

An RNN (e.g., a Bi-GRU) first contextualizes the sentence; the sequence of RNN outputs is then fed to a convolutional layer. Pooling is used to “crisply” select the most salient subsequences, achieving robustness and selectivity in the sentence vector.

f. Attention-Based Models

The “attn1511” approach links the representation of one sentence (e.g., CNN output of $s_0$) directly to the token-level post-RNN representations of the other ($s_1$) via learned attention weights: attention scores are computed across tokens as weighted sums followed by a softmax, amplifying the contribution of tokens deemed most relevant for the pairwise comparison.

3. Robust Evaluation Protocols

The stochastic nature of neural training—random initialization, dropout, mini-batch ordering—can make single-run performance unreliable. To address this, the evaluation protocol involves:

• Averaging over 16 random seeds/runs.
• Reporting 95% confidence intervals using Student’s t-distribution.

Task-specific evaluation metrics include:

• Ranking (e.g., Answer Sentence Selection, Next Utterance Ranking): Mean Average Precision (MAP), Mean Reciprocal Rank (MRR).
• Semantic Textual Similarity: Pearson’s $r$.

The central formalism is reinforced through concise function types: $f_1(s) \in [0,1]$, $f_2(s_0, s_1) \in \mathbb{R}$.

4. Dataset Contributions and Cross-Task Benchmarks

Addressing the prevalence of overused or too-easy evaluation datasets, new baselines such as yodaqa/large2470 and wqmprop are introduced for answer sentence selection, claimed to be substantially harder than the traditional “wang” dataset.

Empirical findings demonstrate:

• BM25 and TF-IDF remain strong baselines, but attention-based and hybrid architectures consistently deliver superior MRR and MAP on answer selection.
• On the Ubuntu Dialogue dataset (challenging due to long, informal sequences), the RNN–CNN hybrid achieves a new state-of-the-art, outperforming memory networks and confirming the practical relevance of unified modeling.

5. Unified Framework and Transferability

An open-source framework supporting modular experimentation—comprising “dataset-sts” for integrating multiple datasets, “PySTS” for standardized data loading, and “KeraSTS” for constructing deep pairwise models—is introduced. This design enables:

• Plug-and-play model and task components.
• Systematic benchmarking of multi-task reusability of trained sentence encoders, including transfer from next utterance ranking (e.g., Ubuntu) to textual entailment, paraphrasing, and other tasks.

The result is a step toward universal, task-agnostic models for text comprehension.

6. Significance, Impact, and Future Directions

Setting a new state-of-the-art on the Ubuntu Dialogue corpus is especially notable given the challenging sequence length and informal language—conditions commonly observed in deployed dialogue systems. The RNN–CNN hybrid, alongside transfer-learning-capable RNNs, underscores the potential for cross-task and cross-domain application.

A plausible implication is that unified sentence pair scoring, with carefully designed architectures and evaluation, enables robust multi-sentence interaction modeling, lessening the need for heavy task-specific supervision and manual engineering. The framework provides a rigorous empirical foundation for future research into universal semantic similarity and relationship measurement across open-domain and task-specific NLU settings.

7. Key Results and Claims

Empirical claims explicitly supported by the findings include:

• Attention-based and RNN–CNN hybrid models generally outperform both traditional IR and simple embedding averages on semantic pairwise tasks.
• On Ubuntu Dialogue, RNN and transfer-learned RNN–CNN hybrids yield state-of-the-art ranking results, outpacing memory networks under identical conditions.
• Statistically rigorous evaluation protocols (multiple seeds, t-distribution intervals) are essential for meaningful benchmarking given model stochasticity.
• An openly released software framework facilitates both reproducible research and practical transfer learning among NLU tasks.

In conclusion, multi-sentence scoring, as defined and operationalized in (Baudiš et al., 2016), provides a unified, effective, and extensible foundation for the systematic paper and construction of models for a broad range of text understanding tasks, offering both theoretical rigor and empirical superiority across major benchmarks.