R-Scorer: Ensemble Triple Scoring

• R-Scorer is a hybrid model that assigns an integer score (0–7) to triples reflecting how characteristic a profession or nationality is for a person.
• It employs a two-branch ensemble with SVR models that combine latent semantic features from word2vec/GloVe and PCA-compressed Freebase attributes.
• The system rounds ensemble outputs to the nearest integer, achieving up to 79.72% accuracy in predicting triple relevance for ranking applications.

to=arxiv_search.search 凤凰大参考 天天中彩票会json content='{"query":"(Chen et al., 2017)", "max_results": 5, "sort_by": "relevance"}' class='json' to=arxiv_search.search {Jsii ിക്കുംjson content='{"query":"(Brumer et al., 2017)", "max_results": 5, "sort_by": "relevance"}' class='json' to=arxiv_search.search 彩神争霸官网json content='{"query":"(Shiralkar et al., 2017)", "max_results": 5, "sort_by": "relevance"}' class='json' R-Scorer is Team Radicchio’s ensemble model for triple scoring in the WSDM Cup 2017. It assigns an integral relevance score in the range $\{0,\ldots,7\}$ to triples from type-like relations, specifically Profession and Nationality, where the score measures how characteristic a relation value is for a person. In the canonical example, $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$ should receive a higher relevance score than $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$ because “politician” more strongly represents Caesar. The system integrates latent semantic representations learned from text with explicit knowledge-base features, and was reported to rank third overall in the WSDM Cup 2017 test set with $79.72\%$ accuracy for predicting within two places of the ground truth score (Chen et al., 2017).

1. Problem formulation and motivation

The task addressed by R-Scorer is to compute a relevance score for triples of the form $$(\text{person}, \text{profession}, \text{profession\_value})$$ or $$(\text{person}, \text{nationality}, \text{nationality\_value})$$. The output is an integer relevance score in $\{0,\ldots,7\}$ indicating how representative the relation value is for the person. In the Cup’s primary accuracy metric, “predicting within two places” means the predicted integer score differs from the gold score by at most $2$ (Chen et al., 2017).

The motivation is explicitly tied to ranking. Accurate prediction of such triple scores benefits applications such as information retrieval and knowledge-base querying, because being able to rank all relations for a person can improve the user experience. The formulation is therefore not merely a classification problem over relation membership, but an ordinal relevance estimation problem over type-like relations.

A central premise of the method is that naïve bag-of-words or TF-IDF/co-occurrence approaches suffer from the vocabulary gap and cannot leverage semantic similarity. The example given in the system description is the mismatch between terms such as “violinist” and “musician.” R-Scorer addresses this by integrating latent semantic representations learned from text with explicit KB features. This suggests that the model was designed to combine lexical-semantic generalization with structured entity metadata rather than relying on either source in isolation (Chen et al., 2017).

2. System architecture

R-Scorer is a two-branch ensemble. One branch uses latent features, consisting of person representations learned via word2vec on a large Wikipedia-sentences corpus together with profession or nationality representations extracted from pre-trained GloVe embeddings. The other branch uses explicit features, consisting of structured, high-dimensional binary attributes from Freebase reduced to a compact representation (Chen et al., 2017).

Each branch is trained as a Support Vector Regression model with an RBF kernel to predict a real-valued score in $[0,7]$. An ensemble layer combines the two branch outputs, and a post-processing step maps the real-valued result to the required integer score. Separate SVR models are trained for profession and nationality, reflecting relation-specific distributions.

The architecture is therefore explicitly hybrid. The latent branch captures semantic proximity in text-derived embeddings, while the explicit branch exploits ontological and attribute-level KB signals. A plausible implication is that the model treats triple scoring as a regression problem over heterogeneous evidence sources, with the ensemble acting as a calibration layer between semantically induced and knowledge-base-derived signals (Chen et al., 2017).

3. Latent semantic branch

The text source for the latent branch is the Wikipedia-sentences corpus provided by the organizers, comprising $33{,}159{,}353$ sentences covering approximately $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$0 persons, with coreference already resolved by canonicalizing mentions to the person entity form. Preprocessing normalizes person mentions of the form $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$1 to the canonical entity token, lowercases all tokens, and strips punctuation. The example $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$2 is converted to “walter_damrosch” (Chen et al., 2017).

Person embeddings are trained with word2vec in gensim using CBOW mode, $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$3 dimensions, window size $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$4, and minimum word frequency $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$5; remaining parameters follow gensim defaults. For multi-word profession or nationality values, the system averages their constituent word vectors to obtain a single $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$6d vector. However, the word2vec vectors for profession and nationality values are replaced by $$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$7d GloVe vectors pre-trained on Common Crawl, while person embeddings remain learned from the Wikipedia corpus (Chen et al., 2017).

The reason for this substitution is particularly explicit for nationality. Nationality and country terms trained only from Wikipedia contexts tend to cluster by similar page contexts, so terms such as “Japan” and “China” can become too close in the word2vec space and therefore hard to distinguish. Replacing relation-value vectors with GloVe improves semantic fidelity, especially for nationalities. The system examines semantic proximity with cosine similarity,

$$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$8

but the actual predictive model is an SVR with an RBF kernel,

$$(\text{Julius Caesar}, \text{profession}, \text{Politician})$$9

which is intended to capture non-linear structure and to help bridge the fact that GloVe and word2vec vectors live in different spaces. For a given triple, the latent input to the word-vector SVR is formed by combining the person’s $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$0d word2vec embedding with the $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$1d GloVe embedding of the profession or nationality value (Chen et al., 2017).

4. Explicit Freebase branch

The explicit branch is built from the Freebase 2014-04-14 dump. Each person is described by approximately $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$2 binary attributes encoding ontological facts such as gender, occupations, nationality, and population-related tags. The raw feature matrix is described as very large, with $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$3 persons and $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$4 features, and many of these features are uninformative for the triple-scoring task because some are active for nearly all persons or otherwise irrelevant for distinguishing person–profession or person–nationality specificity (Chen et al., 2017).

To reduce dimensionality and suppress noise, the system applies Incremental PCA from Scikit-Learn with batch size $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$5, producing a $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$6-dimensional dense representation per person. This compact representation is then paired with the same $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$7d GloVe relation-value vector used in the latent branch and fed into a second SVR with an RBF kernel.

The design of this branch indicates that R-Scorer does not treat KB attributes as a symbolic rule base. Instead, it converts a sparse, high-dimensional binary person profile into a dense low-dimensional representation suitable for non-linear regression. This suggests an attempt to preserve salient variation in structured KB facts while avoiding the computational and statistical burden of the original feature space (Chen et al., 2017).

5. Ensemble learning and score calibration

R-Scorer’s two primary regressors are Word Vector Regression for the latent branch and Freebase Feature Regression for the explicit branch. Both are trained with SVR using RBF kernels and labels in $$(\text{Julius Caesar}, \text{profession}, \text{Author})$$8. Their outputs are combined in an ensemble layer in one of two ways: simple averaging,

$$(\text{Julius Caesar}, \text{profession}, \text{Author})$$9

or a trained linear combiner,

$79.72\%$0

with weights learned via cross-validation using a $79.72\%$1 split into an ensemble-training set and an ensemble-validation set (Chen et al., 2017).

The final step is calibration to the discrete score scale. The system clamps and rounds the real-valued ensemble output to the nearest valid integer in $79.72\%$2:

$79.72\%$3

The description refers to this as “Score Transformation” and notes that it removes the need to engineer a probability-to-score scaling function, which would be required if classification rather than regression were used (Chen et al., 2017).

A concise scoring procedure is also specified. Given a triple $79.72\%$4 with $79.72\%$5, the system obtains a word2vec person embedding $79.72\%$6, a GloVe relation-value embedding $79.72\%$7, and a PCA-compressed Freebase feature vector $79.72\%$8. It then predicts $79.72\%$9 and $$(\text{person}, \text{profession}, \text{profession\_value})$$0 with the two relation-specific SVR models, ensembles them using either equal weights or learned $$(\text{person}, \text{profession}, \text{profession\_value})$$1, and rounds the clamped result to an integer score (Chen et al., 2017).

6. Evaluation, comparative context, and limitations

The supervision available to R-Scorer is minimal: approximately $$(\text{person}, \text{profession}, \text{profession\_value})$$2 labeled tuples across both domains, with unlabeled data comprising about $$(\text{person}, \text{profession}, \text{profession\_value})$$3 person entities. The training sets include $$(\text{person}, \text{profession}, \text{profession\_value})$$4 unique persons and $$(\text{person}, \text{profession}, \text{profession\_value})$$5 unique professions for profession.train, and $$(\text{person}, \text{profession}, \text{profession\_value})$$6 persons with $$(\text{person}, \text{profession}, \text{profession\_value})$$7 nationalities for nationality.train. Evaluation uses Average Score Difference, Accuracy, and Kendall’s $$(\text{person}, \text{profession}, \text{profession\_value})$$8. One common definition of Kendall’s $$(\text{person}, \text{profession}, \text{profession\_value})$$9 is

$$(\text{person}, \text{nationality}, \text{nationality\_value})$$0

where $$(\text{person}, \text{nationality}, \text{nationality\_value})$$1 and $$(\text{person}, \text{nationality}, \text{nationality\_value})$$2 are the counts of concordant and discordant pairwise orderings of relation values, respectively (Chen et al., 2017).

Five-fold cross-validation showed different behavior for the two relations. For Profession, Word Vector Regression achieved $$(\text{person}, \text{nationality}, \text{nationality\_value})$$3, $$(\text{person}, \text{nationality}, \text{nationality\_value})$$4, and $$(\text{person}, \text{nationality}, \text{nationality\_value})$$5, outperforming the explicit branch and also outperforming the ensemble in Accuracy. For Nationality, simple ensemble averaging achieved $$(\text{person}, \text{nationality}, \text{nationality\_value})$$6 and $$(\text{person}, \text{nationality}, \text{nationality\_value})$$7, while each single branch had $$(\text{person}, \text{nationality}, \text{nationality\_value})$$8. On the WSDM Cup 2017 test set, Team Radicchio ranked third overall with $$(\text{person}, \text{nationality}, \text{nationality\_value})$$9, $\{0,\ldots,7\}$0, and $\{0,\ldots,7\}$1 (Chen et al., 2017).

The reported baselines were Random Guess and Majority Vote, both significantly worse. For Profession, Random Guess had $\{0,\ldots,7\}$2 and Majority Vote $\{0,\ldots,7\}$3; for Nationality, the corresponding values were $\{0,\ldots,7\}$4 and $\{0,\ldots,7\}$5. The component comparison further showed that Freebase-only regression was stronger than baselines but weaker than word-vector latent regression on Profession, while ensemble gains were clearest for Nationality (Chen et al., 2017).

Within the WSDM Cup 2017 ecosystem, R-Scorer belongs to a family of systems designed for type-like relation scoring but differs in its particular combination of semantic embeddings and explicit KB attributes. The Cabbage Triple Scorer emphasized neural embedding and nearest-neighbor propagation, reporting official results of $\{0,\ldots,7\}$6, $\{0,\ldots,7\}$7, and $\{0,\ldots,7\}$8 (Brumer et al., 2017). RelSifter emphasized KG-derived activity features from Wikidata and DBpedia, reporting accuracies of $\{0,\ldots,7\}$9 for profession and $2$0 for nationality in its best models (Shiralkar et al., 2017). R-Scorer’s distinctive contribution within that landscape is the explicit integration of word2vec person embeddings, GloVe relation-value embeddings, and PCA-compressed Freebase features in a regression ensemble (Chen et al., 2017).

The limitations noted for R-Scorer are closely tied to the design choices that made the system practical. Nationality embeddings learned only from word2vec can collapse countries with similar contexts; GloVe alleviates but does not fully solve this. Freebase features are sparse and noisy, and their usefulness may remain weak for less-documented persons even after PCA. Minimal supervision constrains calibration and ensemble weight learning and leaves overfitting risks. There is also a feature-space mismatch because person vectors and relation-value vectors live in different embedding spaces, which the RBF kernel only partially mitigates. The future improvements proposed include learning joint embeddings for persons and relation values, exploiting richer textual features such as entity linking and dependency or pattern-based signals, incorporating additional KBs, and refining calibration beyond rounding if more labels become available (Chen et al., 2017).