PMI Matrix Approximations with Applications to Neural Language Modeling (1609.01235v1)

Abstract: The negative sampling (NEG) objective function, used in word2vec, is a simplification of the Noise Contrastive Estimation (NCE) method. NEG was found to be highly effective in learning continuous word representations. However, unlike NCE, it was considered inapplicable for the purpose of learning the parameters of a LLM. In this study, we refute this assertion by providing a principled derivation for NEG-based LLMing, founded on a novel analysis of a low-dimensional approximation of the matrix of pointwise mutual information between the contexts and the predicted words. The obtained LLMing is closely related to NCE LLMs but is based on a simplified objective function. We thus provide a unified formulation for two main language processing tasks, namely word embedding and LLMing, based on the NEG objective function. Experimental results on two popular LLMing benchmarks show comparable perplexity results, with a small advantage to NEG over NCE.