Addressing Complex and Subjective Product-Related Queries with Customer Reviews

Abstract: Online reviews are often our first port of call when considering products and purchases online. When evaluating a potential purchase, we may have a specific query in mind, e.g. will this baby seat fit in the overhead compartment of a 747?' orwill I like this album if I liked Taylor Swift's 1989?'. To answer such questions we must either wade through huge volumes of consumer reviews hoping to find one that is relevant, or otherwise pose our question directly to the community via a Q/A system. In this paper we hope to fuse these two paradigms: given a large volume of previously answered queries about products, we hope to automatically learn whether a review of a product is relevant to a given query. We formulate this as a machine learning problem using a mixture-of-experts-type framework---here each review is an expert' that gets to vote on the response to a particular query; simultaneously we learn a relevance function such thatrelevant' reviews are those that vote correctly. At test time this learned relevance function allows us to surface reviews that are relevant to new queries on-demand. We evaluate our system, Moqa, on a novel corpus of 1.4 million questions (and answers) and 13 million reviews. We show quantitatively that it is effective at addressing both binary and open-ended queries, and qualitatively that it surfaces reviews that human evaluators consider to be relevant.

Addressing Complex and Subjective Product-Related Queries with Customer Reviews

The paper presents an innovative approach to addressing complex, subjective, and product-specific queries using customer reviews. The authors propose a machine learning framework, termed Moqa, which aims to identify reviews relevant to a given query, leveraging the vast corpus of consumer opinions available on platforms like Amazon. This fusion of community Q/A systems and traditional product reviews presents a practical solution to two prevalent issues: navigating large volumes of reviews and answering specific user queries.

The core of the Moqa system is a mixture-of-experts model that treats each review as an expert capable of voting on the relevance of answers to product-specific queries. By training on 1.4 million previously answered questions and 13 million reviews, the model learns a relevance function that surfaces pertinent reviews for new queries. This architecture enables addressing both binary (yes/no) and open-ended queries effectively.

Strong Results

The authors highlight the effectiveness of Moqa through quantitative evaluations showing that it surpasses current best practices for relevance ranking. The model achieved an accuracy of 76.8% on yes/no questions when applied to the electronics category, outperforming traditional techniques such as Okapi BM25+ and ROUGE-based scoring. More significantly, it demonstrated a reduction in error rate by 50.6% for open-ended queries when compared to these baselines. Further qualitative assessment through user studies confirmed that human evaluators preferred the relevance of opinions surfaced by Moqa over state-of-the-art alternatives, with a preference shown in 73.1% of cases.

Implications and Future Work

The implications of this research are twofold. Practically, it offers a scalable solution for online marketplaces and review aggregators to improve customer satisfaction by providing more relevant responses to consumer inquiries. Theoretically, it expands the scope of opinion mining and Q/A systems by demonstrating the utility of subjective data in understanding consumer queries.

Further potential developments of Moqa include integrating user-specific data to enhance predictions and exploring additional textual resources such as product specifications. Another avenue is systematically addressing product compatibility queries that frequently arise in domains like electronics, which could leverage this learning framework for enhanced accuracy. Additionally, handling multiple and potentially conflicting opinions could advance the model's capability to accommodate the multifaceted nature of complex queries.

In conclusion, the paper presents a robust machine learning model that aligns well with the needs of online retailers, enabling them to harness the vast information contained in customer reviews. As a novel interface between opinion mining and Q/A systems, Moqa represents a significant step towards more intuitive and responsive customer interaction tools.