Introduction
This era is not just the era of raging the internet but more specifically that of the ‘Social’ aspect of the Internet. By ‘Social’ we mean the ability of the Internet to connect, network, and influence user behavior. If we think back, the last time we bought some gadget or made any purchase what was a significant influencer in our consideration phase apart from the traditional buying decision parameters? I bet you checked back Customer Reviews or maybe looked at the Facebook Product Fan Page comments.
Unravel beyond the usual
Decisions made nowadays are highly guided by the user reviews/FB comments/tweets or any type of qualitative information easily accessible on the Internet. Be it for e-commerce sites like Amazon and Best Buy or Travel Portals like Trip Advisor, the overall sentiment expressed in the reviews impact decision-making, and believe it or not, they are a gold mine of information that can give significant signals to improve the metrics of a successful outcome. Hence, ignoring such qualitative signals clearly removes the ability to understand the brand sentiment from those who really matter, i.e., the end consumers.
How to mine the data?
We have already stressed on the magnitude of the importance of the qualitative information; let’s briefly dive into how to mine such data. The focus won’t be on the tool used to mine, rather our experience and methodology that we have deployed end-to-end.
Step 1: Data collection and cleaning
As a pre-requisite to any modeling exercise, data cleaning and preparation is a must. We once had the task of mining Twitter data and using relevant keywords for a client, but the exercise still resulted in some unrequired tweets that were extracted. This entailed the exercise of removing the noise (the irrelevant tweets) from the signal. Generally supervised techniques like Naïve Bayes, SVM, or K nearest neighbors do the trick of categorization of the input data set (the models have been pre-trained).
Step 2: Data classification
Once we have prepared the data, the requirement could be to classify them based on the ‘category’ of the qualitative data. In one of our projects for an apparel company, the objective was to first categorize the comments into different buckets like, is the comment talking more about durability or fit or size issues, etc. This step is similar to Step 1 when it comes to the methodology.
Step 3: Sentiment mining
For sentiment mining, there are various standard approaches that are deployed ranging from naïve methods to advanced deep learning recursive models (as shown in the adjacent diagram – not that the latter is necessarily better). Here we will share one of them that we have implemented with great success.
The first step in sentiment mining is to understand that the same word can be used in various connotations, for instance, the word ‘like’ can mean ‘affection’ OR ‘similar’ OR ‘draw attention towards something’ OR ‘as if’, etc., and this is just one example. Consider how many permutation and combinations each word would possibly have, and it’s definitely not intelligent to sit, read, and understand the crux of each word, for 1000s of sentences. So what’s the solution? A machine has to do it. How does a machine understand what exactly is the user conveying when he uses a particular word? Well, researchers have developed algorithms after extensively training them over millions of sets of documents to understand the position and meaning of every word based on the part of speech (POS tags). Example: Battery (NNP) life (NN) is (VBZ) pathetic (JJ); where NNP stands for Noun Proper, NN for Singular Noun, VBZ for third person singular present, and JJ is adjective. Most statistical tools like R, SPSS and SAS are very capable of doing this.
Based on the POS tags there are various add-ons that, given a word with a determined POS tag, tell the sentiment (obviously they can be argued and hence the ones with the greatest probability are returned just like in the case of determining the POS tag). But when thought deeper single words may be insufficient to understand the overall sentiment for example in a sentence like, “I hoped the battery would last long but, unfortunately, it didn’t”. In such cases of the negation of a positive aspect, we had to devise a custom code that neutralized such negations to proper sentiment (this case it would become negative), something like diagram 1.
To conclude
With the hard work done what remains is to articulate the outcome of the mining exercise meaningfully to the client: highlighting issues that have a significant share of mentions that are impacting them adversely; stressing on what exactly are the complaints about (for instance, extracting adjectives from the data set of negative comments). These can make your efforts impactful and actionable.