Remember the last time you wanted to choose a restaurant? After selecting the criteria like location, price range and cuisine, you probably chose one with the highest star rating. A Harvard study revealed that a one-star increase in Yelp rating leads to a 5-9% increase in revenue. The same is true for the enterprise. Temkin Group report shows that Net Promoter Score directly correlates with customer loyalty and business revenue.
So how can a restaurant increase their star rating? And how can an organisation increase their NPS? The answer lies in reviews, customer comments, and answers to open-ended questions. But if you receive hundreds of comments, making sense of them becomes a difficult task. There are plenty of text analytics solutions, also referred to as Natural Language Processing or NLP approaches, that claim to have solved this problem. But in reality there is a disconnect between what such solutions currently provide and what customer insight professionals really need.
This article summarises our findings on what matters when it comes to analysing customer comments using algorithms and provides a guideline to evaluating text analytics solutions.
People’s comments should be the focus, not an add-on
Many companies believe that analysing people’s comments is optional or an add-on to tracking hard data like scores. But imagine being the owner of a restaurant with a three star rating, desperately trying to improve your ranking. Would you rather have 100 ratings that average in the middle, or 10 customer comments explaining the reasons behind ratings?
If you ever left a review yourself, you will know how much value lies in your comment. If you ever left a comment in a survey for a product or survey, you were genuinely hoping that the business reads it and takes it on board.
Three ways comments can improve customer experience
Customer comments can be used to improve customer experience in three different ways:
- Just as scores, comments change over time and can explain why there are rises and drops in the score. If scores stay the same, comments provide a richer insight.
- By looking deeper into the comments, you can find out who should be following up with the customer. For example, a customer could be complaining about an unresolved problem or wanting to cancel the service.
- If you have done any changes to your offering, for example you started to use a new recipe, did your customer actually noticed it and did it affect the score?
Reading people’s comments is the simplest way of learning these insights. However, over time most companies collect hundreds and thousands of comments, and only an algorithm can keep in memory all aspects of the analysis and how it relates to other data such as sales volumes. The analysis of customer comments manually is called coding, or auto-coding if it is performed automatically.
How to summarise comments using text analytics
Ideally, algorithms should be able to make sense of customer reviews, comments or survey responses in a systematic way, without bias, consistently and hopefully accurately. Our conversations with dozens of customer insight professionals revealed that in order for this to happen, a text analytics programme should encompass the following:
1. Capture many ways people may be referring to the same thing
Imagine you have paid for a newspaper delivered to your door. It rained. As you are unsticking the wet pages you are frustrated that you cannot read it. How many ways do you think there are to complain about a wet newspaper? There are dozens possibilities! And if an automated solution cannot capture them accurately, the importance of this issue may be misrepresented.
Many solutions use industry-specific dictionaries or the electronic English language knowledge base called WordNet. But customer comments are messy and synonyms are often specific to your business: company and product names, for instance. Even ‘paper’ and ‘newspaper’ are rarely synonyms outside of publishing. And the word pairs like ‘build’ and ‘buy’ could be either synonyms or antonyms depending on the context: real estate or software. An effective text analytics solution should be learning synonyms from the data itself, merging synonyms, misspellings, typos, autocomplete errors and paraphrases into one single concept, or theme.
2. Capture positive & negative attributes separately
In order to track the positive and negative variations of a common theme, e.g. good coffee/bad coffee, attributes need to be captured separately. This may be relatively easy when the attributes are clear antonyms, e.g. ‘fast service’ vs. ‘slow service’. But negation makes everything much harder.
For example, these sentences use the same nouns and adjectives to comment on two aspects of studying experience at a university, once positively and once negatively, resulting in a total of four themes a human coder decided is worth tracking separately.
Most automated solutions do not deal with negations. Those who do, simply reverse polarity by saying that “not bad” means “good”. But negation is a fascinating phenomenon of human language and has many different purposes:
So, ‘not bad’ does not necessarily means ‘good’, most likely it means a rather neutral statement. When dealing with negation, text analytics solutions need to use parsing in order to determine the focus and scope of what’s being negated. The next step is to actually merge negated statements with non-negated ones correctly. For this, you’ll need some sort of antonym detection. Only then, a solution can help accurately determine how many people liked or disliked a certain aspect of the business.
3. Capture emerging themes
A common approach to summarising feedback, even when done manually, is to use a static set of categories or themes. The first problem with this approach is that it reflects the bias of the person who created them. The second problem is that it is inflexible. Businesses are constantly evolving, as they refine their product offering, or respond to emerging competition. If you want to capture people’s reaction to these changes, you need a solution where themes can emerge over time.
A common solution to auto-coding comments is to apply a technique called supervised categorisation, where each new comment is compared to other comments tagged with each code. However, over time new themes can emerge, meaning that none of the existing codes will apply. As a result, the comments containing new themes will be put into the ‘Other’ category, which can grow over time. Having an ‘Other’ category is unavoidable, but as a rule of thumb, it should not be more than 20%.
4. Link to original for verification and action
There must be a clear link from a code, or theme, to the original comment. Context is king, as they say, and without context it is hard to interpret, understand and act upon the results. Some NLP solutions do not provide this option. Here are the three steps to effective comment analysis:
- Given a new theme of interest, pull out all comments related to this theme.
- Check theme accuracy: Do most comments match that theme.
- Decide on an appropriate action.
Action depends on the theme itself. For example, one company found that wet newspaper is a common reason for subscribers to cancel the delivery, so the action is twofold: initiate the use of a plastic wrap in rainy season and contact customers who left those comments to apologise and inform about the new measure.
3. Ensure transparency and ability to edit
Transparency in how the algorithm came to particular results is also important. Technology is not there to replace people, but to enable them to get to insights quicker. So ideally, customer insight professionals should be able to guide the NLP solution to find what matters to them and ignore what they already know.
Sometimes there is a wrong and right answer. For example, soccer world cup is in many countries the same as football world cup. But in other cases, it depends on the customer’s priorities whether they want to track rugby world cup separately from soccer/football world cup, or as the same thing. And they need to be able to make changes to how the system decided to do the grouping.
6. Work well on small datasets
Small datasets are challenging for data-driven algorithms, which build language models to understand people’s comments. A language model is a machine-readable dictionary that captures the meaning of words and phrases using word occurrence statistics. Such models can infer for example that King – Man + Woman equals Queen, using pure observations of how these words appear in sentences.
Commonly, text analytics solutions use Wikipedia or some other large repository of free text to build a language model, but fail to capture company specific terminology. A better idea is to repurpose a dataset that may not be particularly large, but is thematically more similar to customer comments.
7. Provide actionable insights
Finally, the result of automated analysis should provide information that’s not trivial and easy to act on. Let’s say, an algorithm analysed 500 comments of a software company and returned that key categories like ‘product’, ‘customer service’, and the name of the company. This is not insightful. Similarly, knowing that customer service has poor sentiment is not actionable.
Keeping this in mind, text analytics or NLP solutions can be evaluated according to this diagram.
On the one axis we have language units categorised by how actionable they are. On the other axis, we have trivial knowledge, suspected but needed verification, and finally new insightful knowledge. For example, we can easily guess which words will repeat in customer comments. These words will have zero meaning. Vast majority of solutions today simply capture general aspects and do not return any actionable results.
Ideally, a solution should return a mixture of themes, some of which should be insightful and actionable. In most cases only customer insight managers can judge if something is an insight to them or not, but an ideal text analytics solution should minimise the amount of time from receiving the data to being able to extract one or more valuable insights.