What are the main objections to NPS - and are they legitimate?by
Net Promoter Score has its fair share of detractors. So what's their beef with NPS? And are there legitimate reasons for their concern?
The Net Promoter Score is dead! Or are the reports on NPS simply fake news? Well, the image at the top of the article shows what I found with a simple Google search. So… is the Net Promoter Score dead? Here are the views of two important people:
"The reports of my death are greatly exaggerated" - Mark Twain, 1897
"Everything you read on the internet is true" - Miguel de Cervantes, 1615
Recently, I have been researching what those who do not support NPS have been writing over the years. Here are some of the objections I know about and what I think of each. If you have others in mind, let me know and I will provide my perspective.
‘Compound metrics’ are better predictors of revenue and market share trends than NPS’
Yes, this is true in many cases. It is possible to design and implement more sophisticated measurements that are better predictors and a number of them exist. Reichheld and Markey have never claimed otherwise. Reichheld’s original contention was simply that the recommendation question was the best single-question predictor among those studied, for the majority of industries.
The issue with compound metrics and the reason none have ever caught on is that they are extraordinarily difficult to explain. And because they are not common, every single communication about the results has to start with an explanation of how the metric works. Too boring. Audiences go to sleep, no matter what the value of the sophisticated metric is.
The issue with compound metrics and the reason none have ever caught on is that they are extraordinarily difficult to explain
The recommendation question is not always the best or most predictive single question to ask
Yes, that is true and Reichheld says so in his 2003 HBR paper. Indeed, he refers extensively to Enterprise Rent-A-Car in his paper and the question they ultimately used and that turned out to be a great predictor for them was ‘Please rate your overall satisfaction with your rental’.
When eBay implemented NPS their target customers were people making their living selling things on eBay. They found that when they asked the recommendation question they got answers like “No, I am never going to recommend to a friend that they should compete with me on eBay.” So they implemented “How likely are you to continue selling on eBay?’ and continue to use that question to this day.
Lots of other things affect growth more than NPS
That is certainly true in some industries. Overall, Bain and my own research suggest that NPS scores predict between 20% and 60% of revenue / market share trends, depending on the company. There are clearly industries where external factors drive 100% of the revenue and customer satisfaction will drive market share only.
Take gold sales. Revenue depends on the price of gold. Market share will partly depend on how customers feel about you compared to your competitors. Where there are monopoly providers of essential resources like electricity, customer sat has no effect on revenue, as should be obvious. You can the relevant information from Reichheld and Markey here.
NPS is easy to game
Yes, that is true, and the same is true of every other metric like CSAT or Customer Effort Score. Here is a blog I wrote about how to cheat a couple of years ago: https://customerstrategy.net/gaming-survey-results/
Companies publish fantasy numbers as their NPS score in their annual reports
I wish that were not true, but it is. The only number they should publish is a double-blind competitive NPS benchmark score. Some companies choose to publish numbers that correspond to some positively-perceived subset of their business such as a support desk that only a minor proportion of their customers actually use. Again, the same self-measurement phenomenon could happen with CSAT or other metrics.
NPS is not a good predictor of revenue
While I will go into specific papers about the topic below, I want to make a few general points first. Not all NPS numbers are equal. The only scores that are consistent and reliable predictors of revenue and market share trends are double-blind competitive NPS benchmark trends.
Double-blind means that the people answering the feedback requests do not know who is funding it. Ideally the people analysing the feedback don’t know either, though this second ‘blinding’ may be tricky to achieve.
If, for example, the only NPS numbers you have are feedback from a support desk survey, you should not expect it to predict revenue or market share at all unless yours is simply a support desk business. And of course if your numbers improve but your main competitor’s numbers trend is even better, you will lose market share. I emphasise share rather than revenue because some industries are in decline. If your score is better than your competitors’ scores you may simply decline more slowly in that industry.
We did deep research on this while I was at HP and I have posted the top-level results of the relationships we found between relative NPS trends and revenue in various places.
Since then, Dr. James Borderick of Micro Focus has done even deeper research on the subject across more than 30 global software vendors and once again proven the strong relationship between NPS and market share trends. I don’t know exactly what level of detail he has been allowed to publish as his work does of course include some confidential company data.
If the only NPS numbers you have are feedback from a support desk survey, you should not expect it to predict revenue or market share at all; unless yours is simply a support desk business.
Proof of the relationship between NPS trends and revenue / market share trends has never been published in a peer-reviewed journal
Not true at all. For example, here is one that Andrew Stephenson, Jana Fiserova, Geoff Pugh, and Chris Dimos had accepted for the British Academy of Management annual conference proceedings. It actually won best paper at their 2018 event. The data is from the UK-based multinational furniture retailer that has over $1B in annual revenue. It is quite a ‘deep’ read. To me, the essential point is their choice of using feedback gathered six months after purchase. This seems like a great way of getting a pure brand-level NPS number.
I would like to invite you to take the time it takes to read it, if you are at all interested in the subject. You will see that the relationship between NPS and revenue is established in a quite compelling way. The paper is here. (This should open with a completed search for the name Stephenson, and you can download the paper directly from the search result.)
I suppose you can tell I feel strongly about some of these objections while considering others to be valid. If you have more that you would like me to examine, please feel free.