CEO CCO Global
Share this content
Musk Bezos

The Tesla vs Amazon stats that highlight all of NPS’s flaws

by

Does an analysis of the varying fortunes of these two mega companies in 2019 prove that ranking an entire organisation by a single number is misleading?

8th Apr 2021
CEO CCO Global
Share this content

Let me paint a picture: It’s January 2019, and I’m in the wake of a discussion/debate with some Net Promoter Score (NPS) experts about the rationale for companies investing in NPS.

I’m undertaking a little in-depth research to try and validate my arguments when I come across this tidbit of information that throws everything straight into perspective:

Tesla = NPS score of 98.

At the time, Tesla had never made an annual profit (to that point – apparently has now, with some contentious accounting measures enacted). It was making cars with a vast number of quality control issues and had no centralised CX/CS process to hammer out those issues with its customers (unless you count tweeting Elon to ask him for help and/or praise him for his efforts).

Amazon = NPS score of -26.

Again, at that point in January 2019, Amazon was the first/second most valuable company in the world, depending on the day. It was generally loved by a vast majority of its customers. It had CX woven through and baked into the entire organisation. 

Yes, I appreciate, there’s some bias in my explanations there. But the crucial aspect is the numbers – how can the world’s most (or very close to most, at the time) valuable company have a Net Promoter Score of -26, whilst one of the world’s most financial vacuous companies be soaring above the clouds with a score of +98?

Building the bellwether

First, a bit of backstory. The root cause of my thoughts about NPS most likely emanate from having spent 15+ years bailing out large global consulting companies from their own messes with their clients – and knowing that the “processes” that consulting companies devise all have inherent and critical flaws – NPS clearly being no different.

It was created at one of the larger consulting firms in the world – Bain Consulting – back in 2003, by an executive named Fred Reichheld, when he introduced it in a Harvard Business Review article. It was initially introduced as “Net Promoter Score” – it is now billed “Net Promoter System” as it has evolved to become, some might argue, more bloated and considerably more expensive.

Amusingly, if rumours are to be held true, Fred deeply regrets what NPS has now developed into – a costly, unnecessary bellwether for the current declining status of customer experience in many large corporations – at a time they need good CX the most. 

One need only look at the on-going efforts of large companies who have “tied their wagon” to NPS being a saviour for their business – and it not working out that well (take a look at a certain two large businesses in the UK telecoms and banking sectors, for examples).

But still, the thought that “NPS is something we need for our business and our customers” persists and there are many, many more big businesses wedded to it.

And then we get to the Tesla vs Amazon revelation. How can these scores be representative of the businesses and their (at that time) vastly differing fortunes?

Flaws in the system

One of the main issues with most large, multi-national companies is their complete over-reliance on Net Promoter Score/System (NPS) as a primary measurement tool in determining the company’s perceived “success”. 

A recent University of Cambridge research paper contends that “NPS measurement does not necessarily correspond to actual behaviour”. The researchers advocate for a multi-dimensional model, rooted in big data, that taps behavioural and attitudinal data.

The researchers built a model based on several points of data that represented customer activity (purchases, purchase size, feedback, etc.). This allowed them to compare NPS with customers’ future purchases.

What they found was that many promoters were not promoters (in terms of being the most loyal and best customers), and many detractors were not detractors.

With their model, they eliminated the need for the passives category and split customers into either detractors or passives based on their behaviour.

Another set of researchers, led by Timothy L. Keiningham in 2007, published a paper that argued other metrics were better measures of customer loyalty than NPS.

In their research, they found that NPS was no better than ACSI.

One of the main issues with most large, multi-national companies is their complete over-reliance on NPS.

Other research out of Stanford University also found that other indicators are equal or better measures of customer loyalty and referrals than NPS. The most effective indicator their research found was how much a customer “liked” a company (take that for what you will, given that it is a very wide set of interpretations from customers of what the term “like” might be).

NPS, in its best form, is not necessarily “bad” when seen in its best light, but it is clearly not good enough to even be considered the “best”, nor worth all the effort expended for it -- despite being so widely used. It clearly does not make up for the overhyped, and overly expensive implementation and upkeep costs for something that reveals so little in true value.

Keiningham’s paper goes on to state that managers have adopted NPS because they believe it is superior and that this superiority is based on “solid science”.

The paper notes: “[O]ur research suggests that such presumptions are erroneous.”

The researchers posit a related risk of “potential misallocation of resources as a function of erroneous strategies guided by Net Promoter on firm performance, company value, and shareholder wealth”.

Both Keiningham and other researchers like Morgan and Rego have attempted to replicate the initial claims that NPS outperforms other metrics for customer loyalty.

Improving the customer experience?

So, as we’ve seen in our Tesla vs Amazon example, the main issues remain:

Mobilising an entire organisation to a single number, such is the benefit of the Net Promoter Score/System, can be misleading and can lead to behaviours overly focused on the score and moreover and ironically, completely miss the intended spirit of NPS: Improving the customer experience.

Arguably, banks and telecommunications businesses probably suffer most from an over-reliance on the stated “NPS number” to help model the past attempts at developing a customer experience strategy; and resulting required processes to drive potential success in customer satisfaction, across the scope of their entire retail and business banking divisions. 

It is a common issue for several global companies though – and it is one of the reasons that companies should be evolving away from using NPS as a primary indicator of the success of customer experience (as well as other factors as determining employee bonuses and other compensation formats – which has happened increasingly in recent years) and 0% detractors. Both result in an NPS of +40 which does not seem very logical.

NPS alone does not represent any reality of a business’ success at that given point.

The main thrust of the principal problem with NPS is that although it is a “simple number” that can be easily referenced to show a type of measurement within an organisation – it doesn’t show you how to fix the problems that are at the core of the problems, or even give you a rudimentary idea of when those base complications might arise. It therefore transforms itself into a vanity metric.

The overall NPS score is unlikely to drive any actionable insights, but individual scores, matched with the verbatim feedback from your customers (and other available customer data) is a key to making transformative decisions and progress within your organisation. 

If we continue to look at the stated example given – the wildly varying scores of Tesla vs Amazon in January 2019, it is clear that NPS alone does not represent any reality of a business’ success at that given point.

Businesses must develop a set of relevant customer-focused KPIs that meet all their needs across the range of their customers and effectively measure how customers are interacting with the various parts of the institution and how feedback is handled and processed. 

Each business should have a unique blend of KPIs that matter to both them and their customers. But that takes time, money, and the right set of people to figure out the best path. The thing is that if you invest wisely, the payoff is exponential to both the business and the development of the relationship with the customers. But instead, they continue to invest in lazy, useless things like NPS and wonder why it does not work.

The cost vs. actual value does not add up when the rubber hits the road. And as a final point, it’s worth asking this question:

When was the last time you heard someone say, "I want to buy/work/deal with the company that has the best NPS score..."

 

Replies (9)

Comments for this post are now closed.

avatar
By aki.kalliatakis1
08th Apr 2021 10:18

Hi Dan. I too have spent decades bailing out companies whose customers hate them. It's a great post, and no doubt a great debate will follow. Absolutely brilliant comparison between these two business behemoths. Two quick points...

One, the numerous consultants, MBAs, economists and scientists who advise or run companies think that everything must be measurable and logical . It makes them feel safe that the world is predictable. Not true: 80%+ of decisions are NOT made logically, but emotionally, (see Colin Shaw's and Rory Sutherland's work,) and these emanate from all-too-flawed and unreasonable human perceptions. Ignore these at your peril. You prove the point with the incredible and glamorous hype about Tesla and its founder, Elon Musk.

Two, The various forms of CX and customer loyalty measurement are expensive and time consuming, (as is all this talk of how "analysing big data will be your saviour,") and then you end up with a flawed result anyway. Many years ago, Tom Peters made the point: "Get out there and talk to your customers, dammit!" That, together with being and undercover customer in your own business, should give you the kind of valuable information that you can work with. As for big data... well, my big bank / satellite TV provider / internet company / mobile phone service supplier are utterly clueless about who I am and when my birthday is.

I love the evidence you have cited, and as a fellow marketing maverick, I think more executives should be listening.

Thanks (2)
Replying to aki.kalliatakis1:
Dan Collins
By Dan Collins
08th Apr 2021 17:18

Aki -- thanks for the feedback.

As to the second point -- about what Tom Peters espouses about talking to customers. I was doing that in 1996 -- I was working for a television network out of Atlanta as a 25 yr old. One of the things I insisted upon for the project I was working on was to understand the research they'd paid millions for -- but also talk to other potential sources (customers.)

It led me to hang out in the aisles of Toys R Us asking parents questions about them, their children, and their TV viewing habits. In retrospect, it could be seen as a bit.... how would you call it.... creepy?

that being said, between that and other conversations, I discovered a whole new audience and target for the project. It ended up driving an increase of 40% in the end. It is still why I "poo-poo" surveys and other traditional forms of gathering information -- particularly when the actual discussions reaped so much more valuable and actionable insights -- even if it was just a fraction of the data that the surveys did.

Thanks (1)
linkedin
By LinkedIn Group Member
08th Apr 2021 10:53

From LinkedIn Group CRM & Customer Experience Professionals member, Vincent Amari:

Any analysis using a single number is not an analysis of anything!

Thanks (1)
Replying to LinkedIn Group Member:
Dan Collins
By Dan Collins
08th Apr 2021 17:20

And yet funnily enough a number of global companies, and their executives, are more than comfortable following NPS as the "holy grail" -- when it is obvious it is not, and it bears no relation to the actual customers of the business.

Thanks (0)
avatar
By Ebenezer
08th Apr 2021 15:28

Feel this article is lacking a bit of balance here.

Yes I’ve taken the bait but I feel it’s an important discussion. Possibly should go into a deeper discussion about a) measures why we have them and for what? B) how we drive action from measures?

Perhaps one of The biggest flaws in NPS is also a positive..it’s open sourced so anyone can use it for free if they want. That means there’s been lots of so called experts misunderstand it and misuse it.

The principal problem as you describe is that NPS “doesn’t show you how to fix the problems that are at the core of the problems, or even give you a rudimentary idea of when those base complications might arise”- sales or cost metrics and many others on their own don’t give you that either should we chuck them out too?

It’s more of a risk metric than some realise it looks at likelihood which some people find most difficulties not a static thing, understanding the likelihood is meant to help motivate you to try and influence the situation.

You talk about things like NPS being useless and lazy but then say “ individual scores, matched with the verbatim feedback from your customers (and other available customer data) is a key to making transformative decisions and progress within your organisation.” Am I missing something here?

The metric, question or analysis should never matter to customers what matters to customer. Is what’s changed or not as a result of their feedback. People talk about recommending organisations all the time, TripAdvisor built a business on it, Airbnb grew with recommendations and Groupon with or without NPS was one of the fastest growing businesses ever because of recommendations from friends and family.

It’s oft misunderstood, NPS is not the outcome it supports you on your path to the outcome. In both your examples both organisations had growth that’s a commonality but looking at the score on its own to judge is not much use as oft stated by the people who created NPS.

there have been academic papers that have suggested NPS works too.

Sure it’s poorly applied and used, but why do boards keep using it? What do you say to the many organisations who talk about actual successes as result of properly engaging with NPS?

What’s better?

What measure on its own drives actionable insights?

The cost isn’t in NPS actually the cost is in making NPS work and in making NPS benefit the organisation. If NPS was easy to actually benefit from everyone would be winning and then perhaps maybe it wouldn’t be as valuable. Is there an open source method that has a similar success rate or case studies?

Thanks (4)
Replying to Ebenezer:
Dan Collins
By Dan Collins
08th Apr 2021 17:56

Ebenezer:

It is true that NPS is open source -- you don't have to pay money to Bain directly to use it. That is definitely correct -- but if you hire Bain, you know they'll insist on implementing NPS and trump up the potential of it -- along with dozens of junior consultants in tow -- and the client will end up overpaying for something that is essentially useless at the end of the day -- and that "day", if given a choice will never come for the client because it means the ability to charge fees is complete (I, amongst many others, know this from numerous examples of firsthand knowledge.) It is the reason why NPS morphed from a "score" to a "system" -- because a "system" is potentially never-ending. Branding FTW!

Enacting NPS across an organisation -- in any & every regard does cost an excessive amount of money though -- along with a commitment to extensive time & resources costs -- particularly if you call in the "experts" -- be it Bain or any other consulting resource.

Those related costs are considerably more than the metric is worth in terms of ROI -- or as a measure in regard to Customer Experience KPIs. Which is the premise of this article. (As mentioned on LinkedIn, this article is based on a longer article I recently published there.)

It is about the usefulness of KPIs at the end of the day -- to which NPS falls desperately short. Sales is a great KPI -- directly relates to revenue. Customer churn is a great KPI. Active issues, resolved issues, growth in revenue, client retention, average conversation rate, first contact resolution (if you want to get into call centre specific examples) even employee retention -- all great and useful KPIs to a business.

There are a number of great base KPIs to begin with -- and then another number that is specific to the needs of the customer and company depending on what they want to stress as "important" to them as a business. These vary from company to company -- but that is what companies need to focus on -- not a standard vanity metric such as NPS.

I had the same question posed to me a couple of weeks ago by a senior executive at a global company that is currently utterly devoted to NPS (and they continue to lose hundreds of millions of dollars a year) - "If not NPS, what should we use?"

The logical & right answer:

"There isn't, nor should there be one "holy grail" metric -- and NPS is definitely not that. What is important to you, important to the business, and important to both new & existing customers will all be different. One has to spend the time & effort to understand what "works" for the company. Does that take effort -- [***] yeah it does. But it is worth the time and effort -- like all good things -- CX or not -- the investment can be worth it if done well. It is not going to make the stock analysts happy at the time of quarterly or year-end reporting -- because there is no fantasy number they or a CEO can tout -- but I'd rather make a lot of money, cut costs and have happy customers that try to placate a stock analyst that doesn't understand your business. [***], it worked for Amazon/Costco/USAA/etc. hasn't it?"

Thanks (2)
avatar
By jimmy23
08th Apr 2021 16:50

Do we know what the NPS scores are at Tesla and Amazon right now? That would make for an interesting comparison, based on the 2019 scores highlighted in this article.

Thanks (1)
Replying to jimmy23:
Dan Collins
By Dan Collins
08th Apr 2021 17:26

Jimmy -- I checked yesterday on Tesla -- despite the obvious build quality issues -- they reside at a "37" in one measure. (One of the controversies that remain in the measurement of NPS is that there are a number of different sources that give a number of different numbers -- and it all depends on who you want to believe it seems.)

There are rumours as to why it has regressed -- but I don't have solid proof as to why just yet - so we left it out of the article.

As of a couple of minutes ago, Amazon is now "25" on one measure -- and "69" on a different website. Not sure as to the discrepancy -- but I'm sure Amazon would think the second one is "nicer".

Thanks (1)
Rob Lancashire, CEO, Opinyin - Customer Satisfaction Survey Software
By Rob Lancashire
12th Apr 2021 11:35

If you take a helicopter view of this, a business wants to get somewhere (their vision) and they have a map of what they think is the best route to get there (their mission and objectives). They have determined what they think is the best mode of transport (their business plan) and the mile markers they need to pass to get there (their KPIs). Throughout the journey they have multiple different levers they can to choose to pull or push to change things, which means they pass the mile markers at a different rate; however, if you don't know why things changed its a bit like driving with a blindfold on (I do not recommend this literally or metaphorically speaking).

Any KPI number like NPS, sales, profits, costs are all just mile markers which you can statistically argue about until the cows come in (remember 89% of all statistics are wrong and the other 12% are made up). In my opinion whether NPS is less more or less valid than other measures is academic unless you remove the blindfold whilst driving. I totally acknowledge the summary point of the article is exactly this and the foray into the value of NPS will stoke debate but there is a danger that the real issue of not asking why could get lost. Fred Reichheld has always said your not using the system correctly if you do not ask why. Maybe they should change the open source licence to make the follow up question mandatory?

As children we used to ask why a lot and yet as we get to be adults we seem to get scared of asking that for fear of looking like we don't understand. We all need to embrace our inner child more.

Thanks (2)