Why 'Moneyball' customer metrics must replace the NPS mythby
Businesses must start looking in places and finding stats that don’t appear to interest others, and to turn that data into insights that create rich and contextual customer experiences.
In 1977 an aspiring writer and self-confessed fan called Bill James began self-publishing an annual book titled The Bill James Baseball Abstract.
It was a 68-page compilation of lesser known and loved statistics that James had observed from studying the box scores (a structured summary of the results, interestingly invented by an Englishman called Henry Chadwick in 1845) of the preceding season. He placed a small one-line ad in The Sporting News that garnered scant interest and posed no existential threat to the forests of the world. But somebody was listening – and learning.
Although it took another 20 years before the message got to first base, and James’s statistics and research, that he called Sabermetrics, had a measurable effect on the game. Now there’s no stopping him, with his newest yearly statistical extravaganza, Bill James Baseball Handbook, doing a land office business!
But earlier, many people, both inside and outside baseball, thought of him as an eccentric and misguided journalist, or just a bored number cruncher. That was until Billy Beane became the general manager of the Oakland A’s in 1997. The best-selling author Michael Lewis tells the story of Billy, the Oakland As, and his discovery of James’ statistical approach, in his excellent book Moneyball.
He hits an early and bases loaded home run when he states that Bill James found that, “The statistics were not merely inadequate: they lied. And the lies they told led the people that ran baseball to misjudge their players and mismanage their games.” The ultimate result was that these teams fared poorly in financial terms by overpaying for players who failed spectacularly and consistently, leaving fans disappointed, dissatisfied and planning other activities at World Series time.
This has had, and continues to have, significant parallels in business. By not measuring or clearly understanding what customers really need, many businesses mistakenly believe that they know their customers’ needs and wants better than their customers do, as businesses such as Blockbuster, BHS, Woolworths and Eastern Airlines can tell you. Well, they would tell you if they were still around!
Why single number customer metrics are flawed
In the world of business, and more specifically customer experience, the focus on single number metrics, whether that be NPS, CES or CSAT, has for many reached mythical proportions. But very much like baseball, they rarely tell the whole story, or even the right story, when it comes to measuring true business success through a customer’s eyes, and their resulting emotions.
In baseball, where Bill James started to look wider and deeper, one myth that he exploded was the concept of an error, where a player is judged and statistically disadvantaged, for not doing what he should have done. As an example, not coping with a ball that was hit right at him. James felt that this single metric didn’t take into consideration many other factors and statistics that could both mitigate the players’ supposed failure and provide far greater insight into what actually happened on any given play.
More importantly how to prevent it happening again. He added that “when numbers acquire the significance of language, they acquire the power to do all that language can do: to become fiction and drama and poetry.” In other words, by combining the cumulative effect of things such as environment, game situation, the player’s performance in the game so far, a far different picture emerges with all the colour, pathos and emotion that a single number can’t convey.
Trying to simplify the measurement of your customers’ experiences is often as unreliable and surprising as sporting predictions, election polls or a referendum. The downside of believing what people say they will do and what they actually do can have many politicians and business people going in entirely the wrong way politically and strategically. Using a single question or number, whether It relates to a customer’s willingness to recommend or the ease of the transaction may be mildly interesting but is sadly lacking in the kind of insight that businesses need to deliver consistent, memorable and differentiated customer experiences – and to stay in business.
Really understanding your customers’ needs and wants, the differences between them, and how they truly feel about your company from an emotional perspective are critical, non-negotiable elements in measuring what is important to them and their decisions to continue to do business with you. To paraphrase Bill James, it’s the language not the number that really translates into action.
Why a combination of metrics is important
To see how a Jamesian approach to customer engagement can work in practice our needs are well met by Stephen Hewett of iCustomer, creator of The Customer Needs Score (NetCNS) that captures each customer’s verbatim statements to indicate how well customers’ needs are met across three different dimensions:
- Emotion – how do customers feel when the need is not meet?
- Importance – how important is the need?
- Performance – how well does the organisation currently meet that need?
NetCNS can be calculated by establishing a weighted formula that represents a sum of the experience that relates to the need they have specified. If that need has been met or not and how the customer really feels which is seen as increasingly important metric in any customer experience evaluation.
One of the many well-used quotes in customer experience is about “remembering how people made you feel”. Even if its provenance is unclear, with a number of people being credited, there is no doubt as to the value of the sentiment whereby we can describe feelings with words that can be a far more accurate and valuable measure of how people feel about your business. Whether that’s positive – happy, relieved, grateful, or negative – unhappy, angry, disappointed. These can be charted on an emotional scale and have so much more meaning than trying to simply put a number on the experience without any context.
Feeling the love is at the heart of the eScore developed by Morris Pentel at the Customer Experience Foundation (CXFO). Simply stated the eScore combines Voice of the Customer (VOC) data with an emotion score (eScore) to arrive at a Behaviour of the Customer (BOC) metric, that is based on feelings and emotions and thinking about the actions, not just the voice, of the customer. This is a very interesting play on Bill James’ mantra of numbers acquiring the significance of language and a powerful descriptive language can far more quickly and accurately zero in on areas of customer pain and pleasure.
In those cases where a deeper, more meaningful long-term relationship is desired (Shouldn’t that be most of them?), a single headline metric is wholly lacking in emotional or personal context. Asking a customer whether they would recommend a company to their friends or colleagues (and surely never knowing if they have) is far from conclusive in identifying both current company opinions and future decisive actions.
In those cases where a deeper, more meaningful long-term relationship is desired, a single headline metric is wholly lacking in emotional or personal context.
I’m not alone in this view. A recent poll conducted by Marketforce showed that 66% of respondents believed that a combination of metrics, especially those featuring experiential and emotional values, will be the most widely used method in the next five years.
Understanding the relationship between customer needs, actions, attitudes, responses and value created will identify what really makes a difference in customer terms. It creates Positive Customer Outcomes (PCO) that can be articulated as descriptions, rather than just numbers, that can confirm what great and bad experiences feel like and used to stop, start or change organisational behaviours.
These are metrics based on a combination of values that are developed and used with customers as an effective and enduring measure of success in individual interactions and over the longer term. As it is far more definitive in both name and value, it also reflects positively on the employee or business process as it’s difficult to have a high PCO score without an equally positive employee performance. That in turn can be translated into increased sales, wallet share, customer retention, employee engagement and other more qualitative metrics that actually mean something tangible and can be measured and analysed for their effect on the overall business.
Uncover the real value and what counts most for customers
The early baseball statistics innovators realised that each event on the field had an expected run value and contributed to the overall performance of the team. This in turn showed how to account for a player’s performance by the number of runs scored. But how much each event on the field was worth was much harder to figure out. On drilling deeper, teams found that it contained rich seams of data that could provide answers that had never previously been recorded or investigated fully.
Similarly, in business, every action has a PCO value and the statistics that you need to consider, and combine will change depending on the type of business, the role of the contact centre and/or other interactions across various channels. But they’re worth digging for. As an example: rather than just figuring out success based on overall sales volume or individual purchases, we should look deeper. One organisation I worked with started to take a more in-depth look and went beyond the basics to ask some much more creative and illuminating questions.
- What is the ratio between store visits, web sessions or phone calls per £ of sales?
- How many customers does each customer service agent speak to for every £ of revenue?
- Does wallet share increase with longer calls or online sessions?
- What is the average call length of successful (sale) and unsuccessful (no sale) calls?
- Do customer spend more via digital channels or through assisted service?
- Does a customer buy more when they call early in the day or later?
- Does this depend on hold time, or how easy it was to get to the right place, or speak to the right person?
These may seem inconsequential, irrelevant or difficult to uncover, but in this world of excessive excitement caused by an unrealistic overdependence on often expensive Big Data, this data already exists, is available and at a low cost because you already own it or can quite easily get it.
As Billy Beane discovered, just because nobody else was interested in a certain player, didn’t mean they weren’t valuable. In fact, for him, as time went on and his quirky selections were vindicated, this made them even more potentially valuable.
Bringing data to life – turning insight into action
Maximising the value of your insight may seem obvious, but it’s clear from my own interactions as a customer that very little that I say or do, or how I use products or services, is ever used creatively to deliver a better experience or to achieve a strong PCO.
For whatever reason, many organisations continue to operate with beliefs and biases, many of which are long held, honestly formed, but fatally flawed and operationally inaccurate when used to evaluate performance and determine customer needs and preferences. As the grandfather of customer experience, Heraclitus, once remarked, “The only thing that is constant is change.”
Billy Beane showed that by continuing to expand his statistical view, he found traits and player attributes that everyone else was overlooking. Even when it was clear that the Oakland A’s were onto something, many in baseball derided it as just luck and continued to believe that baseball statistics were the pure accomplishments of men against other men, or perhaps in business parlance, one company against another. But this was wrong, and as Bill James noted, “They are accomplishments of men in combination with their circumstances”. A subtle, but extremely critical difference.
When a customer wins, everyone wins
Innovative journeys into data and insight don’t just benefit customers. In his recent eBook Design Driven Feedback, Max Israel puts it succinctly when he describes the value in combining art with science to elevate enterprise feedback management.
“Over time, design-driven feedback has come to encompass not just how feedback feels from the point of view of a person asked to share it, but also the legions of employees to whom we direct it. In short, design-driven feedback is a movement dedicated to leveraging not just technology and analytical skills but also design and creative ones to make feedback deeply engaging for consumer and worker alike. It has the power to do both.”
These deeper and more personal insights revive and enhance how companies use customer experience data. By ‘humanising’ data, you can also inform staffing decisions, technology investments, new product introductions, and in turn develop huge advances in organisational engagement. This has a waterfall effect and can cascade over many parts of the business refreshing, cleansing and bringing fresh new life to help the business grow and prosper. This is vital as customers are changing fast and conventional wisdom may have had its place in the sun.
Businesses must take a certain leap of faith and start looking in places and finding stats that don’t appear to interest others, and to turn that data into insights that create rich and contextual customer experiences. Much as we hear about ‘intangibles’ among top athletes, there are additional layers of creativity, innovation and personalisation that go beyond just connecting with customers, as there are connecting bat to ball. So, don’t slow down or stop swinging for the fences.
As Babe Ruth said, “Never let the fear of striking out get in your way.”
Gerry Brown, is on a mission to save the world from bad customer service. He helps businesses save customers at risk of defecting and breathes life into their customer service operations and customer experience strategy. Gerry has provided organizational leadership on people development, business transformation, customer engagement and...