How to turn your customer data lake into a manageable flow of insights

We’ve been talking about data overload for a long, long time. A montage of my conference presentations over the past decade or more would surely feature various photos of people being overwhelmed by customer data. You’ve seen those pictures – you could probably even model for one! And the funny thing about all that overwhelming customer data is that when we don’t know what to do with it or how to interpret it, we often turn to our CX programmes, where we find – wait for it – more data.

Not only is the volume of data getting ever closer to infinite, but we have an added challenge: now customers know we have this data. Blame GDPR standards for bringing it to everyone’s attention, but the pressure is on. Today, customers might let you collect their data, but they know you’re doing it. So if you do collect it, but don’t use it to visibly serve customers better, you’ve failed in your customers’ eyes.

So, can we solve this problem of data overload? Yes, we absolutely can. The problem we face is one of separating signal from noise. 

The End Game: Engineering a stream of data

I’ll start by saying that I am not a huge fan of the dreaded 'data lake', but despite the insights it has long promised to deliver, in practice, the data lake is where data goes to stagnate. The impulse behind the data lake is a good one – gather what we know, put it all in the same place so that we can extract the insights we need. The problem is that it doesn’t work. We end up fishing around for insights, but instead of finding the answer to our questions, we find a moldering old wellie on the end of our line.

Instead, what if we created a stream of data? Not a rushing river, mind you, but a controlled and targeted flow so that we can get that data to where it’s needed. I think of it as directing the water – and just the right amount of it – to the parts of the garden that need it. Some plants will need a constant supply, while others do better when they get a little less. It’s much more effective than putting a lake out past the garden fence and hoping that will do the trick.

This brings me to the four key steps I promised at the top of the page….

and I argue that solving it requires us to take four key steps:

1. Identifying the questions we need to answer.
2. Capturing the data that will help us do it.
3. Mapping the data we have to illuminate those original questions.
4. Making sense of the data so we can put it to good use.

1. Begin at the beginning - with a hypothesis

One problem with our traditional approach to customer data is that we don’t give the data enough structure. We need to get better about deciding up front what questions we are trying to answer about the customer experience.

Think back to the old (still sound) advice about designing surveys: if you aren’t going to act on the knowledge you gain, don’t ask the question in the first place. For data, we can modify that: if it’s not going to help you understand how the business can better meet customer expectations, don’t focus on it. You might need to know, for example, how quickly your solution needs to deliver value in order for the customer to become a Promoter. Or you might need insights into how communication affects engagement with your professional services organisation. You know your business, so begin by identifying what the signal should signify.

As a brief aside, I don’t want to minimise the complexity of this step. It’s not something you can accomplish in an afternoon, and it might require quite a bit of iteration over time to get right.

2. Capture and frame your data – in a triangle

Once the questions and hypotheses are in place, it’s time to tackle the challenge of identifying and capturing the data sources you need to understand customers’ experiences. Think about it in this simple framework: to understand customers, we need to know three things: who they are, how they think and feel, and what they do.

• Who customers are is fairly static. You’ll find this information in something like an account or contact record and it won’t change often. You can understand your customer according to what products they own, their service level, or their location, to list a few examples.
   
• How customers think and feel about you gets uncovered primarily by surveys. There are many challenges with our traditional approaches to surveys, as I’ve written about before. But once we navigate around those challenges, surveys regain their value – and then some – for this particular purpose.
   
• What customers do, of course, is indicated by our operational data. It’s a rich, largely unmined source of insight and it gains extra power when we put it in this three-part framework. Keeping these three elements in mind helps us winnow through the vast stores of potentially useful data and focus on what serves our purpose: understanding the customer experience.

4. Build a data map that embraces silos

It seems we’ve been battling organisational silos for even longer than we’ve been overwhelmed by data. We haven’t quite won the fight to break down silos, and it’s a noble fight in many ways, but I argue that we should in fact start by embracing the silos, because that’s where the data lives. Each team (or function, or … dare we say it … silo) has a rich set of operational data that they use to do their work efficiently and well. The problem isn’t that team members have the data, it’s that no one else does. That’s where mapping comes in.

Let’s take the example of operational data indicating a build-up of support cases. This might tell the support team to staff up, or to clarify its customer-facing help documents, and that’s critical, actionable info. But think what other teams could do with the same data, if it were presented to them in the right context, or map.

• The product team might fix or enhance some functionality.
• The marketing team might delay an upsell campaign that assumes satisfaction with a given product feature.
• The sales team might get better at clarifying expectations.
• The customer success team might focus its training efforts.

But to avoid overwhelming all teams with all data (that familiar problem), you need to make sure that the operational data is mapped.  How to map it? First, in relation to who the customers are and what they think and feel, so you know what matters. And second, in relation to the customer journey, so that you know how much a bit of data or insight matters relative to everything else.

4. Make sense if it all with a little help from a machine

No matter how charming the metaphor or how simple the framework, make no mistake: there’s still a lot going on in your customer data. Every human brain I’ve ever encountered, especially my own, needs an assist to understand what data tells us, and that’s where artificial intelligence, or AI, comes in.

Humans tell great stories. This wonderful skill serves us well, until it doesn’t, such as when it prompts us to see patterns and stories in the data that just don’t exist. The machines that support AI are terrible at telling stories, but they are great at ruling out the patterns that our storytelling minds try to see.  

Lets make sure that we use the skills of both humans and AI to get the best results. Firstly, as humans, we create structures and hypotheses for understanding data, respecting (but not blindly following) our organisational structures and drawing on our understanding of our business and industry context. Then, we can apply AI to the data that lives in those structures, for the purpose of sorting out which data combinations deliver meaningful insights, whether they challenge or confirm our prior understanding. Finally bring the human back, to bring their understanding and storytelling capability to verify and activate the right solutions.  From that brilliant combination, we separate the signal from the noise.

Conclusion: A data stream to nurture, not flood

I hope it’s clear: I’m in favour of data! But we generally find in business that we have too much of a good thing, and that can be worse than a shortage. With all this context in mind, let’s revisit the steps for rechanneling the insights hiding in your data lake into manageable flow that you can direct where you need it and make your garden grow.

1. First, figure out what you need to know. If you work at it, you’ll find you can put some clear parameters around the questions you’re looking to your oversupply of customer data to answer.
2. Next, identify the data that helps answer those questions. It might already be there, or you might need to capture something new.
3. Then, create a triangular map of your data that ties together who your customers are, how they think and feel, and what they do.
4. Finally, get a little help from machine learning to help you make sense of it all, so the next best steps become clear.