What is Small Data and why is it the new Big Data?

Some years ago, I noticed something interesting. When I asked innovators, company founders, and entrepreneurs about the breakthrough ideas that led to their killer brands, they didn’t tell me, as I might have expected, that their great ideas had emerged from a well-planned brainstorming session or as the result of years of hard work in the lab.

Rather, their breakthrough ideas came from seemingly insignificant behavioral observations they’d made while interacting with friends, family members, colleagues, or strangers. These key observations often occurred when least expected, revealing an unmet and previously unrecognised consumer need. These were the observations that became the foundation for entirely new, breakthrough brands.

It was a surprising, thought-provoking insight. After all, who would have thought that SnapChat, the social media app that allows the user to create photos with an ultra-short lifespan, was invented when the founder’s friend tried desperately to find a message containing a photo of himself smoking pot? Or that a failed insurance claim on a broken surfboard would lead to the invention of GoPro?

Where Big Data is all about drawing correlations, Small Data is about identifying causation. What has become evident over the years, is that drawing parallels between data and causation always verifies the Small Data hypothesis. And this might be where data mining has hit the brick wall until now, as the outcome never is better than the input. Nor does the input truly allow for human observations.

Where Big Data is all about drawing correlations, Small Data is about identifying causation.

With the introduction of Small Data, we’re hitting the next generation of Big Data, a counter-balance so to speak, where the data mining and research world systematically pick up human clues not necessarily captured via loyalty cards, search algorithms, transaction data, or quantitative studies. Rather, observational insights primarily obtained from interactions between humans give us all the information we need. These seemingly insignificant observations point out hypothesis for Big Data to explore and verify. Don’t get me wrong, this is not a matter of either/or, but merely an elegant combination of both worlds.

The issue is that in our data-obsessed world, we’ve been convinced that billions of data observations drive innovation. However, if you peel the historic onion, you’ll discover that the key to innovation is often a coincidental observation.

A counterbalance

Only a couple of years ago, you wouldn’t be able to attend a conference without hearing “Big Data” mentioned over and over again, nor could you have attended a board meeting at which Big Data didn’t dominate the agenda. Everyone was intrigued by the notion that a black box of data could magically produce deep insight into humans’ deepest needs, thereby revealing billion-dollar innovation opportunities. Like a kid in a candy store, every CEO proclaimed, “I want one of those!”

The term “Big Data” was supposedly coined by John Mashey in the mid-1990s over a lunchtable conversation at Silicon Graphics. Since then, experts have proclaimed Big Data to be some sort of ultimate crystal ball, a window into the consumer’s mind; but in recent years, many perceptive industry experts have began to conclude that the picture is incomplete. Big Data has its value, but something is missing. What’s needed, so to speak, is a counterbalance to Big Data.

Though it may be tiny, the potential impact of “Small Data” is huge.

Just recently a US bank found evidence of churn, a term referring to customers who move money around, financing their mortgages, or generally show signs they are on the verge of exiting the bank. Thanks for the analytics model, the bank promptly drafted letters asking customers to reconsider. Before sending them out, though, the bank executive discovered something surprising. Yes, indeed, Big Data had uncovered evidence of churning. But the churning wasn’t because customers were dissatisfied with the bank or its customer service. No. Most were getting a divorce, which explained why they were shifting around their assets. A parallel small data study would have discovered this in one day.

The missing piece in the puzzle, I’ve discovered, is tiny — and though it may be tiny, the potential impact of “Small Data” is huge. I’m talking about first-hand observations made in consumers’ homes, in restaurants, in night clubs, in sports clubs, when driving or on the phone. These seemingly insignificant, seemingly irrelevant observations, once connected, have the potential to identify the vital causation that Big Data has, so far, seemed to be lacking. What I’ve come to realise is that Big Data and Small Data are partners in a dance, a shared quest for balance.

So next time you hear the term “Big Data”, think “Small Data” too.

Martin Lindstrom is a leading brand strategist worldwide, his clients include Walt Disney Company, Nestlé, PepsiCo, BuzzFeed, and Lego. His new book, Small Data: The Tiny Clues that Uncover Huge Trends (John Murray Learning) is just out: https://www.amazon.co.uk/Small-Data-Clues-Uncover-Trends/dp/1250080681