Six steps to insight advantage in a Big Data world

In the knowledge economy, superior knowledge is the most compelling source of competitive advantage. Despite the self-evident nature of the above statement, few companies have made insight advantage an explicit strategic objective. Many will not even have been considered it a competitive dimension. So before outlining the steps required to achieve insight advantage, we should start with a definition. 

Insight advantage is the benefit gained from understanding better than competitors the complex environment in which the business operates, enabling quicker and more effective decision-making and resulting in superior revenue growth, profitability and return on investment.

The opportunity for insight advantage increases with the breadth and depth of data available, so the explosion of data sources and types created by the digital revolution gives added potency. Making the most of this potential requires a different mindset to traditional approaches to data management and analytics. Specifically, gaining insight advantage requires the following six steps.     

1. Focus on insight required to create and extract value

The first step is defining the insights that will add most value.  This is about asking the right questions – those that have the greatest impact on financial success. 

These questions aren’t the easiest ones to answer. In many cases they may not even have been asked. The importance of know-how – typically described as intellectual property – is widely recognised. But the importance of know-who, know-what, know-why, know-when, know-where and know-how-much have been systematically under-appreciated. It is this knowledge that enables a business to understand its customer and offer the right proposition, at the right time, via the right channel at the optimal price. This is fundamental to attracting, growing and retaining profitable customer relationships. And the same thinking needs to be applied to attracting, developing and retaining talented employees, sales channels, partners and suppliers.    

A business is an ecosystem of different stakeholder groups; and for the business to operate there needs to be a mutually acceptable value exchange between the business (as represented by the management team) and each group. The greater the insight that managers have into the intentions, priorities and latent needs of each stakeholder group, the greater the value that can be created for both sides. Superior understanding of what is valued most highly enables the creation of more compelling propositions; and the more compelling the proposition, the greater the value that can be gained in return. Colloquially speaking, insight increases the size of the pie that can be shared, to the benefit of all parties.

2. Investment in Big Data technologies is necessary for developing an insight advantage, but not sufficient

Analysts’ research suggests that large organisations will be making significant investment in Big Data solutions over the next 2-3 years. Given the reams of data generated by the ever increasing number of digital devices – online and mobile browsing behaviour, social media commentary, smartphone location data and sensor streams – the ability to ingest and digest huge quantities of data and transform it into organisational energy is a prerequisite.

But IT is only one element of capability, the others being processes, metrics, organisation design, culture and competencies. A holistic approach to capability development is required – and focusing on technology without investing in other components almost guarantees failure.

Some gaps in these other elements have been recognised - the lack of data science competencies, for example. The main reason for this gap is that data science encompasses such a wide range of skills – mathematician, statistician, programmer, data analyst and business analyst – so finding them all in one person is obviously a challenge. But advances in tools are enabling these roles to be separated into their component parts, meaning rare skills can be leveraged with less rare. Data preparation can be more easily separated from data analysis. User friendly, drag and drop visualisation tools enables data exploration and predictive model development by analysts with a grounding in statistics rather than programmers with advanced statistical qualifications.       

But that still leaves a number of elements to be addressed – the processes required for rapid iteration, both adding data sources and completing explorations; where analytical teams will sit in the organisation; how other teams will be affected – some positively, some negatively – by the replacement of expert intuition with predictive models; and finally how the culture of the organisation needs to change. Without a holistic approach to insight development that addresses gaps in all operating model elements, IT investments will not generate their predicted returns.   

3. Make a Cinderella of your data

If Pixar were creating a story line about data in business, it might go something like this.

Once upon a time data was unloved. Every day data stewards would strive to ensure quality was as good as it could be.  But there weren’t enough of them and they didn’t have the right tools because no one really cared. That was until reports drawn from the data weren’t showing what someone really important thought they should, at which point the stewards were shouted at for not being good at their jobs. There were some exceptions, but mainly companies which had to submit regular reports to the Royal Inspector and so had begrudgingly made investments in ensuring their data was complete, accurate and up to date.    

Then one day along came the Digital Fairy and she said “Data – you will be Big! And because of that you will have a shiny new carriage called Hadoop and you will have footmen that we shall call Data Scientists. And they will be able to predict the future and make it a better place.  And because of that you will go to the boardroom, where handsome chief executives will fall in love with you. You will be the subject of much hype and hope and everyone will live much more profitably ever after.” 

Flippant, perhaps. But anyone who has worked in data management, reporting and analytics over the last ten years will recognise the truth at its core. Operational (or small) data has been the poor relation of the IT world with insufficient governance and accountability for quality at Board level. And unless all data is valued, investments in Big Data will not deliver value and certainly not an insight advantage. Most companies struggle to get the data basics right due to ingrained cultural undervaluation of its importance. And because culture change is hard to deliver, those with genuine intent to develop superior insight capabilities will find their advantage to be highly sustainable.

4. Keep asking ‘Why?’ 

There are two traps that businesses can fall into with data. The first is having lots of data but insufficient clarity on what should be asked of it with the result that either much time is wasted on investigating irrelevancies or only the shallowest understanding is achieved. With the idea that Big Data renders redundant the scientific method of hypothesise, model and test – the ‘I’ll know it when I see it’ philosophy for data exploration – these risks are much increased. This idea was first expounded in a 2008 Wired article by Chris Anderson called The End of Theory. Central to Anderson’s argument was that causation doesn’t matter, just correlation – what people do is important but why they do it isn’t.   

Anderson’s piece has been roundly criticised, not least by Nate Silver in his best-selling book The Signal and the Noise. But despite that, the argument that only correlation matters continues to gain popular currency to the detriment of curiosity, inquiry and the ability to learn. 

All businesses are systems and causation is pivotal to how that system is managed. All decisions are predicated on achieving an outcome and improving performance on a particular dimension - if we do this, then this will happen. For example, if we re-engineer this process it will improve the customer experience, if we increase marketing spend we will gain market share, if we add these service features we can charge a higher price. All such interventions are designed with an effect in mind. To ignore causality in relationships is counterproductive – indeed businesses should organise reporting so that it can be identified to enable more informed decision-making. 

It could be argued that causation is less important in some activities than others, targeting for example. If you discover that a particular micro-segment is buying your product in unusually high quantities, then targeting those customers is likely to prove effective. Also knowing that beer and nappies are often bought together on a Saturday morning means they can be co-located to provide a stimulus and convenience to those with a high propensity to buy both.  But while that uplifts revenue generation, it doesn’t optimise it. That requires understanding why that micro-segment is buying to identify associated revenue opportunities – for example if it has found a duplicate use that could make the product attractive to other micro segments with related latent needs. Understanding that the beer and nappy relationship comes from fathers with young children shopping on Saturday mornings to give their wives a break suggests a broader set of potential associations between the must-have nappies and more discretionary male-oriented leisure purchases, all of which can then be tested.

If correlation is the sole focus, an organisation’s ability to learn, experiment and generate new data is much diminished. And without creating new data the ability to continually find new patterns – enhancing insight, keeping ahead of competitors – is significantly limited.      

5. Create data

The second trap is the opposite of the first - having clarity on hypothesised relationships but lacking the data to prove them. Given the current data deluge, this may seem an unlikely problem for organisations to face. But as the Ancient Mariner despaired of: ‘water, water, everywhere, nor any drop to drink,’ it is the same with data - volume and universality does not deliver relevancy. With abundance comes the risk of myriad visualisations that fail the ‘So what?’ test. They may answer questions, but not the right questions – not ones linked to value. 

Once information insufficiency is recognised, identifying the additional data required is relatively simple, and follows from 1 above. Locating where it can be found is the first step - for example embedded in online behaviours, in text and video, even in the heads of customers or staff. Collecting and structuring it is more complicated process, requiring decomposition of the typically unstructured data into measurable dimensions, followed by extraction, classification, indexation, summarisation, and interpretation. 

Creating data – or datafication as it has been called – is critical for insight advantage and it will be a topic expanded on in later articles covering gamification, prediction markets and how even something as complex as human behaviour can be structured and quantified.     

6. Blend art, science and engineering to create your vision

Any organisation seeking to generate insight advantage needs to develop a vision for its analytics activities that encompasses all three of art, science and engineering.  Achieving insight advantage requires leveraging a breadth of intellectual capability – not just tapping into one or two strands as has traditionally been the case in the world of reporting and analytics.

Traditional business intelligence was predominantly an engineering discipline – industrialising the process of data consolidation into enterprise data warehouses using ETL (extraction, transformation and loading) tools, with standardised reporting the medium for insight dissemination.  Engineering skills remains as important as ever, not least to meet the challenge of integrating unstructured and ungoverned data stored in Big Data repositories with traditionally warehoused and more governed operational data. And regular addition of new data sources will make data engineering an ongoing challenge.   

As the focus has switched away from reporting towards analytics, the need for data science – data exploration, predictive modelling and decisioning – has increasingly been recognised. Most companies are at the beginning of this particular journey and sophistication is not typically the highest priority when the level of maturity is low. Predictive modelling embodies a trade-off between accuracy and transparency – advanced machine learning techniques such as neural networks are indecipherable to anyone without high level mathematical capabilities. Given the organisational change that comes when decisions traditionally made on the basis of expert opinion are transferred to the realm of predictive modelling, managing the transformation is crucial. In particular, gaining the support of senior managers may require simplifying models in their first incarnation so that they are comprehensible to these influencers. But as models prove their worth and challengers prove their superiority, transparency will become less important and accuracy more. Continually advancing scientific sophistication will also remain important. 

Less recognised is the need for art in analytics. At a simple level this means creating data visualisations that clearly depict relationships and variations. At a more advanced level, the ability to conceive potential relationships prior to evidence being available and generate ideas for how that evidence could be collected is also more art than anything else. Data creation, as per 5 above, requires imagination – thinking beyond current boundaries to invent what is possible.  Imagination helps define the evolving requirements for engineering and science and without it both will suffer. A vision that encompasses all three will be critical to achieving and sustaining insight advantage.

Jack Springman is head of customer analytics, at Sopra Steria.