Big Data: Businesses must "adapt or perish"

The vast majority of companies are unable to effectively use new data to assist their business decision-making, gain competitive advantage, drive productivity growth, yield innovation and reveal customer insights, finds a massive study of the data science community across the United States, the United Kingdom, France, Germany, India and China conducted by EMC Corporation.

At the same time, looming skills shortages and a lack of open data access top the list of factors holding back Big Data, the study has found.

An explosion of digital data created by mobile sensors, social media, surveillance, medical imaging, smart grids and the like — combined with new tools for analysing it all – has resulted in huge demand for data scientists. But 65% of respondents said they believed demand for data science talent would outpace the supply over the next 5 years – with most feeling that this supply will be most effectively sourced from new college graduates.

Commenting on the findings, Andreas Weigend, head of the social data lab at Stanford and former Chief Scientist at Amazon.com said: "We live in a data-driven world. Increasingly, the efficient operation of organisations across sectors relies on the effective use of vast amounts of data.

“Making sense of Big Data is a combination of organisations having the tools, skills and more importantly, the mindset to see data as the new "oil" fueling a company. Unfortunately, the technology has evolved faster than the workforce skills to make sense of it and organisations across sectors must adapt to this new reality or perish," Weigend added.

Jeremy Burton, EVP and Chief Marketing Officer at EMC Corporation, said Big Data offered an unprecedented opportunity to transform business and the way we work and live. “Through the convergence of massive scale-out storage, next-generation analytics and visualization capability, the technology is in place. What's needed to fully realize its value is a vibrant, interconnected, highly-skilled and empowered data science community to reveal relevant trend patterns and uncover new insights hidden within."

The most commonly cited barriers to data science adoption include: Lack of skills or training (32%) budget/resources (32%), the wrong organizational structure (14%) and lack of tools/technology (10%).

Only 12% of business intelligence professionals and 22% of data scientists strongly believe employees have the access to run experiments on data – undermining a company's ability to rapidly test and validate ideas and thus its approach to innovation.