Ford Motor Co. opened a Silicon Valley office in 2012 focused on Big Data, innovation and the user experience, stating, “It's time to prepare for the next 100 years.” Walmart records well over 1 million customer transactions every hour, saving them to databases estimated to have over 200 times the information in all of the books in the US Library of Congress.

SO WHAT IS BIG DATA?

Big Data has moved from technology circles into the business mainstream. But what is it? This article addresses the terms, the business case and management and business issues associated with Big Data.

Big Data describes the volumes of data generated by an enterprise, including Web-browsing trails, point-of-sale data, ATM records and other customer information generated within an organization, plus huge stores of data from new external sources such as social networks like Facebook, Twitter, YouTube and LinkedIn, sensor and even surveillance data and massive public and private databases.

These data sets can be so large and complex that they become difficult to process using traditional database management tools and data processing applications. The data are often unstructured, unformatted and unwieldy. But there can be important business information ready to be unleashed; “the signal in the noise.”

Ever-improving computer hardware tools, such as virtually unlimited storage and continually faster processing speeds, combined with software tools such as artificial intelligence, machine learning and pattern recognition, can be applied to these vast troves of data.

THE BUSINESS CASE

As the examples in the introduction demonstrate, Big Data has the potential to bring about a fundamental transformation of the economy. Almost no area of business activity will remain untouched.

There is a payoff from data-first thinking. Studies show that the more companies characterized themselves as data-driven, the better they performed on objective measures of financial and operational results. Harvard Business Review recently reported that companies in the top third of their industry in the use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors.

McKinsey Global Institute, the research arm of McKinsey and Co., has analyzed Big Data's effect on business, concluding that the use of data and analytics are going to be a basis of competition going forward for individual firms, for sectors and even for countries. The ones that are able to use data effectively are more likely to win in the marketplace.

A NEW CULTURE OF DECISION MAKING

What Managers Must Do

Managers can measure and analyze more precisely than ever before, thus allowing much more insight about their businesses and a better understanding of their customers. This knowledge can translate into more-accurate predictions, wiser decisions and stronger performance.

Managers can analyze vast amounts of data in a rigorous and disciplined way. This enables acting on carefully distilled information, perhaps in place of instinct and intuition. In essence, the manager allows instinct to be overruled by the data.

It is critical to note that Big Data does require management vision and human insight. There needs to be an understanding of how the mathematics differs from reality and an understanding of the business theory underlying the analytical results. Most importantly, the assumptions underpinning the analysis call for thorough understanding and critical evaluation.

Senior leadership identifies the strategic goals and uses the data to make better, faster decisions. The use of the data is directly linked to strategy, with a goal to clarify, define and implement it most effectively.

Challenges for Senior Management

The challenges to a business's culture can be great. A senior executive team might see a shift in who is considered the “expert”. Often, the most senior person or most highly paid person in a business area was automatically considered the expert. With Big Data injected into decision-making, the role of expert could shift. The expert might be a junior person who knows what questions to ask and how to use data to get the best answers.

Senior decision makers have a duty to embrace evidence-based decision-making. The data-driven CEO welcomes, encourages and creates a culture that supports it. For example, when the CEO has a gut feeling about a business trend, the data might not support that intuition. Senior executives that are genuinely data-driven will override their intuition when the data do not agree with it.

The culture should allow a junior manager to use the data to explain results that differ with the boss's instinct. This requires that the junior manager has a thorough understanding of the data and its relation to the business, courage, and a high level of interpersonal skills, in addition to a supportive culture on the part of the organization.

UNDERSTANDING THE PITFALLS

Managers must understand the pitfalls and limitations, as well as the potential, of Big Data. Good data scientists should in part be pessimistic with a great concern about what the information is truly indicating, being sensitive to what can go wrong with predictions and model designs.

Correlation is not causation. With such large data sets and well-honed measurement there is substantial risk of “false discoveries.” Just because elements of the data are highly correlated, does not mean that there is a causal relationship between them; indeed, the correlation might not be meaningful at all from a business viewpoint. Management must use skill and experience to avoid this trap.

Furthermore, any mathematical model inevitably is a simplification. Modeling is used very successfully in the physical sciences; for example, countless phenomena are accurately predictable according to the laws of physics, such as the flow of water or the path of a rocket. This is not the case in substantially more complex systems, such as economics and social systems, which are disciplines directly affecting business.

Managers must understand the observable business theory or insight that explains the statistical inferences. Conclusions are much stronger and more valuable when there is this business insight. The numbers do not speak for themselves; managers speak for them, giving them meaning.

It is human nature to use analysis to confirm one's own biases and prejudices. It can be all too easy to use massive data troves to see what management wants to see without realizing it is doing so. Big data might provide raw material for biased fact-finding ostensibly based on statistics.

CONCLUSION

Big Data is a powerful tool to support smart decision-making. It can allow better understanding of what is most important to the success of an enterprise. Managers must employ it thoughtfully, being fully aware of the obstacles to maximizing its utility, including organizational cultural issues, personal biases and “false discoveries.' With that caveat, management can realize the value of Big Data to better serve customers and the enterprise as a whole.

Stuart R. Levine is chairman/CEO of Stuart Levine & Associates. He can be reached at (516) 465-0800 or stuartlevine.com.