Big Data: The Management Revolution
Smart leaders across industries will see using big data for what it is: a management revolution. But as with any other major change in business, the challenges of becoming a big data—enabled organization can be enormous and require hands-on—or in some cases hands-off—leadership. However, there are three key differences:.
As of , about 2. More data cross the internet every second than were stored in the entire internet just 20 years ago. This gives companies an opportunity to work with many petabyes of data in a single data set—and not just from the internet. For instance, it is estimated that Walmart collects more than 2. An exabyte is 1, times that amount, or one billion gigabytes. For many applications, the speed of data creation is even more important than the volume.
Real-time or nearly real-time information makes it possible for a company to be much more agile than its competitors. Rapid insights like that can provide an obvious competitive advantage to Wall Street analysts and Main Street managers. Big data takes the form of messages, updates, and images posted to social networks; readings from sensors; GPS signals from cell phones, and more. Many of the most important sources of big data are relatively new. The huge amounts of information from social networks, for example, are only as old as the networks themselves; Facebook was launched in , Twitter in The same holds for smartphones and the other mobile devices that now provide enormous streams of data tied to people, activities, and locations.
Thus the structured databases that stored most corporate information until recently are ill suited to storing and processing big data. At the same time, the steadily declining costs of all the elements of computing—storage, memory, processing, bandwidth, and so on—mean that previously expensive data-intensive approaches are quickly becoming economical. As more and more business activity is digitized, new sources of information and ever-cheaper equipment combine to bring us into a new era: one in which large amounts of digital information exist on virtually any topic of interest to a business.
Mobile phones, online shopping, social networks, electronic communication, GPS, and instrumented machinery all produce torrents of data as a by-product of their ordinary operations. Each of us is now a walking data generator. Analytics brought rigorous techniques to decision making; big data is at once simpler and more powerful. We just have more data. But the truth, we realized recently, is that nobody was tackling that question rigorously. We set out to test the hypothesis that data-driven companies would be better performers.
How Data-Driven Companies Perform
We conducted structured interviews with executives at public North American companies about their organizational and technology management practices, and gathered performance data from their annual reports and independent sources. Not everyone was embracing data-driven decision making. In fact, we found a broad spectrum of attitudes and approaches in every industry. But across all the analyses we conducted, one relationship stood out: The more companies characterized themselves as data-driven, the better they performed on objective measures of financial and operational results.
This performance difference remained robust after accounting for the contributions of labor, capital, purchased services, and traditional IT investment. It was statistically significant and economically important and was reflected in measurable increases in stock market valuations. Often someone coming from outside an industry can spot a better way to use big data than an insider, just because so many new, unexpected sources of data are available.
One of us, Erik, demonstrated this in research he conducted with Lynn Wu, now an assistant professor at Wharton.
They used publicly available web search data to predict housing-price changes in metropolitan areas across the United States. They had no special knowledge of the housing market when they began their study, but they reasoned that virtually real-time search data would enable good near-term forecasts about the housing market—and they were right. In fact, their prediction proved more accurate than the official one from the National Association of Realtors, which had developed a far more complex model but relied on relatively slow-changing historical data.
This is hardly the only case in which simple models and big data trump more-elaborate analytics approaches. Researchers at the Johns Hopkins School of Medicine, for example, found that they could use data from Google Flu Trends a free, publicly available aggregator of relevant search terms to predict surges in flu-related emergency room visits a week before warnings came from the Centers for Disease Control.
Similarly, Twitter updates were as accurate as official reports at tracking the spread of cholera in Haiti after the January earthquake; they were also two weeks earlier. So how are managers using big data? One uses big data to create new businesses, the other to drive more sales. Minutes matter in airports.
Moment of Impact Quotes
So does accurate information about flight arrival times: If a plane lands before the ground staff is ready for it, the passengers and crew are effectively trapped, and if it shows up later than expected, the staff sits idle, driving up costs. So when a major U. The pilots made these estimates during their final approach to the airport, when they had many other demands on their time and attention. In search of a better solution, the airline turned to PASSUR Aerospace, a provider of decision-support technologies for the aviation industry.
It calculated these times by combining publicly available data about weather, flight schedules, and other factors with proprietary data the company itself collected, including feeds from a network of passive radar stations it had installed near airports to gather data about every plane in the local sky.
Every 4. This allows sophisticated analysis and pattern matching. When did it actually land? After switching to RightETA, the airline virtually eliminated gaps between estimated and actual arrival times. PASSUR believes that enabling an airline to know when its planes are going to land and plan accordingly is worth several million dollars a year at each airport. Obviously, it would be valuable to combine and make use of all these data to tailor promotions and other offerings to customers, and to personalize the offers to take advantage of local conditions.
Valuable, but difficult: Sears required about eight weeks to generate personalized promotions, at which point many of them were no longer optimal for the company. In search of a faster, cheaper way to do its analytic work, Sears Holdings turned to the technologies and practices of big data. As one of its first steps, it set up a Hadoop cluster. This is simply a group of inexpensive commodity servers whose activities are coordinated by an emerging software framework called Hadoop named after a toy elephant in the household of Doug Cutting, one of its developers.
Sears started using the cluster to store incoming data from all its brands and to hold data from existing data warehouses. It then conducted analyses on the cluster directly, avoiding the time-consuming complexities of pulling data from various sources and combining them so that they can be analyzed. This change allowed the company to be much faster and more precise with its promotions. Because skills and knowledge related to new data technologies were so rare in , when Sears started the transition, it contracted some of the work to a company called Cloudera.
But over time its old guard of IT and analytics professionals have become comfortable with the new tools and approaches. The PASSUR and Sears Holding examples illustrate the power of big data, which allows more-accurate predictions, better decisions, and precise interventions, and can enable these things at seemingly limitless scale.
The technical challenges of using big data are very real.
- The Model Vegan.
- Network Science, Nonlinear Science and Infrastructure Systems: 102 (International Series in Operations Research & Management Science).
- Episode Strategies for Mental Toughness?
- Moment of Impact.
- The Era of Education: The Presidents and the Schools, 1965-2001.
- The Special Smile.
But the managerial challenges are even greater—starting with the role of the senior executive team. One of the most critical aspects of big data is its impact on how decisions are made and who gets to make them. Mauboussin, in this issue. But we believe that throughout the business world today, people rely too much on experience and intuition and not enough on data.
For our research we constructed a 5-point composite scale that captured the overall extent to which a company was data-driven. Executives interested in leading a big data transition can start with two simple techniques. Second, they can allow themselves to be overruled by the data; few things are more powerful for changing a decision-making culture than seeing a senior executive concede when data have disproved a hunch.
When it comes to knowing which problems to tackle, of course, domain expertise remains critical. Traditional domain experts—those deeply familiar with an area—are the ones who know where the biggest opportunities and challenges lie.