Like all senior management responsibilities, making effective use of data is more art than science. Any discussion about using “big-data” technologies has to start with the leadership issue. Someone very high up in the organization has to be an effective advocate for the use of data. Companies should be seeking a leader, a chief data officer — or whatever you want to call the role — who can understand the complexities of the business, recognize opportunities for better use of data, and make the business case for assembling the resources. Just as every function participates in the process of finance or marketing or other areas of senior management in some way, every function would be aware of how they could be improving their use of data.
Sigh. if only this were true.
In reality, the current conversation about big data at most companies is centered around how much to spend on Hadoop or storage or advanced analytics technology, not to find the right people who are artists in the medium of data, people like Hillary Mason of Bitly or Daniel Tunkelang of LinkedIn. There are lots of open positions for people with analysis skills. There are not enough open positions for people with the leadership skills, whose job it is to transform the business.
The lack of attention to the art of data leadership was made profoundly clear to me in an interview with Usama Fayyad, former Chief Data Officer at Yahoo!, who is now chairman and CTO of ChoozOn and CEO of Open Insights. Unlike most of the technology and business worlds, which have only recently become excited about the transformative power of huge data sets, Fayyad has spent his entire career trying to put put big data to work.
Starting as an academic and researcher at places like NASA’s Jet Propulsion Laboratories, Fayyad found that even advanced scientists need lots of help from data experts to finding insights in data that it took billions of dollars in research money to collect. He then joined Microsoft where sought to bring advanced analytics functionality to the world of business. While there, he succeeded in adding advanced data analysis capability to SQL Server.
But it became clear to Fayyad that the problem was not the technology. “It is very rare that you’re able to create a whole solution as a product that can solve important problems,” said Fayyad. “As more and more data becomes available, there has to be a simultaneous progression of two things: first of all, tools to make it usable by other people, and second an understanding by those people of what those tools are actually allowing them to do. Because naïve use of these tools only gives you predetermined answers to predetermined questions. You really have to understand more about what it is these advanced tools can do in order to actually get value out of them.”
Fayyad then left Microsoft and started a series of product and consulting companies that created his professed addiction to startups. These companies (digiMine and Revenue Science) were dedicated to closing the gap between the technology and the art in specific problem areas. He found that over and over again Internet companies and traditional businesses were struggling to find a way to make better use of data. Fayyad joined Yahoo! as Chief Data Officer, after his next consulting company, DMX Group, was acquired by Yahoo!.
I will leave Fayyad’s story of his experience at Yahoo! for another day. While Fayyad was impressed with the desire that Yahoo!’s senior leadership team had to make better use of data, he found it amazing how little both Yahoo! and the entire Internet-based industry understood about how to use data.
“At Yahoo we started identifying a lot of problems that were really economic problems, but they were being decided by engineers, in a suboptimal manner,” said Fayyad.
Since leaving Yahoo! Fayyad has been a Chief Data Officer for hire at Open Insights and has started ChoozOn, a consumer-focused company that helps people find valuable offers. In working with many companies on their most challenging data analysis problems, Fayyad has come to the conclusion that the real challenge is one of leadership. The technology provides the raw material. The art puts that raw material to good use. But the leadership is what puts it on the agenda so action can be taken.
“What you need is a strategic team. The Chief Data Officer has to be very senior. They have to have a voice on the executive table. They have to help set the direction of the company. They have to set other General Managers and VPs and EVPs straight when they sort of go off on the wrong direction,” said Fayyad. “In many, many companies, I’ve seen a very good data infrastructure go to waste because there wasn’t the right representation at the C level. Without credible leadership, the data basically goes unused. Nobody knows how to capitalize on it.”
Fayyad pointed out that the analysis challenge is just getting harder now that the amount of unstructured and semi-structured data is growing. “At this stage, for each problem you have to go on a data expedition,” said Fayyad. “You really don’t know what you’re going to find out, but at the end of the expedition you’re going to discover something, and put some new solutions together. You may end up far from where you thought you’d be or where you aimed to be, but you’re somewhere useful and interesting.”
We can have the science. We can have the art. But without the leadership, nobody goes on the expeditions that are most important. That is the biggest problem facing Big Data today.
Author: Dan Woods