What now seems like a long, long time ago, sharp managers turned to business intelligence tools to find answers to such questions as how a company could profit if it changed its product mix one way or another. Or they dove into data mining, to come up with revolutionary revelations such as the notion of displaying disposable diapers near six-packs of beer or flowers near auto batteries — to boost sales to clueless dads shopping on weekends. Serious stuff, but not life or death, for sure.
Today, however, the prudent manager needs to know which employee out of the 8,000 slated for an upcoming layoff is likely to return the next day carrying a loaded M-16, or which sneaker buyer appears the most inclined to cram explosives into the soles. This too is business intelligence, but not the kind generally taught at business school.
Business intelligence (BI) refers to the process of organizing, filtering, and analyzing volumes of data to come up with useful business insights. BI tools encompass everything from basic query and reporting, which summarizes volumes of data and presents the top-level results, to data mining, which uncovers often-hidden structures and subtle patterns within a large data set. It is data mining’s ability to surface these unnoticed structures and patterns that leads to eureka-type insights.
In terms of technology, BI also encompasses the entire process of capturing, organizing, storing, analyzing, and disseminating corporate data. Data warehousing and data marts play a central role, as do tools to handle data extraction, transformation, and loading (ETL).
Last year, despite the dismal economy, the BI software market grew by 22 percent to $3.6 billion, according to International Data Corporation (IDC), Framingham, MA. Looking ahead, IDC projects this market to grow to $11.8 billion by 2005. As such, BI represents a big opportunity for consulting firms.
Companies are turning BI tools loose on data from a variety of corporate applications, such as enterprise resource planning (ERP), customer relationship management (CRM), and supply chain management (SCM). Similarly, firms are applying BI technology to the massive volumes of data captured in the clickstream of Web site visitors, as managers try to understand who these visitors are, where they come from, what they do (and don’t do) while there, and why
The BI area is saturated with vendors, ranging from small specialty players to industry giants like Oracle Corp. Conventional BI tool vendors include Information Builders, Cognos, and SAS Institute. A new breed of vendors, such as DigiMine, Hyperion, Quadstone, Unica, and thinkAnalytics, offers sophisticated data mining as packaged solutions referred to as analytic applications. Still other vendors offer data warehousing, data mart, and ETL tools. Even packaged application vendors like PeopleSoft are getting into the BI game.
The BI industry, still busy proselytizing the gospel of information-based management for handling the usual business questions, has only just started to think of the new information analysis challenges resulting from the September 11 attacks. Industry leaders, however, are confident that BI can handle the task of sifting through volumes of data to find security threats. “The analytic algorithms don’t have to change. It’s really a question of feeding in the right data and interpreting the results,” insists Usama Fayyad, president, DigiMine, Inc., Bellevue, WA.
Although not quite fighting terrorists, DMR Consulting built a BI solution using Cognos BI tools to help the Quebec Department of Revenue detect tax fraud. The system captures data from 200 government agencies and runs queries against it to identify individuals who fit various fraud patterns. In building the solution, DMR developers had to take into account privacy issues and take care not to violate citizens’ rights. It’s worth the extra effort; the return on investment from the application runs into the billions of dollars, notes Paul Tremblay, alliance manager for DMR’s Business Intelligence Center of Excellence.
When it comes to answering extremely difficult questions, BiosGroup, Santa Fe, NM, draws on nonlinear analysis, chaos theory, and complexity theory. Although its highly sophisticated business intelligence solutions generally are applied to conventional business problems such as determining the optimum makeup of a product line or improving a supply chain, it has also applied its approach in such areas as military and criminal intelligence, notes Bruce Sawhill, senior scientist.
Rather than assume that things evolve in a direct linear fashion, the BiosGroup expects uncertainty. “We know there is noise in the system that flavors the results,” Sawhill explains. Financial traders, for example, tend to work on a very simple principle — buy low, sell high — but when you put together hundreds or thousands of traders following their own strategies based on that principle, the result is very complex and often far different than what you would expect. BI analysis that fails to recognize noise and uncertainty is bound to come up short.
But even recognizing uncertainty, the newest challenges still may be beyond most approaches to BI. Can you, for instance, mine the data on thousands of employees in the HR system to find the ones with the potential to unleash murderous rage?
“You can definitely apply the algorithms, but where would you get the data?” asks William Mcready, of the BiosGroup. Even if you had a rich HR file, the challenge in this case is coming up with the factors that indicate the potential for violence. Despite the obstacles, efforts indeed are under way among those battling terrorists to apply data mining and link analysis tools to these kinds of problems. At this point, however, BI leaves the realm of what rank-and-file consultants are prepared to handle. Even the BiosGroup, which works closely with a handful of consulting firms including Cap Gemini Ernst & Young and Andersen, finds that “the typical consultant has difficulty understanding these algorithms,” says Mcready.
But the BiosGroup, admittedly, focuses on a highly rarefied niche. Consulting firms are quite capable of bringing BI to the great mass of businesses that can benefit from more conventional information-driven management. Consultants have the technical capabilities and business expertise to implement BI solutions, interpret the results, and help clients craft business strategies based on insights gained.
PwC, for example, has gone so far as to set up BI as a service of its Business Process Outsourcing (BPO) operation, a wholly owned subsidiary. “To us, BI is a process for turning data into useful information,” says Kevin Elliott, PwC’s information management leader. In Calgary, BPO partners use technology from Information Builders to perform BI for British Petroleum on an outsourcing basis. BPO collects the data, manages the data warehouse, and transforms the data into something business managers can readily understand, as well as run the BI algorithms and queries.
The biggest value BPO brings to the process is understanding, not technology. “Managers want to know why something happened, not just what happened. But BI algorithms can’t tell you why; to see the why, you need someone who can analyze the results and explain them,” Elliott believes.
In its consulting practice, PwC has adopted the term iAnalytics to describe its efforts to take BI out of the hands of the select few business analysts trained to use the tools and drive it into mainstream management ranks. “Our goal is to move BI from the 10% to 20% who use it to the rest of the employees and other stakeholders who can benefit from it,” says Michael Schroeck, partner-in-charge, PwC’s iAnalytics practice, which relies heavily on Cognos BI tools.
Beyond reaching out to a larger user population, BI is changing in other ways. “We’re now seeing integrated BI. In the past, BI focused on a single area, such as marketing or finance,” Schroeck continues. Specifically, the new corporate focus on the customer requires that BI look across numerous enterprise functions to understand and improve performance in areas such as customer profitability and lifetime customer value. Schroeck also sees BI moving from analysis of historical data to more future-oriented, predictive activities.
At the same time, he concedes that the highly touted ad hoc discovery capabilities of data mining have proven less popular. “Some firms did turn up real nuggets of insight, but ad hoc exploration of data is simply not getting to the masses.”
DigiMine’s Fayyad agrees: “Few business users have the time to do real exploration of the data. They just want to find answers to immediate questions — fast.” DigiMine tries to address this need by pre-mining the data. “There are well-known journeys through the data that managers need to take, so we pave the key roads,” he explains. Essentially, DigiMine provides a packaged data warehousing/data mining solution addressing the most likely questions managers will ask. A number of vendors offer analytical applications that similarly answer common BI queries.
The growing interest in packaged BI solutions and canned analytical applications brings into question the role of the consultant in the BI process. Fayyad however, sees consultants filling several key roles: (1) collecting the data and preparing it for use in BI; (2) interpreting the results of BI queries; and (3) creating business strategies based on insights drawn from BI.
Given the proliferation of Web data, along with growing volumes of enterprise data for CRM, supply chain management, and ERP, the opportunity for BI tools that plow through this massive volume of information appears good. And even if companies don’t use it to track down the next terrorist … well, there still is much value to be gained.
By: Alan Radding