What is the role of the ‘Data Scientist’?

What is the role of the person who analyzes the big databases Internet companies are collecting?

Nicole Laskowski of SearchBusinessAnalytics.com has recently written two very informative news articles:  See “Data Scientists Helping Businesses Navigate ‘Big Data’ Seas” and “Predictive Analytics and ‘Big Data’: The Good, The Bad and the Ugly”.

In “Data Scientists Helping Businesses Navigate ‘Big Data’ Seas” Laskowski describes the role of the ‘Data Scientist’ as being something new, which I think is correct.   Internet based companies collect tera-byte and peta-byte sized data sets.  In parallel, ‘big data’ analysis tools like Hadoop make working with those data sets practical.  The emerging data scientist is able to explore that data to find patterns and answer open ended questions that could not be answered before.  The article stresses the importance of exploration and discovery.  Referencing Usama Fayyad of ChoozOn (and formerly of Yahoo), Laskowski says that when a new problem is assigned to Fayyad’s teams “… the workers spend several days to several weeks studying the data before starting the analytics process.”  Predictive Analytics are one of many approaches that the new data scientist uses.

The new data scientist is doing something fundamentally different from what the traditional Financial Planning and Analysis (FP&A) analysts does when he or she forecasts financial results from sales and cost reports.  Or what the marketing strategists does he or she forecasts demand based on retails store sales reports and promotion plans.  The new data scientist is looking at new data sets for new insights.

Contrast that to Laskoski’s interview with Vantana Research’s VP David Menninger in “Predictive Analytics and ‘Big Data’: The Good, The Bad and the Ugly”.  Menninger views predictive analytics for financials and demand planning as the core discipline.  Big Data augments the traditional methods of statistical sampling and data modeling. Big Data improves the results, but it does not change the game fundamentally.

For the traditional FP&A and store marketing promotions functions, this is probably correct.  But, the new “Big Data” analysis discipline creates an opportunity to understand and market to individual consumers in a way that was impossible before.  Firms developing “Big Data” teams should not limit their scope to traditional problems.  Instead, they should give the team an open ended charter to explore all of the data within the company and to be free to make conclusions about what it means.  This may make creating a budget for the team more challenging as the end product and its value cannot be easily estimated.  But the opportunity to discover important new insights is the real value of the technology and the talent.