When business intelligence (BI) systems first surfaced, their main objective was to empower the business to make insightful decisions and be completely self servicing. However, the belief that IT would completely enable BI hit a brick wall, given that BI has become increasingly complicated along the way, given the multiplicity of tools, staggering amounts of data, need for quick response based on solid data, and the inbred nature of IT teams. It soon became amply clear that the business could not possibly acquire intelligence without the presence of a gamut of BI professionals.
The possibility of hiring a data scientist through an analytics service provider is an intermediary step for many mid-sized and small businesses.
Data is doubling every two years, and everyone has heard of the preposterous growth numbers stated in reports. Against this backdrop, the inevitable outcome is the emergence of the data scientist. A data scientist needs to analyze vast amounts of data and cast the technology map to make the transition of data to insights possible. The scope of the job of a data scientist includes identifying data sources, quality of data, correlations between data points and dissemination to the users of information.
For the moment, the role of a data scientist is played by a combination of people in the BI team, such as the data warehouse architect, business analyst, and others of that ilk. As the situation evolves, the data scientist would work above these professionals to unearth new trends and associations that may be beyond the realm of current models and business problems. The business analyst would work on the data that has been put together by the data scientist. James Kobielus, a senior analyst with Forrester, in his blog, goes as far as to compare the work of a data scientist with the work of scientists in the natural sciences and social sciences, stating that they would need observational data and experimental data to work with. “Historically (data scientists) have had to be content with mere examples.” With a full-fledged career emerging, this will soon change.
Discussions on who exactly is qualified to be a data scientist do not vary too much from the debate held earlier on whether a CIO should have a business background or a technology background. Yet, at the outset, industry experts have indicated that a data scientist should have a master’s degree in mathematics or statistics. Meanwhile, Arun Gupta, group CTO at Shoppers Stop, states, “There is a shortage of professionals who can be called data scientists. At the moment, whoever has a passion to work with data is filling the void.”
A data scientist will work at developing new algorithms and bring forth new patterns and insights into the data that otherwise would stay hidden. “Along with statistics, a data scientist can have a qualification in economics, and most definitely needs a dozen or more years’ experience with working with ten to fifteen BI tools,” says Chuck Hollis, VP for global marketing & CTO, EMC.
Deepinder Singh Dhingra, who heads the innovation and development group at Mu Sigma, a provider of decision science and analytics services says “Data scientists will also dabble in experimental psychology, anthropology and social sciences.” With a need to establish BI centers of excellence (CoE), analytics education for the data scientist will get institutionalized.
The road ahead
With the advent of social media impacting most facets of business, organizations look forward to integrate technology, social software and BI to create a congenial decision making environment. The data scientist will be responsible for providing a social context to information. BI and analytics dance to a new tune when adopting new approaches such as Hadoop. They do not wait for structured, cleansed, pristine data but work with a mixed bag of data to provide real- or near time analysis. Descriptive analytics, inquisitive analytics, predictive analytics and prescriptive analytics are all part of the new paradigm, with the data scientist at the center.
The evolution curve is moving from decision support to becoming increasingly operational, with an imminent progression that will take strategic competence to a whole new level with data scientists in the picture. BI came on the scene fifteen years ago and IT owned these initiatives. Now, BI is a business function involving market research with a central focus on analytics. Companies with large volumes of data (internal and external) would take to the notion of data scientists without batting an eyelid, but smaller companies would think twice before paying someone money to do something their BI vendor told them their BI product would do.
However, it is not as if there are data scientists in abundance, all clamoring for jobs. On the contrary, as Hollis from EMC says, “There is a shortage of talent. For every data scientist out there, there are thirty jobs waiting.”
In answer to this problem, Analytics as a Service presents itself as a feasible alternative. Analytics as a Service is still nascent and evolving; as the complexity grows and mature service models that are linked to outcomes and success emerge, the adoption rate will increase. The possibility of hiring a data scientist through an analytics service provider is an intermediary step for many mid-sized and small businesses.
There is no doubt that the field of analytics is a game changer and that the data scientist will in the next five years be a driver of change and competition. Nevertheless, the rise of the data scientist as a valuable asset will be dependent on how major analytics players — mainly organizations from the BFSI, retail and telecom sectors — push the envelope.
This was first published in December 2011