musings on music and life

August 5, 2014

“Data Science” bubble? Or not?

Filed under: Data Science — Tags: — sankirnam @ 10:15 am

Insight and The Data Incubator are offering bootcamps (similar to the one I mentioned by Zipfian Academy earlier) to train people as “Data Scientists” in the nascent field of “big data”. Supposedly, this is a burgeoning field with vast potential:

The data science education market is far from overcrowded: Demand for data scientists continues to outstrip supply. The McKinsey Global Institute estimates that by 2018, the U.S. will face a shortage of 140,000 to 190,000 people equipped with the deep analytical skills necessary to make sense of big data.”

Does this sound familiar? It sounds like the usual political rhetoric of the “STEM shortage” in the US (which is patently false; there is a shortage of overqualified applicants willing to work for pennies on the dollar).

However, the bootcamps offered by Insight and The Data Incubator are different in that they are free:

Beyond room and board expenses in New York City, the six-week program won’t cost participants a penny. Instead of student tuition, The Data Incubator will charge its employer partners if they decide to hire Data Incubator alumni as data scientists or quantitative analysts (quants).”

This sounds almost too good to be true, so naturally there must be some catch. One is that The Data Incubator’s program is more difficult to get into than Harvard (their acceptance rate is below 5%), but I can’t find anything else…

The other odd aspect of these programs is that applicants must be a PhD candidate or have a PhD. Being a PhD candidate myself, I don’t understand these people’s fixation with the title. A PhD is a very poor measure of intelligence or work ethic. I mean, sure, those with doctorates may be 1 or 2 standard deviations more intelligent than the median population, but if I were an employer, I wouldn’t put too much weight on that myself.

Success in research (especially in an experimental field like chemistry) depends much, much, much more on luck than on intelligence or hard work (I would say that is 99.99% luck and 0.01% intelligence/hard work)! In fact, one of the professors in my building said he would prefer to look at prospective students’ horoscopes rather than their transcripts.

In theoretical fields (such as mathematics, philosophy, and finance to a degree), I suspect this may be different. The key factor there is not so much discovery or observation of some natural phenomena as in experimental fields, but insight, which requires a deep understanding of the subject and a high degree of intelligence. Contrast this with an experimental field like organic chemistry where one may have to try hundreds of different combinations of conditions by brute-force methods before finding one that works. One may get lucky and find it in the first few tries, or may run 500 reactions and still have no useful results (this has happened to me)! Due to this, success in research in a field dominated by stochastic results (i.e., an experimental field) is not a good measure of intelligence or work ethic.


1 Comment »

  1. […] that the situation has not changed significantly since I talked about “Data Science” here a year ago; in fact, the field seems to be growing. Anecdotally, I hear through word-of-mouth that […]

    Pingback by Data Science 2015 | musings on music and life — May 28, 2015 @ 2:19 pm

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: