Ask HN: What's the state of the job market in data science and machine learning?
If one were to use Hacker News as their only source of information, it would seem that machine learning is a very overrated topic. There is something related to it on HN's front page almost every day. This proliferation of courses, resources, books and startups would hint that machine learning is becoming more and more accessible to the average programmer and that the market is on track to getting saturated quickly. Is this the current trend? If yes, is it limited to the US? What about the machine learning scene in Europe? Maybe someone here could provide some perspective.
139 comments
[ 0.20 ms ] story [ 182 ms ] threadProgramming is a tool to create and synthesize. It leads to new products, companies, and solutions. Data science is analysis, not synthesis. You collect data, you interpret it, you move on to other data. Nothing gets created, which for me, is a deal breaker for job satisfaction.
I see. I'm currently in a research lab, and programming is my main day-to-day activity. I have a high degree of autonomy, but have recently considered moving to a product division so I can work on something that actually gets shipped. But it doesn't sound so great, as you describe it. Maybe it's a case of the grass always looking greener on the other side.
[0] http://www.gartner.com/newsroom/id/3412017
State of the art systems are far better than this. Microsoft recently published a paper with a 5.9% word error rate for conversational speech. Speech directed at computers/assistants is already in the high 90s, though I don't have a figure off the top of my head.
Occasionally, android gets the words right (as demonstrated by the onscreen text) and then flubs passing the correct intent because of "loss of connection", which is just about the most frustrating ML fail.
No doubt android's voice recognition is spectacular. I can prompt it in three different relatively orthogonal-sounding languages (English, French, Japanese), and it can figure out which language I'm using and usually get the transcription correct. Notably, I can't activate the italian/japanese pair and get useful results - which makes sense if you know both languages.
Google voice is horrible, however, at transcribing voicemail.
For continuous use, you need 95% at least. See http://anewdomain.net/2014/01/28/lamont-wood-windows-speech-...
http://m.imgur.com/gallery/noBKI
I also believe that most traditional companies do have data scientists, but they havent really start incorporating machine learning into their products, they are analyzing information about their customers, but their products are not reliant on using data. Once that becomes more common, things will pick up.
Soon, however, one could argue that 'traditional companies' will no longer be the norm - data science, ML, etc. will play such a crucial role in the majority of tech firms that the number of companies using it will rise. That's when I expect we'll see a huge portion of software engineers knowing ML concepts. Alternatively, I wonder if we might see the rise of smaller companies contracting out all of their ML to larger ones.
I would also be curious to know if ML background helps one to get a job at a place like Amazon/Google for even 'traditional' positions right now. The amount of data they have now must drive demand for engineers who can write software that takes advantage of it, regardless of position. Of course, like you said, they'll always require engineers to fill more traditional roles with no data interaction.
Maybe if you have a good profile and you get lucky, you'll go interview straight for one group who's interested in you, but I wouldn't bet on that.
Assuming the other people who used that word are me.
Almost no one, except for maybe very high level hires, get a pass for the initial weed-out interviews.
General data science is in need. I can get contracts easily, I know that people looking for competent people need to wait; especially as it is a skill much harder to pick than, say, front-end web dev (unless someone starts from a highly quantitive background like physics, modelling in biology, etc). My general impression are:
- ML (especially practical one, like logistic regression and random forest) is often integral parts of many data analyses (or at least a plus),
- there are not as many jobs solely focused on ML; and if so, often they require some specialistic expertise,
- and even less only for deep learning (also, for DL there is relatively high threshold for having skills at "hireable" level).
Some of my tips on how to learn data science: http://p.migdal.pl/2016/03/15/data-science-intro-for-math-ph... (on purpose I put the emphasis on general data exploration/analysis before machine learning).
One day I want to write how I get started, but I am not sure which steps were essential, which - irrelevant. And many things are not ones one can replicate.
Right now however the theme I've heard from the higher ups has been profitability, and this applies to all tech companies in general. Easy capital is gone and now companies are in the spotlight for not making profits.
So at least from my company's perspective, it's not that data science is saturated, it's that we're trying to not break the bank and hire too much.
In my opinion, you could start by defining what is a data science, a quant, or a machine learning job. Because that's not clearly defined. It means different jobs to a lot of people, jobs that are all hard to learn and absolutely NOT interchangeable.
The major issue is that Data Scientist is a very fuzzy term with it being applied to everyone from undergraduates with Stats degree and to those with PhDs and papers at KDD/ICML/NIPS/CVPR.
However rather than doing a Frontend or Mobile developer coding bootcamp, a data science bootcamp is likely to lead to more transferable skills in case you wish to get an MBA etc.
[1] http://stackoverflow.com/company/salary/calculator?p=7&e=1&s...
They've long understand that there are the finance analysts on the one hand and the software dev on the other. They get both and make them work together.
Looking for 5 rare skills in a single person is bound to disappointment: maths, statistics, programming, large scale systems, production.
Any engineer will quickly figure out that he's out of his depth in the maths & statistics. Any mathematician will quickly figure out that he's out of his depth in the system building.
Can you elaborate on this, and at what level? Are you talking about a PhD level of understanding of cutting edge mathematics, or do you mean understand the basics, or somewhere in between?
1. Asking a candidate to cast a non-standard problem of classification or estimation into a tractable optimization problem. (this is a very valuable skill that someone who has done good studies in numerical linear algebra/machine learning/stat/information theory/control systems/signal processing/math/etc. should be able to do)
2. Asking them to take an algorithm they have used and explain every step in deriving the algorithm. (It will help interviewer calibrate the level of learning in the interviewee. Also helps screen for indisciplined black-box users.)
3. Presenting challenging machine learning scenarios: using customized ensemble learning approaches, imbalanced data sets, noisy labels, multiple instances, different error metrics, etc. and seeing how interviewee approaches the problem from first principles (real world problems almost always involve some of these issues)
4. Testing their intuitions in "feature-engineering" for different types of data. (with the partial exception of cases where rigorous research/successful products show the utility of deep learning, one has to almost necessarily do a fair bit of feature-engineering)
The bias might stem from the fact that we have some huge names in AI doing research here, but the data points seem clear (we say undergraduate education is slow to catch on, right?): the topic as a whole isn't overrated.
However, there seems to be a lack of understanding by people working in tech of the differences (in uses, theory, implementation) between ML, AI, NN, DL, etc. This might stem from a lack of understanding of the foundations of these topics (ex: statistics, vector calculus) or simply because we can abstract a lot of this away (ex: TensorFlow).
That would work up to the point a better abstraction tool/framework comes along. I'd never try to build a career on a single framework, because frameworks come and go.
Theano and TF, for example, both make similar abstractions: graphs and numerical functions on top of the same matrix library, even. I would suspect someone could move between the two fairly easily. The problem is that a programmer can use TF/Theano/etc.'s built-in gradient descent functions pulled from a tutorial with their data subbed in _instead_ of learning the details of backpropogation, end up with decent results, and claim to have a basic understanding of ML - when really, they've managed to avoid it almost completely.
And yet, what is the incessant drumbeat of most job ads, these days - even in data science?
That's right: "N years in framework X"
I don't know where this meme comes from. Six out of the seven jobs/internships I've had (in SF, Seattle, and Toronto) didn't care about any specific framework or language whatsoever and still don't.
Should most ML job postings require n years of experience in some framework? Maybe not - but I can see why a company might see value in it.
There's only in web development where the hype change every year. (And even there there are plenty of companies that are lagging enough behind to still have opportunities in the old thing).
> I'm an undergrad at a big university known for CS in Canada
than the actual name of the university?
> I'm an undergrad at ${name of university}
I think you also need to not confuse the growing ease of machine learning tools with the role becoming more accessible. There is a wide gap between tooling and knowledge to use those tools appropriately and creatively.
And may I never write another HN comment on my cell phone again.
The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it.
The second group has gotten extremely crowded, especially from STEM graduates - usually with a masters or phd - who have completed MOOCs or bootcamps. Supply keeps growing while demand is flat or shrinking (especially as executives get burned by "data scientists" who don't know how to help them build things of value). There's a huge crunch here; a lot of people I know in this group have been searching for jobs for months, eventually settling for a low quality job or giving up entirely :(
The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I described usually had a STEM type PhD.
Basically, everyone has experience when they graduate from a master or a PhD there.
Not all experiences are born equal. e.g. Some work is at top tech companies, some work is at universities or low quality research centers with little work & expectations.
I am curious what countries it works the other way around.
Just find who's looking for economists, specifically. Consulting firms and government agencies hire economists, but they don't call them data scientists. Investment funds may be interested, though hedge funds prefer math and physics graduates because they are easier to train.
Forget about the big companies that everyone knows, forget about the high pay that appear in the newspapers. It's not real, it's not for you.
Get an entry or junior level position and get experience. There are many unknown places that will take people only because they're cheap. That's where everyone started.
Could you elaborate on which elements of vision science are relevant? Do you have any examples of what these people end up working on?
Eh? It's not as easy as it sounds. Search for "mlelr" for a rather detailed illustration of how to code LR in C with only standard libraries. Now that was some fun to put together!
(Edit: I'm not saying MOOC-depth derisively. Serious learners and autodidacts can go pretty deep with just MOOCs for guidance. I just mean prima facie, based on content of some of the lighter and more popular MOOCs like Andrew Ng's. Abu-Mostafa's on EdX is meatier.)
Anyone can be quickly taught how to run a logistic regression by calling one line of code from a high level library. But I'd argue it takes years of study to really know what you're doing.
Would an English MA or PhD, for example couple well with an ML-MOOC for natural language processing jobs? Possibly, but would involve a lot of effort to build the bridge. However, if someone has a mechanical engineering degree or a PhD in something similar, a MOOC-ML can work well to position them for a bunch of jobs in large mechanical engineering companies looking to jump on the ML bandwagon or ones traditionally strong in ML-type mechanical engineering like UTC.
- excellence in being able to clean data at the column level for millions of data points - knowing how to work with such large scale data in a time efficient manner. One of our newer hires worked at the Postgres/SSD level to optimize and got it to where he could produce a full set within 5 minutes. Before that it was once every few months.
Being able to do these things is a prerequisite to building even a prototype of a model, and it requires substantial real-world programming experience to deal with those complexities.
- Research departments that make prototypes that are consistently abandoned and never get any real world usage.
- Actual companies that make prototypes for real business cases that are then shipped to production, maintained and improved as they bring in $$$.
The first experience is of limited value to the world of real business.
I'm continually exposed to new kinds of software engineering roles I never heard of at tech companies. (fwiw software engineer is sometimes still considered an inflated title.)
One data engineering role asked me to implement k-means from scratch and one data science role asked me to do some algorithms whiteboarding. But beyond this, people just asked SQL. From the rest of this thread it doesn't look like people would consider that to be 'SWE ability'.
Should data scientists be expected to do more than that? Or are you expecting a certain level of code quality from them? Within a 48 hour time window?
As far as this thread, don't worry too much about it.
But that's by no means all of the DS field. There are lots of DS jobs where you're collecting and interpreting and communicating about complex data sets. An engineering mindset is occasionally helpful, but a bias towards building versus towards analyzing and writing can just as often be counter-productive. Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution.
The bootcamps have contributed to the problem by focusing so much on building things. The idea that you can go from an econ undergrad to being a self-sufficient member of a production ML team in 6-12 weeks is nuts. What's less nuts (and what I wish programs like Insight focused on) is taking people from having data skills in one domain and with one set of tools (e.g. logitudinal medical record data, stored in CSVs and handled in Stata) to another set of tools (billions of rows of event-based product data stored in a data warehouse, processed in R or Python). But instead the bootcamps behave like the missing skillset is the ability to make a predictive random forest model on some arbitrary data set and build an AWS web app around it. THAT job market definitely doesn't exist and is completely over-saturated.
But people who are smart communicators about data, can manipulate and make sense of massive data sets, can ask incisive questions about their data, and can use data to convince people of a complex argument are always going to have job opportunities, even if they're not production grade engineers. If that sounds like you, I'm hiring - hit me up on Twitter: @drewwww.
I am happily serving a niche market with my own company and I suspect part of the difficulty in finding the skillset is that we can just start our own thing when we find a domain that we like.
Agreed wholeheartedly. Reminds me of another quote from http://www.john-foreman.com/blog/surviving-data-science-at-t... :
""" You know what can keep up with a rapidly changing business?
Solid summary analysis of data. Especially when conducted by an analyst who's paying attention, can identify what's happening in the business, and can communicate their analysis in that chaotic context.
Boring, I know. But if you're a nomad living out of a yurt, you dig a hole, not a sewer system. """
I'm not coming from a CS background and don't purport to know absolutely all of the details of all the mathematics and theory behind many of the machine learning algorithms that I use. I try my best daily to expand my knowledge, understand the algorithms, and apply them appropriately. I would hope that any company who is looking for someone who has a PhD-in-CS-or-ML could weed someone with lesser knowledge out during the interview process.
With that being said, a couple lines of code using SciKit Learn and all the default parameters is enough to impress many non-tech companies that are looking for a way to use 'predictive' in their marketing materials. And they pay very well for it. I get the feeling that provokes the ire of people who think those types of basic implementations belong to the traditional label of 'data analyst'.
For what it's worth, I work with data sets that aren't quite large enough to justify anything more than Python, Pandas, SKLearn, Luigi pipeline, and PySpark. The vast majority of my time is spent cleaning the data and generating features, must less on the hyperparameter tuning, model training side itself.
Anyways, I think I'm rambling a bit.
I just want to say that I LOVE this job, whatever the label is, or whatever the hype surrounding the label is, and I hope it's around for a while before it's automated...
Except for when it's not - those are the cases you're hired for.
But if you have an important data driven decision that is going to cost you a lot of resources (money, people, time, etc), you want the analyst to be able to speak to what the data and/or model is telling you and why. When you have this type of problem and it's "solved" by a few lines of SciKit Learn, you should be prepared to have a really bad day at some point in the future. This is the type of Data Scientist that draws the ire of people who do analysis with more depth.
3 easy steps to get a job in DS if you want them though: Grad Degree in Math/Stats/CompSci; work on a bunch of hard to predict problems and then publish and present them to your local meetup community to gain experience; learn engineering tools and devops and be about 90% as good a software engineer as your team's actual engineers (git, hg, IDEs, java, pig)... your brilliant models are way less important than being able to help the already overwhelmed engineering team make them work.
I think making the transition from the first role to the second role comes with experience, both with the toolsets, and thinking about the problem as a whole.
Isn't that describing a statistician?
This depends quite a bit on critical thinking, a good fundamental ability to analyze a problem and understand its parameters, then manage the logical operations required to deliver the feature and solve the problem.
As for why I think it's on HN every day: I also like to think of an innovation pipeline happening something like this:
We're now in some sort of refinement cycle of innovation, where the current medium has been saturated on some level and there is a lot of push to mine value from the discoveries.I was thinking in terms of the development like this:
1. Observation made 2. Idea created 3. Software/Hardware made 4. Revenue achieved 5. Business parameters tuned using insights 6. Maximum profit achieved
http://www.dice.com/jobs/detail/-/Diceinc/790523?rno=7814317...
The application process via that link is smoother, or so I am told. Please note that while this position is remote, we are only looking for US citizens or Visa holders located within the US right now. Dice.com link:
1 point by simonhughes22 0 minutes ago | edit | delete [-]
This links directly to the dice.com version of the posting (same position). http://www.dice.com/jobs/detail/-/Diceinc/790523?rno=7814317.... The application process via that link is smoother, or so I am told. Please note that while this position is remote, we are only looking for US citizens or Visa holders located within the US right now.
I understand that it's a remote job, but I'm not sure if you'd be considering people from Europe as well.
ps: the link works fine now
If you're serious about machine learning - build a blog or online repository of quality work and use that to get a job instead
If you're hiring: Get the above out of your pathetic small minds and start hiring the smartest people you can find. Look for successes in any industry. Your business isn't that unique. The best people can learn it much faster than you did.
I'm hiring 6 people in a range of roles between "pure" data scientists to more data engineer/SWE roles. The exact mix depends on who we can get.
The ability to find good people is the biggest constraint on the work we do.
Our current team ranges from applied mathematicians (as in they are Math professors) to people with traditional SWE backgrounds. Basically we are a long long way from saturated.
I can't speak to anything regarding ML, but for whatever it's worth in our segment of the market we have seen a lot of competition emerge in a big way the last few years. Former academic-type firms who specialized in bespoke economy analysis reports are starting to build software around all of the data that is out there since it's never been easier to collect and normalize it. I think it's a stretch to say the market is approaching saturation for us, though.