Ask HN: In 2022, what is the proper way to get into machine/deep learning?
By getting into machine or deep learning I mean building upto a stage to do ML/DL research. Applied research or core theory of ML/DL research. Ofcourse, the path to both will quite different.
Standing in 2022, what are the best resources for a CS student/decent programmer to get into the field of ML and DL on their own. Resources can be both books or public courses.
The target ability:
1. To understand the theory behind the algorithms
2. To implement an algorithm on a dataset of choice. (Data cleaning and management should also be learned)
3. Read research publications and try to implement them.
210 comments
[ 3.6 ms ] story [ 216 ms ] threadAfter completing that, I think Kaggle competitions are a great way to master your skills.
[1] https://course.fast.ai
Afterwards, do a statistics class. Most algorithms these days are based on softmax, meaning the cross-entropy between two discrete/continuous probability distributions. There's a lot of choice in which distribution to use to model what and it will have strong effects on your gradients and, hence, training trajectory.
Concepts like shannon information and entropy are also very helpful for you to monitor training progress. Typical loss values will do exponential annealing and it'll be difficult to see further progress. But if you still reduce the bits of entropy in your classifier, learning is still going well. So you need to understand what to visualize and how to calculate that.
As for implementing research publications, maybe start with easy mode and go to paperswithcode.com . There, you will find papers AND their source code, so that you can look at how others implemented their paper.
As for FastAI and Kaggle, my personal impression is that it's mostly for toy problems. No real AI researcher would be willing to disclose their full source code to an international megacorp like H&M for a measly $15k in price money, yet similar terms appear to be the default on Kaggle:
https://www.kaggle.com/competitions/h-and-m-personalized-fas...
https://www.kaggle.com/competitions/dfl-bundesliga-data-shoo...
https://www.kaggle.com/competitions/feedback-prize-effective...
EDIT: Also, I strongly disagree with course.fast.ai on these points: "Myth (don’t need): Lots of math, Lots of data, Lots of expensive computers" To train a state of the art ASR AI, you need roughly 100x A100 for a month, 100,000+ hours of audio recordings, and math knowledge to find a maximum likelihood path through a logit matrix. Unless, of course, you're only working on toy problems.
Do you think there is a distinction between the kinds of problems that take some kind of "raw" signal data (audio, images etc) as input, where deep learning approaches appear to be fruitfully applied, and other kinds of problems that appear in many places in business and the public sector where the input space is not some kind of raw signal data but instead tabular data.
I have heard some people argue that the latter kinds of tabular-data style problems can be effectively tackled with a variety of statistical methods, and that deep learning style approaches do not offer an advantage.
Maybe fast.ai is commenting on the latter class of problem.
So my impression is that fast.ai teaches you how to use AI methods, but with example problems that didn't need AI to be solved well.
They're also very open that areas like ASR and RL take a ton of compute to replicate SOTA. This shouldn't be used to substantiate any sort of criticism of fast.ai the library since it is just wrappers on pytorch. Maybe on the course.fast.ai page they could make some sort of disclaimer that some areas like ASR and yet to achieve the quantum leap of performance that we've seen in other domains.
I disagree completely that fast.ai is for toy problems. It's a development tool that in my experience provides for more rapid iteration than starting in vanilla pytorch. Use it for toys, for challenges, for research, or in industry. You can always rewrite your solution in pytorch once stable if there is benefit in doing so.
I agree with others that fast.ai is a great place to start.
- Text Generation/Transformation: Easy
- Speech Recognition: Medium
- Optical Flow / Depth Recognition: Hard
- Actually controlling robots with AI: Nightmare
I've watched every video in that playlist. He is a fantastic teacher.
Do what the instructor recommends: watch each lesson once in its entirety and then re-watch it while playing along. But don't just type their commands verbatim. Try and do something slightly different.
Sure, most of this sounds as dull as a broken clock, but in my observation it makes the difference between students who can just use machine learning tools by copying textbook cases and adopting a lot of fancy new terminology, and those that understand what they're doing.
That difference really kicks in once you get off the beaten track of popular use-cases, into applying ML to new, unproven applications. Then you need a deeper understanding of why some algorithms may be useful and others are inappropriate.
This. Though I have no textbook I'd recommend; all of the ones I used were a very hard slog to read, let alone grasp the maths in them.
I like Hands-On ML... by Geron as a decent intro to ML book. FastAI seems a bit overrated to me - I didn't like that it uses its own helper library or the teaching style but it obviously works for other people.
Then there's more exhaustive books on theory - Elements of Statistical Learning, Pattern Recognition and Machine Learning, Bayesian Reasoning and Machine Learning, Murphy's books on probabilistic ML etc. But obviously the theory books have a lot of overlap with each other so there will be lots of material to skip after you've read one or two of them.
To really understand what is going on though, the path I am having some early success with (as a long time developer / data pipeline guy, but newly into the standard python / ML practice) is to run through Kochenderfer's "Algorithms for optimization" from 2019 (MIT press), including implementing the exercises, as optimization is the cornerstone of the majority of ML methods. Some of the most fun I've had in a long time.
Freely available here:
https://algorithmsbook.com/optimization/
From there on, I'm less sure, but expect I might experiment with implementing my own deep learning methods just for fun, or similar.
Quotes from the comments:
> For all my hacker news peeps that wants to learn ML and/or DL, you need to drop everything right now, go print this on the office printer, and sit outside with coffee for the next two weeks and read through this entire thing. Turn off the computer and phone. Stop checking HN for two weeks. Trust me, nothing better than this will come around on HN anytime soon.
> The authors are wrong to label this book as useful only to people with a physics background, and in fact it will be useful for everyone who wants to learn modern ML.
As someone who's only ever dabbled into minimaxir's GPT-2 packages this was an extremely approachable exploration (and explanation) of how a neural network works. I can't recommend it enough.
Try the Data/ML Engineer route. Instead of going directly into ML, try to work as a “supporter” of those doing ML. There’s a HUGE gap there, specially if you’re a good programmer.
There are a lot of people in the “pure” ML space, people with science background, with phDs, etc.
But there’s not enough people to support them: taking their models to producing, building their pipelines, etc.
If you get into Data/ML engineer, you’ll be working with these people and learning from them.
It’s a longer route for sure, but I think it can yield the highest success rate.
Chicken and egg problem. Probably the optimal way is just aiming for top ML research teams right from the start as a new grad. No other way is better.
At what?
If you want to be an SRE for a data platform, sure, but this pretty thankless work:
- cleaning up dodgy data
- cleaning up behind low code data pipelines and other painful integration work with systems that suck and sometimes just don’t work (like PowerBI).
- cleaning up behind data scientists that create models in arcane and imaginative ways and expect you to “productionise” them.
- cleaning up behind brain dead scheduling systems that fail unexpectedly.
- constant churn with partners and cloud products for whatever the latest hotness is.
If you want to be solving actual problems with ML, this is a dead end.
It’s IT support for the people doing real work.
…so, it depends on your goals. Getting a job? Sure! Everyone wants a workhorse who they can dump all the annoying problematic on-call tasks to.
Learning ML, contributing to research, building models?
This isn’t a path that leads there.
This is an appalling perspective. Good MLE skills seem a lot harder to find that good ML ones.
My apologies; It is real work; the point I was making is it’s not ML work, any more than writing a yaml file is ML work.
If you want to write yaml files, any number of possibilities exist.
If you want to work with machine learning, then don’t become a data engineer. The skills are, mostly, not related ML, and more closely aligned with SRE / devops.
It’s not infrastructure and helping build models as you mature and advance: it’s almost literally just infrastructure and fire fighting… in my, limited, 3 years of experience as such.
Yes, don’t get into it if you want to do ML research or apply ML, but if you are interested a bit in it and find building models the least creative, most boring shit ever like I do, and prefer traditional coding, it’s a nice spot to be in.
A lot of the hard part isn't the model, and especially in a world where bert, xgboost, optuna, pytorch, etc have solved much of the classic problem and forced 'real' DS to specialize on either the business consulting side (not math/engineering) or theory side (barely implemented). The rebrand of 'data analyst' (SQL, powerbi, . ..) to 'data scientist' by even top tech companies underscores this. It's not yet to where web dev has gotten in terms of global $20/hr fiverrr contractors, but already at say $40/hr for someone who can build real production models for more boring scenarios.
The result is the vast bulk of data scientists (phd, self-trained, consulting, ...) we interview are weak engineers, so going from a make-believe notebook to a trickier production scenario requires the data engineer / MLOps / etc to solve a lot that a typical DS doesn't really understand in practice. Scale, latency, distributed systems, testing, etc. Likewise, the part the DS solves has little to do with the latest neuroips paper, and more just about lifecycle tasks like getting better data, which the other folks on the team will often be involved with as well.
So 2 natural high-paying paths here:
data engineer / MLOps -> MLEngineer -> DS
data engineer -> all-in-one data analyst/scientist -> ML/AI data scientist
The DS will likely:
- be close to the business case & business stakeholders to ask questions a normal lead can't
- know the relevant math + ML algorithms, and build up specializations pairing DS niches ("time series forecasting") with industry niches ("supply chains in manufacturing")
- enough engineering & performance understanding to work with a DE on going from small data sets to big ones
- have an intuitive feel for all of the above - how data/usecases/etc. go right/wrong
That's a lot!!
One path is jumping in as a low-paid intern or new grad and doing your time. But a pivot is different, esp. to get paid along the way. Most CS grads had little math ("intros to stats, combinatorics, & algs; dropped linear algebra"), weak ML ("did algs; intro to ML only covered kmeans & bayes; tried running a BERT model on some data"), and little intuition for how ML typically goes wrong ("what's class imbalance?"). So if they do get hired directly as a mid-level DS, it's probably on a team of the blind-leading-the-blind. Oops.
BUT SQL/Spark/K8S/pandas/regex are real skills. Doing the data engineering, ML operations, etc., around making an ML pipeline more than a fanciful notebook that wouldn't last a minute in production is real work. That stuff does pay well, and by working with the ML folks, you'd naturally get pulled into the ML tasks as well. DS write all sorts of bugs that surface as production evolves and the full team works together on, and new features that needs a team to make real. So taking a job that mixes engineering specialties with ML specialties is a smoother pivot path for the typical CS backgrounds I've seen. Over time, drift to more ML-y aspects of the projects happening until you can do the full hop. (Nit: That won't teach the math & deeper intuition, so I'd still do courses + projects on the side.)
Am I understood correctly that there is much more demand for DE than for DS?
- a data analyst role rebranded as a DS role will be lower paid than a DE role, maybe 50% diff
- an actual DS role is probably higher paid than a DE role, but really depends on the job+co
- a great DS role and a great DE role are both super well compensated. Though maybe again DS higher than DE in most just b/c ability to more directly drive $. Unless something like an infra company, the DS will be inherently closer to the business & outcomes. ("I did this clever thing that netted 2% revenue spike that adds up to $40M/yr in new revenue, what did you do?")
It doesn't mean that there aren't data scientists who have some engineering experience as well, but this seems to be rare. For that reason, getting those ML models that they painstakingly build to where they'll generate some real value is super hard. They just don't know where to start. Working across multiple teams and multiple functions is very challenging and it often creates friction. Therefore, creating tools and systems that will enable those data scientists to see the actual value of their labor is paramount.
That's why we're seeing a huge resurgence of so called MLOps tools and platforms that aim to solve all or some of the problems of the entire stack. We are very very early in this journey, but I believe 2020's will be for ML and AI what 2010's were for the cloud and data, ie. new Snowflakes and Databricks but for the actual ML apps. It's exciting.
I absolutely think MLEng is important and much needed, but too often under appreciated. Being this half breed part engineer part ML leaves you on a lonely island often in many orgs. The ML managers don't really understand what you do and neither do the engineering managers. It is kind of thankless unless your management really understands your role and appropriately advocates for you.
MLEng is often an engineer who wanted to get into the sexy ML space and since it is in the title it feels cool. Then you realize you're more an Ops engineer who deals with the inane code of many "true" DS/ML scientists. Thankless, indeed.
Stuff to do could include: - Getting a network architecture to run. - Applying optimization depending on target arch (pruning, quantisation, custom cuda kernels, etc). - Integrating models (rule of thumb: a product is 95% ordinary code, 5% is ML related). - Constructing benchmarks, monitoring
From the boss's perspective, the grungy IT support stuff is closer to real work (although the only thing that's actually respected in management, because it's what they do) than the shiny ML stuff that one star hire is allowed to do because it makes the company look cooler than it actually is.
If you've ever crafted a pipeline and tuned it to hum along, then watched it break with new/more/messier data, then figured out creative ways to fix it or replace parts of it with more robust parts, iterating on that and scaling it up, you would know some of the fantastic pleasure of whatever you call it, data engineering.
This may indeed be how the term "data engineer" is used sometimes, but I have my doubts that it was originally created with this meaning. Not really sure where/when the term "data engineer" was actually created, but ICDE started in 1984 [1] and the Data Engineering Bulletin was renamed in 1987 [2] (from "Database Engineering"). It seems likely that the term "data engineer" has also been used since at least then.
Of course ML did also already exist then, but it's certainly a while before the current "big data" / "deep learning" time. And regarding the topics considered "data engineering" at that time, this is from the foreword of the December 1987 issue of the Data Engineering bulletin:
> The reasons for the recent surge of interest in the area of Databases and Logic go beyond the theoretical foundations that were explored by early work [...] and include the following three motivations:
> a) The projected future demand for Knowledge Management Systems. These will have to combine inference mechanisms from Logic with the efficient and secure management of large sets of information from Database Systems.
Which sounds just as relevant today as it did back then. It also does sound like a rather challenging task, and not exactly like "unpleasant repetitive work". Or at least not any more repetitive than: change some model parameters / retrain model / evaluate results / repeat ;)
[1]: https://ieeexplore.ieee.org/xpl/conhome/1000178/all-proceedi...
[2]: http://sites.computer.org/debull/bull_issues.html
I used to be a member of SIGMOD and the "data engineering" you mentioned was just an academic term.
In Star Wars, it takes Luke Skywalker years to learn to use a light saber skillfully. Then in The Force Awakens, some ex-Stormtrooper with no training picks up the light saber and within 5 minutes is a pro. Kinda ruined the mystique of Star Wars, just like people jumping into ML with no training ruins the mystique of ML.
Creative and critical thinking is not exclusive to people with a PhD. The ability to understand ones strengths and weaknesses is not exclusive to a PhD.
I would never attempt to write or publish a paper without help of somebody with stronger mathematical or statistical knowledge. On the other hand they should not write source code for a paper without consulting somebody with a strong background in sw engineering. You complement each other. Power is in recognizing that.
You would be surprised how many software bugs I have found that invalidated entire (draft) papers. A PhD in ML doesn't save you from that.
Years and years ago in the movie universe, Luke had to painfully learn to use his lightsaber from a mentor who passed down techniques and philosophy to the student.
In the current day of the movies, lightsabers are understood to be powerful, yet temperamental and exotic, weapons. Mildly trained individuals can use them, even if it's to a limited extent (i.e. flick switch; shiny side is the business end; heat bad, ouch, no touch).
To belabor the metaphor, you've also had a tradition of people, from all walks of life, using vibroblades (basic to advanced standard statistical analysis and regression) in order to achieve some level of parity against users of lightsabers.
Smart people will be able to contribute even if they don't have a PhD. Some PhD are useless and everyone is wondering how the hell they go through that.
What is unique to a PhD is that you took a very long time to master a small slice of the knowledge pie. The emphasis here is on long time: Most people simply aren't willing to go for years on a low salary and tedious job.
It doesn't mean PhD's are more (or less) creative, top coders and whatnot; it means we took the time to read all the papers, to know all previous solution attempts and who all the big players in the field are ("all" w.r.t. our niche). It also mean we can read papers much faster than other people because that is basically what we do all day.
There you have it! Now don't send me ML job offers, cause I gotta read this next obscure paper to figure out if they are legit. :D Just kidding.
From one perspective, it is disheartening that a deep understanding of the underlying methods doesn't necessarily win the day. From another, it is quite remarkable that having good implementation skills and a methodical mindset can get you quite far.
Most of the people doing real ML in industry are prestige hires--they're hired because their names draw people in, but basically get to work on whatever they want--and you need a top-10 PhD at an absolute minimum to be eligible for those.
The ugly truth about industry is that 99.9997% of it is flow capture based on power relationships, found artifacts (i.e., corruption opportunities) within the state, and the implementation of very simple processes but in a way such that the threat to executive reputations as a first priority, and profit as an important second one, are minimized. This doesn't exactly make a market for ML innovation, unless your boss for some weird reason still cares about being a co-author on your papers (which his bosses will pressure him not to let you publish, because after all, this publishing is a distraction from your paid work).
On the other hand, if you want to be able to afford a house in the Bay Area, and to be tapped for (indeed, most likely forced into, both due to losing interest in and being unhireable for IC work) management in your mid-30s... then go for industry. The poison carrot will make you sick but it will kill you more slowly than poverty, so that ain't so bad, now is it?
There's a vast amount of work that doesn't involve unethical recommendation systems.
Expand your horizon outside the Bay Area.
The plurality of work I see is straightforward computer vision/NLP applications.
The landscape is varied. There are companies doing real actual big leading edge stuff, there are companies where ML is sprinkled onto projects as a buzzword but no real interesting work happens, and companies that just need a practical small solution like the ones you mentioned, and could get by with Core ML, but don't because they hire a PhD who isn't aware of Core ML.
I’m interested in “flow capture based on power relationships”.
Do you have any recommended reading on this?
Only the ones which have an unkillable cash cow. So, I suspect Google or large banks are mostly like that, but places like SpaceX or even large consulting firms (Delloitte, IBM etc., where managers essentially eat what they kill) cannot allow themselves to degenerate into a Chinese court.
The large-scale failure of democracy that's happening all over the world is something different, though. Regulation is struggling to keep up with technology, and it doesn't help that nation-states have already been doing a piss-poor job of protecting people from their employers. If the US falls in the next 20 years, it won't be due to Covid or Trump or nation-level adversaries; it'll be due to the obscene power given to employers, who can literally ruin an employee's life--not just fire him, but anally ravage him in perpetuity with bad references--for any reason or none. Eventually, unless national governments start dropping serious lead pipe on employers' heads, people are going to tire of paying 30+ percent of their incomes to a government that lets bosses get away with this shit.
How do you define "US falls"?
>Eventually, unless national governments start dropping serious lead pipe on employers' heads, people are going to tire of paying 30+ percent of their incomes to a government that lets bosses get away with this shit.
People endured much worse in medieval times, and endure much worse right now in China.
Really? The USA has hollowed out portions of the country equal/worse than the worst 3rd world countries.
Can you elaborate on that? I've never heard that parcitular thing about the US. For reference, In Kongo, there are 5 year olds today carrying heavy buckets in makeshift cobalt mines, a'la XIX century England or France (plus the toxicity of cobalt, people who work in these mines get cancer if they don't die in an accident first). Even with whole families working in such conditions, the pay is not enough and not stable enough to sustain the family, and they are often working while hungry. Is there anything comparable going on in the US?
The places where the police do “bother” are, ipso facto, also run by local gangs.
The US does seem to lag the UK and Europe in terms of employment law in some cases (no formal employment contracts for most employees, can be fired without notice, little statutory holiday, maternity or paternity leave entitlement, etc. etc.)
It has been argued that the union movement — while susceptible to corruption — was a hugely positive force in economic and political terms for American workers. Unfortunately thatcher and Reagan saw this as such a threat that they attempted to destroy their own manufacturing base in order to smash the unions.
There are exceptions - where someone principled dictator impose a VC style model where teams basically become independent startup and die or succeed. 100 fails, one becomes the next revenue maker for the company. That's how AWS was born.
Accountability comes from elections. If managers in companies had to be re-elected it would be interesting.
Companies are fantastic at making it look like accountability exists, because people at the bottom get punished for even the smallest mistakes, but avoiding any consequences that would affect high-ranking members or force the organization to change how it does business.
While data engineer is an excellent role for a fresh graduate, the data engineering profession shares many similarities with the SRE/IT professonional.
The best data engineers are the folk who had job title of DBAs of yonder year.
You will always be the supporting cast, rarely the star.
The best way is to take Stanford Deep Learning courses at SCPD, build a reputation, do real ML work (even if it's not a PhD, it's the same courses Stanford PhDs take).
If you want to do real ML work, you pretty much need the PhD. This is a hard thing for people who have 140+ IQs but do poorly for whatever reason with formal education to accept, but it's true. Even if you get one real ML job without a doctoral degree, you won't get a second one.
Sure, other 140+ IQs can recognize very smart people with only (or not even) a bachelor's degree, but (a) your career in industry will be influenced by the opinions of not-smart but politically empowered people who rely on heuristics like educational prestige because they can't judge the genuine article, and (b) some of those 140+ are nevertheless scumbags and will use (a) against you.
If you want to be a serious player in an academic field like ML, you need to not only get the degree but start publishing and never stop. It doesn't matter all that much if your papers are any good; no one in industry will ever read them. But you need the image of a successful academic who's just slumming it and can go back any time.
Government may be an option, although it doesn't pay as well as industry and you can end up in a comfortable but stifling role.
In corporate, though? Yeah, you pretty much need to have the appearance of star power, which means degree prestige matters. Whether you're actually any good (and, trust me, there are plenty of mediocre people from top schools) doesn't really matter, because the decision-makers are too stupid to know the difference.
There are ways to play this, though, if you're aiming at industry. Harvard isn't a top-10 CS department, but the people in corporate aren't going to know that, and so "Harvard PhD" is going to make them fellate you just because it's Harvard. That may be an avenue. Or, better yet, get a PhD in something that sounds technical but is easier and less selective.
That said, if your goal is to play the corporate game and make a lot of money, you should probably forget about ML and focus on becoming a manager as quick as possible. If your goal is to do intellectually stimulating work, you should probably not consider corporate, because your work is going to be evaluated by people who are literally 50 IQ points too dumb to do so, and while this noise factor is manipulable rather than truly random, the people who have the skills to do perform said manipulation tend to go into management, not technology.
Maybe if you consider doing "real ML" exclusively working with Deep Mind or on Meta core research team, but there's a lot more to "real ML" than just these teams.
I'm curious how long you've been in industry and what types of orgs you've worked at to get this impression?
edit: In general the comments in this post are bizarrely out of touch with reality and have really shifted my perception of the avg HN commenter.
Even if you find such a role it will be chaotic and not meaningful.
It will give you a ton of exposure to ML techniques and infrastructure practices, but the true modeling work is still done by PhDs with the prerequisite background/knowledge. You will be taking black boxes -- pre-built models -- and doing the data cleaning, fine-tuning, and experimentation.
I know a handful of people who transitioned to full-on ML from this role, but those individuals were quite gifted and hardworking, and probably could have learned without this exposure. There's a significant gap in knowledge that can only be attained through academic experience.
So many PhDs I've worked with have known the literature but couldn't produce something valuable to save their life. That might have been the nature of them being over billed on projects and spread too thin, but I do not think that is a requisite for building models. It certainly isn't for the rest of the engineering pipeline.
If you want a role in a small company building best-effort, out-of-the-box models, then the courses are plenty fine.
Is there difference and who gets more and how much more?
You need to be not afraid of doing proofs of theorems (most of them have to do with stats because machine learning is basically stats on steroids).
I started out without *any* background knowledge a few years ago. Found the Data Scientist career track of Datacamp pretty helpful, since it goes beyond programming and includes the mathematical and statistical theories as well. (https://www.datacamp.com/tracks/data-scientist-with-python)
It's basic, but a solid foundation to build upon.
If you're already familiar with most of these topics, fast.ai is the way to go!
If you want something more theoretical there is a book by Hopcroft et al. that was released in draft form a number of years ago. It appears to be out for real now: Foundations of Data Science, by Avrim Blum, John Hopcroft, and Ravindran Kannan. Blurb and video lectures: https://www.microsoft.com/en-us/research/publication/foundat... I just found these so haven't looked at them yet. The book draft (2014, wow) is here: https://www.cs.cornell.edu/jeh/book11April2014.pdf I didn't stick with it long enough to make much progress, unfortunately.
Kaggle.ai problems are a good set of practical projects even if you're not aiming to be competitive at them (which takes a lot of effort and resources). The Fast.ai vids are ok as preparation for them.
I would start with math foundations: basic linear algebra, stats, probability and some analysis. CS undergrad level is plenty of math for start.
Then I would try to understand back prop on intimate level: learn how to calculate gradients, maybe take a look on how autograd works as well.
Then you should know a bit to pick your next steps by yourself.
One has nothing to do with the other.
That said data engineering at scale pays a lot more than deep learning but is also a lot less fun. Figure out which you'd rather do.
In my opinion, jump straight into this! Learn prerequisites as you need them.
I found Goodfellow's book [1] to be helpful to learn some basics.
But don't think you need to read the whole book before you start reading and implementing research papers.
If you try and build up all the fundamentals thorougly, you run the risk of going down a very deep rabbit hole e.g. learning real analysis so you can learn measure theory so you can learn measure theoretic probability theory so you can learn stats properly etc.
You can be a productive researcher and patch up the fundamentals over time.
[1] https://www.deeplearningbook.org/
1) Get some statistics/probability basics. It's full of people (you can see a lot of analyses on Kaggle) that "do machine learning" but make very silly mistakes (e.g. turn categorical data into a float and use it as a continuous variable when training a model).
2) take a look at traditional machine learning approaches. Nowadays you're swamped by DL (a lot of good suggestions on this thread, I won't chime in), and you miss the fact that, sometimes, a simple decision tree, or dimensionality reduction approaches (e.g. PCA or ICA) can yield an incredible value in a very short time on huge datasets.
I had written a fairly short post about it when I finished my georgia tech path https://www.franzoni.eu/machine-learning-a-sound-primer/
3) It can take a lot of time to become effective in ML, effective as in, what you _manually create_ is as effective as picking an existing trained model, fine tune it, and use it. This can be frustrating: low hanging fruits are pretty powerful and you don't need to understand a lot about ML algorithms to pick them up.
4) Consider MOOCs or online classes. I took Georgia Tech OMSCS, I can vouch for it and some classes force you to be a data scientist and read papers as well, and you can have "real world" recognition and discuss with your peers, which is useful!
Your first model should always be something that predicts a constant value, or maybe in really complicated cases something like a linear/logistic regression. Then you have a baseline to compare more advanced approaches to. But in order to understand how to use linear regression well, you need to understand how it works in the first place.
Also experiment structure, sampling design, hypothesis testing, etc. will tell you a lot about what conclusions you can and cannot draw from observational data, which is what a lot of ML is about.
One clarification: you don’t need an extreme understanding of stats, probability, linear algebra, imho. If you already took college level classes, you’re likely to be fine.
So my point is, if it isn't making sense the way you are being taught, go explore other avenues. There's no way these machine learning algorithms would have made sense to me as a 20-something undergrad, but as a 40-something who can explore them via software rather than a whiteboard, they really aren't that complicated to get started.
This example: > It's full of people (you can see a lot of analyses on Kaggle) that "do machine learning" but make very silly mistakes (e.g. turn categorical data into a float and use it as a continuous variable when training a model).
Doesn't seem related to stats or probability at all to me. Just critical thinking skills
But if anyone wants to become an expert into ML/DS, learning statistics and probability is fundamental. Books like A First Course in Probability, Introduction to Statistical Learning and Elements of Statistical Learning, to name a few, are very important.
A lot of the mistakes done in practice are based on a lack of understanding in sampling techniques, how statistical metrics can be misleading and so on.
First thing I learned in statistics is the difference between quantitative and qualitative information. If someone knows this before hopping into Kaggle, they know that categorical features can't be used as continuous features.
Though for applied papers that's sometimes enough.
But heed Larry Wasserman's advice: "Using fancy tools like neural nets, boosting, and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a band-aid."
For the 101 level, I agree; I’d say you need a good understanding of basic probs and stats, rather than a vague understanding of advanced topics.
If you can spot when somebody says something about a dataset, the assertion is true only if the data is normally distributed, but there’s no checking about the actual distribution, you’re probably good to go.
[1] https://www.youtube.com/watch?v=j9WZyLZCBzs&list=PLUl4u3cNGP...
Do you want to:
a) Become an academic in mathematics/statistics.
b) Become an academic in computer science with a focus on artificial intelligence.
c) Become a MLE in "regular" statistical applications. Aka bayesian classification, "core" statistical principles.
d) Become a specialized computer vision/natural language processing focused MLE.
e) Become a generalist software engineer who can whip out the above if needed.
In no way is e) the inferior option.
Generalists who can write code fast with 100% test coverage and pristine logging are by far the segment the industry has the shortest supply of.
There are TONS of math guys. Vanishingly few Principal Engineers who can write a design document and lead a project.
(Machine learning customers are OBSESSED with test coverage and verifiability. Believe it or not, multinational corporations generally don't want to unleash a {your_adjective_here}ist algorithm on the world.)
2. Study the above, properly.
To study the math, Elements of Statistical Learning/Algorithms by Goodfellow.
Start on page 1, do every second exercise. Publish a summary of every chapter you finish with your answers to GitHub.
3. Pursue your goal in a publicly verifiable manner.
See:
https://news.ycombinator.com/item?id=32071137
https://www.deeplearningbook.org/