Ask HN: How to get back into AI?
I was involved in machine learning and AI a few years ago, mainly before the onset of the new diffusion models, large transformers (GPT*), Graph NNs and Neural ODE stuff.
I am comfortable with autograd/computation graphs, PyTorch, "classic" neural nets and ones used for vision-type applications, as well as the basics of Transformer networks (I've trained a few smaller ones myself) and RNNs.
Do you know of any good resources to slowly get back into the loop?
So far I plan on reading through the original Diffusion/GPT papers and start going from there but I'd love to see what you think are some good sources. I would especially love to see some Jupyter notebooks to fiddle with as I find I learn best when I get to play around with the code.
Thank you
146 comments
[ 0.33 ms ] story [ 191 ms ] threadIt sounds like you're into it already. And you already know which new papers are interesting to you.
Quite honestly, the opportunity all seems to be on the front end. The idea that you are going to airdrop yourself as a hands on AI programmer into this market doesn’t make a huge amount of sense to me from a career perspective.
The opportunity is with the tools and how they are applied. Building front end experiences on ChatGPT and integrations and applied scenarios.
You actually doing the AI yourself means competing with PHDs and elite academics immersed in the field.
I think knowledge of AI is far less valuable than knowledge of the emerging landscape combined with a broad understanding of different tools and how they are applied.
The new trend here is very strongly Large Language Models (LLM). You should be far more specific with what your goal is and where to spend your time.
A lot of the “AI” you are referring to seems to be no longer relevent or interesting to the market.
If you are spending time with Jupiter notebooks I would say you are probably completely wasting your time and heading in the wrong direction.
LLM is the major trend. Focus entirely on that and the tools landscape and how to integrate it and apply it. It feels like you are navigating using an out of date map.
>Fine tuning models is where the future will be
Who would request fine tuning? Businesses?
Are you suggesting to build some sort of data set broker?
I think it'd be fun to use vision models to pick up the interesting parts of an image, describe only the high-yield context as English text, and put them in a sort of Unix pipeline or node graph to connect it to other models that can input/output text. With fine-tuned or prompt-engineered LLMs as the intelligent centerpiece.
Rather than jump on the same horse that everyone is jumping on, maybe one should start looking at where language models fail -- and from the very nature of how they are conceived and what they are made of -- will most likely never be good at.
For anybody intrigued, here are those Reith Lectures:
https://www.bbc.co.uk/programmes/m001216k/episodes/player
https://wikipedia.org/wiki/Reith_Lectures
This would have been influential to me when I was a teenager and nihilistic about accelerating technology in the hands of distinctly-not-developing humankind (reading all about the Manhattan-project forebears to AI), but after devoting a decade to science and being fascinated yet boycotting of AI research and it's implications, I think it has ended up a bit too late for a change of heart and direction to have as much positive impact.
For those fortunate enough to be in a position of even trivial influence over the cutting-edge, and whose moral sophistication can therefore matter more than the next person, I hope you don't take the implications of whatever agency you may have lightly!
I wouldn't be surprised if AI conferences have a minimum of 10 spooks from every major western intelligence agency in attendence.
In general, it is thought that physics knowledge transfers better to ML than math because it's less abstract and physicists are more likely to be used to dealing with large datasets and software.
If you're already in the field you'd likely have to take a pay cut, but the federal government is always interested in research physical scientists and will snatch up anyone who knows the physics and also knows ML.
Also you‘re wrong. Look at what the OP wrote and then look at how the latest models are actually built and you would see at least 2/3 of their knowledge is relevant.
"knowledge of the emerging landscape combined with a broad understanding of different tools and how they are applied."
MS and PhD students start out behind and they’ll spend most of their time during the next 2-5 years on irrelevant things (e.g. 95%+ of their graduate coursework will be behind the state of the art; most PhDs will focus on a niche project that fails to have major impact or relevance by the time they graduate).
It sounds like OP just wants to learn out of general interest, which is fine. But others shouldn’t be discouraged. A sufficiently-dedicated person with talent and a strong classic ML foundation can catch up reasonably fast, to the point of getting their foot in the door professionally.
And this: "If you are spending time with Jupiter notebooks I would say you are probably completely wasting your time" And how do you suggests one performs data analysis on any problem that's not an LLM -- data analysis of any kind, such as "is the model I am trying to build a front-end for even works for my problem?"
Example 1: have a look through here: http://synapticpaint.com/dreambooth/browse/ for some examples of dreambooth models people have created
Example 2: you can merge different dreambooth models together to varying degrees of success (the idea being, you train model A on subject A, model B on subject B, and now you want to generate pictures of A and B together). My understanding is that this doesn't work too well at the moment, but it's possible that a different interpolation algorithm can yield better results.
I do agree with the general sentiment that you wouldn't necessarily be training your own models or creating your own architecture, just want to provide the perspective that understanding the AI side is valuable because it can lead to different capabilities and products.
Can you explain the value of these models, please?
This is a serious question. My eyes just see one horrendously ugly (and dystopian) eyesore after another.
Personally, I see this tech capable of destroying and recreating the advertising industry completely via inserting everyday consumers into ad media, depicting them as happy consumers of a product they've not used yet - while celebrity spokespeople, appearing as their personal friend, inform them how much the celebrity idolizes them for using said product. This is an obvious non-subtle application. There will be many, many more.
My impression is that the field is more disciplined in terms of knowledge now than it was ~8 years ago - the fundamentals are better understood and more clearly expressed in literature.
Also there are still plenty of topics on which the new techniques can probably be fruitfully applied, especially if you have some domain knowledge that the math/CS PhDs don’t have.
For OP - I’m in a similar situation and have been going through Kevin Murphy’s “Probabilistic Machine Learning”, which is pretty massive and dense but also very lucid.
Is that really true? That's not my impression at all (though to be fair I haven't been keeping up with current research as much as I used to). My understanding is that there is still hardly any knowledge on what deep learning models (and large language models in particular) actually learn. The loss surfaces are opaque, one still doesn't know why local minima reached by gradient descent tend to generalize fairly well for these models. The latent representations that generative language models learn is, with the exception of the occasional paper that finds some superficial correlations, hardly investigated at all and overall extremely poorly understood.
Very much interested in any references that contradict that sentiment.
The previous version of this book was from 2012 though and I'm not 100% sure how much of the material in the current edition is new (there is definitely a _lot_ more deep learning stuff in it).
So yeah it could be that my impression is wrong, or that I made the scope at which it applies sound bigger than it is.
Almost all of the content that the new book covers, with the exception of the third part on deep learning, is about theory that was almost exclusively invented before 2012. Classical ML (non deep learning) is actually very rigorous compared to modern ML. There exist good theorems (statistical learning theory) for most of the classical models I'm aware of.
Yes, LLMs are amazing but they won't be winning every single Kaggle competition, displacing every other ML algorithms in every setting.
Also, are you aware that one of the most prominent AI tools of this month (ChatGPT) was obtained with RL!
Upgrading existing systems with AI is probably where it's at using existing models like stable diffusion, GPT-3 or some of the smaller downloadable language models if the task is very simple and the economics of using GPT-3 don't make sense.
https://course.fast.ai
I just read Francois Chollet's Deep Learning with Python and found it to be a fantastic high level overview of all the recent progress. There's some code, but not a lot. I mostly just appreciated it as a very straightforward plain-language treatment of RNNs, CNNs, and transformers.
Now I'm going through Stanford's CS224 lectures.
I'm sort of planning to read papers but as some other comments have pointed out, I'm less sure of the ROI on that since I'm not sure how feasible a future in AI is for me
Some of the stuff I'm currently reading/watching or have recently
Practical Deep Learning, though it sounds like you may know this stuff already (https://course.fast.ai/)
Practical Deep learning part 2, more about diffusion models. Full course coming early next year (https://www.fast.ai/posts/part2-2022-preview.html)
Hugging Face course (https://huggingface.co/course/chapter1/1)
Diffusion models from hugging face https://huggingface.co/blog/annotated-diffusion https://huggingface.co/docs/diffusers/index
Andrej Karpathy's Neural Networks: Zero to Hero. He goes from the basics up to how GPT etc, so you can start wherever suits you (https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...)
3blue1brown's videos. I've found all his videos on neural networks and math worth watching, even for stuff that I already know, he sometimes has some new perspectives and nice animations.
brilliant.org. Nice math refresher and the courses there are almost like fun little games.
Looks like it's a companion to this YouTube series that looks pretty interesting https://youtube.com/playlist?list=PLQVvvaa0QuDcjD5BAw2DxE6OF...
I will check it out for sure
https://compneuro.neuromatch.io/
Recent research like "Relating transformers to models and neural representations of the hippocampal formation" might make it more relevant though (https://arxiv.org/abs/2112.04035v2)
quote from the abstract of that paper: "Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension."
For example, do you want to develop models as a hobby? Make models or software for a living? Use AI in some particular problem domain?
If you're the same, is there something more specific to help you focus and put in the time?
Odd take to be honest. Like suggesting you cannot believe someone is a good materials scientist, because their CAD/mechanical design skills are not up to par with a mechanical engineer.
If you always throw over the wall, you probably don't understand how results that look good in a Jupyter notebook can easily fall apart in the real world.
Most medium and low-quality papers are full of errors and noise, but you can still learn from them.
Get your hands dirty with real code.
I would take a look at those:
https://github.com/geohot/tinygrad
https://github.com/ggerganov/whisper.cpp
I've done some work in the field, and I've seen errors in almost all papers.
Sometimes they are not really errors, but they give a solution that only works in the very specific context they tested it and completely falls apart anywhere else.
I've lost countless hours and days studying and replicating papers. Of course, ML could be an entirely different experience.
I am a research engineer/applied ai person building vision models in healthcare domain. I am currently preparing to transition to engineering roles like you did. For that, I am currently going through web dev - both frontend and backend. Would love to get some pointers from you on my approach and any recommendations from your side. Thanks!
Build out your portfolio so you have projects that you can speak to and show your tech depth.
FWIW, I do AI backend at FAANG and mentor as tech lead.
As someone with 10 years of professional experience in software, I find every AI "trend" that has come up in that time to be incredibly odd. It is certainly remarkable what chatGPT, StableDiffusion, and other examples are doing today... Ultimately people are giving waaaaaaaay too much credit without understanding the technical details. These are pidgeon-holed examples that still aren't solving any real problems.
AI is still just statistics with marketing.
I’ve found that what ChatGPT comes up with is often a great start and has already saved me hours of time. Is it as good as paying a professional? Probably not. But I think it’s fair to say these models are already solving real world problems, even if they still need a bit of a helping hand. Just my thoughts, I’m not an expert on the models themselves.
> mostly as a favor to folks I know personally
Scale is where the rubber meets the road. I'm glad ChatGPT has been helpful to you, but it is still a gimmick for the majority.
Now I just ask ChatGPT.
I would say AI is the opposite of statistics, for good or for bad.
Maybe it is and he's simply somebody with an interest for the topic and some progress to catch up with.
Considering OP mentioned a lot of recent hype trains relating to AI, it's perfectly reasonable to assume they are chasing dollars, rather than contributing to the underlying technology without further expectations.
So, OP comes off as very disingenuous when they lead with the hype and provide nothing of substance.
Just use Hugging Face.
The OP wants specifically sentence-transformers, which are hosted on huggingface but it's a separate library.
You'll want to go with whatever models that the sentence transformers documentation recommends.
For some context, something I should have mentioned in the original post but failed to do: I was not intending to do a professional pivot to an AI role; it is more of a personal interest. I used to be really excited about this stuff and am looking forward to getting involved in it again just because I find it interesting.
Thank you, I really appreciate everyone's responses.
I am an ML researcher working on generative modeling. I think you have enough experience that you'll catch up quickly. But the question is most what you're interested in? With that I can give better advice. Don't let anyone stop you from learning just to learn. Not everything has to be a career. A lot of us got here because it's fun.
I do think you'll pick up diffusion models quickly. I like the explicit density side of things more and density estimation. So I like Song's works and similar with Kingma. Also check out Lilian Wang's blogs. They are a wealth of material and sources. Can't go wrong there. You'll find that diffusion and VAEs are kinda similar. The difficulty you'll have in understanding something like stable diffusion is actually the programming (at least this was the hardest part for me).
Good luck and let me know if I can help.
There are a few non equivalent universal approximation approaches. I’m not sure I fully understand why this is will end up being the one even on a 10 year horizon.
But understanding AI fundamentals gives me a fresh perspective on how to build applications that leverage ChatGPT (for example).
Crafting the inputs to achieve desired outputs. Training the models with a corpus of data relevant to a niche industry, etc...
When this course becomes public next year, I think it will be a great way to get caught up. In the meantime, you might still be able to pay the AU $500 fee and watch the course content, which was all recorded, if you are anxious to get going.
This is the way to start a PhD research (assuming OP is trying to stay up-to-date in this area by "get back").
Also there's no cost, no book to buy, no email signup, it's just a guy sharing knowledge like the old days. Great course.
Also, if they ever make another dataset like the pile, don't expect to get your own data into it (even if they say that they will add it).
I'd work with other groups if I were you.