This is really cool! I especially like that there's a premade colab notebook that lets you play with it: https://colab.research.google.com/github/openai/clip/blob/ma... . I'm a little surprised that the paper doesn't…
I don't think you could get access to the actual models that are being used to run e.g. Google Translate, but if you just want a big pretrained model as a starting point, their research departments release things pretty…
> Is it always good enough to take the outputs of the next-to-last layer as features? It usually doesn't matter all that much whether you take the next-to-last or the third from last, it all performs pretty similarly.…
There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems:…
Yeah, I think this pattern is pretty common. (Basilica's main business is an API that does deep feature extraction as a service, so we end up talking to a lot of people with tasks like yours -- and there are a lot of…
I think if you have a small to medium sized dataset of images or text, deep feature extraction would be the first thing I'd try. I'm not sure what the most interesting problems with that property are. Maybe making…
I hadn't read it before! That's a fascinating result, actually. They emphasize interpretability in the paper, but I find it more interesting that you can do so well with only local information. My first thought is that…
I don't work with time series data much myself. I would imagine you can get at least some transfer learning, since there are patterns that show up across different domains. It looks like there's been a little bit of…
Definitely. There's been a lot of exciting work recently for text in particular, like https://arxiv.org/pdf/1810.04805.pdf .
Linear Algebra Done Right would be my recommendation.
Hi everyone! Author here. Let me know if you have any questions, this is one of my favorite subjects in the world to talk about.
> He's almost describing a future where we might buy/license pre-trained models from Google/Facebook/etc that are trained on huge datasets, and then extend that with more specific training from other sources of data in…
Really cool idea. I hope you manage to get into a sustainable cycle of people you've helped with bankruptcy getting back on their feet and donating to help others in the same position.
I would second this; sentence embeddings outperform word embeddings on basically all tasks where you actually have sentences to work with. The only downside is that they're significantly more computationally intensive,…
Interesting. It took me a while to figure out what the main contribution here is, since doing dimensionality reduction on embeddings is fairly common. I think the main contribution is an empirical measure of A) how…
A word embedding transforms a word into a series of numbers, with the property that similar words (e.g. "dog" and "canine") produce similar numbers. You can have embeddings for other things, such as pictures, where you…
It's really difficult to overstate how important embeddings are going to be for ML. Word embeddings have already transformed NLP. Most people I know, when they sit down to work on an NLP task, the first thing they do is…
This is really cool! I especially like that there's a premade colab notebook that lets you play with it: https://colab.research.google.com/github/openai/clip/blob/ma... . I'm a little surprised that the paper doesn't…
I don't think you could get access to the actual models that are being used to run e.g. Google Translate, but if you just want a big pretrained model as a starting point, their research departments release things pretty…
> Is it always good enough to take the outputs of the next-to-last layer as features? It usually doesn't matter all that much whether you take the next-to-last or the third from last, it all performs pretty similarly.…
There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems:…
Yeah, I think this pattern is pretty common. (Basilica's main business is an API that does deep feature extraction as a service, so we end up talking to a lot of people with tasks like yours -- and there are a lot of…
I think if you have a small to medium sized dataset of images or text, deep feature extraction would be the first thing I'd try. I'm not sure what the most interesting problems with that property are. Maybe making…
I hadn't read it before! That's a fascinating result, actually. They emphasize interpretability in the paper, but I find it more interesting that you can do so well with only local information. My first thought is that…
I don't work with time series data much myself. I would imagine you can get at least some transfer learning, since there are patterns that show up across different domains. It looks like there's been a little bit of…
Definitely. There's been a lot of exciting work recently for text in particular, like https://arxiv.org/pdf/1810.04805.pdf .
Linear Algebra Done Right would be my recommendation.
Hi everyone! Author here. Let me know if you have any questions, this is one of my favorite subjects in the world to talk about.
> He's almost describing a future where we might buy/license pre-trained models from Google/Facebook/etc that are trained on huge datasets, and then extend that with more specific training from other sources of data in…
Really cool idea. I hope you manage to get into a sustainable cycle of people you've helped with bankruptcy getting back on their feet and donating to help others in the same position.
I would second this; sentence embeddings outperform word embeddings on basically all tasks where you actually have sentences to work with. The only downside is that they're significantly more computationally intensive,…
Interesting. It took me a while to figure out what the main contribution here is, since doing dimensionality reduction on embeddings is fairly common. I think the main contribution is an empirical measure of A) how…
A word embedding transforms a word into a series of numbers, with the property that similar words (e.g. "dog" and "canine") produce similar numbers. You can have embeddings for other things, such as pictures, where you…
It's really difficult to overstate how important embeddings are going to be for ML. Word embeddings have already transformed NLP. Most people I know, when they sit down to work on an NLP task, the first thing they do is…