with hf transformers, you still need to manage GPUs
Why did the share of childless women 35-44 suddenly drop 5 points around 2012? Such a massive difference must be a change in measurement methodology?
The llama1 team did not have a validation set. I don’t know what the Llama2 team did - I left before seeing any of the details. My guess is Llama2 upsamples Wikipedia a good bit, but given they didn’t report any…
One noteworthy thing is that no one is posting validation curves, only training curves. All these models will happily bring training loss eventually to near zero with infinite compute, as the model overfits to the…
Updating the model happens offline to ensure protections of this. As part of the research on BlenderBot 3 we also have improved detections for safety and can learn from "trolls." https://parl.ai/projects/trollhunting/…
We trained on Reddit comments and HackerNews comments.
I run a fairly large open source project (https://github.com/facebookresearch/ParlAI/) and we use mypy. Our experience has been that it can be quite difficult to placate, so we usually treat it only as a warning.…
https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327
Support for this was also added to [Fairscale](https://fairscale.readthedocs.io/en/latest/) and [Fairseq](https://github.com/pytorch/fairseq) last week. In particular, the Fairscale implementation can be used in any…
They don't. But half the people got a placebo and half got the vaccine. 185 in the placebo group got covid, with 30 having severe covid, but 11 with covid and 0 severe in the vaccine group got it. So do you really think…
Streets would be much more useful than a true north compass.
Perhaps you are outside the United States? PBS is the public television channel, so perhaps they geolock :(
The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia. You can find a very comparable corpus open sourced and easy to use on the [T5…
No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future. On another note: At 175B parameters, with float16…
I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still nuances around safety, controlling…
(BlenderBot original developer here) You can read the paper here, which contains much more technical details of how we built it: https://arxiv.org/abs/2004.13637
Hi there, thanks for asking! It's very cool to make research more accessible. Would you be able to rename your experience to "Modified Small BlenderBot without Safety Layer"? And I think it’d be helpful to call out what…
Hi there. One of the original authors of the BlenderBot paper here. This cocohub website is not what we released. It appears that cocohub has removed our safety layer and provided other modifications to the bot. We…
Embeddings tables aren't hard on the GPU (being only a lookup table), and the output softmax still requires you do the full matrix-multiply. The label may be sparse, but the computation is far from sparse.
We limited the scope of our study to the PersonaChat dataset, which is a standard benchmark in dialogue. In PersonaChat, both participants are presented with Personas and asked to roleplay as that person. (e.g. "I love…
Yes, that's a great response for one of the models to have. :)
We originally set out to see whether we could use conditional generation to sort of adjust the behavior of these neural generative dialogue systems. Much of the original idea was about sort of "planning" our response to…
Co-author of the original paper, if you have any questions.
with hf transformers, you still need to manage GPUs
Why did the share of childless women 35-44 suddenly drop 5 points around 2012? Such a massive difference must be a change in measurement methodology?
The llama1 team did not have a validation set. I don’t know what the Llama2 team did - I left before seeing any of the details. My guess is Llama2 upsamples Wikipedia a good bit, but given they didn’t report any…
One noteworthy thing is that no one is posting validation curves, only training curves. All these models will happily bring training loss eventually to near zero with infinite compute, as the model overfits to the…
Updating the model happens offline to ensure protections of this. As part of the research on BlenderBot 3 we also have improved detections for safety and can learn from "trolls." https://parl.ai/projects/trollhunting/…
We trained on Reddit comments and HackerNews comments.
I run a fairly large open source project (https://github.com/facebookresearch/ParlAI/) and we use mypy. Our experience has been that it can be quite difficult to placate, so we usually treat it only as a warning.…
https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327
Support for this was also added to [Fairscale](https://fairscale.readthedocs.io/en/latest/) and [Fairseq](https://github.com/pytorch/fairseq) last week. In particular, the Fairscale implementation can be used in any…
They don't. But half the people got a placebo and half got the vaccine. 185 in the placebo group got covid, with 30 having severe covid, but 11 with covid and 0 severe in the vaccine group got it. So do you really think…
Streets would be much more useful than a true north compass.
Perhaps you are outside the United States? PBS is the public television channel, so perhaps they geolock :(
The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia. You can find a very comparable corpus open sourced and easy to use on the [T5…
No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future. On another note: At 175B parameters, with float16…
I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still nuances around safety, controlling…
(BlenderBot original developer here) You can read the paper here, which contains much more technical details of how we built it: https://arxiv.org/abs/2004.13637
Hi there, thanks for asking! It's very cool to make research more accessible. Would you be able to rename your experience to "Modified Small BlenderBot without Safety Layer"? And I think it’d be helpful to call out what…
Hi there. One of the original authors of the BlenderBot paper here. This cocohub website is not what we released. It appears that cocohub has removed our safety layer and provided other modifications to the bot. We…
Embeddings tables aren't hard on the GPU (being only a lookup table), and the output softmax still requires you do the full matrix-multiply. The label may be sparse, but the computation is far from sparse.
We limited the scope of our study to the PersonaChat dataset, which is a standard benchmark in dialogue. In PersonaChat, both participants are presented with Personas and asked to roleplay as that person. (e.g. "I love…
Yes, that's a great response for one of the models to have. :)
We originally set out to see whether we could use conditional generation to sort of adjust the behavior of these neural generative dialogue systems. Much of the original idea was about sort of "planning" our response to…
Co-author of the original paper, if you have any questions.