> After hours of training on the Shakespeare database, on a 300k parameter model, this has been the output
I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?
I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).
So what are the expectations of this GPT model? Should it be bound by the same limitations?
4 comments
[ 2.6 ms ] story [ 25.1 ms ] threadEverything is implemented from scratch, including the tensor processing logic along with training/inference code of a minimal GPT architecture.
I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text?
I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had).
So what are the expectations of this GPT model? Should it be bound by the same limitations?