[–] gunalx 1y ago ↗ If all the training data is in the txt files, it is obviously trained on copyrigthed material, and immensly low amounts of text. Im impressed if the outputs even start to make sense at all. [–] nickpsecurity 1y ago ↗ True. The best solution for small models is to use Project Gutenberg to reduce risk:https://huggingface.co/datasets/manu/project_gutenbergIt’s also enough data to be Chincilla optimal in under one epoch for SLM’s. [–] [deleted] 1y ago ↗ (comment deleted) [–] [deleted] 1y ago ↗ (comment deleted)
[–] nickpsecurity 1y ago ↗ True. The best solution for small models is to use Project Gutenberg to reduce risk:https://huggingface.co/datasets/manu/project_gutenbergIt’s also enough data to be Chincilla optimal in under one epoch for SLM’s. [–] [deleted] 1y ago ↗ (comment deleted)
[–] nickpsecurity 1y ago ↗ Are you the author of the GitHub? If so, I might have a few suggestions. [–] [deleted] 1y ago ↗ (comment deleted)
[–] parpfish 1y ago ↗ “Small LLM” means “Small Large Language Model” [–] [deleted] 1y ago ↗ (comment deleted)
[–] burgerrito 1y ago ↗ ....is that the whole Harry Potter book in one .txt file, hosted on GitHub!? [–] ClearAndPresent 1y ago ↗ That is all the Harry Potter books in one .txt file, hosted on Github.
[–] itishappy 1y ago ↗ I'm no lawyer, but this repository sure appears to be relicensing the Harry Potter series under the GPL. [–] [deleted] 1y ago ↗ (comment deleted)
[–] cjtrowbridge 1y ago ↗ Bro use fine web. Random books are not objectively good training data. [–] [deleted] 1y ago ↗ (comment deleted)
17 comments
[ 3.6 ms ] story [ 47.1 ms ] threadhttps://huggingface.co/datasets/manu/project_gutenberg
It’s also enough data to be Chincilla optimal in under one epoch for SLM’s.