I'm downloading the torrent right now and it's 1.2TB of mostly zip files each containing many txt files. So the 1.2TB does appear to be the compressed size (unless the zip files aren't actually compressed, but that would be weird)
Downloading this right now, but I have a few problems with it.
One is that the books are organized into thousands of zip files, while I'd personally rather they just be organized into directories. I get that most people will benefit from the compression, but I suspect that relying on btrfs's transparent zstd compression will do just fine for me, so I'd rather have a simpler access pattern that doesn't involve extracting or scanning through a zip file whenever I want to look for a book. Not a huge deal. I'll just extract them all I guess, just feels like a waste of IO.
Another thing, why txt? I'd much rather this be a collection of epubs instead, so normal ebook readers can open them with proper formatting, and they'd already be similarly compressed. There'd certainly be some increase in overall size, but I think it'd be worth it.
Another problem is that the index files don't seem to be clean? There are references to books that don't exist, apparently because if whatever automated tool was used to create this couldn't extract the text from a book, the book simply isn't present but it still has an ID and an entry in the index files. Another thing that isn't a huge deal and that I can fix locally, but is a bit of an annoyance.
Once I have this all set up in a way I want, I expect to just be able to serve it with a simple web server and a url rewrite rule instead of the software shipped as part of this project. Nothing against that software, but I'd rather just use a filesystem to organize text files. I guess I'll have plenty of time to think about how best to do it, because the current download ETA is over a week.
> if the book is missing (15%), that means the text could not be extracted.
> if the book is missing (10%), that means the text could not be extracted.
===
| why plain text?
===
- text is smaller (figures are for old .tar.xz, compressed .zip is about double the size)
+----------+------------------------------------+------------------------+
| | size | number of files |
+----------+----------+-------+-----------------+-----------+------------+
| data set | original | text | compressed text | original | text |
+----------+----------+-------+-----------------+-----------+------------+
| lg | 32TB | 0.8TB | 0.32TB | 2,450,000 | 2,220,000 |
+----------+----------+-------+-----------------+-----------+------------+
| ff | 2.4TB | 0.5TB | 0.18TB | 2,180,000 | 1,890,000 |
+----------+----------+-------+-----------------+-----------+------------+
- text can be viewed on every device.
- plain text cannot contain malicious code. pdfs can.
- text can be searched and indexed more easily.
- why csv? anyone can use csv. it can be read raw in a text editor, opened in excel, or imported into a database. database dumps can be used only by experts.
I don't see how plain text in zip files is any more searchable than epubs (which are each just plaintext-heavy XML files wrapped in a zip file). And PDFs potentially containing malicious code is an argument against using PDF, not an argument for using plaintext specifically. Being viewable on every device is a decent reason, I just don't personally think it's worth the degradation in reading quality.
> if the book is missing (15%), that means the text could not be extracted.
I saw that, I just don't understand why a book that isn't included has an entry in the CSV file. If the automated tool fails to extract the text from the book to include in the archive, then why put it in the index file?
Is this the answer to the question of where to get enough data posed in the article about Chinchilla[0]?
If tokenized with the GPT-3 tokenizer, about how many tokens would this be?
Your article mentioned that Chinchilla is trained on 1.4T tokens. So there is quite some difference.
The article also mentions different mysterious book data sets with 27B tokens, 560B tokens, or 390B tokens.
The latter datasets were made by Google. So you are still behind Google massive book dataset even if you use probably the largest book dataset "available" to people or instituions outside of Google.
EDIT: I thought I made a mistake, but T stands for trillion or tera which are both 1E12.
>change from .tar.xz to .zip format. this doubles the size, but speeds up file access from 30s to 0.01s
Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.
ZIP is seekable -- you can get a list of files and/or read individual files out of the archive without decompressing the whole thing. TAR+(stream compressor) isn't; the only way to get a list of files or to decompress individual files is to read through the whole thing.
The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.
I did something similar 3 years ago. My goals were minimal file size, as well as fast read times.
I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially.
At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.
The start of the files had a header with pointers to the manifest.
The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.
Edit: If there's some interest, I can try to convert the dataset to this format.
I built an offline wikipedia reader that heavily relied on compression while giving nearly instantaneous access to an arbitrary article (< 100ms). It uses sqlite FTS for searching article titles. I did not enable FTS over the body of the articles.
28 comments
[ 3.8 ms ] story [ 66.7 ms ] threadHowever, it should be of great use for indexing and searching in them!
I'm downloading the torrent right now and it's 1.2TB of mostly zip files each containing many txt files. So the 1.2TB does appear to be the compressed size (unless the zip files aren't actually compressed, but that would be weird)
One is that the books are organized into thousands of zip files, while I'd personally rather they just be organized into directories. I get that most people will benefit from the compression, but I suspect that relying on btrfs's transparent zstd compression will do just fine for me, so I'd rather have a simpler access pattern that doesn't involve extracting or scanning through a zip file whenever I want to look for a book. Not a huge deal. I'll just extract them all I guess, just feels like a waste of IO.
Another thing, why txt? I'd much rather this be a collection of epubs instead, so normal ebook readers can open them with proper formatting, and they'd already be similarly compressed. There'd certainly be some increase in overall size, but I think it'd be worth it.
Another problem is that the index files don't seem to be clean? There are references to books that don't exist, apparently because if whatever automated tool was used to create this couldn't extract the text from a book, the book simply isn't present but it still has an ID and an entry in the index files. Another thing that isn't a huge deal and that I can fix locally, but is a bit of an annoyance.
Once I have this all set up in a way I want, I expect to just be able to serve it with a simple web server and a url rewrite rule instead of the software shipped as part of this project. Nothing against that software, but I'd rather just use a filesystem to organize text files. I guess I'll have plenty of time to think about how best to do it, because the current download ETA is over a week.
About the missing books fiction and non fiction
> if the book is missing (15%), that means the text could not be extracted.
> if the book is missing (10%), that means the text could not be extracted.
> if the book is missing (15%), that means the text could not be extracted.
I saw that, I just don't understand why a book that isn't included has an entry in the CSV file. If the automated tool fails to extract the text from the book to include in the archive, then why put it in the index file?
So now you have a list of stuff that should have been included but isn't? Then maybe chase down why and fix?
[0]: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
1) I assume 1TB (not TiB) of uncompressed (?) text
2) I assume one character is one byte
3) I assume 5 (actually it seems to be 4.7 in English) characters per word
So 1TB/1B/5 = 1.0E12/5 ~= 2.0E11 = 0.2T = 200B words.
Your article mentioned that Chinchilla is trained on 1.4T tokens. So there is quite some difference.
The article also mentions different mysterious book data sets with 27B tokens, 560B tokens, or 390B tokens.
The latter datasets were made by Google. So you are still behind Google massive book dataset even if you use probably the largest book dataset "available" to people or instituions outside of Google.
EDIT: I thought I made a mistake, but T stands for trillion or tera which are both 1E12.
Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.
* Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.
People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone".
> Yes, there's a lot better compression available these days
Tell me more!
Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.
I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd.
The start of the files had a header with pointers to the manifest.
The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary.
Edit: If there's some interest, I can try to convert the dataset to this format.
Outline of the technique is here https://news.ycombinator.com/item?id=28013347