21 Gigs in total for everything, ignoring errors (files too big / broken links)
I'm really happy their share it though. The collection is just amazing - it's a good idea for "what books to buy for a personal library when you start making decent money". Definitely would spend money on things like…
I'm at 300 right now, 9.5 gigs. I think some are skipped cause the download times out (can redownload manually I suppose)
P.S. it will take hours or days =))) <3 <3 <3
Please use responsibly. Recommended to delay by a few days - it seems like there is an initial surge going on right now. Increase download delay to 20 sec - 1 minute. Be respectful.…
I think someone else is messing with it - got slower for me too.
It's slow cause things are quite large there (just saw a 2 gig book). I'm using delays, don't worry - hence "in a few hours" <3 <3 <3
<3
Uploading a scrapy crawler that downloads PDF books to github in a few hours, gonna post the link here
some "ethical" measures may do the trick to. scrapy has a setting to integrate delays + you can use fake headers. Some sites are pretty persistent with their cookies (include cookies in requests). It's all case by case…
depends on the task. For example they have a decent file/image downloading middleware.
scrapy has a pretty decent parser too
21 Gigs in total for everything, ignoring errors (files too big / broken links)
I'm really happy their share it though. The collection is just amazing - it's a good idea for "what books to buy for a personal library when you start making decent money". Definitely would spend money on things like…
I'm at 300 right now, 9.5 gigs. I think some are skipped cause the download times out (can redownload manually I suppose)
P.S. it will take hours or days =))) <3 <3 <3
Please use responsibly. Recommended to delay by a few days - it seems like there is an initial surge going on right now. Increase download delay to 20 sec - 1 minute. Be respectful.…
I think someone else is messing with it - got slower for me too.
It's slow cause things are quite large there (just saw a 2 gig book). I'm using delays, don't worry - hence "in a few hours" <3 <3 <3
<3
Uploading a scrapy crawler that downloads PDF books to github in a few hours, gonna post the link here
some "ethical" measures may do the trick to. scrapy has a setting to integrate delays + you can use fake headers. Some sites are pretty persistent with their cookies (include cookies in requests). It's all case by case…
depends on the task. For example they have a decent file/image downloading middleware.
scrapy has a pretty decent parser too