This looks interesting, how would you compare it with colly? I’ve used colly for a while and overall am pleased with it, but things can always be improved.
Colly is a great library for builing scrapers if you are familiar with Go.
Flyscrape a cli tool that lets you define your scraping logic with a declarative JavaScript API, while it takes care of issuing requests, caching, rate limiting and more.
Hey, quick question: Is there a way to handle async behavior? Like in puppeteer we had something like waitFor helper to wait while loading the page. I couldn't find a similar thing in your readme. Or is this framework for only static pages?
5 comments
[ 3.4 ms ] story [ 26.5 ms ] threadFlyscrape a cli tool that lets you define your scraping logic with a declarative JavaScript API, while it takes care of issuing requests, caching, rate limiting and more.
The data extraction function of your script also runs aside the crawler, not inside the page.