[–] jshchnz 2y ago ↗ This is cool, seems like a nice way to easily add context for ChatGPT at the least [–] cpeffer 2y ago ↗ * Creator here - Thats the goal! [–] donohoe 2y ago ↗ And you honor or ignore robots.txt? [–] cpeffer 2y ago ↗ It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
[–] cpeffer 2y ago ↗ * Creator here - Thats the goal! [–] donohoe 2y ago ↗ And you honor or ignore robots.txt? [–] cpeffer 2y ago ↗ It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
[–] donohoe 2y ago ↗ And you honor or ignore robots.txt? [–] cpeffer 2y ago ↗ It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
[–] cpeffer 2y ago ↗ It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
[–] altdataseller 2y ago ↗ What user agent does it use and do you obey robots.txt?* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK [–] super256 2y ago ↗ > FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and cachingThey probably ignore robots.txt
[–] super256 2y ago ↗ > FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and cachingThey probably ignore robots.txt
[–] pryelluw 2y ago ↗ Does this respect any anti AI related scraping rules set forth by website owners? [–] duskwuff 2y ago ↗ It doesn't even respect general robots.txt exclusions, so... I doubt it.
[–] nyolfen 2y ago ↗ https://gist.github.com/yoki/fd2a3642b059703529d158de8233bbc... [–] cpeffer 2y ago ↗ It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.If only that script worked for every website. But, alas, it does not. [–] [deleted] 2y ago ↗ (comment deleted)
[–] cpeffer 2y ago ↗ It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.If only that script worked for every website. But, alas, it does not.
[–] padolsey 2y ago ↗ Do you have pay-as-you-go pricing? I feel that’s always missing from things like this. Cool otherwise.
15 comments
[ 4.1 ms ] story [ 36.8 ms ] thread* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK
They probably ignore robots.txt
If only that script worked for every website. But, alas, it does not.