Built an open-source web scraper for AI agents – seeking feedback (github.com)

Anyone else here building AI agents that need web data?

So I built *AnyCrawl* specifically for AI use cases:

Handles JS-heavy sites (React, Vue, etc.) 20x performance boost through multi-processing Support proxies (no more blocks!) Returns LLM-friendly structured data Open source (MIT) + optional SaaS

It doesn't seem to respect robots.txt.