Show HN: MrScraper – A visual web-scraping tool (mrscraper.com)
Two months ago, I started building this side-project in the morning, before my full-time job.
A visual and easy-to-use web scraping app.
Please, roast it a bit so I can work on improving it. Thanks.
84 comments
[ 3.2 ms ] story [ 163 ms ] thread"With MrScraper, you won't be blocked.
We use real browser instances to perform fast but human web scrapings, resulting in a much lower block ratio."
"won't be blocked" implies a zero block ratio. (I do a lot of work with Puppeteer and Playwright, and some larger websites are pretty advanced at their heuristics at catching automation, so true zero really isn't a defensible claim)
It looks interesting. I tried puppeteer and playwright but never got the hang of it, so I might be a client for one of these scraper services one day. The first time I tried it I got immediately blocked (probably because it had no agent, which was a raspberry pi)
I feel like this is saying your systems have perfect security, which itself is not a defensible claim
Anyway, if you want to be technical about it, the marking is correct. YOU won't be blocked. The agent running on your behalf might be blocked, however...
But from a marketing perspective, this "you won't be blocked" falls into the acceptable simplification category. Maybe they could add a * footnote, giving some more detail elsewhere. But at this point in the landing page, it wouldn't make sense to try to state it more accurately as that would require too many words.
How block you’ll be “blocked less” or some variation of that form?
Still simple, less risk of disappointment/trust issues.
Or I might have some kinda of addin/setting configured from hacking around on something over the years.
I will rewrite the copy to make better statements. Thank you so much
Your assumption that blocking is somehow ethical by default is not unproblematic.
There's a world wide web built by academics for free exchange of information and there's a closed garden web built by major capitalists.
Just how free that exchange of information should be is not a settled problem. Some very libertarians argue along the lines of information "wanting to be free". Some commercial entities seem to identify copyright and trademark law with moral doctrines. There are plenty of arguments for in-between positions as well.
If we look at less democratic societies, the efforts made to circumnavigate state censorship are publicly lauded as morally good actions by the international community. Could an analogy be drawn to large corporations censoring the less fortunate in a economically uneven societies too, for instance?
Your assumption that blocking is somehow ethical by default is not unproblematic.
is itself an assumption.
The problem I'm concerned with is aggressive (either deliberately or ignorantly) crawling/scraping of non-commercial sites which often lack the financial resources to defend against activities enabled without apparent concern by tools like the site here.
If a site allows reasonable access in good faith, then subverting those limits and constraints for self-serving reasons is ethically dubious at best, and any service not addressing that while promising to enable that subversion should be questioned.
Are you using a service like 2captcha to auto-solve captchas?
I always wonder what web scraping tools use as their proxy solution, because afaict they tend to be quite expensive, especially for residential IPs. How are you handling that?
In my case I have a landing page (with pagination) with a list of records I want to extract. However, to extract the full information I need for each record, I need to click on each item and navigate to a detail page to extract further info.
Looking at your app and docs you don't seem to support this either. Is this something you are considering?
I'm currently working on standard pagination (click next page button) and click button + infinite scroll.
What you comment is not currently possible with a single scraper, you would need to send one to collect links and then scrape those links. But I'm also working on "nesting data" feature, and what you comment should be possible in an ETA 2-3 weeks max.
Thanks for commenting!
The biggest complication with visual scrapers is all the edge cases. The selector algorithms usually become a mess on any complex website especially if there's uneven data.
Then you have css selectors no longer working and so on. Very brittle.
{ "new_tests": "{\n \"puppeteerEvaluationScript\": \"OK\",\n \"webdriverPresent\": \"FAIL\",\n \"connectionRTT\": \"FAIL\",\n \"overrideTest\": \"OK\",\n \"puppeteerExtraStealthUsed\": \"OK\",\n \"inconsistentServiceWorkerNavigatorPropery\": \"OK\",\n \"inconsistentWebWorkerNavigatorPropery\": \"OK\"\n}", "detection_tests": "{\n \"intoli\": {\n \"userAgent\": \"OK\",\n \"webDriver\": \"FAIL\",\n \"webDriverAdvanced\": \"FAIL\",\n \"pluginsLength\": \"FAIL\",\n \"pluginArray\": \"FAIL\",\n \"languages\": \"OK\"\n },\n \"fpscanner\": {\n \"PHANTOM_UA\": \"OK\",\n \"PHANTOM_PROPERTIES\": \"OK\",\n \"PHANTOM_ETSL\": \"OK\",\n \"PHANTOM_LANGUAGE\": \"OK\",\n \"PHANTOM_WEBSOCKET\": \"OK\",\n \"MQ_SCREEN\": \"OK\",\n \"PHANTOM_OVERFLOW\": \"OK\",\n \"PHANTOM_WINDOW_HEIGHT\": \"OK\",\n \"HEADCHR_UA\": \"OK\",\n \"WEBDRIVER\": \"FAIL\",\n \"HEADCHR_CHROME_OBJ\": \"FAIL\",\n \"HEADCHR_PERMISSIONS\": \"FAIL\",\n \"HEADCHR_PLUGINS\": \"WARN\",\n \"HEADCHR_IFRAME\": \"FAIL\",\n \"CHR_DEBUG_TOOLS\": \"OK\",\n \"SELENIUM_DRIVER\": \"OK\",\n \"CHR_BATTERY\": \"OK\",\n \"CHR_MEMORY\": \"OK\",\n \"TRANSPARENT_PIXEL\": \"OK\",\n \"SEQUENTUM\": \"OK\",\n \"VIDEO_CODECS\": \"OK\"\n }\n}" }
and ip range of "us" geo proxy gives is_abuse: true. Consider this to be okayish though, given that this is a default proxy pool.
Last time I needed it I just used Python + Selenium
Obv at the downside of needing to code but I don't mind that.
CF is Cloudflare, which offers an anti-scraping protection for websites (among other things): https://www.cloudflare.com/
But if I have to be honest, I can not guarantee it at the present time.
This app is the side-project I starter 2 months ago, it's evolving fast but I still need to add some key features for enterprise customers.
The knowledge base and API documentation is good to me, but maybe not ideal for your target customer, the person looking for a no-code solution and probably somewhat intimidated by anything beyond a CSV. I think you should add a step-by-step or maybe a video showing how the HTML selectors and rules work in outline. When I first got interested in this topic there were two main stumbling blocks: cursors/pagination, and how to identify selectors on a page with multiple similar but distinct items (social media mutuals lists, product catalogs etc.). Since you're aiming at a non-technical audience, I think you need to give them feel of a walkthrough before downloading the app.
I've noted down your comment and I'll be improving things for next week!
I've noted down your suggestion and I'll make this more clear or add field edition.
Thanks!
I also had a real problem naming the "Store as" field for my data extractor. It didn't seem to like things in the format "foo_bar_baz56" (i.e. ending with digits). This page https://mrscraper.freshdesk.com/support/solutions/articles/1... says "The variable name can not contain special signs" but doesn't explain what special signs are. Anything other than [A-Z_]?
Now that I've finally set up and tested my first scraper, I'm really impressed. It was much easier to set up than I would have guessed, and specifying a selector made it dead simple. Results worked out of the box, on a site that is super touch about being scraped.
However, now that I'm viewing my scraper, I see no way of editing the scraper or data extractor. What's the trick to editing a scraper once you've saved it and gone back to view it?
I've noted everything down and I'll improve the onboarding experience, fix and make variable names more understandable and improve the edit button.
A scraper is not editable once is queued to run or currently running, then, you need to reload the page for the edit button to appear again. I will improve this.
Thank you again
I'm aware there are pdf2html toys, and sometimes they do something reasonable, but just like with web scraping the markup of the target matters a lot and so, too, would the "markup" of the target PDF
Further, just like often it is better to go after the underlying XHR instead of trying to de-React the HTML, I'll offer that when possible it would be far better to try and identify the upstream source of the information in the PDF than trying to reverse engineer a postscript VM
""" What happens if my scraping fails? Not to worry! We will make every effort to determine the cause of the problem and assist you in resolving any issues with your scraper.
Additionally, please note that unsuccessful scrapings will not be included in your monthly quota. """
I'm curious the feedback mechanism for failed scrapes. Is there any validation configuration or an email notification I can configure in the event the target changes their page layout or DOM or whatever happens to cause interference.
But you gave me the idea to send an alert or email notification if a scraper stops returning data or content changes.
Thanks
If running the model was cheaper I would even say run the whole page through ChatGPT and ask it to format the information on the page for you.
But I'm currently tinkering with other applications of AI and MrScraper. Will ship something when I think it's reliable enough.
https://app.mrscraper.com/onboarding
<input x-data="{}" wire:model.defer="name" type="text" dusk="filament.forms.name" disabled="" id="name" class="block w-full transition duration-75 rounded-lg shadow-sm focus:border-primary-500 focus:ring-1 focus:ring-inset focus:ring-primary-500 disabled:opacity-70 border-gray-300" x-bind:class="{ 'border-gray-300': ! ('name' in $wire.__instance.serverMemo.errors), 'dark:border-gray-600': ! ('name' in $wire.__instance.serverMemo.errors) && false, 'border-danger-600 ring-danger-600': ('name' in $wire.__instance.serverMemo.errors), 'dark:border-danger-400 dark:ring-danger-400': ('name' in $wire.__instance.serverMemo.errors) && false, }">
Other users have also reported that this is a bit confusing, so I'm going to start working on improving this.
Thanks for you feedback!