HN Explorer

Ask HN: Created Scraping SaaS, best way to avoid misuse?

2 points by agibsonccc ↗ HN
In light of the last few days, and all the scraping topics popping up. I figured I'd ask how to approach this.

I created a service that's in very early alpha that's able to use lots of different techniques (pattern matching, NLP,javascript) to get data out of webpages.

For data gathering, I planned on having an interface to different kinds of APIs, along side the scraping, that users might need to do mashups.

It's already paying the bills for me, so I'd like to start scaling this out now. There can be a lot of potential for misuse here.

Besides the obvious things like robots.txt, naming my user agent, and users being able to opt out of things like emails gathered by the system and throttling. What other things should I should be aware of?

3 comments

[ 5.6 ms ] story [ 9.9 ms ] thread
thecommentator
I'd be interested in using this type of service; I've tried a few but they were mostly too complicated to use, failed to work as advertised, or cost more money then hiring someone off vWorker to write a specific one-time solution.
agibsonccc
That's what the NLP is for. It's a three step process: Specify what you want, where you want it from, come back later and download your spreadsheet.

I planned on adding more advanced features later for users to specify css selectors and stuff.

The main ability of this is a preview and adjust feature I created in there that highlights the data it "thinks" it would grab for each website.

I took a lot of notes as I was creating this.