HN Explorer

Ask HN: How to Make Money from Web Scraping?

17 points by pythonbase ↗ HN
What are some ways to make money from web scraping?

Few options that I can think of, are:

1. Scrape and sell data

2. Scrape public data and publish as a website. Make money with ads / subscriptions

3. Build price comparison websites

4. Offer web scraping services.

Any real life examples / use-cases out there?

12 comments

[ 3.0 ms ] story [ 39.1 ms ] thread
jppope
So not to be decisive... but if you have a skill such as a talent for scraping why not look for a value-add for the world compared to primarily "rent-seeking" activities like the above. (Advertising is never a good way to make money)

Why not set up a scraping system that helps people improve accessibility by alerting them to flaws in their sites?

pythonbase
Good idea. Can you elaborate this a bit?
rdbell
If any HN community members are doing web scraping that may benefit from residential proxies rather than datacenter IPs, I can get you set up with some free test credits on our residential proxy network: https://packetstream.io

We tunnel requests through real opt-in residential ISP users.

Send us a message via our contact form and we'll load up your account with some test credits.

Happy scraping!

nickthemagicman
That's badass but how do they stop from being banned?

Like if I scrape Craigslist enough and get that IP banned the resident can't visit Craigslist?

I don't know enough about this to understand.

Or would it cycle through the proxy IPs real fast?

surewhynat
usually these are controlled via an api of some sort. depending on the distributed system. another popular solution is hola which offers basically the same thing, albeit I think worse prices when I was using it than this one...
rdbell
> Or would it cycle through the proxy IPs real fast?

Exactly. PacketStream lets you change your visible IP with each request. If you need a persistent IP/geolocation that can be configured as part of the request.

jackcodes
The coolest use of a botnet I’ve seen in a while!
gitgud
Possibly one more item is "scraping to gather analytics and trends", the revenue comes from using machine learning to predict trends based on the scraped data from social media, news etc... in most cases the raw data isn't as valuable as the patterns or predictions you can see in it...

Example -> https://meetglimpse.com/

pythonbase
Very interesting. Your project?
gitgud
Not my project, was on "Show HN" a few months ago.

But I did try to use Web-Scraping a few years ago, when I was trying to set up a crypto-currency [1] arbitrage... but there were a lot of risks and challenges... could be another option for you though.

[1] https://en.wikipedia.org/wiki/Arbitrage