Ask HN: Can web scraping be the basis of a viable business model?
Without getting into the details yet: it aims to make web data collection a little bit easier for non-devs. I'll soon have an MVP and will start pitching to investors: aiming for an open-source business model (after a few months of stealth development) and eventually a typical SaaS offering for extra functionality.
At this point I'm trying to consolidate and counter the steel-man counter-arguments I should expect from investors. The most obvious one: as one can imagine, the product it's not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for.
I have done some preliminary analysis of the space of potential competitors (think import.io, Apify, Zyte/ScarpingHub, etc.) and described opportunities for differentiation. What I'm afraid of is getting sidetracked in a discussion of "um, this is web scraping and it's hard to make a business on top of it".
I understand that there's not much context now and one could easily say "well yeah, anything could be possible with a good team, product...", but I'm reaching out to the HN community to gather some considerations, mental models and pointers, I may not think of myself at this point.
113 comments
[ 0.82 ms ] story [ 185 ms ] thread* What features are common among my competitors?
* What features are unique?
* Who are target customers and users? Is there any overlap, or do some competitors target unique market segments?
This last question ties in to a discussion I was having with a friend recently. In B2B sales, your customers are businesses, but your users are people in those businesses with certain roles and responsibilities. Understanding the difference is key, because you will often need to develop your sales and marketing strategies based on the business/customer profile, but your UX will depend on the needs of the users within those businesses.
In my opinion you are more likely to be successful if you can get an initial foothold in a market by identifying a specific target of customers and users, solving their use case very well, developing a moat, and then growing out from that foothold to provide a wider set of options. Web scraping is just a tool. You need to find businesses who can gain value from scraping or from scraped data. Are there businesses who, for whatever reason, would not be able to adopt one of your competitors' products, or would find that adoption difficult? Maybe you could specialise in scraping a particular kind of data, or providing a full-stack solution for companies with limited in-house technical expertise (like some kind of consulting, you hop on a call with the client, they tell you what they want to scrape, and you set up a hosted solution which provides a SQL or Excel interface to the data).
In short, successful product development is all about understanding customer and user pain and needs. If you can find pains or needs which are a common theme for a particular demographic of companies and roles, you can work with those people to understand their problems and make a product which is very valuable to them.
You're exactly right on the tricky relationship between developers as internal ambassadors to businesses - customers. I think it somehow applies to almost any developer-facing tool.
Your recommendation to focus on a specific vertical at first makes sense too. Helps prioritising the backlog as well.
>the product it's not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for.
Can you make it magic or maybe develop end to end solutions for your first "customers" using your product? That sounds like the schlep you need to do.
Sounds promising! Find yourself a customer or two!
If you really want to go the open source route, just focus on that and then see if people pick it up and use it. Then you'd offer the SaaS.
Wasn't aware of the Ticketmaster v. Tickets.com case. Will have to print it along the SCOTUS LinkedIn v. hiQ ruling.
Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.
:-)
Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag.
> There's tremendous value in restoring some of it.
To this comment and OP, my startup is using web scraping to pre-populate machine-readable data for a DNS-based protocol called NUM [0]. So as others have said, whilst web scraping itself may be difficult to build into a viable business, it can be a key component of a viable business. Email in my profile if you want to discuss.
0. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-...
The Semantic Web would have given much better search among other benefits.
Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying that a person is, in essence, the world's largest mouth.
I think it's incredibly difficult to build a profitable business in this space. The number of customers who a) need to scrape the web b) aren't sophisticated enough to do it themselves and c) are big enough to make $$$ from are small. The important bit is always processing the web pages for whatever content is salient to the given customer. Which means that you need to deliver the web pages to them. So effectively your business is providing nothing more than URL lists and potentially some additional metadata compared to what the customer would get if they fetched the pages themselves. There are definitely some other complexities you could resolve, but it's hard to imagine that those benefits would be enough to build a business on.
Just scraping upon scraping.
Makes the founder a hefty sum.
Hedge funds actually call this "alternative data".
Of course it is.
Finance/banks are especially... inconsistent, to say the least.
There are a handful of companies doing very well with models similar to what you’re describing. I can’t mention specific customers, but I see some of them doing very large scraping volume through our network.
It’s an industry where having a good product is more important than the amount you’re spending on marketing. If developers are happy with your product they’ll take it with them to future companies/projects and share it with colleagues.
It can be a cat-and-mouse development cycle where the sites you target break your functionality and you’ll have customers that will want fixes to be implement ASAP because they rely on your tools to make money.
I don’t know what you’re building exactly, but keep in mind there’s a good chance that you’ll need to commit to long-term, continuous, rapid development cycles if you want to retain customers.
Best of luck!
I just signed up, got sent to a dashboard…NO API DOCUMENATATION, just a link to download an app (!) for people who want to sell their residential bandwidth.
See ScraperAPI for a company that does API documentation well in this space. (I've spent well over $100K with them.) Or Stripe. PUT THE CODE UP FRONT.
There's also no API docs here, under "Support": https://packetstream.io/support/faq
Update: I see it's hidden under "Network Access." This kind of thing should be OBVIOUS, not the sixth option (looks like all of the others) on some random SaaS dashboard.
Instead, think about what people want to do with the data. For example, if you are going to scrape diamond prices, don’t try to sell that feed. Set up a website with a UI so people can research diamond prices, and get alerts when specific thresholds are met or items come in stock. Monetize with ads.
I wrote the scraping code. Had a list of sites and macros for extracting quotes, updated every day to every customer. If one quit working (the site attempted to prevent scraping) the app would use another and give a notice back to me. I'd tweak the macro for that site, and we'd be back scraping it the next day.
We eventually hired a finance student (Josh Hatwich, now a fellow at Adobe) to parse a Comstock satellite feed we put on the roof. That ended the era of scraping at StockPoint.
https://www.screendaily.com/features/how-uk-data-company-app...
Applaudience’s algorithms trawl through every exhibitor website, looking at every showtime of every film, and tracks the auditorium layout as each seat flips from available (unsold) to unavailable (sold).
It is an arms race, since many people don’t want you to scrape. We tried hard to respect robots.txt, but we still got angry cease-and-desist emails from people who’d malformed or misconfigured the file.
You will have a scale problem: it’s a lot of data. You’ll have parsing problems: live HTML is about the dirtiest data set I’ve ever seen. Refresh rate can be a major competitive advantage: how often can you scrape, store, diff, and report? These days you’ll need first-class JavaScript execution to catch dynamic content.
But the biggest problem isn’t the scraping tech, it’s the use case — what uses cases are you going to afford your early users? You don’t mention this in your post, and it will non-trivially affect what you scrape and how you report it. I’d encourage you to find users who have business problems that can be solved by paying money for scraping. Otherwise you’ll be another interesting open source tool that no one’s figured out how to monetize. Do this _before_ you talk to investors or take their money.
Consider, apart from a tool for general purpose scraping, what information a specialized scraper might obtain for a valuable but underserved industry that can profit from the data.
I have a buddy that scrapes data specifically for the tanker/shipping industry, for example.
General purpose scraping will involve a lot of competition and a bit of an arms race. Niche scraping lets you fly a little below the radar.
I think you'll have a lot more luck finding 2-3 initial customers before you try to raise money. It's always easier to explain what your product does, and who the target market is, in terms of actual customers; instead of hypotheticals.
Remember, the goal is to build a business. If you fall in love with the code, it's too easy to build something you enjoy working on, but has limited commercial value.
Good point though. Bootstrapping is a viable option too, but fundraising has certain no non-financial benefits that can be appealing.
It's very hard to fundraise without some kind of market validation.