Two years ago we showed HN Apifier - a hosted web crawler for developers - during our time in the Y Combinator Fellowship.
Today we're launching the largest upgrade of Apifier to date - a new product called Actor and a complete redesign of our website. Also, we changed our name to Apify.
Actor is a new serverless computing platform that enables execution of arbitrary pieces of code in the Apify cloud (we call them "acts"). For example, you can have an act to run a web automation job in headless Chrome with Puppeteer, to post-process data from the crawler in order to remove duplicates or to upload new contacts into your CRM. The possibilities are unlimited.
We also launched a library (https://www.apify.com/library), where you can find acts and crawlers built by other people and share yours to support the web scraping community. There are already several acts and people are adding new ones almost every day.
We're really looking forward to hearing what you think about Actor and to seeing what you can built with it. If you have any ideas, questions or feedback, just let us know on support@apify.com or here.
We consider ourselves only as service providers. We provide technical means to perform the crawling but the responsibility for the actual crawling is with our users. It's similar to Amazon AWS - they only provide infrastructure and it's your responsibility not to use it for anything illeagal.
(co-founder here) We have hundreds of customers, half of them on recurring subscriptions, half just one-time customers paying for crawler configurations.
Gonna check it out. I always use request and cheerio on node so it's almost identical. Is this an actual headless browser though? When I have to go that route it's painfully slow.
Very cool. I'm looking at it more in depth and I'm pretty impressed. This is basically all things I've been doing for years, but encapsulated in a way that I'm pretty sure end users will be able to use. I've always built these beasts and tried to evangelize for a MEAN stack as the best option but people haven't been ready. I have never met a problem I felt promises were a good solution for, but you've done a really great job of using them to let people who can't think asynchronously get things done.
Pseudo URLs are also something I've done before but are never the optimal solution. Again though, this is something that will allow a total beginner to catch up to me about 75%, and I've been developing for 25 years.
This is the best pricing page I’ve seen on mobile, hands down. Pricing pages are so often broken completely, or a list of boxes (difficult to compare plans). This beautifully handles both. Well done.
Currently I use phantomjs via selenium hub for a product and would like to migrate to chrome but couldn't much information on how to run a chrome cluster. So it will be great if you can shed some light on how you have set this up.
21 comments
[ 6.0 ms ] story [ 55.7 ms ] threadTwo years ago we showed HN Apifier - a hosted web crawler for developers - during our time in the Y Combinator Fellowship.
Today we're launching the largest upgrade of Apifier to date - a new product called Actor and a complete redesign of our website. Also, we changed our name to Apify.
Actor is a new serverless computing platform that enables execution of arbitrary pieces of code in the Apify cloud (we call them "acts"). For example, you can have an act to run a web automation job in headless Chrome with Puppeteer, to post-process data from the crawler in order to remove duplicates or to upload new contacts into your CRM. The possibilities are unlimited.
We also launched a library (https://www.apify.com/library), where you can find acts and crawlers built by other people and share yours to support the web scraping community. There are already several acts and people are adding new ones almost every day.
We're really looking forward to hearing what you think about Actor and to seeing what you can built with it. If you have any ideas, questions or feedback, just let us know on support@apify.com or here.
[1] http://www.reuters.com/article/us-microsoft-linkedin-ruling/...
Pseudo URLs are also something I've done before but are never the optimal solution. Again though, this is something that will allow a total beginner to catch up to me about 75%, and I've been developing for 25 years.
Wondering alternative to Selenium
This is the best pricing page I’ve seen on mobile, hands down. Pricing pages are so often broken completely, or a list of boxes (difficult to compare plans). This beautifully handles both. Well done.
https://imgur.com/gallery/i5WvP
I'll admit it's well done, and a simple solution, though I'm not sure it deserves the rave review.
What ones do you hate so much?
BTW we're working on migration of Crawler to headless Chrome.
If you could provide more details about your use case, I'm sure we will figure out how to do it with Chrome.