HN Explorer

16 comments

[ 2.6 ms ] story [ 45.5 ms ] thread
Pmop
Nice. I'm stealing this concept for my scraping engine.
r_singh
Anyone here using this in production (even for a side project)?

What are some of the challenges you've faced as opposed to just using ScrapingHub's Scrapy Cloud service?

skykery
I am using this in production inside a docker container. It does the job pretty well.
jatinshah
All such admin tools use scrapyd which is not maintained and not good enough to be used in production.
gregsadetsky
What do you recommend using instead?
rozhok
One of the companies I've worked for developed in-house solution (with some domain-specific stuff).

I've had to write some kind of admin tool for one of my clients.

And then I've used to start another project with scrapy and I feel like I need to develop another one tool for scrapyd management :)

[dead] shiblukhan
See also the list of contributors who participated in this project.
oqulie
SpiderKeeper least received an update 2 years ago. Scrapyd last received an update 9 months ago. Scrapy (the actual spider) is being actively maintained.
madc
Loads of not maintained open source projects make it to the front page lately..
slightwinder
How does this compare with a general solution like Apache Airflow?
troymc
The following is a simplification but should suffice.

Airflow is for hooking together and managing a bunch of data sources, data-processing nodes, and data sinks.

This project seems like a GUI for Scrapy, which is a web crawling framework, i.e. one kind of data source.

dzink
Are these spiders able to fetch content that loads in segments after scroll events?
jsmith99
Scrapy doesn't run javascript so it can't scrape client rendered content at all. There are projects that combine scrapy with a headless browser or javascript engine.
cs64org
I think it can faster than the normal then a basic function. ti increase the case memory.