Any ideas how to pivot a web crawler/scraper project?
There is a quite powerful crawler and scraper engine. We developed it for internal needs at happystayapp.com. We used it as a hotels reviews meta search and hotels aggregator from sites like booking.com, expedia, etc.
But I'm feeling it's a bit underused and I'm thinking how else it can be used?
It supports many of features modern web scraping tools have. The engine is scalable and easy extendable.
The technologies: - MongoDB - C# - Microsoft Azure - JSON
Where and how else the technology can be used? Any ideas?
0 comments
[ 0.21 ms ] story [ 11.8 ms ] threadNo comments yet.