Dude, you're making ScraperAPI requests (with your API key in plain text as a parameter) directly from the client app...this thing is a nice idea but it'd probably be a good idea to clean it up and repost.
Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around.
They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.
Which was a shame, because padmapper stuck it on a map and linked to Craigslist, instead of just taking the content. If Google Maps did it nobody would think twice.
I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.
Proof of Concept is a vague term and doesn't give you any sort of protection or defense. If it's against Craigslist's terms you're taking on some risk by making and public and promoting it.
I, too, entered a search query, pressed go, and witnessed absolutely nothing happen. Turns out, you just have to wait a bit. It scrapes Craigslist in real time from the client, so you have to wait a bit for results to pop up. It'd be nice if they had a loading/progress indicator.
would this not fall into a similar category of happened to linkedin? The data is publicly facing, and not behind a paywall or a password, therefore, basically fair-use of internet?
ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.
Cool idea, but the implementation is kinda crazy. Good enough to just test the idea at home on your own machine, but pretty risky to expose it to the world and let a ton of other people go nuts making bazzilions of requests.
That's basically what happened to Padmapper, sadly.
With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.
Maybe timeliness is the important thing to Craigslist.
If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL.
I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.
You should really consider caching results and only fetching them again after a certain amount of time has passed since the last time you scraped them. This way you're not blowing through your ScraperAPI limit.
Its cool but the main problem I am facing with searching for remote jobs. Is that most remote US jobs are not technically remote. They are for US citizens only, so I can't work on it from Europe.
This is an amazing project! Probably one of the most useful ways to actually find quality job listings. If only Craigslist would offer site-wide search natively.
Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.
56 comments
[ 3.4 ms ] story [ 85.2 ms ] threadconsole.log(items);
console.debug(items);
They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.
Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
What sort of defenses are they using?
Then it sent 503 XHR requests:
https://i.imgur.com/R9D23iU.png
ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.
Also, exposed API key in the request.
With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.
If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL.
I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.
"You're sending requests a bit too fast! Please slow down your requests, or contact support@scraperapi.com with any questions."
Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.