HN Explorer

Ask HN: Craigslist and crawlers

1 points by kinkdr ↗ HN
I wrote a small crawler for Craigslist making absolutely sure that it obeys the robots.txt rules.

Sure enough, after a few hindered of requests my crawler got banned.

Doing a simple search at Google, I can see from the results I get that CL is happily allowing Google bots.

I strongly believe that this discrimination against small players is highly unethical, but my question is, is it breaking any laws or maybe anti-monopolistic rules?

6 comments

[ 2.7 ms ] story [ 27.7 ms ] thread
IanDrake
Stay away from doing this. I built a small business doing real-time alerts on CL and eventually got a C&D letter from their lawyers. I gave up.

I got around IP restrictions by essential having SETI like software running on my client's computers.

They will threaten you saying you're committing a felony by violating the CFAA. They are insane, but have deep pockets.

After I shut down, others stood up against CL's ridiculous claims and lost. Search for 3taps.com and padmapper.com to learn more.

kinkdr
Wow! Thanks for the info. So sad that they lost. How is this not illegally thwarting of competition?

I wonder how can they claim that what you did was illegal or against their TOS, but what search engines are doing is not.

I guess I am lucky I didn't spend too much time on this. Thanks again!

IanDrake
In the epic 3taps vs. CL battle, CL claimed copyright on all posts, which wasn't supported by their TOS. So they updated their TOS to give them worldwide exclusive rights to ads.

CL finally backed down when the EFF told them to, which says a lot because the EFF is practically owned by CL.

jacquesm
> because the EFF is practically owned by CL.

Sorry?

I'm totally floored by that, did I miss something?

I know that CL donated some sum of money to the EFF but that was a settlement.

IanDrake
The EFF is an outspoken critic of the CFAA and has admonished multiple entities for threatening criminal prosecution. When Craigslist threatened me the first thing I did was contact the EFF. They didn't ignore me. They told me there is nothing they were willing to to at the time, which in my eyes was worse than just ignoring me.

Only until it became a high profile issue with 3taps did the EFF speak out against CL.

That was about a year after I contacted them and they only did after 3taps started getting press.

To me it was pretty clear that they didn't want to bite the hand that fed them, until they had to or else lose all credibility.

kinkdr
> Craigslist is a privately owned website, they can ban you at will.

I don't think it is as simple. Wouldn't selectively banning competitors of Yahoo search, break some competition/anti-monopoly rules/laws?