Twitter bans unauthorized search engines from expanding shortened links (t.co)
Content of robots.txt is:
User-agent: * Disallow: /
No robots should try to expand twitter shortened links. So search engines have to pay Twitter (for firehose, for instance) to get better results.
0 comments
[ 3.4 ms ] story [ 8.4 ms ] threadNo comments yet.