The odd thing, is that getting a list of all profiles correctly doesn't wor. If you go to Google's robots.txt and scroll to the bottom to find the profiles sitemap is located at:
Because I spent too much time trying to figure out why the index sitemap XML file was returning a 404. I'm really surprised that Google, one of the inventors of sitemaps and big proponent of using them, can have such a high-profile error. I dug around to see if I could find what's going on.
By the time I got to figuring out how many text files there are, I was getting bored.
sitemap-3278 is the upper limit, so there appear to be roughly 16.4 million profiles available. I would be interested to see how many of these have public information (name, address, etc.).
Those profile lists aren't complete. For more discussion, see the Google Buzz list where, until the Buzz team released their own firehose, any of us who wanted a firehose had to build our own. Also noteworthy, it's easier to just download and parse those XML files -- why use a search engine for that to begin with when it's already aggregated for you?
No, harder. Your link would require you to crawl thousands of Google results pagea, while the OP's link gives you sitemap URLs of many profiles at once.
I don't think it's a big deal but I guess the author wanted to point out an easy way to crawl this public info. There are similar ways to crawl public Facebook profiles too, so it's really not a big deal.
Have you tried it? This was posted in January, but doesn't seem to work anymore.
curl -A Opera 'http://www.facebook.com/ajax/typeahead_friends.php?u=4&__a=1'
for (;;);{"error":0,"errorSummary":"","errorDescription":"","errorIsWarning":false,"silentError":0,"payload":{"friends":[]}}
pete warden tried that and got sued by facebook. they have since changed their robots.txt to forbid public crawling of their site although they still want the google juice so they make exceptions for major search engines.
I said you can _crawl_ in the sense that it's technically possible & easy. Also I don't think you'd get into trouble if you use it for your own research and not disclose your results.
It's interesting that the profiles list the public followers and following stats. With some deep crawling, you could build a huge social graph from this data.
you need to be signed in to google to see the followers/following data. it shouldn't be hard to pass a valid cookie along, but it might get a little suspicious for one user to make ~60m requests.
When I filled out my Google Profile I was told by Google that the information would be added to their search and would make me more easily discoverable.
Are people going to act all shocked that means it's indexable? gasp
The thing that really bugs me about Google profiles is that they're directly connected to email addresses. I guess Google thinks their ability to deal with spam is good enough that putting millions of their user's email address publicly available isn't a concern.
The email address is not displayed on the profile page.
A user can customize his or her profile URL with an email address. The settings page clearly states that the email address is publicly discoverable if the email address is used in the profile URL.
Shouldn't I be allowed to customize my profile URL with a string other than my email address? Why force me to expose my email if I want something other than a 21-character numeric id?
46 comments
[ 3.9 ms ] story [ 96.8 ms ] threadhttp://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Which 404s.
Luckily, it's quite easy to reverse engineer. Start with:
http://www.gstatic.com/s2/sitemaps/sitemap-000.txt
and increment 000 till you 404. From quick testing, it's less than 3500 but more than 3200.
By the time I got to figuring out how many text files there are, I was getting bored.
curl http://foo.com/[1-1024] -o "#1.txt"
That's files sitemap-000 through sitemap-3278. They contain a total of 16,256,271 profile URLs.
Just to do it?
http://www.google.com/profiles/112195136944637672112
When I filled out my Google Profile I was told by Google that the information would be added to their search and would make me more easily discoverable.
Are people going to act all shocked that means it's indexable? gasp
A user can customize his or her profile URL with an email address. The settings page clearly states that the email address is publicly discoverable if the email address is used in the profile URL.
Point being to follow like-minded people on Buzz to gain exposure to your own profile.