"The Click Dataset is large (~2.5 TB), which requires that it be transferred on a physical hard drive." Don't expect it to be hosted online for download any time soon.
This would be a direct violation of the Data Security Agreement (https://protect.iu.edu/system/files/Data-Security-Access-Agr...) that all persons requesting the data must agree to. Furthermore, they seem to indicate that they will only be releasing the data to researchers or large organizations.
That being said, feel free to request the data from them.
How did they collect this data without someone raising privacy flags? Releasing this data is almost certainly a bad idea, since it will likely reveal who the people are who made those requests. Anonymized data usually isn't.
> Additionally, while the dataset has been approved by the Indiana University IRB for “non-human subjects research” (protocol 1110007144), it might potentially contain bits of stray personal data. Therefore we require that you follow these instructions to request the data. You will have to sign a data security agreement.
> Data Transfer: If your request is approved, you will send a blank 3TB hard drive to the address below.
> We will return the loaded drive to the address specified in your request within 10 business days of receiving it. The data on the drive will be encrypted using TrueCrypt. The password to decrypt the data will be emailed to address that you specified in your request. It will be your responsibility to install and configure TrueCrypt (http://www.truecrypt.org/) on the system where you will be accessing the data.
> 4.) Data Removal: When you have finished using the data, you are responsible for securely and
permanently removing the data including the drive that was used to transfer the data. For more
information about secure data removal please see: (https://protect.iu.edu/cybersecurity/data/secureremoval) .
> I have read and agree to abide by all University data security practice related to access to University confidential data. To the best of my ability, I will comply, keep secure, or return all information provided to me.
TL:DR they're just crossing their fingers and hoping for the best.
Oh boy, this is going to end badly. Remember when AOL released a huge anonymized dataset of their searches? People were still identified because naturally people searched for their own names, the names of their friends and families, local businesses, personal websites, etc.
This dataset is even worse since it includes both referral and the destination.
Keep in mind websites often put the usernames within the URL
The data doesn't seem to contain enough information to cluster unrelated URLs (e.g. by User Agent). Each record contains 1) a timestamp, 2) the requested URL, 3) the referring URL, 4) a boolean classification of the user agent (browser or bot), 5) a boolean flag for whether the request was generated inside or outside IU.
Although you could reconstruct who's looking at Facebook from inside IU, it'd be difficult to further associate them with some other website like Hacker News. On the other hand, the timestamps do leak some information which could potentially be used to identify patterns of activity.
Yes, but they aren't sharing the user agent. They're just sharing a boolean derived from the user agent which says whether they think it was a bot or a browser.
If you are interested in this kind of data, it's worth noting that there are some older, but, in a sense, more manageable datasets at the Internet Traffic Archive [1]---the data there can be downloaded and does not require being physically shipping through the post.
The largest dataset consists of 1.3 billion requests (for the 1998 World Cup website).
From a research standpoint this data set is much less interesting than a bunch of students/faculty/bots/apps clicking and surfing their way around the whole Internets.
Marc Smith at Microsoft Research had a Usenet dB for research porpoises created about 6 years ago or so, and provided it to any researchers who wanted it. Although I didn't care about Usenet for my stuff, it was a good and useful offering for various researchers, and I hope this newer dB also proves useful! Thanks to Indiana for going to the trouble.
17 comments
[ 2.4 ms ] story [ 50.0 ms ] threadThat being said, feel free to request the data from them.
http://carl.cs.indiana.edu/data/webtraffic/click-dataform.pd...
> Data Transfer: If your request is approved, you will send a blank 3TB hard drive to the address below.
> We will return the loaded drive to the address specified in your request within 10 business days of receiving it. The data on the drive will be encrypted using TrueCrypt. The password to decrypt the data will be emailed to address that you specified in your request. It will be your responsibility to install and configure TrueCrypt (http://www.truecrypt.org/) on the system where you will be accessing the data.
> 4.) Data Removal: When you have finished using the data, you are responsible for securely and permanently removing the data including the drive that was used to transfer the data. For more information about secure data removal please see: (https://protect.iu.edu/cybersecurity/data/secureremoval) .
> I have read and agree to abide by all University data security practice related to access to University confidential data. To the best of my ability, I will comply, keep secure, or return all information provided to me.
TL:DR they're just crossing their fingers and hoping for the best.
This dataset is even worse since it includes both referral and the destination.
Keep in mind websites often put the usernames within the URL
Eg: http://www.facebook.com/Your.Name
http://www.reddit.com/user/USERNAME/
http://slashdot.org/~USERNAME
http://news.ycombinator.com/user?id=USERNAME
So no matter how much you think you have it anonymized, a person's browsing history could reveal a lot more than you think.
Although you could reconstruct who's looking at Facebook from inside IU, it'd be difficult to further associate them with some other website like Hacker News. On the other hand, the timestamps do leak some information which could potentially be used to identify patterns of activity.
https://www.eff.org/deeplinks/2010/01/tracking-by-user-agent
- The potential upside is a few citations in research papers.
- The potential downside is a widescale invasion of privacy of IU students and staff, and a huge PR disaster.
The largest dataset consists of 1.3 billion requests (for the 1998 World Cup website).
[1] http://ita.ee.lbl.gov/html/traces.html