HN Explorer

Ask HN: Where can I find Internet browsing data?

2 points by gregw134 ↗ HN
Somebody posted a large dataset from Indiana University a while back, but it's several years old and only available to researchers.

http://cnets.indiana.edu/groups/nan/webtraffic/click-dataset

Does anyone know of another source of browsing data? Is it possible to get this data in real-time?

4 comments

[ 5.1 ms ] story [ 16.4 ms ] thread
Irishsteve
To get it in real time you most likely have to pay someone who is already collecting said data (Don't know who) or write software yourself and pay people to install it.

Is there any particular type of data you are looking for? Or is it simply anon user x went to page y on domain z

gregw134
>you most likely have to pay someone who is already collecting said data (Don't know who) or write software yourself and pay people to install it.

Yeah that's what I was afraid of :(

I'm looking for basic, anonymized browsing history for users over a period of time. The idea I'm trying to test is whether you can identify the users who are "trendsetters"--people who visit sites or posts before everyone else--and whether you can sniff out links that will be popular based on their behavior.

I've found a few more links, but most of them require you to be a researcher to access the data:

https://www.predict.org http://www.caida.org http://crawdad.cs.dartmouth.edu/

ig1
Why not do it based on tweets ?
Irishsteve
Spotting trend setters on the web is going to be a little tough because of how large and segregated it is.

Some datasets people use is Wiki based around comments / edits or who created a page first which became popular in the end.

You could also use Facebook likes (facebook.com/me/likes) where each hlike is time stamped.