Free SQL dump with 200 million tweets from 13 million users
About the data:
- DB Size: 543 million rows
- Data Size: 173GB (uncompressed)
- Stored in mysql
- 200+ Million tweets from 13+ Million users
- Collected in 1 week
- Operation costs: 100+ dollars
- Rackspace Cloud - 1 CentOS 8GB Ram server
- Java, memcache, mysql and perl for core processing
- js, php for analytics & visualization
* Download the data at this url http://www.archive.org/details/2011-06-calufa-twitter-sql
40 comments
[ 3.9 ms ] story [ 68.1 ms ] threadTwitter search still only goes back 10 days in 2011, so how deep is this data?
So you caught about 2 days worth but randomly in time.
Feel free to follow me http://twitter.com/calufa.
In either case, as Sethish said, distributing dumps like this is against the new ToS.
I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.
- You can cluster users based on tweet data, links relationships &/or even user-to-user relationships
- Understand how retweets work and how fast they propagate.
- Sentiment analysis based on a specific keyword.
- Trend analysis.
There are N number of ways this dataset can be helpful. You have 200MM tweets. Enough for a quick experiment using real data.
* Its true that is "random" data. Just unrandom it!
Propagation of retweets really needs a near complete dataset of those tweets/retweets. A steaming sample of the dataset really isn't great for this.
Sentiment analysis can be done to determine the overall feeling on a topic, but I'd feel really incomplete doing it on this dataset. Again, pulling the stream for the term or keyboard you're looking to sample is much better. Most sentiment analysis on Twitter is pretty flawed anyway.
Trend analysis works on this dataset ok, but measuring the true magnitude of an event would be hard (like Osama being killed) since you don't know what portion of the tweets you've actually got.
I worked with Sethish on the Web Ecology Project. I wouldn't call your dataset useless, but it really would be more useful generally to have a question, then pull the best possible data that will help you answer that question. Otherwise there's going to be a lot more unknowns that make it a weaker piece of research.
I want to clarify that this dump is for learning purposes due the lack of "open data".
If people can play with real data from real people and get "real" inputs, that can encourage curious programmers to join the data-mining party. I know there are other dumps out there, np with that. This is just another dump, it may help people come with ideas without the need of coding a a multi-threaded scrapper.
With that said...the data is now unavailable?
(Anyway, I guess that in practice, if a torrent with a dump of tweets appears somewhere, it's pretty hard to find out who did it. Yes, Twitter could do some clever watermarking of the API results or correlate the dump contents with the server logs, but it would probably be a lot of work.)
bunzip2 < my_database.sql.bz2 | mysql -h localhost -u root -p my_database
"Msbuild seems to limit to 100 files on a cl command line, which introduces noticeable sync losses when parallel building on 24 threads."
It's not all meaningless, you just choose to follow meaningless users.
edit reply via twitter: "they asked me to remove the dump due TOS" (http://twitter.com/#!/calufa/status/78556903772393474)
which I guess is what I expected.
But are scrapers subject to TOS?