Python is perfectly fine for these type of tasks. I ingest all of Reddit in real-time (https://pushshift.io) and also ingest Gab.com and several others (Stackoverflow, etc.) and at most one or two CPU cores are at…
When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution.…
It's variable based on what's going on at the time but I've seen upwards of 7k tweets a second for the sections of the timeline that I've ingested using this technique. Someone suggested trying it when the New Year…
This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
I am the author of this document. If anyone has any questions, I'd be happy to answer them!
Python is perfectly fine for these type of tasks. I ingest all of Reddit in real-time (https://pushshift.io) and also ingest Gab.com and several others (Stackoverflow, etc.) and at most one or two CPU cores are at…
When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution.…
It's variable based on what's going on at the time but I've seen upwards of 7k tweets a second for the sections of the timeline that I've ingested using this technique. Someone suggested trying it when the New Year…
This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
I am the author of this document. If anyone has any questions, I'd be happy to answer them!