It’s 12 AWS Snowmobiles, which are container trucks full of hard drives. A corporation that generates that much data probably has more container trucks of furniture.
I don't think they factor in compression here. It's likely the 1.7MB they're talking about is raw data. Google's location history for example can be compressed down to very manageable single digit percentages even with fast compression.
Edit. I'm talking about Grab, the "transportation, e-commerce, payments, and financial services" company in the article, which sounds like it collects little if any multimedia from users. Also thinking more of data the company generates from the user (usage history, metadata and collection), not what the user generates and uploads.
The 1.7MB figure can't really be extrapolated for a Facebook-like data set (or worse, YouTube). Social networks have a lot of mostly multimedia content created and uploaded by the user, and lower volume of company generated content like location tracking and lists of interactions.
If it’s raw data and not simply actual disk use divided by users, I would guess the savings from compression likely pale in comparison to metadata such as indices, backups and other overhead.
With some basic compression, this daily 1.7 MB will fit on a 1.44 MB 3.5" floppy... Our Father, who art in The Cloud, give us this day our daily floppy ! Don't forget to swap your floppy every morning. https://i.imgur.com/ntrjkV7.jpg !
Here in Thailand I use grab at least once every day. Sometimes up to 4 times a day.
Taxi to office, from office, meals, groceries, courier service. Grab would not work without the army of motor bikes.
Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data.
Transactions are probably the smallest amounts. Another big one is probably all the inventory records from supermarkets, convenience stores and restaurants in probably the number one foodie city in the world - Bangkok. Then there is taking photos to prove delivery pickups and drop offs. Damn…
Then travel to the Philippines, Vietnam, Indonesia - grab also just works. It’s actually quite an impressive operation and they’ve improved the app quite a bit in the last year alone.
Now with the lockdowns at the moment they are super busy more than usual - they probably up that amount of daily data by amount 50%
> Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data.
Logging gps once per second every second of the day using two doubles will get you 691Kib of data. Unless you have two deliveries in flight 24/7 the math does not check out.
I believe precise GPS coordinates is 12 bytes per sample, so that’s ~1MiB/day. However, it compresses extremely well even with lossless content-unaware compression techniques (though I don’t have any real numbers), and if you’re serious about compression you can do even better still, with http://cs.joensuu.fi/~mchen/GPSTrajComp.htm showing a technique that gets you down to ~3–6KiB per day if you’re happy with up to 10m of error.
You would only need to store the relative differences between the coordinates, so the actual precision is much lower than 12 bytes, even for a naive scheme.
For correctness, I must note that if you’re using the usual static-size data types like floating- or fixed-point numbers, deltas take up roughly the same amount of space unless you impose speed limits, which allow you to get a bit cleverer about the space consumed, though it’s still complicated by the world being a sphere rather than flat.
Notwithstanding that, when you get to compression, delta encoding does make it losslessly compress a bit better, though not as much as you might think, especially with the better compression algorithms. I simulated some stuff for this last year, but didn’t keep the answers I came up with. My extremely vague notion is that zip or gzip got something like a 10% improvement, but LZMA only saved another 1–2%. Don’t trust my numbers, though. And yeah, if you’re caring about this stuff you can do way better by a more deliberate encoding.
If you have a list of coordinates, then you know a priori what the maximum precision of the deltas is. If your delivery driver doesn't teleport at any point in the journey it's likely that you can shave quite a few bits off the top of each delta.
For example, if your coordinates are precise to 1 angular meter, and the maximum delta in the list is 200 angular meters, then you only need 2x1 bytes for each coordinate.
Obviously you can't say clearly all the data is valuable, but you can say clearly some of the data is valuable because without any data they would not have any business.
A useful reference for magnitude of "1MB" is that a typical (english) novel contains around 0.5 to 1 MB of ASCII text. Neuromancer is just under 500kB; Snow Crash is ~900kB. So 1.7MB/user/day is roughly equivalent to 2-3 novels written about each user every day.
Number of values would be much more interesting. An image is big, a bloated json somewhat big, and compressed or not compressed? (actually the compressed size would be more telling)
28 comments
[ 4.8 ms ] story [ 73.2 ms ] threadI used the number 2 billion because it's approximately the number of people on Facebook. Not vetted, just for a quick calculation.
No wonder why HDD maker are happy.
1.2 Exabytes a year is a lot.
Edit. I'm talking about Grab, the "transportation, e-commerce, payments, and financial services" company in the article, which sounds like it collects little if any multimedia from users. Also thinking more of data the company generates from the user (usage history, metadata and collection), not what the user generates and uploads.
The 1.7MB figure can't really be extrapolated for a Facebook-like data set (or worse, YouTube). Social networks have a lot of mostly multimedia content created and uploaded by the user, and lower volume of company generated content like location tracking and lists of interactions.
Here in Thailand I use grab at least once every day. Sometimes up to 4 times a day.
Taxi to office, from office, meals, groceries, courier service. Grab would not work without the army of motor bikes.
Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data.
Transactions are probably the smallest amounts. Another big one is probably all the inventory records from supermarkets, convenience stores and restaurants in probably the number one foodie city in the world - Bangkok. Then there is taking photos to prove delivery pickups and drop offs. Damn…
Then travel to the Philippines, Vietnam, Indonesia - grab also just works. It’s actually quite an impressive operation and they’ve improved the app quite a bit in the last year alone.
Now with the lockdowns at the moment they are super busy more than usual - they probably up that amount of daily data by amount 50%
Logging gps once per second every second of the day using two doubles will get you 691Kib of data. Unless you have two deliveries in flight 24/7 the math does not check out.
Notwithstanding that, when you get to compression, delta encoding does make it losslessly compress a bit better, though not as much as you might think, especially with the better compression algorithms. I simulated some stuff for this last year, but didn’t keep the answers I came up with. My extremely vague notion is that zip or gzip got something like a 10% improvement, but LZMA only saved another 1–2%. Don’t trust my numbers, though. And yeah, if you’re caring about this stuff you can do way better by a more deliberate encoding.
For example, if your coordinates are precise to 1 angular meter, and the maximum delta in the list is 200 angular meters, then you only need 2x1 bytes for each coordinate.
What do they say about correlation and causation?
Obviously you can't say clearly all the data is valuable, but you can say clearly some of the data is valuable because without any data they would not have any business.