Show HN: Gargantuan Takeout Rocket – Google Takeout Transloader to Azure (github.com)
Gargantuan Takeout Rocket (GTR) is a toolkit to make the pain of backing up a Google account to somewhere that's not Google a lot less. At the moment the only destination supported is Azure.
It's a guide, a browser extension, a Cloudflare worker to deploy, and Azure storage to configure. This sounds like buzzword creep, but believe me, every piece is extremely important.
It's very cheap to run/serverless. You can backup a Google account at about $1/TB.
Compared to renting a VPS to do this, it's much more pleasant. You aren't juggling strange URLs, needing big beefy boxes to buffer large data, or trying to login to Google or pass URLs through a VPS. Unfortunately, not everything about the procedure can be automated. But whatever can be, is.
It's very fast. 1GB/s is the stable default and recommended speed. However, you can have about 3 of these going at a time for about 3GB/s+ overall. This trick is accomplished by making Azure download from Google to a file block, a unique API not seen in S3 or S3-like object storage.
Unfortunately, Azure has URL handling bugs and only supports HTTP 1.1, greatly limiting parallelism. We can use Cloudflare Workers to work around these issues.
I use GTR myself with a scheduled Google Takeout every two months to backup 1.5TB of data from Google. This can be photos, YouTube videos, etc. I can finish my backups to safe non-Google storage in 15 minutes after I get an email from Google that my Takeout is ready to be downloaded.
Unfortunately the only destination is currently Azure. There's also no encryption support. And also Cloudflare is involved. That said, if you're fine with this, this is a fine way to backup a Google and Youtube account as-is.
42 comments
[ 2.6 ms ] story [ 108 ms ] threadIt simply had the least amount of danger or cost for hosting a public demo. The bandwidth costs that the Big cloud providers like AWS, Azure, and GCP have simply wasn't there.
It also deploys in milliseconds, which was a great developer experience.
Copied and pasted from https://github.com/nelsonjchen/gtr-proxy
* Cloudflare does not charge for incoming or outgoing data. No egress or ingress charges.
* Cloudflare does not charge for CPU/Memory used while the request has finished processing, the response headers are sent, and the worker is just shoveling bytes between two sockets.
* Cloudflare has the peering, compute, and scalability to handle the massive transfer from Google Takeout to Azure Storage. Many of its peering points are peered with Azure and Google with high capacity links.
* Cloudflare Workers are serverless.
* Cloudflare's free tier is generous.
* Cloudflare allows fetching and streaming of data from other URLs programmatically.
* Cloudflare Worker endpoints are HTTP/3 compatible and can comfortably connect to HTTP 1.1 endpoints.
* Cloudflare Workers are globally deployed. If you transfer from Google in the EU to Azure in the EU, the worker proxy is also in the EU and your data stays in the EU for the whole time. Same for Australia, US, and so on.
What this project is doing is adding another potential point of privacy failure. My suggestion would be for users to not use the public proxy, but to modify their own proxy to GPG encrypt each Takeout file with their public key as the it is passed through to Azure Storage.
Unfortunately, even if people did modify their own proxies, it would quickly blow through any CPU budget on Cloudflare workers. When you finish processing and return some stream objects or whatever on Cloudflare Workers, Cloudflare basically unloads the worker, stops CPU billing, and just handles shoveling bytes between two objects/sockets.
The closest thing I think would work is using Azure Storage's encryption. It's still theoretical though and I haven't tried it with block by block transloads much less uploads. Unfortunately, it's symmetric and Azure will hold the symmetric key but pinky promises to wipe it after the transfer is done. This should deter most adversaries.
https://learn.microsoft.com/en-us/azure/storage/common/stora...
https://github.com/nelsonjchen/gargantuan-takeout-rocket/iss...
There's also the obvious thing of just downloading the archives via an Azure VM, encrypting it, and throwing it back in the storage/bucket, and deleting the unencrypted transloaded bytes. Downloading from Azure Storage to an Azure VM should be very quick. You might even be able to do it "stateless" without intermediary files in the VM! There is still a window where it's unencrypted but you can close it up afterwards.
The bytes become way more handleable once they're outside Google's URLs and on Azure. Use GTR to get the bytes out of Google Takeout's quagmire.
Worker was supposed to run for free as the files were streaming and no CPU time was used but in practice CF workers were stopping because I was exhausting the free limit so I guess something in my code did end up using CPU
So, I guess the TL;DR (heh, quite literally in this case) is that if Wasabi allows you to download something like an Ubuntu ISO to Wasabi storage by giving it the ISO URL from your phone, it'll likely work for you. Otherwise, you're back in the VPS setup TFA was seeking to avoid
Wasabi allows cloud-to-cloud migrations[1].
Disclaimer: I'm an official partner in Brazil.
[1] https://wasabi.com/cloud-migration-tools/
Hypothetically one could trick the Cloudflare workers into exposing those Takeout URLs under fake GetObject request-response handshake, but ... what a lot of work just for Wasabi
From the outside it's a lot closer to the storage transfer job stuff I've seen on GCP and yet also kind of far as it's gated behind those infernal "talk to our team" for access gates.
That said, I've tried PUT-ing directly the Azure but quickly discovered that Azure actually has an ingress(!) quota on bytes, not requests like S3 or S3-likes I've seen before.
I have not implemented a S3 target yet. I might try. I'm particularly interested in targeting Cloudflare R2 as a temporary staging area so I can download the takeout files via a resumable or bulk download manager for a local backup without heavy bandwidth costs. AWS S3 also has their own competitive Archive tier too.
Maybe keep an eye on https://github.com/nelsonjchen/gargantuan-takeout-rocket/iss...
Yes, so is Wasabi. It doesn't charge you for ingress/egress data, so there is a clear cost advantage over similar services.
Disclaimer: I'm a Wasabi official partner in Brazil.
As for that special-ness you've mentioned, Wasabi is actually falling behind on that "charge you for ingress/egress" point since has a serious asterisk. One of the reasons I'm far more interested in Cloudflare R2 for a possible future temporary staging platform for local downloads is because it's genuine not "charge you for ingress/egress" as Wasabi says "don't egress more than you have stored" or we'll ban you. R2 does not have that limitation there. In fact, I've used it to serve a 28GB file that have been downloaded many, many times with only 28GB stored in R2. Such a thing is against Wasabi's use and the threat makes it unsuitable for GTR.
As for cost, like a final destination, there is no option to do Deep Glacier or Archive tier like on AWS or Azure. It is significantly more expensive.
I'm also absolutely hate the Chrome Marketplace for prohibiting the existence of such tiny use cases (although kind of understandable)
But I'm also aware that Azure can't be that cheap. And thankfully, the repo includes a description of the catch:
> Restoration: For 1TB, this will cost about $100.88 (rounded up). Small price for salvation.
For those who don't know, Clouds separate costs based on readiness (Hot, Archive) and operations (Read, List, Write), unlike Google Drive, which has fixed pricing per TB
So, this makes use of the Archive's low cost, which won't work well for backup use cases that require random access (such as looking up an old photo), but will be great for an additional backup that comes after a local hard disk backup.
Great solution though! Maybe with some adjustment someone can make it Support random access in the future (making an index to restore a single file to "Hot Storage" again)
https://github.com/nelsonjchen/gargantuan-takeout-rocket/iss...
Direct Google Takeout archives URLs are only valid 15 minutes.
I'm not going to make any effort until R2 gets lifecycle rules though. It is too easy to accidentally leave too much there and get a large bill.
I am hesitant to use any backup solution that doesn't have an equally well tested restore path. Ideally a restore path to another provider too - eg. migrate google photos to icloud photos, with as many tags/labels/metadata preserved as possible.
And those other already existing projects are useless without the bytes :).
> Pre-GTR, the backup procedure would have taken at least 3 hours even with a VPS Setup facilitating the transfer from Google Takeout as even large instances on the cloud with large disks, much memory, and many CPUs would eventually choke with too many files being downloaded in parallel.
("VPS" above links to https://sjwheel.net/cloud/computing/2019/08/01/aws_backup.ht... , which describes spinning up an EC2 machine with enough local storage to download the whole takeout, and using that to download from google and then upload to s3)
I think one could do this "transloading" comfortably inside the AWS Lambda limits. The golang AWS sdk's s3.s3manager.UploadInput takes an io.Reader and puts its contents into an S3 bucket+key. You could make a lambda function that downloads a Takeout part and chunks it into s3. Lambda scales to zero and will be patiently waiting for you two months from now when you need it again.
Unfortunately, product-wise, they seemed to have missed the memo about making an S3 compatible interface like GCS. Oh well. At least we got GTR!
GTR was organically grown from seeing Azure's Blob Storage download API. I got disappointed at the bugs seen and then just patched over it with Cloudflare Workers, but didn't change up to just AWS/S3.
I could see myself also going this route too in a clean slate. With the 100MB limit gone, and having to pay for CPU anyway, maybe we could also get proper GPG public key encryption in it too.
It's documented here:
https://developers.cloudflare.com/workers/platform/limits/#r...
This limit actually isn't specific to Workers. It's a general Cloudflare thing, because Cloudflare buffers uploads to disk in order to defend against slow loris attacks. That is, this is actually meant to protect your origin. We've occasionally discussed letting people turn this off if they don't need it but it hasn't come up very often...
(I'm the lead engineer for Workers.)
Outgoing requests from Workers destined for external servers (i.e. not other sites on Cloudflare) really have no reason to be buffered and thus no reason to impose these limits. We should fix that. I also suspect most Workers-based apps don't really need incoming buffering either.
Have you considered streaming the files from there to Azure Storage ?
That said, I'm pretty happy I got acknowledgement on a Workers issue though. Wasn't sure if it was a bug, but it was a bug! https://news.ycombinator.com/item?id=34891237