"The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."
It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.
We made the same decision for 's3cmd'.
All of that to say:
The biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky auth workflow for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.
[1] rsync.net
[2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.
Rclone is clunky because it works with everything, even things like OneDrive on Linux, which is not supported officially. Love rclone. Also just a great copy / move utility.
I think there are several workflows for doing this - I know in the past we used something called 'cx_Freeze' but I think the person in charge of this is now using 'py2exe' ?
I think the 'borg' project packages up their single file .exe distribution with 'pyinstaller' but I may not fully understand that part of their process...
Pyinstaller is the most actively maintained and in-use, but nuitka is on the rise as well. There are less-active but still serviceable ones like py2exe as well
I'd also be curious about how rclone's parallelization compares to gcloud storage's in practice:
> Second, it utilizes a new parallelization strategy that treats task management as a graph problem, which allows more work to be done in parallel with far less overhead.
> ... In gcloud storage, all operations happen in parallel. As an example, parallel composite uploads are enabled automatically based on bucket configuration. This is a vast improvement compared to gsutil, which requires the -m (parallel operations) flag to improve the performance for uploads and downloads.
how faster is rclone compared to s3cmd ? We're using s3cmd to sync some stuff for backup purpose (many many small-ish) files and performance has been generally mediocre
I have also had persistent issues with gsutil and have used rclone instead, rclone is clunky but it has been an ever increasing part of our toolkit and I don't see that changing soon.
gsutil hardly ever finished when trying to download a couple-hundred google workspace takeout, preferring to consume all available memory instead - so that's a pretty easy thing to beat.
This was inevitable it seems. If you look under the hood gsutil is a wrapper around a vendored version of boto2. boto3 is a big departure and never got support for Google Cloud Storage.
Also the fact that it was separate from the rest of the gcloud CLI was a weird quirk. This fixes both of those issues.
"These tests have been performed on Google Cloud Platform using n2d-standard-16 (8 vCPUs, 32 GB memory) and 1x375GB NVME in RAID0 in us-east4."
Am I missing something? Is there some configuration of RAID0 that allows you to use a single drive? Or is this just a confusing way of saying they have one logical drive available, because the two attached NVME drives are configured in RAID0?
This could just be an ambiguous wording thing, i.e: it's one 375GB effective volume, backed by NVME in RAID0. I dunno how it works, but depending on how it's sold (e.g: you choose the size you need, then the redundancy/performance you want, rather than having to define how that is constructed at a physical level), it may make sense to talk about it that way.
The original intention for most of the flags was to keep them aligned with rsync. The feature only compressed the data in flight, as opposed to -Z/z which stored the compressed version. This didn't directly align to anything in rsync, so I picked an unused letter in rsync (-J/j).
In retrospect, it might have had a better home as some option on -Z/z, or if I had the tool figure out if you were bottlenecked on bandwidth, spare compute, and your data compressed well, and apply it automatically.
You're totally right! If anything the gsutil's -Z/z doesn't align with rsync's -Z/z, and gsutil's -J/j is a better match for what rsync's -Z/z does. I added -J/j well after -Z/z was there so it's something that we have to life with unfortunately.
I initially misunderstood the article's explanation of the gsutil shim: namely, I thought it was "turned on" by default, but that is not the case. One either needs to apply the shim, or to start using `gcloud storage` instead of `gsutil`.
I'm interested to see more benchmarks. The posted numbers in that blogpost seems too good to be true, is tooling really so far behind that a single tool could surpass the transfer time by that much?
It “cheats” by setting up a VM in GCP and another in AWS and does a direct transfer between the two instead of having to upload and download on whatever local connection you’ve got.
Faster than the old gsutil utility. The speed increase is real and easily measurable, but will depend on your files (lots of small ones vs a few huge ones etc).
I have not worked with gcloud storage command as I am only familiar with gsutil, but is the syntax the same? Also, is there a comparison with Terraform?
44 comments
[ 3.4 ms ] story [ 113 ms ] thread"The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."
It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.
We made the same decision for 's3cmd'.
All of that to say:
The biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky auth workflow for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.
[1] rsync.net
[2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.
[3] https://rclone.org/
I think there are several workflows for doing this - I know in the past we used something called 'cx_Freeze' but I think the person in charge of this is now using 'py2exe' ?
I think the 'borg' project packages up their single file .exe distribution with 'pyinstaller' but I may not fully understand that part of their process...
https://pyinstaller.org/en/stable/usage.html
https://www.nuitka.net/
Haven’t heard of rclone before so can’t compare yet but s5cmd is supposed to work against GCS and other s3 compatible apis.
> Second, it utilizes a new parallelization strategy that treats task management as a graph problem, which allows more work to be done in parallel with far less overhead.
> ... In gcloud storage, all operations happen in parallel. As an example, parallel composite uploads are enabled automatically based on bucket configuration. This is a vast improvement compared to gsutil, which requires the -m (parallel operations) flag to improve the performance for uploads and downloads.
https://github.com/mishudark/s3-parallel-put
does it outperform rclone?
Also the fact that it was separate from the rest of the gcloud CLI was a weird quirk. This fixes both of those issues.
Am I missing something? Is there some configuration of RAID0 that allows you to use a single drive? Or is this just a confusing way of saying they have one logical drive available, because the two attached NVME drives are configured in RAID0?
In retrospect, it might have had a better home as some option on -Z/z, or if I had the tool figure out if you were bottlenecked on bandwidth, spare compute, and your data compressed well, and apply it automatically.
> -z, --compress compress file data during the transfer
https://medium.com/@paras_jain/skyplane-110x-faster-data-tra...
The architecture goes through some of the methodology: https://skyplane.org/en/latest/architecture.html which is what I'm currently reading through and trying to understand.
[1] https://docs.globus.org/globus-connect-server/v4/man/globus-...