The screenshot on the repo only shows a comparison to the AWS CLI. Here is a comparison for uploading a single 1KB file with the AWS CLI, a minimal AWS API client and raptor:
~ $ time aws s3 cp data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
real 0m0.720s
user 0m0.593s
sys 0m0.070s
$ time ./s3upload data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
Uploaded data/1kb/file_1.bin
real 0m0.327s
user 0m0.136s
sys 0m0.062s
~ $ time ./raptor data/1kb/file_1.bin $RAPTOR_ENDPOINT --server-key $RAPTOR_SERVER_KEY --client-key $RAPTOR_CLIENT_KEY --silent
real 0m0.303s
user 0m0.013s
sys 0m0.014s
AWS S3 CLI or API, it's the transport and protocol overhead that are burning the extra CPU time (power). At least, that's my conclusion.
The raptor tool has another nice advantages not possible with the AWS API:
Optional Acknowledgement. The `--confirm` option allow skipping waits for ACKs, which takes the run time down to just what it takes to put the outbound packets on the network. That makes uploads very fast, and the reliability is tunable with the `--overhead` option. On networks with significant packet loss this can make urgent uploads much, much faster.
The `--rate-mbps` controls the transmission rate. Sometimes slower is better, and having this option means not needing to setup bandwidth control using `cgroup` or similar.
Interesting, I actually had to solve this problem at work. I wonder for use cases where users really need to upload that many small files, whether providing more SDKs gives more benefits than a CLI. (I see you provide a C# library, unfortunately that's out of my expertise). If your benchmark can include stats for uploads that normally take more than 15 min, that'd be much more eye-catching at least to me. Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
Some more meta-level questions if don't mind:
1. People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether. If somebody cannot use this approach, I guess their use case involves much more real-time reading of each file after creating it. Your pipeline involves quite a lot of moving parts. Does the tool/sdk have enough support for observability/failure recovery?
2. Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
2 comments
[ 2.5 ms ] story [ 8.1 ms ] threadCode for the s3upload tool is here: https://gist.github.com/mlhpdx/aaf99a468c06747dbea12f9639102...
The raptor tool has another nice advantages not possible with the AWS API:
Optional Acknowledgement. The `--confirm` option allow skipping waits for ACKs, which takes the run time down to just what it takes to put the outbound packets on the network. That makes uploads very fast, and the reliability is tunable with the `--overhead` option. On networks with significant packet loss this can make urgent uploads much, much faster.
The `--rate-mbps` controls the transmission rate. Sometimes slower is better, and having this option means not needing to setup bandwidth control using `cgroup` or similar.
Some more meta-level questions if don't mind: 1. People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether. If somebody cannot use this approach, I guess their use case involves much more real-time reading of each file after creating it. Your pipeline involves quite a lot of moving parts. Does the tool/sdk have enough support for observability/failure recovery? 2. Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Anyway, interesting read. Learned something new.