S3cmd S3 sync memory usage solution?

1 points by jaddison ↗ HN
I'm sure I'm not the only one to encounter OOM (MemoryError) issues using `s3cmd` to sync files to AWS S3. I believe that the culprit is that the local/remote file lists are stored in memory.

Has anyone got any novel solutions to this? As for fixing s3cmd itself, I've suggested the use of sqlite3 as a replacement for in-memory lists: https://github.com/s3tools/s3cmd/issues/408

A bad path? Any better options?

1 comment

[ 3.1 ms ] story [ 18.9 ms ] thread