HN Explorer

Ask HN: Any good reads, articles or books on file migration?

4 points by bosch_mind ↗ HN
I’m looking to build a tool to migrate files between cloud providers (ie blob storage, s3, etc).

I’m looking to learn best practices for doing so. The volume is a few petabytes but can be done incrementally.

I was thinking some queue that handles N transfers concurrently running in a k8s cluster. Queue I was thinking Redis or Kafka. However, I’d love to build something nice and resilient

4 comments

[ 4.2 ms ] story [ 23.8 ms ] thread
warrenm
Don't have any articles handy (beyond the various cloud providers' offerings which you can find with a quick "s3 to s3 migration" search)

But the egress and ingress fees, added-onto the middleman position your tool would be occupying sound heavy :)

bosch_mind
Big company wallet :)
warrenm
why not aim for something that doesn't hurt the "Big company wallet" :)