Ask HN: Why does S3 still not support append?

Google Cloud Storage does not support append. Their docs: "... you cannot make incremental changes to objects, such as append operations or truncate operations."

I don't think it has a traditional filesystem. It probably just writes all puts sequentially as fast as possible and stores the location and then replicates. The easiest way to append would be to read the object, append, and then write to a new object. If they did that internally there would be no transfer out and no revenue although they could probably charge for the internal expense. Another reason is that people would probably think that appends are no big deal and try to append continuously to multi-gigabyte files. If this is the case then it is best to let the client handle appends where costs are out in the open.

    aws s3 combine --target s3://bucket-name/output-file.txt \
      s3://bucket-name/input-file-1.txt \
      ... \
      s3://bucket-name/input-file-n.txt

  1. Start a multipart object upload
  2. Issue "Upload Part - Copy" requests for each part of an object ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html )
  3. Complete the multipart object upload

  1. Enable bucket versioning
  2. Download the parsed S3 objects
  3. Start a multipart object upload to S3 with the specified target object as the object name
  4. Reupload the parsed S3 objects as parts of a single multipart object upload
  5. Delete the previous parsed objects once the multipart object upload is complete (a delete marker should be added to the top of the version stack, but the previously stored version should still be available if you specify its handle/version id).

I mostly miss a MoveObject operation to rename files myself, but I guess they are keeping things simple and scalable etc. on their end and requiring us to work around it with the existing lower level operations.

FWIW, Azure supports Append Blobs: http://blogs.msdn.com/b/windowsazurestorage/archive/2015/04/...

Google Cloud Storage does not support append.

One reason these things don't support append is at some point they need to choose the "version" of the object. Usually this is done when an object upload has completed.

If they allow arbitrary appends to objects, then they would have a hard time assigning any type of ordering to them, as the concept of an object being "complete" would be thrown out the window.

(EDIT: and what does it mean to have a GET on an object, if you don't know the latest version to return?)

I think something like this could be implemented, but it would probably be an entirely different product that supported some specific traditional file operations (rename, ftruncate, link, etc) but had different scaling properties.

You mean appending things on the end of files or what? If so, probably because its trivial to work around by storing the data in new files - and large data where this would be valuable should be broken down into pieces for n different reasons anyway.

Why do people who ask questions fall slightly short of providing enough information to meaningfully answer them?

Because reconciling 2 separate appends to 2 separate nodes which have different copies of the data would be a huge mess.

S3 is a key/value store. Appends don't make sense in that context. If you think of it as a key/value store, then a lot of their constraints start to make more sense.

S3 is eventually consistent, appending an eventually consistent file is going to get very messy, very fast - what happens when an append reaches a replica node before an earlier one does?

If you're happy with out of order appends, just use a container file format like Parquet where appends are actually additional file creations

After a decently large RAID failure, I needed to gzip and send as many large files and send it over to S3 as quickly as possible on the risk of another failure. The script would gzip the file and then sync it up to s3, all in its own backgrounded processes. If two large files would get sent at the same time, both would die, then /leave incomplete files/.

After leaving that running over night, all of the files appeared to be uploaded... until the owner of the company needed to use them.

I'm still not sure if that's an exceptional use case, but it left a pretty bad taste in my mouth about S3 ever since.

Well answered by other comments here, the whole "eventually consistent blob" vs "directed graph of mutation operations" problem. FWIW, this is a good distributed systems interview question :-)

S3 is more of a simple key-value store than a full filesystem (and for good reason). I suspect the reason their docs push the filesystem metaphor so much is because filesystems are more familiar to many people, and most filesystem semantics can be implemented using a key-value store. In that sense, there is no update() or append() in S3, just a simple set().