Additional correction: S3 is consistent after new object puts only if you haven't already asked for that object. Otherwise you may get a stale "no object" response.
But I still can't set a bucket-wide policy for caching to be applied automatically to new files. So in other ways, consistency is inconveniently eventual.
This is pretty easy to do with a cloud function. I was a little apprehensive at first, but have been using it across many buckets and haven't seen any issues. Here's a version of the script I'm using:
https://gist.github.com/justinph/acb6301d63b407e1fb205cbb42e...
Google recently announced some changes with cloud storage and pubsub triggers that potentially means that you could only have this trigger when a file is newly created, not every change. I haven't looked into that yet.
The code I've picked up from a predecessor uses a Concourse task to achieve a similar end. But it's just annoying that I can't set a policy on the bucket.
As an aside, it looks as though Google's golang API for GCS has a way to set the policy when you provide the file, so maybe at some point I can close this issue: https://github.com/concourse/semver-resource/issues/61
While there is not an option to set a default cache policy on a bucket, you can specify a cache policy as part of object uploads, although I admit that this can be inconvenient if you upload from multiple clients.
In addition, caching only applies to objects that are anonymously readable. Private objects are never cached.
14 comments
[ 2.9 ms ] story [ 43.7 ms ] threadThese are connected because it means that it's tough to prototype on something cheaper and then switch over.
If you want strong consistency for S3 consider something like JuiceFS: https://JuiceFS.io
It wraps your S3 in a strongly consistent and fast POSIX API (aka regular mountable filesystem) and its way less expensive than EFS.
Correction: S3 is strongly consistent after new object puts, but not lists, modifications, and deletes.
Definitely not as convenient but you can get strong consistency if you only rely on new objects.
That being said, I am curious as to how available/consistent S3 is versus Google cloud storage.
Google recently announced some changes with cloud storage and pubsub triggers that potentially means that you could only have this trigger when a file is newly created, not every change. I haven't looked into that yet.
As an aside, it looks as though Google's golang API for GCS has a way to set the policy when you provide the file, so maybe at some point I can close this issue: https://github.com/concourse/semver-resource/issues/61
In addition, caching only applies to objects that are anonymously readable. Private objects are never cached.
A great example is Google needed a determinate network so standard TCP/IP would not cut it. But then other things in the infrastructure was needed.
This is the ultimately full stack solution.
https://research.google.com/archive/spanner.html