S3 (and similar storages) have caused plenty of security issues in several occasions (usually because of misconfigured buckets, making all contents available to the public). Given this, it would be expected that companies would pay a bit more attention to the security of these data storage methods.
However, in Digital Ocean, by design, you can't restrict keys to certain buckets. Once you issue a key, it can access all buckets within your project, with all operations (list, read, delete) files.
The issue has been reported and is known for a long time. It's even the top voted "idea" in the company portal.
Posting this here in hopes to bring awareness of this issue.
Things like these are why I stopped using digitalocean.
They had issues with DNS PTR records too for years. Same for initial auth tokens being global. I had hopes for them and used them exclusively at 1 time.
My read on this situation is that DO is holding out until they have a comprehensive IAM solution. If you need IAM, don’t use DO. If you can live without IAM, DO is pretty nice.
I think DO is doing fine addressing their target market. I also wish they had IAM but you have to admit IAM is huge and messy and is often more about managing your employees than your infrastructure. I’m okay waiting for them to get it right rather than rushing out a quick half baked idea that ruins their entire product stack err platform.
I don't care for the full on flexibility of IAM, but I'd kill for a simple "this key is for this bucket" feature. That doesn't seem so far fetched, I don't need per-object permissions within a single bucket!
As a user, you can fix this by having a proxy that fits in front and manages ACL's.
Obviously you have to pay for resources for that proxy, and it's probably going to want a large bandwidth allocation. Lucky because DO doesn't charge for bandwidth.
If I have to implement and manage an ACL myself (such a basic feature!!), it's better to go ahead and implement something like min.io and ditch DO entirely.
This is mostly true, but there is a case where you use your transfer pool.
From their product docs:
Droplets have their own transfer allowance, independent of Spaces. Traffic from Droplets to Spaces does not count against your Spaces transfer allowance (because inbound bandwidth to Spaces is free), but does currently count against your Droplets’ outbound transfer allowance.
I'm afraid you broke the site guidelines by using the submission title to editorialize. Please see https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait; don't editorialize."
If you want to say what you think is important about a page, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so.... Alternatively, you could have made a text post to communicate what you think is important about the issue, and then your title would have been fine.
What's not ok on HN is to use the title field as a way to comment on an issue, because then the submitter gets a privileged position relative to other commenters. Since the title is by far the largest influence on a thread, we want to avoid that. Being the submitter of a page shouldn't confer any extra special power over the discussion.
I haven't used DO Spaces for a good while as I've shifted to B2 as cheaper S3-compatible storage that suits my archival workload.
However I absolutely would be using Spaces as storage alongside any apps running on DO, and would have assumed that all the usual per-bucket permissions I'm used to elsewhere were present, so thanks for the heads up!
DO has had a terrible attitude to security for as long as they've been around. I reported a major data leak vulnerability to them and they told me it was operating as intended, so I published it, and then they accused me of irresponsible disclosure, while simultaneously claiming that there was no security issue. This was a dozen years ago and the founders were involved in the response.
They are at the top of the list of companies with which I will never do business.
On delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning.
I wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.
The fact that you think it should be the customers’ responsibility to make sure that DO doesn’t accidentally give your private data away to unknown third parties is an absolutely minblowing take on this matter.
I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data.
If anyone can suggest any alternatives that are not Hetzner, I would be interested.
Well first, I've not worked for DigitalOcean in a long time, I left in 2015. I don't really wanna re-hash something from 2013, but again: when you deleted a VM you could click the scrub button or pass the scrub flag in the API when you issued a delete call. It was very well documented and the scrub button wasn't hidden anywhere in the interface. The fog platform wasn't using the scrub flag in their app for whatever reason. As I recall, we did change it to scrub by default shortly after. From what I remember, the reason it wasn't originally default was the zero'ing process tried up a lot of resource on the box and it was hard on the drives and we were very much in startup mode at that time with limited capacity, so it was better to only zero if the customer requests it.
You still don't appear to get it though. The fact that there even exists an option (never mind a default) which does not scrub storage between use by different tenants is completely unacceptable in a multi-tenant environment.
Imagine if your bank told you they only close the vault when a customer asks.
And they give you an excuse that it’s time consuming for them and you should respect their time.
Would you bank with them?
This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center.
It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)
I was always very clear at that time "this is a startup" - "do not use DO for mission critical work" - "the product is not very mature yet" - we had no sales team, nothing in our marketing was gear towards business and we primarily catered to students/personal sites/etc. I'd accept your analogy except we regularly said "use AWS for important stuff, not us".
>the controller must implement appropriate technical and organizational measures to protect personal data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure or access, in particular where the processing involves the transmission of data over a network, and against all other unlawful forms of processing.
Do you know what else ties up a lot of resources? Civil action, negligence claims, PR disasters.
This specific GitHub issue is one of several why I will never use DO: language, official response, and attitude all matter. I'd be curious to know how much revenue DO has lost because of this one issue.
There is precisely zero circumstance where it is ok to give private customer data to another customer.
The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it.
The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.
Unless I misunderstood something and one has to explicitly opt out of scrub for this to occur, I agree 100%. This is surprising. And they were mad that a customer thinks this is undesirable?
I was using DO. It is not a serious project but I think I'll be moving on nonetheless.
Yeah, I read a bit further and it was not scrub by default but had to be opt-in. And today is the first I heard that all API keys have ridiculous access. I'm done.
Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.
Why on Earth would someone have to opt-in to scrubbing with a flag vs. opting out? That's surprising. Is the flag on by default and they specifically opted out before complaining about it?
> there isn't actually a security issue, the fog app wasn't passing the scrub flag
At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.
You're right. I hadn't really thought about it that way. My response at the time actually came from confusion on my part (that is my own fault I suppose). I got woken up at 4am on a Sunday by a flood of calls saying there is a post on the top of HN about DO leaking data between VMs. I got out of bed to see if I needed to wake up the engineering team, checked the support system and the disclosure emails because my first thought was shit someone has pwned the LXC/escaped the jails or something. Then when I read the github, I thought the scrub flag wasn't being respected when it was passed. That was the type of security issue I had originally taken it to mean, both I would have woken up engineering for. It was a someone is in the hypervisor type of security issue I thought we were dealing with. Once I realized it was fog didn't pass the scrub flag, I wish I'd just gone back to bed and dealt with it on Monday instead of trying to do it in real time, that was stupid of me. I also accept your point that an unexpected behaviour can in itself be a security issue.
While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem.
> 10 years later and I still remember how pissed off I was that day, hah.
I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.
Consider the multiple S3 leaks that caused AWS to change their defaults to block public access and disable ACLs. You could say that since buckets were always default private, any misconfiguration was simply user error and users complaining were doing so in bad faith. Or you could understand the implications and help your customers protect their data.
Reminds me of a hosting company in my country. When you use their spaces thing, they generate a password for you (not being able to set your own is already a red flag for me, but ok). It was "hidden" in the UI to look secure. However, the password itself was passed as an argument that was part of the URL. So everyone who logs access urls could see the password.
I reported it, they fixed it quickly, but never acknowledged that it was a "security" issue.
A flaw was found in Ceph RGW. An unprivileged user can write to any bucket(s) accessible by a given key if a POST's form-data contains a key called 'bucket' with a value matching the name of the bucket used to sign the request.
The result of this is that a user could actually upload to any bucket accessible by the specified access key as long as the bucket in the POST policy matches the bucket in said POST form part.
We have assigned it a CVE of CVE-2023-43040 and the patch is attached.
Nope, this post is from 2021 pretty much begging that DO provide per-bucket level access keys for Spaces. Otherwise, right now, all keys created for an account can access all buckets on that account.
This is correct, Ceph's RadosGW doesn't support this feature.
For people trying to figure out if their provider is using Ceph, make a request for https://<s3-endpoint>/test and see if the x-amz-request-id header begins with "tx000", if it does, then they're using Ceph.
The point is that there are no bucket keys. Only Access keys for all buckets. There is no way to issue bucket specific keys. I'm not sure I follow how the title could be misleading.
The lack of ACLs or comparable permissions is by far the biggest thing that prevents me from recommending DO for production workloads. This kind of thing is absolutely essential. You can't even separate dev resources from prod resources, every API key has godmode on your whole account. This is a security disaster.
For a simple example, I'm running externaldns on a kubernetes cluster. For production use, I'd want to at least have an API key that can only access the DNS domains and nothing else. As is, the key can create compute resources or delete anything (including object storage buckets!). Again with no dev/prod isolation, so a leaked dev key that looks innocuous means your prod resources are now compromised.
Some second tier providers do better at this, OVH and scaleway both have some concept of ACLs, but DO just don't even try.
Apologies for the slight rant but it's frustrating to see something so basic overlooked by a company the size of DO.
I agree, probably the biggest point Digitalocean is lacking. I think the only workaround is creating several projects? But that probably becomes annoying and hard to manage quite easily if you want a lot of separation.
IIRC projects only group resources under the same account. API keys are still account-wide, so projects provide no isolation there.
There is a other mechanism DO has to allow multiple accounts under one billing context (I think it might be called 'teams'?) but sadly this is still extremely coarse grained, and doesn't allow you to lock a key down to a particular resource, or even a class of resources.
Agreed. We had solved it at PhishDeck back in the day through the use of multiple projects, i.e., Platform Production, Platform Staging, Tooling Production, Tooling Staging etc.
Still not ideal, but it minimised security risk and kept things organised.
Wait till you hear that when you generate a DO container registry key it includes a key that gives you access to all of the resources, not only the registry.
It's not about the API key. It's about the container registry credentials. Which you would expect to be able to only interact with the container registry. This is not the case. If you go to the container registry and click to download credentials you will get JSON like:
If you decode the base64 encoded credentials it will be a string like "<token>:<token>" with either a read only or read/write token to all of the resources on DO.
> It's not about the API key. It's about the container registry credentials. Which you would expect to be able to only interact with the container registry.
think my reply shortness maybe has you misunderstanding me?
The fact that an API token can only have read/write project wide basically results in everything you said.
No RBAC on services. No RBAC on specific API actions. Anyone with read+write can do/nuke everything.
That’s why I’m migrating $COMPANY off to AWS as fast as I humanly can by myself.
Ok, I see what you mean. Still, even knowing that general access tokens are all or nothing it was surprising that exactly the same thing was used for containers. Like, if you generate a separate credentials for containers it strongly suggests there are for containers only.
Yes. And when you generate the container registry token it doesn't mention any of it and it's hard to notice, cause docker auth is base64 encoded. But after you decode the credentials you get a regular DO token that you can use with the API
I don't know if they've improved it, but 5 years ago I basically couldn't delete a DO Space because they had no way of doing a mass deletion. I even got technical support who basically told me I was doing it the best way possible. My delete operation took several months to run.
That is a limitation that comes along with having to be S3 compatible; the best you can do is to combine ListObjectsV2[1] + DeleteObjects[2] (1000 objects in a single API call).
While S3 doesn’t have a REST API to do it, you can trigger it through the console and it applies pretty quickly if you don’t have a lot of objects (hundreds of thousands if I recall correctly) and will apply in the background otherwise. The way to do it via the s3 API is to install a lifecycle policy to delete everything.
Lifecycle requests are a good call, but being an asynchronous process, it may take some time to be processed, even when using S3. On a particularly large bucket, I've observed it taking over a week in one instance.
The console isn't magic however, it makes the same ListObjectsV2 + DeleteObjects calls in parallel.
I used to use DO Spaces and Backblaze, but then I found Wasabi and have switched everything over since. It pretty much has full S3 compatibility, and doesn’t use Minio or Ceph. Mostly just keep adding data, very rarely delete data.
I unfortunately had to ditch DigitalOcean because of a number of small problems like this (not taking permissions seriously is a symptom of a larger problem). Other issues included how you can't assign a floating IP to a load balancer, the managed kubernetes is embarassing (to resize nodes, you must destroy and recreate the whole cluster), paltry volumes IOPS, minimal support for DNS stuff, and more.
Aside from this, their Spaces service has had multiple outages, the service seems to be pretty unreliable in general (definitely compared to S3). A quick glance of their NYC3 status suggests they may have finally improved this recently, but I don't know what use case I would have for a version of S3 that goes down multiple times a year.
Indeed - I used to use Spaces for a project but found not just the uptime but teh overall reliability and performance was, at the time at least, markedly worse than AWS.
DigitalOcean should not be used for production use, or the sole network.
There is a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.
Another reason DigitalOcean should not be used for production use, or the sole network.
I am looking into blackblaze and other providers.
As seriously, Digital Ocean contains a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.
103 comments
[ 3.0 ms ] story [ 391 ms ] threadHowever, in Digital Ocean, by design, you can't restrict keys to certain buckets. Once you issue a key, it can access all buckets within your project, with all operations (list, read, delete) files.
The issue has been reported and is known for a long time. It's even the top voted "idea" in the company portal.
Posting this here in hopes to bring awareness of this issue.
They had issues with DNS PTR records too for years. Same for initial auth tokens being global. I had hopes for them and used them exclusively at 1 time.
Obviously you have to pay for resources for that proxy, and it's probably going to want a large bandwidth allocation. Lucky because DO doesn't charge for bandwidth.
From their product docs:
Droplets have their own transfer allowance, independent of Spaces. Traffic from Droplets to Spaces does not count against your Spaces transfer allowance (because inbound bandwidth to Spaces is free), but does currently count against your Droplets’ outbound transfer allowance.
https://docs.digitalocean.com/products/spaces/details/pricin...
If you want to say what you think is important about a page, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so.... Alternatively, you could have made a text post to communicate what you think is important about the issue, and then your title would have been fine.
What's not ok on HN is to use the title field as a way to comment on an issue, because then the submitter gets a privileged position relative to other commenters. Since the title is by far the largest influence on a thread, we want to avoid that. Being the submitter of a page shouldn't confer any extra special power over the discussion.
However I absolutely would be using Spaces as storage alongside any apps running on DO, and would have assumed that all the usual per-bucket permissions I'm used to elsewhere were present, so thanks for the heads up!
They are at the top of the list of companies with which I will never do business.
https://github.com/fog/fog/issues/2525
https://news.ycombinator.com/item?id=6983097
https://news.ycombinator.com/item?id=6983520
https://github.com/fog/fog/issues/2525#issuecomment-31336855
I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data.
If anyone can suggest any alternatives that are not Hetzner, I would be interested.
And they give you an excuse that it’s time consuming for them and you should respect their time.
Would you bank with them?
This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center.
It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)
Before the GDPR the EU had the 1995 Data Protection Directive, which you would have to comply with to have EU customers.
You can’t just decide to not follow the law because you’re a startup.
In the event your google is broken:
----
https://en.wikipedia.org/wiki/Data_Protection_Directive
>the controller must implement appropriate technical and organizational measures to protect personal data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure or access, in particular where the processing involves the transmission of data over a network, and against all other unlawful forms of processing.
Should I explain this too?
Do you know what else ties up a lot of resources? Civil action, negligence claims, PR disasters.
This specific GitHub issue is one of several why I will never use DO: language, official response, and attitude all matter. I'd be curious to know how much revenue DO has lost because of this one issue.
There is precisely zero circumstance where it is ok to give private customer data to another customer.
The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it.
The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.
I was using DO. It is not a serious project but I think I'll be moving on nonetheless.
Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.
At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.
The official blog from DigitalOcean is here:
https://www.digitalocean.com/blog/transparency-regarding-dat...
While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem.
> 10 years later and I still remember how pissed off I was that day, hah.
I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.
This issue is the reason why I will never use DO or recommend it to a customer.
I wonder how one could test for that “automatically” on other providers.
I reported it, they fixed it quickly, but never acknowledged that it was a "security" issue.
https://news.ycombinator.com/item?id=22490390
Tailscale uses it as their main host for the repos, snd its down right annoying with no CDN and slowdowns occuring often.
https://www.openwall.com/lists/oss-security/2023/09/26/10
---
A flaw was found in Ceph RGW. An unprivileged user can write to any bucket(s) accessible by a given key if a POST's form-data contains a key called 'bucket' with a value matching the name of the bucket used to sign the request.
The result of this is that a user could actually upload to any bucket accessible by the specified access key as long as the bucket in the POST policy matches the bucket in said POST form part.
We have assigned it a CVE of CVE-2023-43040 and the patch is attached.
Credits to Lucas Henry of Digital Ocean.
For people trying to figure out if their provider is using Ceph, make a request for https://<s3-endpoint>/test and see if the x-amz-request-id header begins with "tx000", if it does, then they're using Ceph.
$ curl -sSi https://nyc3.digitaloceanspaces.com/test/ | grep x-amz-request-id
x-amz-request-id: tx00000947bc21a401c0f02-00651981d1-4b6a0-nyc3d
Access request needs to be created, validated, logged, processed, expired or revoked.
Or just validated, processed, and logged.
So the title is ambiguous, it could mean that keys have to be unique across buckets since they share the same namespace.
I honestly find it hard to understand why this kind of basic security feature wasn't in GA.
[1]: https://community.cloudflare.com/t/r2-token-per-bucket/38906...
For a simple example, I'm running externaldns on a kubernetes cluster. For production use, I'd want to at least have an API key that can only access the DNS domains and nothing else. As is, the key can create compute resources or delete anything (including object storage buckets!). Again with no dev/prod isolation, so a leaked dev key that looks innocuous means your prod resources are now compromised.
Some second tier providers do better at this, OVH and scaleway both have some concept of ACLs, but DO just don't even try.
Apologies for the slight rant but it's frustrating to see something so basic overlooked by a company the size of DO.
There is a other mechanism DO has to allow multiple accounts under one billing context (I think it might be called 'teams'?) but sadly this is still extremely coarse grained, and doesn't allow you to lock a key down to a particular resource, or even a class of resources.
Still not ideal, but it minimised security risk and kept things organised.
Whilst AWS is more expensive and perhaps over-engineered for our use-case, it was absolutely essential to have some kind of ACLs.
I mean, I can't exactly give my junior dev access to DO if it means he can also delete the entire production database and all backups with a misclick.
There is basically no granularity.
think my reply shortness maybe has you misunderstanding me?
The fact that an API token can only have read/write project wide basically results in everything you said.
No RBAC on services. No RBAC on specific API actions. Anyone with read+write can do/nuke everything.
That’s why I’m migrating $COMPANY off to AWS as fast as I humanly can by myself.
> you can simulate what `doctl registry login` does by using an API token string as the username and password when calling `docker login`
https://docs.digitalocean.com/products/container-registry/ho...
[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
[2] https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteOb...
The console isn't magic however, it makes the same ListObjectsV2 + DeleteObjects calls in parallel.
There is a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.
I am looking into blackblaze and other providers.
As seriously, Digital Ocean contains a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.