Docker Hub Hacked – 190k accounts, GitHub tokens revoked, builds disabled
"On Thursday, April 25th, 2019, we discovered unauthorized access to a single Hub database storing a subset of non-financial user data. Upon discovery, we acted quickly to intervene and secure the site.
We want to update you on what we've learned from our ongoing investigation, including which Hub accounts are impacted, and what actions users should take.
Here is what we’ve learned:
During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as Github and Bitbucket tokens for Docker autobuilds.
Actions to Take:
- We are asking users to change their password on Docker Hub and any other accounts that shared this password.
- For users with autobuilds that may have been impacted, we have revoked GitHub tokens and access keys, and ask that you reconnect to your repositories and check security logs to see if any unexpected actions have taken place.
- You may view security actions on your GitHub or BitBucket accounts to see if any unexpected access has occurred over the past 24 hours -see https://help.github.com/en/articles/reviewing-your-security-log and https://bitbucket.org/blog/new-audit-logs-give-you-the-who-what-when-and-where
- This may affect your ongoing builds from our Automated build service. You may need to unlink and then relink your Github and Bitbucket source provider as described in https://docs.docker.com/docker-hub/builds/link-source/
We are enhancing our overall security processes and reviewing our policies. Additional monitoring tools are now in place.
Our investigation is still ongoing, and we will share more information as it becomes available.
Thank you,
Kent Lamb Director of Docker Support info@docker.com"
269 comments
[ 3.2 ms ] story [ 262 ms ] thread- Change your password on https://hub.docker.com
- Check https://github.com/settings/security
- Reconnect oauth for Automated Builds
- Roll over effected passwords and API keys stored in private repos / containers
Quick take:
- Password hashes
- Github tokens
- Bitbucket tokens
- Your Automated Builds might need new tokens
Checking my github logs - It looks like they've known about this for at least a full 24 hours. Most people aren't going to have this looked at until Monday which kind of sucks. Hopefully there is more of a postmortem coming.
Is anyone from github able to comment on this as well?
There doesn't seem to be a way for us to tell if a repo was read by these keys over that time period.
EDIT: it finally worked, 4th attempt, and very slowly. Looks like something isn't working 100% as it should
EDIT 2: aaaand I can't login now with the new password. A password reset did work, but it looks like their password database is under some stress at the moment.
Specifically to make the password database more secure, the generation of password hashes is very computationally intensive by design (e.g. that's the whole point of something like bcrypt vs. sha1)
Password systems really shouldn't be designed to handle a 10x or 100x load without some slowdown. If they could handle that, it means their password DB probably isn't as hardened as it should be.
https://developer.github.com/apps/differences-between-apps/
https://www.netlify.com/docs/github-permissions/
It seems like Docker Hub is implemented as an OAuth app [2], where these granular options are not available and you have to grant access to all your repositories.
[1] https://developer.github.com/apps/differences-between-apps/
[2] https://docs.docker.com/docker-hub/builds/link-source/
honest question, what's the point of using OAuth when the Authz is so coarse? Why not augment to have scopes per repo? Is it considered bad practice to have have a variable (repo name) as a scope?
I almost gave up on automatic Docker image push on one of my projects [1] earlier this year, when GitHub services were deprecated.
Luckily I found a way to have Travis push the image instead [2]. It involves giving Travis my Docker Hub password (encrypted), but it's certainly better than granting some service full access to my GitHub account.
[1] https://github.com/mpolden/echoip
[2] https://docs.travis-ci.com/user/docker/#pushing-a-docker-ima...
Also not sure what access permissions you need but deploy keys are repo level.
https://developer.github.com/v3/guides/managing-deploy-keys/...
Machine users are another option.
https://developer.github.com/v3/guides/managing-deploy-keys/...
- retrieve a list of all repos to display in the autobuild setup page
- setup webhooks for the gh repo that should be built via dockerhub autobuild
- setup a deploy key for said repo, so that it can be cloned
I removed the dockerhub oauth on github side, after setting up autobuild. My builds on push to master and tag are still working. So it seems possible to remove dockerhubs write access to your github repos after the autobuild setup, which really seems to be a good idea.
> Service user (or machine/bot account) suggested
> Attaching your personal GitHub or Bitbucket account to this Docker Hub organization will allow other organization owners to create builds from your private repositories. We suggest using a service user (also referred to as a machine user or bot account).
c.f.: https://docs.docker.com/docker-cloud/builds/automated-build/...
Seems worthwhile to do this, if you're an enterprise or otherwise have sensitive private repos. But I agree that it would be better to have an easier per-repo authorization system, since many users won't bother going through the hassle of setting up a service account.
> c.f.: https://docs.docker.com/docker-cloud/builds/automated-build/....
Did they remove this language from your link? I don't see it anymore.
I hope this doesn't hurt docker too badly. I really like the hub / auto build service.
With how much of the internet blindly pulls images from it, the potential gain from hijacking just one high-profile one would be monumental.
But no, not Docker.
You're totally right; with as important as their registry is to well funded attackers, and as startup-y and "agile" as they are, and as godawful as the security practices are that underlie their tools and standards... they hadn't a chance. They still don't.
There is no reason to expect them to get better.
[0] https://twitter.com/WHHackersBR/status/1118393568656334850
(that said, google main page vulnerable to xss is kind of like... what, we're afraid someone will take over google and put some cryptominers on the google.com main page?)
6% of the people received a specific email saying the body of their email was accessed and they had to backtrack.
> This unauthorized access could have allowed unauthorized parties to access and/or view information related to your email account (such as your e-mail address, folder names, the subject lines of e-mails, and the names of other e-mail addresses you communicate with), but not the content of any e-mails or attachments, between January 1st 2019 and March 28th 2019.
Notice it says your email account. The whole email is about the account of the recipient, not those of other recipients. Given that they explicitly worded it this way and people clearly misinterpreted it to mean something else, I hope you can forgive me for being a little skeptical of third-party anecdotes that suggest Microsoft claimed nobody's email contents were accessed...
Why wasn’t that the case before?!
There's always a way to enhance your processes, monitor more indicators, etc. or otherwise improve your security.
It wasn't authentication credentials, but still.
> The bigger problem for Google isn’t the crime, but the cover-up. The vulnerability was fixed in March, but Google didn’t come clean until seven months later when The Wall Street Journal got hold of some of the memos discussing the bug. The company seems to know it messed up — why else nuke an entire social network off the map? — but there’s real confusion about exactly what went wrong and when, a confusion that plays into deeper issues in how tech deals with this kind of privacy slip.
(https://www.theverge.com/2018/10/9/17957312/google-plus-vuln...)
as long as you're using software somewhere in the stack that isn't like maturity level 5, AND you don't have constant audits looking for novel attacks on working-as-intended systems, you're pretty much guaranteed to inherit (or create) a vulnerability at some point, and if you're important enough it will get exploited. the reason that doesn't mean we should start modeling computer systems as "living organisms that eventually get old and die" and should keep modeling security like war is that when you get hit, you can respond. all the layers matter, and insofar as Microsoft or Google do it right, they primarily do it right by having a mature process for monitoring, patching, isolating, etc.
as for docker hub though, yeah i'm totally with you. i'm just saying we shouldn't overestimate the preventive capacity of anyone, honestly. if you're doing anything important over the internet at all, you're making some compromises somewhere.
here are 2 links to things i handwaved at above, for example's sake:
https://www.wired.com/story/microsoft-email-hack-outlook-hot...
https://www.forbes.com/sites/kateoflahertyuk/2018/10/09/goog...
As I understand it, there's no element of signing from the actual devs of an image, just from the central trust service of Docker Hub.
There already have been questionable images hosted there ... just by users uploading compromised images. No hacking needed.
Shortcuts all around -- kind of reminds me of MongoDB. Sad it's the primary player...
How is this issue specific to Docker? Anyone can download a random library off github, use a shady linux distribution, or install utility tools loaded with spyware.
I don't think Docker aims to solve issues relating to trusting upstream software. It's a tool to help package applications, just like how tar allows you to package files. What you put in it is up to you.
How is this github any different?
I wonder if docker allows this and on the other hand if that's even feasible for say application images, given that applications must be updated a lot for security reasons. Of course if the Dockerfile's parent reference is not pinned, that does only help to some degree...
docker pull ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2
To be fair, the docker.io/library/* images are signed but no other images are and there are a bunch of issues with how the signing policies work for users that want to enforce that some images must be signed.
Installing known-vulnerable old versions of legitimate software can be just as bad as installing custom malware.
And as I said, only official-library Docker images are signed. All other images are unsigned and even for third-party repos you can't force Docker to verify all images from a given repo (you have to enable it globally, which breaks the utility of a local "docker build").
[+] Arch is the only counterexample I can think of and I'm not even sure if my memory is correct.
If Github was compromised, it would be easy and obvious to insert malicious code in a repository, but hide those changes from anyone on the github website.
Images on Docker Hub don't even need to share their Dockerfile, to talk of all the source/etc that went into their build.
> Which you can avoid by forking the mainline repo and depending on your fork.
If github was compromised, the it would be pretty easy to generate forks with the same compromised code.
Nobody read the source code for this exact reason: “the community is here to read it so I won’t".
For docker (and npm for all that matters) _a lot_ of important dependencies are basically simple one-off "developments" with a single developer and no userbase at all caring for them, because they don't really solve any consistent problem, being basically just created to increase the visibility of its creator on primitive metrics. The community is there for high-level packages, but the dependencies lurk in test-scripts and seldom-used functions carefully placed by some idiotic digital nomads for their personal CV-polishment (ehm, not looking at you: https://github.com/sindresorhus/shebang-regex). Have a look at where this package is used (basically only in cross-spawn, where there are 10 other similar dependencies), then think about, how much effort creating the dependency hierarchy was, then look up who contributed the changes, where this micro-package was required and finally decide whether this was some thing sane people would do or if it's just for personal gain...
In Debian we review and vet packages.
Strawman. Anyone can use Debian Stable or at least Testing.
And there's Docker Notary.
Actually only short names go to docker hub, one can setup their own registry and use it via dns names.
Example: docker pull quay.io/letsencrypt/letsencrypt
One day, there's going to be a colossal compromise, and that might finally change where we place security in the priority chain.
I do wish developers would be a little wiser around these things, especially when they see companies taking such huge amounts of capital. I found it quite depressing to watch the unquestioning way development communities assimilated the docker worldview.
I’m just hoping that I was using a password manager by then.
Any word as to the cause of this? Was something important stored in plaintext, etc.?
This going by the way it's worded "single hub database with a subset of non financial data"
Eh? Doesn’t let you use what without an account?
Anyone can pull images anonymously. An account is only for publishing.
Notice the big "Please Login to Download" button.
The problem is more that they make it harder to find if you don’t log in, which is really not great. But if you don’t want to create an account there is certainly no need to do so.
https://github.com/docker/docker.github.io/issues/6910
Mac: https://download.docker.com/mac/stable/Docker.dmg
Windows: https://download.docker.com/win/stable/Docker%20for%20Window...
NPM is bad because the Javascript ecosystem is fast-moving with loose builds that have thousands of dependencies that are all bundled and run insider consumer's browsers.
It'd be a legit criticism of ruby gems or CPAN, but linux distros are an entirely different kettle of fish, and most of the mainstream distros take security pretty seriously
https://www.theregister.co.uk/2019/01/22/debian_package_mana...
https://whydoesaptnotusehttps.com
[1] https://whydoesaptnotusehttps.com
Now, should apt use TLS by default? Ideally, yes. A secure transport is better than an insecure one regardless of what you're sending through it. But unfortunately it's not as simple as that. Most CDNs charge extra for TLS, and many existing free mirrors of packages don't provide TLS at all. Also, using HTTP allows for proxies to cache packages.
Unfortunately, as we discovered recently, apt had not been distrustful enough of HTTP metadata (which was a pretty big mistake since the entire design of package managers is that they must distrust the transport, especially if it's completely insecure like HTTP).
Not impossible, nothing ever is, but fairly difficult.
(Side note: this obviously wouldn't have prevented the current attack)
If they had write access, then leaked personal data is the least of anyone's worries. The real concern is how close the hackers came to infiltrating the image source for virtually every modern microservices system. If you could put a malicious image in say alpine:latest for even a minute, there's no telling how many compromised images would have been built using the base in that time.
Even then just read access to code often allows enough info for leveraging/escalating privilege.
I'm not 100% sure if Docker hub uses deploy keys for repos it has access to thru the integration, but at least previously there was an option to manually add one to repository if it couldn't access it otherwise.
Their newer GH apps permission model allows fine-grained access to only specific repos (and also only read access e.g.). However their older Oauth flow only allows full access to everything. And 99% of GH integrations still seem to use the older authentication method.
This is also something that many CI providers suffer from. There are only few that already support GH apps.
Definitely wouldn't have helped prevent the compromise.
Why not?
Use proper cryptography and don't roll your own!
I guess if you were really sure you had obtained a checksum prior to the service compromise, then that would give reasonable assurance the image was not tampered with.
Assuming you have fetched a given image and captured its sha in a config file in your version control (e.g. a kubetnetes manifest), then whenever you deploy a container you are sure that you're not affected by exploits happening _after_ you saved the fingerprint.
For example, when Linux Mint's ISOs were briefly backdoored, the attackers also changed the checksum shown on the website: https://www.zdnet.com/article/hacker-hundreds-were-tricked-i...
Nowadays, it's arguably a best-practice when designing a new protocol or storage format to simply make all checksums cryptographically strong unless there's a reason not to. I think that might be where the confusion is coming from.
Using SHA-256 as you describe works well and is widely used by package systems to ensure that source code being built does not change. Signatures can be better for determining if the initial image is valid if the signing is done away from the distribution infrastructure since development machines might be more secure than distribution infrastructure (and if not you will have problems either way). You still need to get the correct public key to start with. However, if you do have a good image by luck or actually secure enough distribution infrastructure then SHA-256 will certainly let you know that you get the same one later. Signatures almost always sign a cryptographic hash of the message and not the message itself.
Regardless, I think it's certainly an excellent hardening step.
They'd really need to be signatures attached to the images, not just hashes.
And even if you hosted your own distribution and notary (like we now do for openSUSE and SUSE), you can't force Docker to check the signatures of all images from that server!
Only docker.io/library/* has enforced image signing and the only other option is to globally enforce image signing which means "docker build" will result in unusable images out-of-the-box.
If you look at something like OBS (the Open Build Service that openSUSE and SUSE use to provide packages as well as host user repos), the signing story is far better (and OBS was written a very long time ago). All packages are signed without exception, and each project gets it's own key that is managed and rotated by OBS. zypper will scream very loudly if you try to install a package that is unsigned or the key for a repo changes without the corresponding rollover setup. And keys are associated with projects so a valid rpm from a different project will also produce a warning and resolution prompt. That's how the Docker Hub should've been designed from the start.
(Disclaimer: I work for SUSE, on containers funnily enough.)
See this issue: https://github.com/docker/hub-feedback/issues/873
IMO, dockerhub should publish:
* exact time when attack happened, so we know exactly what builds might have pulled malicious images
* degree of access the hackers gained (whether it had ro or rw access, and to what exactly)
* mechanisms used for password/tokens/pii/etc. security (salt, encryption, hashing alg...)
* list of affected public accounts/images
Instead of last point, or complementing it, it would be useful if image providers would confirm on their official channels if their images were or weren't compromised, and which tags&shas are (un)safe to use.
EDIT: https://success.docker.com/article/docker-hub-user-notificat... is now updated with additional information:
Q: What happened?
There was a brief period of unauthorized access to a Docker Hub database. During this time some sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of users as well as GitHub and Bitbucket tokens for Docker autobuilds. All these tokens have been revoked.
Q: Were any of the Docker Official Images impacted by this incident?
No Official Images have been compromised. We have additional security measures in place for our Official Images including GPG signatures on git commits as well as Notary signing to ensure the integrity of each image.
And the main usecase is k8s. So docker is just an implementation detail its relevancy is waning imo
Some organizations took the risk of running docker taking images directly from docker hub. They were relaying the security of the images to them.
Some organizations are going to panic now and host their own registry. Which they need to protect as well. But in general it will create a better decentralized ecosystem.
I think this is good for the docker community in general.
Nevertheless: The base images will be pulled from dockerhub by default and I am not sure we should trust them. Do we have any better alternatives for this?
By replicating the images (or packages) that you need into your own account, you can minimise the possibility of a bad actor replacing a well-known image with something untrusted.
An alternative is to side-cart a service like Notary (https://docs.docker.com/notary/getting_started/) in order to establish a chain of trust for images. If an image gets changed, Docker will refuse to use it and you will be warned that it is untrusted.
Biased opinion on an alternative registry:
- Cloudsmith: https://cloudsmith.io/l/docker-registry/
But you've got other options, such as:
- Self-hosted: https://github.com/docker/distribution)
- Cloud-specific (e.g. ECR, GCR, ACR, etc.)
- Sonatype Nexus: https://www.sonatype.com
- ProGet: https://inedo.com/proget
- Gitlab: https://gitlab.com
- Artifactory: https://jfrog.com/artifactory/
If you're missing the auto-build functionality, this can be achieved reasonably easily with any of the mainstream and awesome CI/CD services out there, such as:
- SemaphoreCI: https://semaphoreci.com/
- CircleCI: https://circleci.com/
- DroneCI: https://drone.io/
Disclaimer: I work for Cloudsmith, and still think Docker Hub is great. :-)
Back to the images bit first:
Base images are only referenced/pulled at build time. So if you've already built your own image and stored it, it'll contain all of the layers necessary to run it without explicitly pulling from Docker Hub.
In the case that you're building new images (likely), it'll need to pull the base images from Docker Hub. However, if you pull the base image(s) from Docker Hub first, you can tag them and store them in your local (or hosted) registry, then refer to those explicitly instead.
For example (using a Cloudsmith hosted registry):
Now, instead of the usual FROM directive: You can refer to your own copy of alpine: As you can see Docker's syntax doesn't make this extremely pleasant, and you'll have to change existing Dockerfiles to point at the base images, but it's certainly possible to mirror your dependencies without rebuilding.Caveat: The downside is that you have to trust those dependencies at the exact point you pull them down, so I concede it is still not perfect without rebuilding the lot. :-)
> Data includes usernames and hashed passwords
How are they hashed? And specifically, can we expect them to be already cracked?
It may also explain some suspicious behaviour / source of compromise in the past (we know when the issue was uncovered, not when the first dump was taken)
> I imagine it is a one way hash
All hashes are one way. If it's lossless and can be reverted, it's a compression algorithm or isomorphism or encryption or cipher or any of a number of other things, but not a hash.
> I don’t even know how it can get decrypted.
It is not decrypted, but brute forced. For example, even if you can't algorithmically figure out what the input to md5sum is that gives you '1f3870be274f6c49b3e31a0c6728957f', you could apply md5sum to every word in the dictionary in a matter of seconds and find out that 'apple', when md5summed, has that output. You would then have one possible password for that hash (though technically there are infinitely many inputs that have that output).
The only way we can know how computationally difficult it is to brute force the password hashes is if we know the following: 1) the hash algorithm used (and other inputs like cost factor) and 2) the entropy of the salt used. Those two together lets us calculate the amount of computation needed to try one brute force "guess". Individual password's difficulty to be brute forced can then be calculated from their entropy (e.g. 'apple' has less entroy than '2SEZb'), to determine the average number of inputs needed to be tried, multipled by the cost of each attempt. Given that difficulty, you can then estimate how long an attacker will take to find your password by estimating how much computational power they have at their disposal.
In general, if you randomly generate 10+ character passwords and docker used best practices, the answer is that any attacker will not get your password in under a thousand years, and if you use a password which has been leaked before or is a dictionary word (or simple variation), it can be found on the order of minutes to days.
The most famous application of rainbow tables is one of Microsoft's family of terrible password hashes LM. But the reason it breaks that wide open is not just the lack of salt, it's also that LM hash only works for 7 character passwords, and up to 14 chars are supported by doing two entirely separate hashes - so you can craft rainbow tables for all possible 7 character inputs and then reverse the hash.
Also, why rely on users to change their passwords? Is there a security log I can check?
"Sorry, it's not you. It's us, but we are working on it!"