And this would be a cool feature from github too. A link mentioning "we found something in your code that looks like a secret, please know people will use it."
They do this for all of their own API keys already. They not only notify you but instantly invalidate a key pushed to a public repo.
Annoyingly there is no way to turn it off even when you explicitly want to share an API key knowingly. But i'm more than fine with needing to "obfuscate" an API key or manage secrets correctly knowing it saves TONS of people.
It's been a while, but IIRC it was a key with no permissions used on a CI server to get around github's API usage limits.
It probably wasn't the best idea, but it was the only "secret" needed in the whole project and I didn't want to maintain a way of managing secrets in a public project for a pointless key.
In the end I did just that, and looking back it was the better choice, but at the time it was annoying.
At my work one of my coworkers accidentally put a secret token in a GitHub issue. Couple hours later he got an email from the sysadmin at the parent company saying his token finding script went off. He probably wouldn't have noticed for a long while if that script wasn't running.
2. It's embedding the token as query-string parameter in a GET request, which is much more likely to be logged (than sending it as data in a POST request), and more likely to be available to less-privileged/less-trusted staff.
3. If the request is sent to a non-HTTPS endpoint, the query can be MITMd, revealing the token.
I'd be very wary of setting up a token-finding script, it feels like it adds more risk than it saves.
You can scrape the issues without exposing the token. You could probably do it by just subscribing to all of them and parsing emails. No one(especially in security) should be using a third party search to match sensitive data. It's like searching Google for your social security number.
Maybe they search the tokens public key and not the token itself. Then if the public key is found, then they download the repo and do scanning for the private key.
A group at BigCorp Inc. was sharing a tool they'd written to ThirdParty Ltd. As part of this, they transferred documentation, including how to configure the tool. Including an example. With a real AWS key. For a dev too, so the key had no restrictions.
I once pushed my Amazon S3 key to GitHub accidentally. Realized instantly what I'd done, and while in the process of feverishly regenerating a new key, my cell phone rings. It's Amazon telling me I pushed my S3 key to GH.
It's cheaper for them to give a few engineers a web crawler project that's this specific than it is to refund people. Im just surprised they don't have an "auto revoke access key if found on interwebz" setting in the AWS account settings actually.
- a key is made public, and we have to call a user or refund them (for retention purposes)
- a key is made public, and we revoked the key, potentially breaking the customers builds/deploys and potentially knocking a customers stuff out (if, for example, a key is disabled during a push to production).
Github provides a public firehose for events[0]. So it's possible to hook a process to read from the firehose, and look for commit events and then match file contents against the list of API keys.
The interesting thing is that there is also an evil crawler that will automatically launch thousands of windows vms to mine bitcoins (that's all they do). Amazon told us that we have leaked our account id and secret but also they notice the other crawler has launched a lot of VMs and they did a refund to us. yes, we love amazon.
Lesson learned: you never put the account id and secret in your code, not only that you should not hardcode it, but there is no need to even read that from the environment etc.
Don't do something like this `new S3({accountKey: ..., accountSecret: ..}` instead you do `new S3()` and that's it. Every AWS SDKS is smart enough to find the keys in the environment following a series of steps:
- environment variables
- ~/.aws/credentials
- and when your code is run on ec2, lambda, etc. you should use IAM Roles.
So, in addition to not hardcoding an AWS secret, your code should not even pass the secret to the SDK.
Consider also enabling CloudTrail and have alerts on that.
There is also a way to not have ~/.aws/credentials in your machine and have another thing that requires MFA. I am not familiar how this work yet but we started to use it.
MailChimp has something like this. A few years ago I accidentally committed and pushed an API key, and I got an email from them a few minutes later saying that they had found the key and already invalidated it, so it couldn't cause any damage. Very proactive and smart, especially for an email service which is likely a huge target for abuse around this sort of thing.
I'm embarrassed to say that it's happened to me. I deliberately created an IAM user that was 'public', for purposes of the application. Amazon shut down that user and everything (not much) that it had access to. This was in 2014 or 2015.
I should be amazed at how prevalent this is but after almost two decades in IT/IS, it's no more than the equivalent to the Post-IT on a monitor, but more accessible. Dumb, but business as usual.
Writing passwords down on a piece of paper, and keeping that in your wallet or locked desk drawer is actually one of the more secure ways of storing passwords these days.
No risk of electronic compromise, and its highly unlikely that people who would steal your wallet or break into your home are also interested in your online accounts.
To be honest, and not that I ever would, but, if I stole your laptop, the moment I would become interested in your online accounts would be the moment I found your paper with all of your passwords on it.
2.You work for a large corporation and we'll you can't say you can trust everyone there.
3. Small company of 10 (company I work at for example) could be compromised by the weekly janitor.
4. Someone could break in and make it look like a robbery all while stealing your critical infrastructure.
I recently helped research a bit about internal security for our office and sticky notes are still a very common place for credentials to be compromised.
I notice in many instances Github showing an error.
We could not perform this search: Must include at least one user, organization, or repository
But if you change anything in the URL, it works again. Such as adding &p=2, which lucideer tried. But I got the error on his link. So I changed it to p=3, and it worked.
So I'm guessing Github has an autodetection for a particular global code search getting high hits (for something like this, I assume) that locks people out.
Seems more like a band-aid on a broken leg, though.
The worst part in that is that it provides tons of passwords to analyze and detect recurring words or schemes. This probably also will hurt people that never commited their passwords in public repos. Github should probably filter out such searches.
Yeah, indeed. I guess we can expect password schemes to change over time, so it's still a good idea to prevent it. Not sure in which proportion it helps, though.
git is a great tool, unfortunately there's no standard of storing sensitive info (like passwords).
some store it as ENV variables, some hard copy the files to servers using custom scripts etc..
would love to see an easy tool that developers can use to manage this info like people manage files on git.
This is a good example of the increased risks from doing your development out in the open, any mistakes are exposed to a much wider group of potential adversaries.
On an internal VCS, this would still be a problem, but a bit less visible/exploitable...
On an internal VCS this may be a deliberate decision: Secrets need to be stored somewhere and a cost-risk analysis can result in "this is the best place that we currently have at our disposal". That obviously won't fly if your threat model includes "adversary may attack our github account from within GH" or if you ever plan on opening up that repo, but if neither applies this may be the best place to store some sorts of secrets.
I've gone through the process of open-sourcing previously closed codebases, and in virtually all of them a decision is made to make a single "genesis" commit to start the public exposure because there's just not enough manpower (or I don't know git well enough) to go through and ensure there not only aren't any secrets now (meaning passwords, or info the company doesn't want to release), but also there weren't at any point in the past.
Sure there's always a cost / benefit balance to take into account.
That said I'd say putting secrets in a git repo is a pretty risky thing to do. By the nature of the tool that means that the secret ends up on the device(s) of every developer who checks out the codebase, so the security of the secrets is equal to the security of the worst secured device in question.
> That said I'd say putting secrets in a git repo is a pretty risky thing to do
Storing and keeping secrets is a pretty risky thing in general. Think: small team, small app, everybody has the secrets anyways for deployment purposes. Sure, setting up vault is superior - but how much effort does that cost that could be invested in a better solution. Or a puppet repo that you use to provision your machines, shared in the ops team: small team, everbody has root - on each machine there might be an ssh key that gives away all your secrets. So better invest in solid FDE and maybe tie that to a TF device, a yubikey that is required to decrypt the disk etc. Not perfect by all means, but there's limited time to go around and you really should think about what threats you want/can defend against. (for example, for most projects, I'm not wasting any thought about defenses against a nation state actor, that's a threat that I won't be able to meaningfully counter anyways).
FDE's and Yubikeys are nice and good controls for some classes of risk, but distributing your passwords onto dev laptops via a git repo. opens you up to a wide range of risks that those won't help you with.
Unless you got super-corporate lockdown with the end point devices you have risks like "A user with access to the repo. installs software which turns out to be malware", "A user with access to the repo. leaves their laptop in a coffee shop unlocked", "A user with access to the repo. puts it on a USB key and loses the key". None of these are nation state level concerns, they're things that could impact the project purely by accident, or at the hands of low-skill attackers
The point is once you've allowed secrets to be in a distributed system like this you have very little control over what happens to them, which is why I'd recommend using a secrets management system where there's more control (e.g. vault from hashicorp) in almost all circumstances.
We've published internally developed projects on github after removing anything sensitive and initializing a new repo from the latest version of the code base.
You lose your development history, but you ensure you won't get bitten by stuff like this.
I pretty much stick to private repos to avoid this being a problem. I'm still generally pretty careful about it, but in case I slip up it's nice to know the info isn't just floating around out there for anyone to grab.
If you found a similar mistake in your repository, you can delete commit from history using: `git rebase --onto <commit-id>^ <commit-id>`. Or if you want actually rewrite it, see git rebase -i` documentation.
It's better to just revoke and not re-write git history in a public repository. Re-writing history is pointless after the credentials are revoked, and causes a headache to others using your repository.
I find force pushes acceptable for topic branches of public repos. In fact, I use them a lot to leave behind clean history. Same as with squash merges, which technically also lose history.
A single person who checked out your repo before the force push will still have the credentials. Once this has been pushed to a public repo, assume that the credentials are burned and revoke them.
if you revoke the credentials, removing them from the git history serves no purpose but disrupts everybody that has a clone of the repo. So you're doing harm for little benefit other than covering up the incident. A net loss, if you ask me.
It serves the purpose of removing a hint on your password patterns from public availability.
E: Oh, and just to preempt this, even saying "i use only random passwords with no pattern" is useful information, as is having a ballpark password length.
Don't have password patterns, problem solved. Knowing that my password is 20 random characters of all possible symbols will not reduce your search space by any significant amount.
Still useful, means there's no point in checking any < 20, which halves the search space. Or, on the outside, can be useful to abort any attempts at bruteforce by way of cost evaluation and move on to another target.
My goal when choosing a password (generator) typically isn't "what will tie up an adversary for the longest, preventing them from moving on to attack someone else".
Call me selfish, but if my password is known to be too tough to bother, so Eve moves on to someone else's weak password, great.
Halves the search space, so in other words it only reveals 1 bit of information about the password, a completely insignificant amount. For a 64-symbol alphabet, a 20 character password is 120 bits long, so you'll still have to brute force 2^119 passwords on average. The sun will swallow the Earth before then.
Too little, too late. Plenty of people watch the stream of recent commits to github, and can snatch an API key as soon as it's pushed. Removing the compromised, revoked key from your git history is like making sure your front door is closed properly after coming home to find you've been burgled.
I can think of a legitimate use case for that: cleaning up a that's about to be opened to the public (so no one who shouldn't have access to sensitive info has checked it out yet).
Searching filename:id_rsa also yield a rather interesting result. Alongside with "BEGIN OPENSSH PRIVATE KEY". I wonder how much of these also contains ssh_config.
This is the main reason I use gitlab, because they have free private repos. When your trying to smash out code as fast as possible in a startup you don't want to have to worry about acccidently checking a secret in
Computers are supposed to be good at what imperfect humans are not. This only proves how primitive the tool is.
That is, for example, if Gmail can ask "it looks like you forgot the attachment" why can't Git say "this is a public repo and you're about to commit and push passwords. Are you sure?"
It's going to be easier to fix the tool than it is to make humans be perfect.
I believe Django's logging framework will automatically replace strings in your settings.py file (basically a dict) with '*' if the key "looks like" a secret (contains the word 'secret' or 'key' or 'passw' etc).
What would I do? I'd take ALL those incidents on GitHub and I'd run them thru some sort of AI pattern recognition algorithm. That would become my identification "engine" (?).
It might not catch everything all the time - since humans are pretty creative when it comes to fucking things up - but I bet it would be pretty effective. Certainly more effective than what we have now. Then if it can keep learning going forward, all the better, eh.
While there may well be common trends, Git is a tool for arbitrary content - it's going to be pretty hard to accurately find passwords/secrets being committed. There are tools out there for more specific sets of stuff, but expecting git to catch anything is a little much.
Regardless. The broader point is, Git is a screwdriver and what is needed at this point is a hammer. Sure, we can keep trying to pound nails with a screwdriver but that's harder work and is far less productive.
We. Need. A. New. Tool.
p.s. But there are how many CSS pre-processors? And how many JS frameworks? Etc. Things we don't need. Go figure.
I mean, I'd argue it's a screwdriver and you want an electric screwdriver. Sure, I applaud that effort, but it doesn't mean the original thing is bad, just that it could be improved.
Gmail can make a simple keyword search for a handfull of phrases in something that's known to be text
Git would have to first decide whether a file is a textfile or binary file, a decision that can be done reasonably well heuristically but that is undecidable in the general case. Then it has to parse text files for a long, curated list of known keywords that are only used for storing API keys and are not (usually) used in normal code. I'm not sure if that's even feasable.
And then of course git has no concept of "public" and "private" repos, so the entire task can't be handled well by git.
Good idea! Shouldn't be so hard to implement a simple prototype.
if "PASSWORD=xxx" in text => prompt alert or ask confirm
Next step would be to take this list (search result), make a curated list of 100-1000 unencrypted password (text/line + files + infos of repo), and then hard code some rules to detect +80% of cases.
Careful. If you have already committed it, and you ever push the repo to a public GH repo, your key is compromised. Just because some benevolent slacker-attackers on HN aren't sniffing the PSHB event queue, doesn't mean no one is. If you ever send the secret to Github, criminals have it. If you ever committed it, and then you ever push, then you've sent it to GitHub.
So yes, add the file to gitignore and git rm it, but also invalidate your keys and get new ones.
For anyone wondering, if you want to remove a file or secret you've already committed, you can use BFG Repo-Cleaner to go through your commit history and completely remove any trace of it.
Just note that if it's a public repo, it may not help you, due to attackers scraping Github's API and mirrors like GHTorrent. From "Why Deleting Sensitive Information from Github Doesn't Save You":
Warning - in the HN comment that is linked, don't click the link, is a browser popup spam which is actually hard to close (url has been dropped and picked up by a spammer?)
Why would they if the tip does not have any passwords in it? It's not like a potential contributor will search the commit log to see if there were once passwords around. Besides, making such changes in public changesets is rude, to say the least.
What I mean is, lots of folks seem to be saying to ignore the presence of the file and just change the password where it's used.
Removing the file, or the password and adding a comment, as well as changing the password where it's used is much less likely to end up with a re-added password later.
Of course, removing the file, adding it to .gitignore and changing the password makes it even harder as a contributor would have to work to add the password back, which is even less likely to happen.
265 comments
[ 4.6 ms ] story [ 379 ms ] threadThere are just so many of those it's crazy:
Also replace "remove" with delete/rm/replace/etc.And replace "YOURFAVORITEAPI" with CircleCI, Travis, Mailchimp, Trello, Stripe, etc, etc.
Also, companies I contacted consider it the customer fault and basically don't care.
However it should be pretty easy for them to set up a script to search github for this kind of stuff and automatically invalidate keys
2) Search if found keys are actual valid keys
3) Expire key, send explanatory email, issue new key
(I think that's what AWS does)
Annoyingly there is no way to turn it off even when you explicitly want to share an API key knowingly. But i'm more than fine with needing to "obfuscate" an API key or manage secrets correctly knowing it saves TONS of people.
It probably wasn't the best idea, but it was the only "secret" needed in the whole project and I didn't want to maintain a way of managing secrets in a public project for a pointless key.
In the end I did just that, and looking back it was the better choice, but at the time it was annoying.
I wonder why they'd want to invalidate that. :)
$key = "BAAD" + "F00D" + "CAFE" + "BABE";
If the token is XYZ and the script is searching https://github.com/search?utf8=%E2%9C%93&q=XYZ&type=Commits&...:
1. It's sharing the token with GitHub.
2. It's embedding the token as query-string parameter in a GET request, which is much more likely to be logged (than sending it as data in a POST request), and more likely to be available to less-privileged/less-trusted staff.
3. If the request is sent to a non-HTTPS endpoint, the query can be MITMd, revealing the token.
I'd be very wary of setting up a token-finding script, it feels like it adds more risk than it saves.
- a key is made public, and we have to call a user or refund them (for retention purposes)
- a key is made public, and we revoked the key, potentially breaking the customers builds/deploys and potentially knocking a customers stuff out (if, for example, a key is disabled during a push to production).
[0] - https://developer.github.com/v3/activity/events/
Reminds me of the water pipeline in Finding Nemo with the crabs above it.
Does that imply that they are not hashing secret keys, or did you also push the account key (allowing for a single auth test on their side)?
The interesting thing is that there is also an evil crawler that will automatically launch thousands of windows vms to mine bitcoins (that's all they do). Amazon told us that we have leaked our account id and secret but also they notice the other crawler has launched a lot of VMs and they did a refund to us. yes, we love amazon.
Lesson learned: you never put the account id and secret in your code, not only that you should not hardcode it, but there is no need to even read that from the environment etc.
Don't do something like this `new S3({accountKey: ..., accountSecret: ..}` instead you do `new S3()` and that's it. Every AWS SDKS is smart enough to find the keys in the environment following a series of steps:
- environment variables
- ~/.aws/credentials
- and when your code is run on ec2, lambda, etc. you should use IAM Roles.
So, in addition to not hardcoding an AWS secret, your code should not even pass the secret to the SDK.
Consider also enabling CloudTrail and have alerts on that.
There is also a way to not have ~/.aws/credentials in your machine and have another thing that requires MFA. I am not familiar how this work yet but we started to use it.
https://news.ycombinator.com/item?id=7411927
By just looking quickly, it seems that you can still find many recent live keys...
It gets a little scary when it veers from professional security to individual personal privacy https://github.com/search?p=2&q=smtp.gmail.com+pass&ref=sear...
Writing passwords down on a piece of paper, and keeping that in your wallet or locked desk drawer is actually one of the more secure ways of storing passwords these days.
No risk of electronic compromise, and its highly unlikely that people who would steal your wallet or break into your home are also interested in your online accounts.
1. New guy gets hired.
2.You work for a large corporation and we'll you can't say you can trust everyone there.
3. Small company of 10 (company I work at for example) could be compromised by the weekly janitor.
4. Someone could break in and make it look like a robbery all while stealing your critical infrastructure.
I recently helped research a bit about internal security for our office and sticky notes are still a very common place for credentials to be compromised.
I notice in many instances Github showing an error.
We could not perform this search: Must include at least one user, organization, or repository
But if you change anything in the URL, it works again. Such as adding &p=2, which lucideer tried. But I got the error on his link. So I changed it to p=3, and it worked.
So I'm guessing Github has an autodetection for a particular global code search getting high hits (for something like this, I assume) that locks people out.
Seems more like a band-aid on a broken leg, though.
On an internal VCS, this would still be a problem, but a bit less visible/exploitable...
Also, you can hide your crimes and not show off all your "TODO: put more stuff here" commits to the world.
That said I'd say putting secrets in a git repo is a pretty risky thing to do. By the nature of the tool that means that the secret ends up on the device(s) of every developer who checks out the codebase, so the security of the secrets is equal to the security of the worst secured device in question.
Storing and keeping secrets is a pretty risky thing in general. Think: small team, small app, everybody has the secrets anyways for deployment purposes. Sure, setting up vault is superior - but how much effort does that cost that could be invested in a better solution. Or a puppet repo that you use to provision your machines, shared in the ops team: small team, everbody has root - on each machine there might be an ssh key that gives away all your secrets. So better invest in solid FDE and maybe tie that to a TF device, a yubikey that is required to decrypt the disk etc. Not perfect by all means, but there's limited time to go around and you really should think about what threats you want/can defend against. (for example, for most projects, I'm not wasting any thought about defenses against a nation state actor, that's a threat that I won't be able to meaningfully counter anyways).
Unless you got super-corporate lockdown with the end point devices you have risks like "A user with access to the repo. installs software which turns out to be malware", "A user with access to the repo. leaves their laptop in a coffee shop unlocked", "A user with access to the repo. puts it on a USB key and loses the key". None of these are nation state level concerns, they're things that could impact the project purely by accident, or at the hands of low-skill attackers
The point is once you've allowed secrets to be in a distributed system like this you have very little control over what happens to them, which is why I'd recommend using a secrets management system where there's more control (e.g. vault from hashicorp) in almost all circumstances.
You lose your development history, but you ensure you won't get bitten by stuff like this.
Just revoke the password/secret/whatever.
Squash merges are just bad. They destroy all the info that makes git handle branching and conflicts better than svn.
E: Oh, and just to preempt this, even saying "i use only random passwords with no pattern" is useful information, as is having a ballpark password length.
Call me selfish, but if my password is known to be too tough to bother, so Eve moves on to someone else's weak password, great.
If the keys/passwords are already pushed to GitHub or other public hostin, they should also be revoked.
<commit-id>^ <commit-id>
is
<commit-id>^!
I use it all the time with:
git diff <commit-id>^!
I use either `git-crypt` [1] or `ansible-vault` [2].
1: https://github.com/AGWA/git-crypt
2: http://docs.ansible.com/ansible/playbooks_vault.html
Advantage of this approach is it encrypts the values individually instead of per file. This way the secrets files are git/review friendly.
Github, like SSH, uses an asymmetric authentication scheme. They even publish everyone's public keys. It's much more secure than passwords.
[1] https://github.com/awslabs/git-secrets
That is, for example, if Gmail can ask "it looks like you forgot the attachment" why can't Git say "this is a public repo and you're about to commit and push passwords. Are you sure?"
It's going to be easier to fix the tool than it is to make humans be perfect.
In comparison Gmail doesn't catch all cases either, if you say something like "here are" instead of I've attached it misses it.
Passwords, look for variables with the name password, passwd assigned strings.
Like Gmails attachment, it'll get stuff wrong, just make it easy to continue on.
It might not catch everything all the time - since humans are pretty creative when it comes to fucking things up - but I bet it would be pretty effective. Certainly more effective than what we have now. Then if it can keep learning going forward, all the better, eh.
We. Need. A. New. Tool.
p.s. But there are how many CSS pre-processors? And how many JS frameworks? Etc. Things we don't need. Go figure.
Git would have to first decide whether a file is a textfile or binary file, a decision that can be done reasonably well heuristically but that is undecidable in the general case. Then it has to parse text files for a long, curated list of known keywords that are only used for storing API keys and are not (usually) used in normal code. I'm not sure if that's even feasable.
And then of course git has no concept of "public" and "private" repos, so the entire task can't be handled well by git.
That's all I'm trying to say
We keep expecting a cat to bark, and then we're shocked and disappointed that it doesn't. So let's stop asking and find / build a better tool.
if "PASSWORD=xxx" in text => prompt alert or ask confirm
Next step would be to take this list (search result), make a curated list of 100-1000 unencrypted password (text/line + files + infos of repo), and then hard code some rules to detect +80% of cases.
add password / add passwords
* https://github.com/search?utf8=%E2%9C%93&q=add+passwords&typ... * https://github.com/search?utf8=%E2%9C%93&q=add+password&type...
add secret / add secrets
* https://github.com/search?utf8=%E2%9C%93&q=add+secret&type=C... * https://github.com/search?utf8=%E2%9C%93&q=add+secrets&type=...
That people took the trouble to use PGP but then go and do something this silly?
search for code making connections to db
ruby https://github.com/search?q=DBI.connect&ref=searchresults&ty...
java https://github.com/search?p=2&q=DriverManager.getConnection%...
and so on...
http://stackoverflow.com/questions/1139762/ignore-files-that...
The other alternative I can think of is to hide sensitive values in environment variables
So yes, add the file to gitignore and git rm it, but also invalidate your keys and get new ones.
https://github.com/search?p=2&q=remove+password+oops&ref=sea...
https://rtyley.github.io/bfg-repo-cleaner/
http://jordan-wright.com/blog/2014/12/30/why-deleting-sensit...
The top HN comment on the article details their experiences with getting hacked this way:
https://news.ycombinator.com/item?id=8818035
Removing the file, or the password and adding a comment, as well as changing the password where it's used is much less likely to end up with a re-added password later.
Of course, removing the file, adding it to .gitignore and changing the password makes it even harder as a contributor would have to work to add the password back, which is even less likely to happen.
Edit: I had similar objections to "why not rework databases as an immutable diff history?" https://news.ycombinator.com/item?id=13581096