I find it hard to believe that they could have made $25k with this. There are companies that scan all commits on gh for secrets, using similar techniques for finding secrets in files.
An interesting look at one of the consequences of using git and public repo's.
Does leave me wondering how long before someone has a setup which detects and tries to exploit these in real-time, which feels like it could be nasty.
Also a challenge with these posts is they were unlikely to have been able to contact all the affected developers who have got exposed secrets, meaning that any that were uncontactable/non-responsive are likely still vulnerable now, I'd guess that means they're about see what happens if those secrets get abused, as people start exploring this more...
Maybe I missed it but the article doesn't mention the even easier way to see this: the activity tab.
It has everything. Any force push to hide ugly prototype code is kept forever which annoys me. I wish we were able to remove stuff from there but the only way to do it is to email support it seems?
All devs should run open-source trufflehog as a precommit hook for all repositories on their local system. It’s not a foolproof solution, but it’s a small time investment to get set up and gives me reasonable assurance that I will not accidentally commit a secret. I’m unsure why this is not more widely considered standard practice.
What I've never understood is, how is this an issue with private repos? Aside from open source projects I can't see the problem with accidentally doing this, even though it is a smell.
Unfortunately for those of us without a Google account, it seems one is required to download the mentioned SQLite database (force_push_commits.sqlite3.)
The thing that people miss out is Git is really a content addressed storage. This means all commits, even the ones not linked to any refs are still stored and addressable.
p.s: If you run OSS project, please use Github Advanced Security and enable Push Protection against secrets.
Probably worth mentioning that force is a ref-related activity not a snapshot related activity. Garbage collection might remove unreferenced commits.
This should be done through history rewrites but as other commenters mention - GitHub has its own rights (and GitHub != git).
I’d recommend looking at simpler alternatives. IMO Jujutsu is mature enough for daily usages, and Fossil is a neat alternative if one wants to drop GitHub completely (albeit not very easy to use).
So the question is: after I orphaned a commit how do I _truly_ make sure it's not visible anywhere on github? Is there no way short of contacting customer support to GC a repo? Shouldn't this just basically be a button on the repo, in the "danger zone" area of the repo maintenance?
I got tired of "oops" over time and started abusing environment variables. If you have enough discipline to spend 10 seconds configuring them, you'll never have to worry about magic strings accidentally getting sucked up into source control.
The other upside with environment variables is that they work across projects. Set & forget, assuming you memorized the name. Getting at tokens for OpenAI, AWS, GH, etc., is already a solved problem on my machine.
I understand why a lot of developers don't do this though. Especially on Windows, it takes a somewhat unpleasant # of clicks to get to the UI that manages these things. It's so much faster (relatively speaking) to paste the secret into your code. This kind of trivial laziness can really stack up on you if you aren't careful.
For a long time and probably still today, Google AppEngine kinda encouraged storing secrets in the YAML, which is easy to accidentally git-commit. There's no easy way to pass secrets to your services otherwise, unlike Heroku etc where it's always been a single command to put them into envvars on the jobs.
Crypto miners have been doing this for 10-12 years at least. I used to speak on this topic regularly. Deleted commits are insufficient, you must rotate your secrets
22 comments
[ 6.7 ms ] story [ 100 ms ] threadCheckout to event, commit in clean state with prior log history, overlay the state after the elision and replace git repo?
When I had to retain log and elide state I did things like this in RCS. Getting date/time info right was tricky.
Does leave me wondering how long before someone has a setup which detects and tries to exploit these in real-time, which feels like it could be nasty.
Also a challenge with these posts is they were unlikely to have been able to contact all the affected developers who have got exposed secrets, meaning that any that were uncontactable/non-responsive are likely still vulnerable now, I'd guess that means they're about see what happens if those secrets get abused, as people start exploring this more...
- Once it is on the internet - it is always there so Rotate the key/secrets FIRST.
- Never think secrets are gone because of you have recommited .
- Deleting a commit is not enough , use BFG Cleaner - https://rtyley.github.io/bfg-repo-cleaner/ , and force commit to change history.
Edit- Forget to add most important thing - rotating the key.
It has everything. Any force push to hide ugly prototype code is kept forever which annoys me. I wish we were able to remove stuff from there but the only way to do it is to email support it seems?
Here it is for the test repo mentioned
https://github.com/SharonBrizinov/test-oops-commit/activity
Not if you contact customer support and ask them to garbage collect your repo.
What I do when I accidentally push something I don’t want public:
- Force push;
- Immediately rotate if it’s something like a secret key;
- Contact customer support to gc the repo (and verify the commit is gone afterwards).
(Of course you should consider the damage done the moment you pushed it. The above steps are meant to minimize potential further damage.)
So I just hard coded the key. The key was rotated after the presentation.
Does not look very good on a repo.
p.s: If you run OSS project, please use Github Advanced Security and enable Push Protection against secrets.
This should be done through history rewrites but as other commenters mention - GitHub has its own rights (and GitHub != git).
I’d recommend looking at simpler alternatives. IMO Jujutsu is mature enough for daily usages, and Fossil is a neat alternative if one wants to drop GitHub completely (albeit not very easy to use).
The other upside with environment variables is that they work across projects. Set & forget, assuming you memorized the name. Getting at tokens for OpenAI, AWS, GH, etc., is already a solved problem on my machine.
I understand why a lot of developers don't do this though. Especially on Windows, it takes a somewhat unpleasant # of clicks to get to the UI that manages these things. It's so much faster (relatively speaking) to paste the secret into your code. This kind of trivial laziness can really stack up on you if you aren't careful.
Last time I tried, the default suggestion was Cloud KMS (yeah), now there's some new secret manager that also looks annoying: https://stackoverflow.com/questions/58371905/how-to-handle-s...