I was building a github app and was scratching my head about where I messed up. The website and front end were loading but github isn't sending webhooks.
That's great. 2 minutes before it went down a pipeline workflow ran in Heroku deploying a bug to my app. Now I can't deploy the hotfix since it depends on Github and I'm not able to revert to the last successful build on the Heroku interface.
This is a good point, but it could be worded better.
One of the things a site reliability engineer should think about is how well the site can be operated when dependencies have issue. After an incident like this, even if you were able to recover, it's worth thinking about how things could have gone better.
In the past I had a painful experience with one application I was supporting that needed to install NPM packages on deployment. We couldn't successfully deploy (or scale up) for the duration of that outage. In that case we realized it was safer to switch to server images with all assets pre-installed and an NPM cache to give the build a better chance of succeeding. The next NPM outage we only noticed after the fact :)
Not certain how this particular deployment pipeline is failing due to the GitHub outage, but a post-mortem to discuss may be helpful and protect against future issues.
To your point on NPM dependencies, CDNs for JS libraries is another thing that I recently learned is common practice, which I don't fully understand. It seems like if the CDN goes down then your application stops working, but I am convinced I'm missing something here because it seems like such poor engineering judgement. This seems to be really common in SPAs, which is precisely where (it seems to me) you shouldn't be using a CDN.
It seems the convenience of cloud based deployment pipelines is not really worth situations like this.
Specifically for the javascript frameworks, the use of a popular CDN increases the chance that the browser will have the asset in the cache already. Browser cache is a huge win for load times.
Static file hosting via a managed CDN is a fairly reliable option, better than many companies can build on their own.
Lol. Couldn't work on github, I thought it was just a slow connection so checked HN instead to see whether the page loads - and now I guess it wasn't just an issue on my end :D
Edit: I can see where your solution has an advantage. When one accepts contributions, all of that can be automated with web hooks via github. Still, having to run lambda/similar for this is a bit heavy weight.
Edit 2: Considering that for a release, I would opt in for release from a tag or explicit commit, that would be a non issue. Pull when needed, for redundancy, use another, self hosted bare repo. Push there in case of github down.
Wouldn't replacing the github hook with a git-hooks script incrementally improve the solution as it currently exists?
Then, combine that with a git-hook script that designate all non-release branches push to team's Keybase repo as well? Maybe including a custom prefix in the branch name to avoid need for conflict resolution?
I'm also thinking that it would be easier to use git-hooks to push to AWS CodeCommit (in a random region that GitHub is unlikely to use) paired with CloudWatch events to invoke service backup to Keybase
Of course, ideally it would be a Terraform (or similar open source stack) script so it's not dependent on AWS as a vendor... I have experienced the real pains of vendor lock-in, even with cross-region there is opportunity to improve robustness
I'd imagine the higher volume due to remote work (e.g. people collaborating more on issues and such who may have communicated in person previously) is also a contributing factor.
From the sound of their earlier 'not postmortem but we'll do one soon', it sounded like they currently have a database tier that's creaking at its foundations, and I'd guess any time there's extra strain or someone pushes a fix to the GitHub codebase that adds a tiny bit of unoptimized code, it strains it past the limit.
I don't see the point for major open-source projects or even company projects to depend on a third-party provider for git services in the cloud that is not on-premise. It's fine if the source code is self-hosted or on premise and the mirror is hosted on some cloud VCS like Github, but not the other-way round, which for the latter you risk getting problems like this.
The case for self-hosting a VCS server for serious projects is made once again.
I can understand this sentiment, but a lot of projects and companies (especially startups) want something that "just works" and not something that they'll need to invest resources into maintaining. Something like this can happen on your self-hosted git server and you won't have an SLA or anything to fall back on.
Github (and gitlab) has a large value add with regard to discoverability, issue tracking, user management, hosting simplicity, and usually solid uptime.
GitHub addiction is not just about git itself. It's about the entire ecosystem of pull requests, issues, forking, linking to related projects, CI integrations, hosting for docs, and so on, all integrated under one sign-on umbrella. It is awfully convenient. Yes, for sure you can self-host one of the alternatives like GitLab, or assemble an equivalent set of services from smaller parts. It is absolutely reasonable to suggest that people should - I'm even inclined to agree - but comparing against git all by itself is the wrong comparison.
It is indeed convenient but the whole ecosystem now runs the risk of being a single source of failure if even one part of the service goes down and it is out of your control to do anything about it and also defeats the purpose of decentralisation of using git. It may not matter for personal or toy projects, but for company or serious ones, it does.
In the case of other risks of only using GH to host your project, Its like hosting your encrypted private key on Keybase because it is temptingly convenient but both run the risk of going down or getting compromised in a security breach.
Serious projects like the Linux kernel, WebKit and Chromium aren't primarily hosted on Github but have Github mirrors instead which makes more sense.
Unless you have the (ops) capacity to host it yourself reliably, chances are GitHub will be up more than your self-hosted solution.
As I'm typing this, I'm waiting for GitHub to recover so I can deploy an update to a rather important project for a big client. However, I'm happy it's their engineers to be scrambling to fix their git hosting, instead of me context switching from my work and rushing to duct-tape piece of plumbing I really don't care to think about.
> self-host git over ssh more reliably than github.
Host where? That host may be down. If it's local, then you run into other problems trying to access it remotely.
Still, I find github to be down quite a lot lately which is concerning. It doesn't stop me from working, I just can't review PRs and get changes.. but I would be more worried if I were using github actions stopping me from releasing a hotfix.
The nature of our product is to follow 100s of projects. And most of the interesting stuff is actually not in the code but in the issue tracker, release notes etc (for us).
This is why it's a good idea to have multiple remotes and advertise those remotes in
the project README (my examples: [0]). You'll lose centralized issues during
downtime, but you'll at least be able to interact with some remotes and do basic
collaboration with `git format-patch` and `git send-email`.
If something like sourcehut [1] goes down, nothing will change; users will still be
able to push/pull to mirrors and work with mailing lists and git's built-in
email-based pull-requests.
Yeah I gotta agree with you. How is this any different from the other 10s of thousands of variations of phishing attempts?
If it's not signed and it's not from github.com or a registered subdomain, or if the URL of the action isn't _explicitly_ github.com... it's not legit. It doesn't matter if it slipped through GMail's filter or not...
I don't mean to shame people for falling for phishing. I fell for a convincing Steam phish once when I was a young teenager after a friend's account was compromised and messaged me with a link to a perfect clone of the login page and a similar domain.
But as far as phishes go, this one just seems not too convincing.
In the case of the github phishing it does have a valid certificate and the host name is something that looks legit enough it will probably fool a lot of people
92 comments
[ 2.8 ms ] story [ 150 ms ] threadsilver linings
ssh -T git@github.com throws the same permission denied error.
Any ideas?
One of the things a site reliability engineer should think about is how well the site can be operated when dependencies have issue. After an incident like this, even if you were able to recover, it's worth thinking about how things could have gone better.
In the past I had a painful experience with one application I was supporting that needed to install NPM packages on deployment. We couldn't successfully deploy (or scale up) for the duration of that outage. In that case we realized it was safer to switch to server images with all assets pre-installed and an NPM cache to give the build a better chance of succeeding. The next NPM outage we only noticed after the fact :)
Not certain how this particular deployment pipeline is failing due to the GitHub outage, but a post-mortem to discuss may be helpful and protect against future issues.
It seems the convenience of cloud based deployment pipelines is not really worth situations like this.
Static file hosting via a managed CDN is a fairly reliable option, better than many companies can build on their own.
I think read replica mirrors are good practice for fault-tolerant systems
Anyway, here's how you connect Keybase to GitHub:
https://blog.codefor.cash/2019/08/30/free-automatic-github-b...
https://github.com/codeforcash/github-to-keybase-mirror
Note: this was a v1, and I'm sure there are more resilient ways to do this. Critical feedback strongly encouraged (ideally along with suggestions).
Edit: I can see where your solution has an advantage. When one accepts contributions, all of that can be automated with web hooks via github. Still, having to run lambda/similar for this is a bit heavy weight.
Edit 2: Considering that for a release, I would opt in for release from a tag or explicit commit, that would be a non issue. Pull when needed, for redundancy, use another, self hosted bare repo. Push there in case of github down.
Then, combine that with a git-hook script that designate all non-release branches push to team's Keybase repo as well? Maybe including a custom prefix in the branch name to avoid need for conflict resolution?
From the sound of their earlier 'not postmortem but we'll do one soon', it sounded like they currently have a database tier that's creaking at its foundations, and I'd guess any time there's extra strain or someone pushes a fix to the GitHub codebase that adds a tiny bit of unoptimized code, it strains it past the limit.
Mind your own codes, they said.
The case for self-hosting a VCS server for serious projects is made once again.
In the case of other risks of only using GH to host your project, Its like hosting your encrypted private key on Keybase because it is temptingly convenient but both run the risk of going down or getting compromised in a security breach.
Serious projects like the Linux kernel, WebKit and Chromium aren't primarily hosted on Github but have Github mirrors instead which makes more sense.
The issue is that people start using the CI/CD, the PRs, the issues and... yeah.
As I'm typing this, I'm waiting for GitHub to recover so I can deploy an update to a rather important project for a big client. However, I'm happy it's their engineers to be scrambling to fix their git hosting, instead of me context switching from my work and rushing to duct-tape piece of plumbing I really don't care to think about.
For an individual it's absolutely trivial to self-host git over ssh more reliably than github.
It just costs a bit of money.
Host where? That host may be down. If it's local, then you run into other problems trying to access it remotely.
Still, I find github to be down quite a lot lately which is concerning. It doesn't stop me from working, I just can't review PRs and get changes.. but I would be more worried if I were using github actions stopping me from releasing a hotfix.
But 9/10 every project we use is on GitHub and our work is stalled. Centralization will be our downfall :(
https://docs.gitlab.com/ee/user/project/repository/repositor...
Congrats on the self-hosted uptime :)
If something like sourcehut [1] goes down, nothing will change; users will still be able to push/pull to mirrors and work with mailing lists and git's built-in email-based pull-requests.
[0]: chattiest-channels, dotfiles, term-dmenu, and mpd-scripts at https://git.sr.ht/~seirdy
[1]: https://sourcehut.org/
EDIT: (21:00 UTC ) Just got in :D
https://twitter.com/PonteIneptique/status/783190337602846720
Phishing email looks like a "review your suspicious activity" alert, but the alert is the suspicious activity.
https://imgur.com/a/zdtWmuN
If it's not signed and it's not from github.com or a registered subdomain, or if the URL of the action isn't _explicitly_ github.com... it's not legit. It doesn't matter if it slipped through GMail's filter or not...
But as far as phishes go, this one just seems not too convincing.
Amazing!