cd /[mason@IT-PC-MACPRO ~]$ cd Code/rollerball
[mason@IT-PC-MACPRO rollerball (master)]$ git pull
remote: Internal Server Error.
remote:
fatal: unable to access 'https://masonmark@github.com/RobertReidInc/rollerball.git/': The requested URL returned error: 500
[mason@IT-PC-MACPRO rollerball (master)]$
Luckily that came back up 5 or 10 minutes after the outage began. And they updated it with a down notice.
A side project of mine called StatusGator [1] monitors status pages and alerts you when something is posted on them. I built it to handle the use case that you're trying to diagnose an issue only to remember to check the status pages of your dependencies and notice that they are already working on it.
It's pretty useful for those services that keep their pages up to date. But not very useful in cases like this when it's not updated.
Huh, TIL. I always assumed that they meant that the errors themselves are unicorns because they rarely happen and even when they do, you usually don't see them.
I'm guessing the parent doesn't mean only bower, but bower uses github as the source of packages/code (I don't use bower, so I'm only going from memory). On a build that downloads everything 'fresh' it's not going to be able to get sources that are only on github.
Bower certainly isn't the only thing that does it. It's a trend that's becoming more and more common.
I was in a go tutorial a couple of years ago where the presenter was including directly from github.
And sitting in a ruby talk when the rubygems compromise happened just after everyone had been told to go upgrade a gem because of flaws in a specific gem.
Well, it's not like we have invented binary and source packages two decades
ago, so people don't need to re-download and re-build their build-time
dependencies every time they start compilation.
Yep, but nobody ever includes them. Nice stuff to automate, gotta chek if there is already DevOpsy stuff to fix that, and if there isn't, got me a new weekend project...
It is obviously gonna be hosted on GitHub. Oh, the irony...
You shouldn't notice a webhook deluge because the site isn't generating events. I'm watching our webhook services though and will let you know if that changes.
(Positive in this sense has a similar meaning in behavioral psychology's terminology for operant conditioning, adding punishment.. which both come off as odd with our common use of positive as good and beneficial :/ https://en.wikipedia.org/wiki/Operant_conditioning)
...Carrying on with your analysis, since uninterrupted positive feedback loops result in explosion, I guess that would mean either (for this outage) github's fallback status page would go down or the parts of the network carrying that traffic, which could not handle their increased load, would go down.
Probably better we would rework these control mechanisms and reroute \ rewire \ entirely change these processes.
Self-hosted isn't 100% reliable either. Nobody is that good. I've done serious high availability (financial system that would land on the front page of the New York Times the way Github lands on HN), and it wasn't 100%. And the overhead of high availability is incredible.
No way would I do self-hosted version control. I have better things to do than babysit servers for commodity services.
"Fool me once, shame on you. Fool me twice, shame on me."
I can learn from my mistakes and improve progressively.
If I rely on another instead, I cannot, and I cannot (necessarily and confidently) see to it that they learn from their mistakes and improve.
By relying on another (at least an unreliable \ uncommunicative \ uncooperative one), I cannot improve my chances of them not making those same mistakes again, taking me down with them.
When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.
I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login...
I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.
So... you were doing a production redeploy, and it crashed?
The URL "git://github.com/components/ember.git" [1] suggests this is an internal GitHub Bower build log, but your post history doesn't mention anything about GitHub (let alone whether you work there), so I'm not 100.00% sure.
Assuming this is, in fact, a GH Bower log, the first thing that came to mind was that this architecture isn't (and possibly should be) using a dual-silo approach: when you upgrade, the upgrade gets loaded into a new blank namespace/environment, tested, and if it worked (passes CI test coverage or something like that), the main entry point is switched to the new environment (maybe with a web server restart or config rehash) and the old environment gets purged (possibly after a trial period). The current stack looks quite akin to "click this button to flash the new firmware and DO NOT UNPLUG your device or you'll brick it."
But then I realized... wait. You guys have like... isn't it like, a few dozen RoR worker boxes? Was this crash on the inbound router or something? xD
It's all good though; consider this "curious criticism" - like constructive criticism, but with extra sympathy. And hey, GH's never broken on my watch before (not that I need it atm... hopefully); this is interesting =P
Shit happens some time. Who cares who broke it- what matters is they're fixing it. I hope github does make public a official reason tomorrow. Not looking to blame anyone- I just want to know what they're doing internally to make sure it does not happen again... adaptation. Even if it's from some external hack or ddos attack- how can they plan on building redundancies in. I am pretty sure GitHub's dev team is talented- I'm being such an anti-troll.
> I hope github does make public a official reason tomorrow
I do too, but, mostly I find post-mortems quite interesting to read.
If your services are at some 95% uptime or lower, you're doing something (serveral things) very wrong and its probably not that interesting to me.
Getting from 95% -> 99% you probably did some interesting things there.
Going into multiple .9s beyond that, you're likely doing quite a bit of interesting stuff but what I find more interesting is where you went wrong. Figuring out not just where you went wrong, but, what your incorrect assumptions were and WHY they were wrong. "We believed X could never fail because of Y, and even if X did fail, it would not cause production impact because of Z!"
Ah, glad I didn't presume it was a valid message! I've heard people joke about this kind of thing exactly this way in the past.
But now my "technical breakdown info" box has no tidbits in it. :P
I'm glad you're back up now (sortakinda - what sort of traffic are you sustaining right now? :D), but a rough idea of what asploded would be really cool to know about.
Speaking of which, I'd like to take a moment to make a strong point about the fact that disaster-recovery situations don't get blogged about enough. Vague "we fixed it" datapoints get buried in status update logs like it's something to hide and hope nobody brings up.
In situations like these, the only constructive perspective is for everyone to accept that something went horribly wrong and not make a fuss about it, and if such a mentality can be established, this creates an environment within which we can share technical breakdowns of "we found ourselves in XYZ position and then we did these thirty highly specific things in heroically record time to be up and running again", and I think sharing this type of info would potentially be more educational than setup tutorials or the "we switched to X and it improved Y by 1400%" type things the Net's full of. Sure, you'd have to generalize and probably give a lot of backstory about infrastructure, but it's becoming trendy (in a sense) for companies to describe their operations in precisely this way, so it's not completely nonviable.
(Note my use of the angle of "we found ourselves in XYZ position" - maybe a small highlight of what led up to the disaster would be included (worth considering if the information would be educational), maybe not. In a blog context, moderating comments to keep the discussion on-track and constructive may be necessary, but IMO would be worth it.)
I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened.
Regardless, most dependencies are on GitHub which breaks bower install for most people. It's crazy how much infrastructure relies on this single point of failure.
What would that look like? Can you describe it? It's not very useful to say stuff without providing some sort of useful idea. You can't just say "someone should write some sort of stupid content tracker, and give it some random three-letter combination that is pronounceable," or whatever.
Completely agree. But whoever makes it, they should make it free and open source and designed to handle everything from small to very large projects with speed and efficiency.
Additionally, it should be easy to learn and have a tiny footprint with lightning fast performance. It should outclass SCM tools like Subversion, CVS, Perforce, and ClearCase with features like cheap local branching, convenient staging areas, and multiple workflows.
That sounds very cool. I was just pointing out how divorced these ideas still are from the way we actually program and implement things on the web.
In an ideal world, a link to a source code repository (or a link to anything for that matter...) would never fail because there would be automatic mirrors to at least provide read-only access to it.
It sort of forces one to ask whether this is the result of fundamental mistakes within HTTP / DNS itself. It's not realistic for a web designer to put time into making external links fault tolerant by running some query to switch the routing link to an available node.
Content addressed links at least make this possible. With http you have to reach a particular server. If that server is down, the link is broken. With ipfs all you need is at least one machine on the entire network to be serving a file and the link will work and as an added bonus you can verify the hash of what you get so a mitm attack on that link is impossible.
npm i semantic-ui
npm ERR! fetch failed https://github.com/derekslife/wrench-js/tarball/156eaceed68ed31ffe2a3ecfbcb2be6ed1417fb2
npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 503
So, in this case, it looks like github isn't acting as a version control server, but rather a static server... It turns out that github isn't just a single point of failure for some application, but rather multiple points of failure for slightly different reasons...
I changed jobs recently- previously used Github, now on BB- glad I'm not at my old job, I'm sure several people's phones are blowing up from auto-emergency fail measures.
If the internet would hire people to advertise for them, it would bring marketing to the next level. Take for example if BitBucket hired someone to go online, and compare the two: GitHub and BitBucket. Of course the comparison would have to disclose that thy work for BitBucket, but it would be a very thoughtful response.
We would get these detailed as fuck responses on why one project (BitBucket) is worth more to the developer than GitHub instead of these little jabs at why it is better. I want thoughtful responses.
If there was a duplicate response, the employee that responds to various threads online can link them to their answers.
This isn't even directed at you @ntaylor. I think it's an exploitable marketing strategy. A clear example of this is Katie from PornHub. /u/Katie_PornHub (or whatever the user name is) posts on reddit, which in tern gets more interest in PornHub. PornHub is basically using reddit for free advertising.
Basically. I want to know WHY something is better than something else, and that WHY is with as much detail as possible with a lot of thought put into it. Give me a pros and const list between the two. Anything but two good things about it.
- - -
Holy fuck, this would just add a human element to advertising. You hire humans to serve ads to people. Whenever "GitHub" is mentioned, and you have a competing better service, their only job is to advertise your service to anyone have problems with the other service.
In cases like these, it always seems to happen naturally. Humans are willingly advertising for free, when they could be paid for it.
The downfall of the internet:
> We make up the ad network
> It carries us far;
> however---giants must fall.
Thinking about this more, this could be automated to some degree. You could have a bot that listens and responds when key words appear somewhere online. You could use Google's API for when it's bot finds something of interest to you "As it happens"
This would only get you so far. What if a user has a question? How will you help them out if it's a bit. The solution?
Write API's for every site that you want to use. You can either scrape every site for responses that fit your criteria, or let google do the digging for you with the "as it happens" notifications which would indicate that you have to scrape that site for the information.
If the information you get from the site is something you want to respond to, you can.
- - -
Holy fucking shit.
I just described a system that would allow for you to use all websites as inboxes for chat messages.
- - -
Refine:
With this early model, we would have one program on our machine that would understand X websites. It can parse user comments directed at us from a website (A). It can send responses back to A, and so on.
Now, users on Program X can chat with theoretically any
person on any website (A, B, ...), but users of website A can only chat with other people on website A OR people using Program X.
Why doesn't everyone use program X? Because it does not exist yet.
We use Bitbucket in our ~30 member academic robotics research lab, it's wonderful because they are nice enough to supply us with as much as we need (private repos, teams, etc.) for free!
Does this have something to do with the constant DoS attacks they've been getting from a certain country for hosting certain open source projects that defy the censorship authority of said certain country?
jim% git clone git@github.com:pfsense/pfsense.git
Cloning into 'pfsense'...
fatal: remote error:
GitHub is offline for maintenance. See
http://status.github.com for more info.
296 comments
[ 5.0 ms ] story [ 380 ms ] threadhttps://status.github.com/
edit: Now says "19:32 Eastern Standard TimeWe're investigating connectivity problems on github.com."
edit2: "19:47 Eastern Standard TimeWe're investigating a significant network disruption affecting all github.com services."
https://www.dropbox.com/s/ap1nqogmqza2e12/Screenshot%202016-...
And:
Nothing for this yet, though.
EDIT: There it is: https://twitter.com/githubstatus/status/692505376554618883
A side project of mine called StatusGator [1] monitors status pages and alerts you when something is posted on them. I built it to handle the use case that you're trying to diagnose an issue only to remember to check the status pages of your dependencies and notice that they are already working on it.
It's pretty useful for those services that keep their pages up to date. But not very useful in cases like this when it's not updated.
1. https://statusgator.io
[1] http://unicorn.bogomips.org/
http://aokp.co/
(they're using a different but similar image now, but they used to use that exact same logo)
Bower certainly isn't the only thing that does it. It's a trend that's becoming more and more common.
And sitting in a ruby talk when the rubygems compromise happened just after everyone had been told to go upgrade a gem because of flaws in a specific gem.
It's not unique, but it is terrible.
Oh wait, we have.
It is obviously gonna be hosted on GitHub. Oh, the irony...
It looks like webhooks are wedged.. no?
No word on twitter yet: https://twitter.com/githubstatus
git:(master) ok this is actually really annoying now
zsh: command not found: ok
git:(master) fuck
look this is actually really annoying now [enter/↑/↓/ctrl+c]
look: is: No such file or directory
git:(master) fuck
> No fucks given
That's kind of perfect.
(Positive in this sense has a similar meaning in behavioral psychology's terminology for operant conditioning, adding punishment.. which both come off as odd with our common use of positive as good and beneficial :/ https://en.wikipedia.org/wiki/Operant_conditioning)
...Carrying on with your analysis, since uninterrupted positive feedback loops result in explosion, I guess that would mean either (for this outage) github's fallback status page would go down or the parts of the network carrying that traffic, which could not handle their increased load, would go down.
Probably better we would rework these control mechanisms and reroute \ rewire \ entirely change these processes.
No way would I do self-hosted version control. I have better things to do than babysit servers for commodity services.
I can learn from my mistakes and improve progressively.
If I rely on another instead, I cannot, and I cannot (necessarily and confidently) see to it that they learn from their mistakes and improve.
By relying on another (at least an unreliable \ uncommunicative \ uncooperative one), I cannot improve my chances of them not making those same mistakes again, taking me down with them.
Shit happens.
I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.
Recovered all the data by using open file handles in /proc/.
Not a fun two hours.
Shit happens. Live and learn.
The URL "git://github.com/components/ember.git" [1] suggests this is an internal GitHub Bower build log, but your post history doesn't mention anything about GitHub (let alone whether you work there), so I'm not 100.00% sure.
[1] https://webcache.googleusercontent.com/search?q=cache:3e00jl...
Assuming this is, in fact, a GH Bower log, the first thing that came to mind was that this architecture isn't (and possibly should be) using a dual-silo approach: when you upgrade, the upgrade gets loaded into a new blank namespace/environment, tested, and if it worked (passes CI test coverage or something like that), the main entry point is switched to the new environment (maybe with a web server restart or config rehash) and the old environment gets purged (possibly after a trial period). The current stack looks quite akin to "click this button to flash the new firmware and DO NOT UNPLUG your device or you'll brick it."
But then I realized... wait. You guys have like... isn't it like, a few dozen RoR worker boxes? Was this crash on the inbound router or something? xD
It's all good though; consider this "curious criticism" - like constructive criticism, but with extra sympathy. And hey, GH's never broken on my watch before (not that I need it atm... hopefully); this is interesting =P
Source: I work at GitHub.
So, you're saying you broke it.
Everybody, over here, gang up on this guy.
> I hope github does make public a official reason tomorrow
I do too, but, mostly I find post-mortems quite interesting to read.
If your services are at some 95% uptime or lower, you're doing something (serveral things) very wrong and its probably not that interesting to me.
Getting from 95% -> 99% you probably did some interesting things there.
Going into multiple .9s beyond that, you're likely doing quite a bit of interesting stuff but what I find more interesting is where you went wrong. Figuring out not just where you went wrong, but, what your incorrect assumptions were and WHY they were wrong. "We believed X could never fail because of Y, and even if X did fail, it would not cause production impact because of Z!"
But now my "technical breakdown info" box has no tidbits in it. :P
I'm glad you're back up now (sortakinda - what sort of traffic are you sustaining right now? :D), but a rough idea of what asploded would be really cool to know about.
Speaking of which, I'd like to take a moment to make a strong point about the fact that disaster-recovery situations don't get blogged about enough. Vague "we fixed it" datapoints get buried in status update logs like it's something to hide and hope nobody brings up.
In situations like these, the only constructive perspective is for everyone to accept that something went horribly wrong and not make a fuss about it, and if such a mentality can be established, this creates an environment within which we can share technical breakdowns of "we found ourselves in XYZ position and then we did these thirty highly specific things in heroically record time to be up and running again", and I think sharing this type of info would potentially be more educational than setup tutorials or the "we switched to X and it improved Y by 1400%" type things the Net's full of. Sure, you'd have to generalize and probably give a lot of backstory about infrastructure, but it's becoming trendy (in a sense) for companies to describe their operations in precisely this way, so it's not completely nonviable.
(Note my use of the angle of "we found ourselves in XYZ position" - maybe a small highlight of what led up to the disaster would be included (worth considering if the information would be educational), maybe not. In a blog context, moderating comments to keep the discussion on-track and constructive may be necessary, but IMO would be worth it.)
1. https://bitbucket.statuspage.io/
Additionally, it should be easy to learn and have a tiny footprint with lightning fast performance. It should outclass SCM tools like Subversion, CVS, Perforce, and ClearCase with features like cheap local branching, convenient staging areas, and multiple workflows.
That would be a significant upgrade.
[1] https://ipfs.io
In an ideal world, a link to a source code repository (or a link to anything for that matter...) would never fail because there would be automatic mirrors to at least provide read-only access to it.
It sort of forces one to ask whether this is the result of fundamental mistakes within HTTP / DNS itself. It's not realistic for a web designer to put time into making external links fault tolerant by running some query to switch the routing link to an available node.
https://ipfs.io/ipfs/QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5...
:-) IPFS includes an optional http gateway to give the traditional web access to IPFS before browsers support the IPFS protocol natively.
We would get these detailed as fuck responses on why one project (BitBucket) is worth more to the developer than GitHub instead of these little jabs at why it is better. I want thoughtful responses.
If there was a duplicate response, the employee that responds to various threads online can link them to their answers.
This isn't even directed at you @ntaylor. I think it's an exploitable marketing strategy. A clear example of this is Katie from PornHub. /u/Katie_PornHub (or whatever the user name is) posts on reddit, which in tern gets more interest in PornHub. PornHub is basically using reddit for free advertising.
Basically. I want to know WHY something is better than something else, and that WHY is with as much detail as possible with a lot of thought put into it. Give me a pros and const list between the two. Anything but two good things about it.
- - -
Holy fuck, this would just add a human element to advertising. You hire humans to serve ads to people. Whenever "GitHub" is mentioned, and you have a competing better service, their only job is to advertise your service to anyone have problems with the other service.
In cases like these, it always seems to happen naturally. Humans are willingly advertising for free, when they could be paid for it.
The downfall of the internet:
Me.This would only get you so far. What if a user has a question? How will you help them out if it's a bit. The solution?
Write API's for every site that you want to use. You can either scrape every site for responses that fit your criteria, or let google do the digging for you with the "as it happens" notifications which would indicate that you have to scrape that site for the information.
If the information you get from the site is something you want to respond to, you can.
- - -
Holy fucking shit.
I just described a system that would allow for you to use all websites as inboxes for chat messages.
- - -
Refine:
With this early model, we would have one program on our machine that would understand X websites. It can parse user comments directed at us from a website (A). It can send responses back to A, and so on.
Now, users on Program X can chat with theoretically any person on any website (A, B, ...), but users of website A can only chat with other people on website A OR people using Program X.
Why doesn't everyone use program X? Because it does not exist yet.
We use Bitbucket in our ~30 member academic robotics research lab, it's wonderful because they are nice enough to supply us with as much as we need (private repos, teams, etc.) for free!
1. Commit fails
2. Try again, see if it fails twice
3. Check internet connectivity
4. Try github in browser
5. Try github in a different browser
6. Go to HN to see if it's down for everyone
7. Write snarky comment
8. Go try again...
git clone git://github.com/MunGell/Codeigniter-TwitterOAuth.git .
where git clone https://... failed. YMMV, of course.