This is pretty generic -- isn't that how virtually any decent footprint site with physical infrastructure operates (at least from ~2000 onward)?
I'd be more interested in blog posts about either what they were doing before which was different/worse and got upgraded, or how they decided on specific things.
Their in-house tool gPanel is pretty cool, though (https://githubengineering.com/githubs-metal-cloud/) -- I've seen a lot of people build their own of that in ways which are far less sophisticated, or used one of the ~10 other tools for it. How much effort an org puts into making sure firmware is all the same is probably a good indicator of how competent they are, IMO.
>In memtesting we boot a custom MemTest86 image and monitor it while it completes a full pass. Our custom version of MemTest86 changes the color of the failure message to red which allows us to detect trouble. We’ve hacked together a Ruby script that retrieves a console screenshot via IPMI and checks the color in the image to determine if we’ve hit a failure or not.
Wow. Why is this done this way? Is there no other way to receive hardware memtest info? I would have thought that there was some way to revive machine serial output status in a datacenter.
Based on reference to iDRAC, certainly seems that way. Presumably they 1) don't have a huge amount of hardware (in the sense a commodity hosting provider or cloud provider would) and 2) get a decent deal from Dell.
- No mention of an European data center (data sovereingty and regulation might be something they don't care about, but I deal with it every day).
- No mention whatsoever of virtualization (although there was an older post mentioning a move to Kubernetes for a part of their stack).
I wonder what the economics of taking this to a public cloud provider would look like (full disclosure - I work on Azure), and how much they could do architecturally to benefit from that.
At a scale of Github’s it’s highly unlikely that this would be a cost effective maneuver - see Gitlab’s effort to move off of cloud providers as an example. There comes a point where you get so big that it no longer makes sense to use third party hosting. Dropbox is another example of this.
When you're using full racks of hardware for basically one product, what does a virtualization layer add?
Virtualization is useful for doing many small tasks on a single machine, especially if those tasks are owned by different companies (and you are ok with the tenant separation you get), but if you're using full machines it adds overhead.
1) They probably have a huge amount of data. Moving it without downtime might be an extremely costly and long process.
2) Cloud providers aren't generally that great for IO intensive application, I don't know if they use something like FusionIO cards, but if they do, it might be hard to get analog IO performances on cloud providers (i3 on AWS are quite good at that however).
3) If you can front the cash, manage the logistical aspects and have a relatively constant load and defined core functionality, bare metal is way cheaper.
24 comments
[ 6.2 ms ] story [ 74.5 ms ] threadI'd be more interested in blog posts about either what they were doing before which was different/worse and got upgraded, or how they decided on specific things.
Their in-house tool gPanel is pretty cool, though (https://githubengineering.com/githubs-metal-cloud/) -- I've seen a lot of people build their own of that in ways which are far less sophisticated, or used one of the ~10 other tools for it. How much effort an org puts into making sure firmware is all the same is probably a good indicator of how competent they are, IMO.
https://m.imgur.com/r/cableporn/X1HoY
Wow. Why is this done this way? Is there no other way to receive hardware memtest info? I would have thought that there was some way to revive machine serial output status in a datacenter.
Over serial? Physical serial is rare. SOL would work, but since it is connection-based you can't poll it.
This seems like a simple and reliable method to me.
- No mention of an European data center (data sovereingty and regulation might be something they don't care about, but I deal with it every day).
- No mention whatsoever of virtualization (although there was an older post mentioning a move to Kubernetes for a part of their stack).
I wonder what the economics of taking this to a public cloud provider would look like (full disclosure - I work on Azure), and how much they could do architecturally to benefit from that.
Dropbox was primarily about moving storage off S3 which makes sense given their business.
*: That is, work that does not interact heavily with content
Virtualization is useful for doing many small tasks on a single machine, especially if those tasks are owned by different companies (and you are ok with the tenant separation you get), but if you're using full machines it adds overhead.
2) Cloud providers aren't generally that great for IO intensive application, I don't know if they use something like FusionIO cards, but if they do, it might be hard to get analog IO performances on cloud providers (i3 on AWS are quite good at that however).
3) If you can front the cash, manage the logistical aspects and have a relatively constant load and defined core functionality, bare metal is way cheaper.