I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?
GitLab CEO here. Hardware and hosting are certainly not our core competencies. Hence all the questions in the blog post. And I'm sure we made some wrong assumptions on top of that. But it needs to become a core competency, so we're hiring https://about.gitlab.com/jobs/production-engineer/
I think you're under estimating the number of people required to run your own infrastructure. You need people who can configure networking gear, people swapping out failed NICs/Drives at the datacenter, someone managing vendor relationships, and people doing capacity planning.
I think you could probably get the IO performance you're talking about in your blog post from AWS instances or Google Cloud's local NVMe drives, but if you truly need baremetal, I'd recommend Packet or Softlayer. Don't try to run your own infrastructure or in a year you'll be: https://imgflip.com/i/1fs7it
I love that this question baited your well-known introduction out again. Wasn't there a post at some point that searched for 'GitLab CEO here' on this very site, just for kicks and giggles?
I would challenge your assertion that you need a core competency in bare metal. AWS and GCE are performant enough--you're just not using them correctly. Invest in IaaS expertise and be successful; invest in bare metal at this day and age and live to regret it forever.
You are ruling out providers like OVH because lack of sufficient SLA but then you are replacing it with your own solution that will have no SLA.
Lower your SLA requirements and go with multiple providers like OVH. Make your site work at multi datacenters. At the end of the day your users will be much happier.
Moving to your own hardware will almost certainly improve performance, reduce incidental downtime, and cut costs substantially. Including hiring more engineers, you might expect total costs to be ~40-50% of what you would have spent on cloud-based services over the first 24 months. If your hardware lifecycle is 36-48 months, you will see large savings beyond 24 months.
A few things to watch out for, if your team doesn't have a lot of experience in direct management of dedicated servers/datacenters:
- There is an increased risk of larger/longer very rare outages due to crazy circumstances. Make sure you have a plan to deal with that. (I've had servers hosted at a datacenter that flooded in Hurricane Sandy, and another where an electrical explosion caused a >3 day outage..)
- It's easy to think you'll rely on managed services, but that rarely works out well. It also can become very, very expensive -- possibly more so than cloud-based hosting.
- Specifically, regarding H1, H2: Dedicated hardware is substantially cheaper than cloud-based hosting, but if you rely too much on managed services you negate a large portion of the savings. Consider that most service providers will be both more expensive and less competent than doing it yourselves. Also, having your own team have direct knowledge + their own documentation of the setup will be beneficial.
- I'd recommend budgeting for and ordering some extra parts to keep on hand for replacement, if you are having datacenter ops handle hardware or can have an engineer located relatively close to your datacenters. (A few power supplies, some memory, a couple drives - nothing too crazy.)
- Supermicro's twins systems are great. In the past I've gone with their 1U models vs. 2U to slightly reduce the impact of unit downtime. (Having to take one 2U Twin down affects four nodes. It sounds like you'll have to decide on balancing that against the increased drive capacity, in your case.)
In a long run, probably. Immediately after deploying, unless they hire very experienced, I expect quite a few "never seen before" issues (may not result in publicly visible downtime though). Monitoring for "thermal events", weird and hard to debug issues requiring firmware updates, "bad cable" issues, etc. are not what you have to deal with in the cloud.
Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also. There must be a margin in it since the big players are making money at it.
I'm interested to see what they end up with in the end.
It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.
I've been following the technical discussions around this move, and I'm wondering if you guys looked at making architectural changes to shard your data into more manageable chunks?
Naively it seems like you should be able to reduce your peak filesystem iops by sharding the data at the application layer. That does introduce application complexity, but it might shake out as being less work than the operational complexity of running my own metal.
Of course, easier said than done -- I just didn't spot any discussion of this option, and it seemed like the design choice of having one filesystem served by Ceph was taken for granted.
Then we have to think about redundancy. The simple solution is to have an secondary NFS server and use DRBD. For the shortcomings of that read http://githubengineering.com/introducing-dgit/
The next step is introducing more granular redundancy, failover, and rebalancing. For this you have to be good in distributed computing. This is not something we are now so we rather outsource it to the experts that make CephFS.
The problem of CephFS is that each file need to be tracked. If we would do it ourselves we could do it on the repository level. But we rather reuse a project that many people have already made better than go through the pain of making all the mistakes ourselves. It could be that using CephFS will not solve our latency problems and we have to do application sharing anyway.
That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way.
Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with.
(For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)
Think about it this way: your EE customers probably have the easier bare metal knowledge, but would be willing to pay for you to solve the distributed system problems for them :)
Would love to chat with you to see if a switch to GCP might solve your performance and pricing issues. It's also always interesting to see how GCP vs AWS vs bare metal fairs.
email me at bookman@google.com and I'll be sure to get you in contact with the right people at google.
Have you considered hosting with Packet.net? You'd be on bare metal, thus solving your performance problems, but you'd still be renting by the hour as you are now, and you wouldn't have to deal with buying your own hardware and all the complexity that comes with that.
I looked at their site and they talk about bring your own block, anycast, and IPv6. But I can't find any information about networking speeds. What if we end up needing 40 Gbps between the CephFS servers?
> Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also
I wouldn't be so quick to jump to that conclusion. It's not just the cost of owning and renewing the hardware, it's everything else that comes with it. Designing your network, performance tuning and debugging everything. Suddenly you have a capacity issue, now what b/c you're not likely to have a spare 100 servers racked and ready to go, or be able to spin them up in 2m? Autoscaling?
Companies spend enormous amounts of engineering hours to maintain their on-premise solutions. And sometimes that's fine b/c you have requirements that you can't easily do in the cloud (think of high frequency trading for example). However, once you tally all that up, plus all the value added services you can buy in the cloud (just take a look at the AWS portfolio for example) the price might well be worth it. That's not to say you won't need engineers to help you with cloud stuff, but you'll probably need less and they'll be able to focus on solving a different class of problems for you.
> There must be a margin in it since the big players are making money at it.
From what I've seen the players aren't making (lots of) money on providing compute power. They're basically racing against each other to the bottom. What they're making money on is all the value added services, the rest of the portfolio AWS/Google Cloud Platform/Azure offers.
So in gitlabs case, they have a load that they can monitor and predict. They are looking at 60+ processors, so they can plan to add 10% (6 procs at a time) and grow. They know their load, so the likely need to spin 150% of current capacity isn't something on their plan. I'll give you that there are companies that have erratic loads that are hard to predict, they make sense to place in something that can grow 100% on an email.
Big companies, most of their servers have a pretty stable load, it's unlikely things like internal email, Sharepoint, ERP/MAP systems will take a spike. It's only things like front end order processing that takes the hit.
There are lots of businesses that make sense and some that don't
I like the concept of "racing to the bottom" but they are still making money. But lets take your comment of the Value Added Services other than the ability to spin up capacity. What's the cost to Gitlab to pull this together and keep it running? There is an overflow every day on HN articles about operations monitors, containers, network monitoring. The tools are there, its an effort to glue them together, but then they are there.
So I'll still posit there is cases that the dollars to own are less than the dollars to rent. And I'll agree with your cases of rent because of capacity blowouts is key. The issue, is your ops team savvy enough to figure out what to keep/own, what to rent?
If you are looking for performance, do not get the 8TB drives. In my experience, drives above 5TB do not have good response times. I don't have hard numbers, but I built a 10 disk RAID6 array with 5TB disks and 2TB disks and the 2TB disks were a lot more responsive.
From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys have experienced, is something you want to get right on the first try.
Edit:
N1: Yes, the database servers should be on physically separate enclosures.
D5: Don't add the complexity of PXE booting.
For Network, you need a network engineer.
This is the kind of thing you need to sitdown with a VAR for. Find a datacenter and see who they recommend.
Thanks for commenting! You're right that 2TB disks will have more heads per TB and therefore more IO. We want to increase performance of the 8TB drives with Bcache. If we go to smaller drives we won't be able to fit enough capacity in our common server that has only 3 drives per node. In this case we'll have to go to dedicated file servers, reducing the commonality of our setup. We're using JBOD with Ceph instead of RAID.
If the 8TB drives don't offer the IO we need we'll have to leave part of them empty. I assume that if you fill them with 2TB they should perform equally well. Word on the street is that Google offers Nearline to fill all the capacity they can't use because of latency restrictions for their own applications.
It is an interesting idea to do SSD cache + SSD storage + HDD instead of just SSD cache + HDD. I'm not sure if Ceph allows for this.
It doesn't work like that. 2TB drive will always be faster than an 8TB drive. The amount of data has no effect when compared to the physical attributes of the drive. More platters will increase the response time.
Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.
I'm not sure I understand your question. A 2TB/2TB disk will have the same number of requests as a 2TB/8TB disk, as they both have the same amount of data.
If you are talking physically, 2TB/8TB would theoretically be faster than a 2TB/2TB disk if the performance attributes were the same. But a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially filling a drive would probably be offset by the slower overall drive.
> a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially filling a drive would probably be offset by the slower overall drive.
I'm skeptical here. So your minimum seek time goes up on the 8TB because it's harder to align. But your maximum seek time should drop tremendously because the drive arm only has to move along the outer fifth of the drive. And your throughput should be great because of the doubled linear density combined with using only the outer edge.
I'm not saying that you wouldn't see any performance gain. 2TB on the outer track will be faster than 8TBs on the same 8TB disk, but I'm saying any gains will be lost due the dense nature of the drives.
A quick google search shows that there are marginal gains on the outer track vs the inner, but that is only on sequential workloads. For something like GitLab, the workloads would be anything but.
Ignore the part about where the partition is, then.
1. If I look at 2TB vs. 8TB HGST drives, their seek times are 8ms and 9ms respectively. But if you're only using a quarter of the 8TB drive, the drive arm needs to move less than a quarter as much. Won't that save at least 1ms?
2. The 8TB drive has a lot more data per inch, and it's spinning at the same speed. Once a read or write begins, it's going to finish a lot faster.
> Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.
By "Tiering", is this moving data between different drive types? Or by moving the data to different parts of the platter to optimize for rotational physics?
By Tiering, I'm talking about moving blocks of data from slower to faster or visa verse. If I have 10TB of 10k IOPS storage and 100TB of 1k IOPS storage in a tiered setup, data that is frequently accessed would reside in the 10k IOPS tier while less frequently accessed data would be in the 1k tier. In this case, the blocks of popular repositories would be stored in SSD, while the blocks of your side project that you haven't touched in 4 years would be on the HDD. You still have access to it, it might just take a bit longer to clone.
That would be something I would test. I don't know Ceph, so I would be taking a shot in the dark. I would guess it would not make much of a difference as everything is block level. I, personally, would do 1x PCIe SSD for cache, 1x 2/4TB SSD, and 2x 4TB HDD for each storage node.
Edit. If Ceph is smart enough, it would be aware of the tiers present on the node, and would tier blocks on that node. So a Block on node A will stay on node A.
The fact that you're asking this question on Hacker News leaves little doubt in my mind that you and your team are not prepared for this (running bare metal).
I read the entire article, and while you talked about having a backup system (in a single server, no less!) that can restore your dataset in a reasonable amount of time, you have no capability for disaster recovery. What happens when a plumbing leak in the datacenter causes water to dump into your server racks? How long did it take you to acquire servers, build the racks and get them ready for you to host customer data? Can your business withstand that amount of downtime (days, weeks, months?) and still operate?
These questions are the ones you need to be asking.
In other words, double your budget because you'll need a DR site in another datacenter.
They don't need another datacenter. AWS has had more issues they NYI's datacenters have had in the past year. There are many companies that are based out of a single datacenter that haven't had any major issues.
I recommend testing and evaluating Bcache carefully before planning to use it in a production system. I found it unwieldly, opaque, and difficult to manage when testing on a simple load; Benefits were mild.
I'm sure it has its uses! But I'd emphatically advise anyone to obtakn hands-on experience with Bcache before planning around it! As I'm sure you are already doing or considering of course :)
So, the 8TB drives are fine if you're doing sequential writes, mainly doing write-only workloads, or have a massive buffer in front.
In my experience, the PCIE drives they mention (the DC P3700) are incredible. They're blazing fast (though the 3D X-Point stuff is obviously faster), have big supercaps (in case of power failure), and really low latency for an SSD (150 microseconds or so). They're a pretty suitable alternative to in-memory logbuffers where latency is less crucial. Much cheaper than the equivalent memory too. Having a few of these in front of a RAID 10 made up of 8TB drives will work just fine.
FWIW your experience with a RAID 6 of big disks is unsurprising. Raid 6 is almost always a bad choice nowadays since disks are so big. Terrible recovery times and middling performance.
True about RAID6, I was just commenting on performance. A 2TB will spank an 8TB in any configuration.
I would think that GitLab's workload is mostly random, which would pose a problem for larger drives. The SSDs are a great idea, but I've only seen 8TB drives used when there are 2 to 3 tiers; with 8TB drives being all the way on the bottom. I'm not sure how effective having a single SSD as a cache drive for 24TBs of 8TB disks will be.
C1: The SuperMicro offers blade-server densities without the ridiculous prices that proprietary blade-server chassis come with. It's a pretty good system. Please make sure you have assessed your power requirements correctly and communicated that to your datacenter.
If you were larger, Open Compute Platform might be a way to go. Maybe next generation.
Yeah, link aggregation doesn't work how they think it does. And not having a separate network for Ceph is going to bite them in the arse.
GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).
Seems this is one of the goals of the article: "We're hiring producton engineers and if you're spotting mistakes in this post we would love to talk to you (and if you didn't spot many mistakes but think you can help us we also want to talk to you)".
They'd save themselves a whole lot of time, effort, and money if they looked at partitioning their data storage instead of plowing ahead with Ceph. They have customers with individual repositories. There is no need to have one massive filesystem / blast radius.
I don't know much about this stuff, but won't that stop working if they ever decide to expand to multiple geographical sites, to reduce latency to customers in different locations? In that case, different sites can receive requests for the same repositories, and ideally each site would be able to provide read only access without synchronization, with some smarts for maintaining caches, deciding which site should 'own' each file, etc. They could roll their own logic for that, but doesn't that pretty much exactly describe the job of a distributed filesystem? So they'd end up wanting Ceph anyway, so they may as well get experience with it now.
They said very little about how they think link aggregation works. Just that they can send packets on both and continue working with only one port. That's basically the definition of link aggregation. So what's wrong about the post?
Ceph should get a separate network which is only used for re-replication in case something happens. Consider a node goes down.
Another thing I might recommend is a third network (just a simple 1GB and a quality switch) for consensus. Re-replication can max the network out and further cause consensus fails, causes more re-replication winding down everything ... If that's not possible, add firewall rules to prioritize all consensus related ports high.
Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be < 200ms. Russia is 40-50ms and Asia should be between 100ms and 200ms. Australia will probably have the worst ping with around 200-300ms.
Just as a suggestion since you seem to be so certain about the location. I basically know every DC in Frankfurt, so if you need any help or info in that regard feel free to contact me :-)
Frankfurt surely has a great IX. Even if Frankfurt would make everyone better off (which I'm not sure about) there is another problem. People in the US are used to lower latencies because most SaaS services are hosted there.
Some financial institutions require all sensitive* data to be stored/hosted in the same country/state(archaic yes).
It's real hard to actually define sensitive data but the IP* in some code a quant wrote can totally be considered a trade secret by a non technical person
Please don't get me started on how stupid I think it is that people consider code to be IP
Definitely not archaic. If the FISA courts taught you anything, it should be that the country your things are living in determines which entities can tap your traffic/hardware.
It really depends on where most of their high-paying customers are. Which is, very likely, in the US/Canada and some of EU. So US East Coast makes sense.
To that end I'd be cautious of facilities in NJ. If you are dead-set on the US, Ashburn, or even better further inland such as Chicago or Dallas, may be better suitable if you do not plan on having redundancy in another facility.
Frankfurt may be a better option solely on the basis that it is more "environmentally" stable (in terms of events that may disrupt the operation of the facility).
If you can only deploy one place in the world, east coast US is generally the right fit (if you have typically distributed user traffic, of course). You can cover western Europe and the major population centers of eastern North America, and still have reasonable performance in western North America as well. While it's increasingly less true over time, keep in mind that this is the place that other countries all prioritized building connectivity to, because historically this is where early commercial websites lived.
Of course there's the issue of potential legalities for your business, but don't kid yourself that you're safe from prying eyes by deploying in a particular country.
The next step (and particularly important for a business like GitLab) is to land a second site in either western Europe or west coast US. Honestly you should be thinking about this right away, and look to sign leases on both spaces simultaneously with a 3-mo delay built in for the second site. This should help you negotiate price as well, if you're able to go with the same dc provider, but be aware that that itself is a single point of failure as well. Make absolutely sure you negotiate your MSA to the n-th degree, get a good SLA, etc. You can still get burned, but do your legal due diligence now because you won't have a chance to change terms later.
Then continue to optimize by having multiple sites per-region (so that failover doesn't involve a big performance hit), adding APAC / AUNZ regions, and so forth. For a service like GitLab, I wouldn't think that time to sync the repo is hyper important, but responsiveness of the web interface is fairly key. So that may lead to a hub/spoke design where there are a few larger sites storing the bulk of the data and more small sites to handle metadata and such to present the web views.
That's all years down the road though. For a first pass, I can't see a problem with northern Virginia as the first site.
There are basically a shitload of datacenters in Frankfurt so I won't talk about every DC, but more of a general overview. It doesn't matter what your budget is or what your requirements are, you will find a DC in Frankfurt.
You basically have the big brands like Equinix (7 facilities in Frankfurt), Interxion (They are currently building their 11th facility in Frankfurt), Telecity (now part of Equinix) but also some local ones like e-Shelter, First Colo or Accelerated. There is also a DC only meters away from the Interxion campus that is called Interwerk and is basically the cheapest DC in Frankfurt, but it has an awesome price/performance ratio and you have access to all peering/transit options of Interxion (via CWDM) and can get a 1/1 rack for under 200 EUR/mo. I have been in that DC for several years and never had a single issue, so if you don't rely on any certificates this is a cheap option.
I was also colocating in the First Colo DC for some time, they are also in the locality around the Interxion campus. It's a pretty small DC but it is more premium than the Interwerk and also is kinda cheap. I personally wouldn't go with Accelerated, since they had multiple issues in the past.
For the big brands I would definitely go with Interxion, they are great, have all the certificates and are premium while not going crazy with their pricing like Equinix does. DigitalOcean, Twitch, etc. they are all in one of the many Interxion facilities in Frankfurt. If Price isn't a concern I would probably go with Equinix FRA5.
DE-CIX is present in around 7 facilities, 3 of them are Interxion and I think 2 Equinix.
If the customer base has lots of US enterprise customers, that choice will cost you money. Ex-US data residency is an issue for many compliance standards.
Isn't that the same vice versa? Sure, if the majority of the customers sit in the US _and_ care about that: Fine. Otherwise I felt that the online community rather likes to avoid the US for data if that is an option.
(I'm from Germany, but I couldn't care less about the country per se or GitLab being co-located in Frankfurt)
It's a problem IMO that they are trying to pick 1 site. Geodiversity is necessary for any serious enterprise, and given that, I would suggest 1 site on the US west coast to please users in SF/LA/SEA (and most of Asia) plus 1 site somewhere in Europe (AMS, FRA, etc). Once you can afford 3 sites, add Asia or US east, depending on where the users are.
It seems based on this post that all the hardware is going to end up in one datacenter. that seems like a big risk if that datacenter ends up having issues.
Perhaps look into how to have the nodes split between 2 or 3 different datacenters?
We've considered having part of the servers in a different location. But it is a big risk to see if Ceph can handle the latency. We're also considering using GitLab Geo to have a secondary installation. It seems that data centers can have intermittent network or power issues but they are less likely to go down for days (for example Backblaze is in one DC). At some point we'll likely have multiple datacenters but our first focus is to make GitLab.com fast as soon as possible.
You will have to go out of Ceph and build a middleware to route the git operations on the right shard. Take a look at libgit2[0] for that.
It is pretty easy to build such middleware both for the git over ssh (a simple script performing a lookup of where the shard is and then you connect to shard to operate there) and just a little bit more for the http part. At the webapp level, you will have a kind of RPC to run the git related operations which will connect to the right shard to run the operations.
When you use Ceph you are basically running a huge FS at the full scale of your GitLab installation, but practically, you have many independent datasets within your GitLab installation and you do not need to pay the cost for the global consistency of Ceph. You have many islands of data.
Do you have a disaster recovery plan that starts with "A meteor has destroyed our primary data center."?
I do; that's my default scenario. If you can survive that, you can survive all sorts of smaller issues like network congestion, data center power problems, grid power problems, and zombie plagues (or flu, which is more likely.)
Depends on what you mean by 'survive'. I'd call a backup in google nearline sufficient for the meteor scenario, but that's going to be very slow and unpleasant to depend on for milder problems.
It's not sufficient. How quickly could you procure new hardware, install that in a datacenter, make it fully functional, and restore your backups? The answer is likely weeks/months. Could your business survive being offline that long? It sounds unlikely.
In an emergency you don't need new hardware. You can get cloud servers in minutes. If people have been practicing restores then it should not take particularly long to get the containers working again. A couple days to get things working-ish. That should be survivable while everyone focuses on the news coverage of the meteor.
But that's a true emergency situation. Don't go offline for multiple days for something that's reasonably likely to happen.
I have a Cisco c220 m4 (http://www.cisco.com/c/dam/en/us/products/collateral/servers...). I was surprised at the complexity of the configuration options. For example, it has 24 memory slots, but if you fill them memory speed is 2133mhz, whereas if you only use 16 memory slots, max memory speed is 2400mhz.
I've ordered a pair of Intel 750 Series 1.2TB NVMe SSD's for it, can't wait to try that out. Still waiting for the SSD drives to arrive.
N1/N2: I think you are making a mistake by concentrating on the advantage of having a single node type. Databases really aren't like other systems.
SuperMicro has a 4 node system available in which each node gets 6 disks, 2028-TP-xx-yyy. Get two of these, populate 2 nodes on each as your databases, and you can grow into the other spaces later. Run your databases on all-SSD; store backups on spinners elsewhere.
Having two node types is not a calamity compared to having just one.
"Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters."
I'm getting the same feeling I get when I'm watching those "Hold my beer and watch this..." videos.
From my vantage point they look to have 0 experience in building and running infrastructure... and asking advice on HN. They might ask well post a Ask Slashdot thread if they want armchair advice. Genuinely, I think they've crunched some numbers and think they can run their stuff cheaper and faster in-house... but probably underestimated the human-experience angle.
For just 10-20 physical servers, this is going to be either extremely expensive (if they hire right) or extremely painful (if they don't).
Then prototype first. Rent a server or two somewhere for 3-6 months and run a shadow first. Once you're confident that you understand all the "other 80%" stuff that is involved running your own infrastructure and don't lose data, then think about doing it yourself.
A service providers' biggest responsibilities to its customers are security, durability, availability and performance -- in that order. You guys are vastly underestimating the complexity involved in getting first 3 right.
D5: You want a local boot drive, and you want it to fall back to PXE booting if the local drive is unavailable. Your PXE image should default to the last known working image, and have a boot-time menu with options for a rescue image and an installer for your distribution of choice.
It's better to have it the other way around. Attempt to boot off the network and fall back to local drive. That will allow you to reimage a node without having to fiddle with the BIOS. For regular boot there should be no image configured. Network boot will fail and the node will boot off disk. However if an install image is configured for that node, you can reimage it at will. The install image should reset to having no image for that node, once it's done.
You'll need to maintain an association between - dns, ip, mac address, ssh keys etc.
Hardware break-fix workflow is usually ignored by most production engineers. You'll be doing that a lot. You want to get your hw back into use as fast as possible.
Have you thought about how many spares (CPU, RAM, disks) you'll have to keep at your datacenters ?
> We want to dual bound the network connections to increase performance and reliability. This will allow us to take routers out of service during low traffic times, for example to restart them after a software upgrade.
does not really agree with
> Each of the two physical network connections will connect to a different top of rack router.
Sure, you can do it with something like MLAG, but that's really just moving your SPOF to somewhere else (the router software running MLAG). Router software being super buggy, I wouldn't rely on MLAG being up at all times.
> N1 Which router should we purchase?
Pick your favorite. For what you're looking for here, everything is largely using the same silicon (broadcom chipsets).
> N2 How do we interconnect the routers while keeping the network simple and fast?
Don't fall into the trap of extending vlans everywhere. You should definitely be routing (not switching) between different routers. You can read through http://blog.ipspace.net/ for some info on layer 3 only datacenter networks.
You'd want to use something like OSPF or BGP between routers.
> N3 Should we have a separate network for Ceph traffic?
Yes, if you want your Ceph cluster to remain usable during rebuilds. Ceph will peg the internal network during any sort of rebuild event.
> N4 Do we need an SDN compatible router or can we purchase something more affordable?
You probably don't need SDN unless you actually have a SDN use case in mind. I'd bet you can get away with simpler gear.
> N5 What router should we use for the management network?
Doesn't really matter, gigabit routers are pretty robust/cheap/similar. I'd suggest same vendor as you go for whatever your public network routers.
Also, consider another standalone network for IPMI. I can tell you that the Supermicro IPMI controllers are significantly more reliable if you use the dedicated IPMI ports and isolate them. You can use a shitty 100mbit switches for this, the IPMI controllers don't support anything higher.
> D5 Is it a good idea to have a boot drive or should we use PXE boot every time it starts?
PXE booting at every boot is cool, but can end up sucking up a lot of time. If you have not already designed your systems to do this, and have experience with PXE, then don't.
> The default rack height seems to be 45U nowadays (42U used to be the standard).
You may not have accounted for PDUs here. Some racks will support 'zero-U' PDUs, but you'd need to confirm this before moving on.
> H3 How can we minimize installation costs? Should we ask to configure the servers to PXE boot?
Assume remote hands is dumb. Provide stupidly detailed instructions for them. Server hardware will PXE by default, so that's not really a concern. IPMI controllers come up via DHCP too, so once you've got access to those you shouldn't need remote hands anymore.
> D2 Should we use Bcache to improve latency on on the Ceph OSD servers with SSD?
Did you consider just putting your Ceph journals on the SSD? That's a lot more standard config then somehow using bcache with OSD drives.
>Each of the two physical network connections will connect >to a different top of rack router.
>
>Sure, you can do it with something like MLAG, but that's >really just moving your SPOF to somewhere else (the router >software running MLAG). Router software being super buggy, >I wouldn't rely on MLAG being up at all times.
I would strongly consider doing this via pure L3 routing. This is a scale at which the benefits of L2 fabric switching vs L3 multihomed routing (yes, routing decisions on every node) begin to be interesting decisions.
We're already planning a separate router for the management network ("Apart from those routers we'll have a separate router for a 1Gbps management network.").
If you're doing this right, the management network will not actually be accessible via the normal operating system. Most IPMI controllers support sharing a normal nic and management (meaning both the IPMI controller and host OS can access it), but I wouldn't recommend doing this.
>> N1 Which router should we purchase?
>Pick your favorite. For what you're looking for here, everything is largely using the same silicon (broadcom chipsets).
For switches, yes. Many of the switches share the same merchant silicon Broadcom Trident-II, Tomahawk, et al., however there are switches like the Juniper EX9200 which isn't baed on merchant silicon. Routers (N1) are also not typically based on merchant silicon (Juniper Trio-3D for example).
For such a big expenditure, some prototyping first? Maybe buy a mainboard or two and some CPUs/HDDs/SSDs and benchmark them on your specific workloads. Also look into using something like bcache if going all-SSDs is too expensive.
We're in a rush because we need GitLab.com to get fast now. We might shoot ourselves in the foot by not prototyping first but we're taking the risk. We're hiring more consultants to help us with the move.
Bcache is mentioned in the article under the disk header.
If you're committed to having a robust architecture (this may not be financially viable immediately) you should study the mistakes that Github have made, e.g. https://news.ycombinator.com/item?id=11029898
Geo-redundancy seems like a luxury, until your entire site comes down due to a datacenter-level outage. (E.g. the power goes down, or someone cuts the internet lines when doing construction work on the street outside).
(This is one of the things that is much easier to achieve with a cloud-native architecture).
This is definitely what perked my ears up when they mentioned the US East Coast, especially when you consider the risk that a natural disaster might take out the facility.
Exactly this. I don't think that if you take into account __all__ parameters bare metal is cheaper. My other problem is that they are moving from cloud to bare metal because of performance while using a bunch of software that are notoriously slow and wasteful. I would optimise the hell out of my stack before commit to a change like this. Building your own racks does not deliver business value and it is extremely error prone process (been there, done that). There are a lot of vendors where the RMA process is a nightmare. We will see how it turns out for Gitlab.
The RMA process is pretty much a moot point at this scale. It simply doesn't make sense to buy servers with large warranties. Save the money, stock spares, and when a component dies replace it. In the long run it will come out to cost a lot less. I'd much rather pay much less knowing a component may fail here and there and you buy a new one.
As far as moving from cloud to bare metal, another thing to take into consideration (that this was replied to), if you don't architect your AWS (or other cloud solution) to take advantage of multiple geographic regions, the cloud won't benefit you.
I 100% agree that there should be more than one region deployed for this service. As others have said, all it takes is 1 event and the site will be down for days to weeks to months (It may not happen often, but when it does, you go out of business). The size and complexity of this infrastructure will make it nearly impossible to reproduce on short order in a new facility. If I were the lead on this I would have either active / active sites, or an active / replica site.
I would also have both local (fast restores), and off-site backups of all data. A replica site protects against site failure not data loss and point-in-time recovery.
"As far as moving from cloud to bare metal, another thing to take into consideration (that this was replied to), if you don't architect your AWS (or other cloud solution) to take advantage of multiple geographic regions, the cloud won't benefit you."
Yep, this is why scaling starts with scalable distributed design. We were moving a fairly large logging stack from NFS to S3 once, for the same reason Gitlab is trying to move to bare metal now. Moving off cloud was not an option, moving to a TCO efficient service was. NFS did not scale and there was the latency problem. I think moving to bare metal cannot help with scale as much as a good architecture can. We will see how deep the datacenter hole goes. :)
Agreed. Application Architecture is far more important than Cloud vs. Bare Metal. It is just easier and more cost effective to through more bare metal hardware at the problem than it is cloud instances. For some this does make bare metal the better option.
To add to my previous comment though, AWS (and cloud in general) tends to make much more sense if you are utilizing their features and services (Such as Amazon RDS, SQS, etc.), and if you aren't using these services I can absolutely guarantee I can deliver a much lower TCO on bare metal than AWS. (Which is why I offered to consult for them) I see this all the time. Company moves from bare metal to AWS as bare metal is getting expensive, then they quickly find out AWS can't deliver the performance they need without massive scale (because they aren't using a proper salable distributed design and can't afford to re-architect their platform)
There's no reason you can't have a hot cloud site, and operate in lipo mode (or even scale it up temporarily) if you lose your DC. Best of both worlds.
I'm surprised to see that GitLab is using Unicorn. Isn't Unicorn grossly inefficient, because each of the worker processes can only handle one request at a time. Are web application processes actually CPU-bound these days?
I don't know much about the Ruby web server landscape, but might Puma (http://puma.io/) be better?
For "grossly inefficient" I imagine it depends. If you're loading most of the app prefork then the memory overhead for more processes is pretty low. (Disclaimer: Dunno much about ruby deployments)
But the number of Unicorn processes that GitLab runs is "virtual cores + 1". It seems to me that this only makes sense if web application request handling is actually CPU bound. Maybe it is if you have a really good caching implementation and hardly ever have to query the DB.
Puma is something we want to eventually run, but right now it's not really a priority. Instead we're focusing on response timings and memory usage. See the following links for more information:
H1: I don't see a discussion of getting an ASN and doing BGP with a number of upstreams. You say you want a carrier-neutral facility, but that won't buy you much unless you have your own AS.
@sytse: If this (BGP + ASN + /24 IPv4 + /n IPv6) is your plan, I'd encourage you to get started on the process now.
Go ahead and apply for your ASN now and an IPv6 allocation. Then start working on the paperwork for an IPv4 allocation. Because there is no more IPv4 to allocate you'll have to go through the auction process and then the subsequent transfer process.
You'll easily be able to find a provider that can give you a /24 if you buy transit from them, but you don't wanna go through the trouble of renumbering into your own IP space if you can avoid it.
The raw performance benefits of bare metal vs cloud are incredible, but why does that necessarily mean building & maintaining your own hardware when you can lease (or work out whatever financing you want, but still let the hosting company maintain a lot of the responsibility for HW)? And besides financing, taking on all the HW maint? I'm not sure your needs are so unique as to require custom hardware.
You're talking about only 64 nodes right now. Your storage and IOPS requirements are not huge. A lot of mid-size hosting companies will give you fantastic service at the 10-1000 servers range. If I were you I'd talk to someone like https://www.m5hosting.com/ (note: happy dedicated server customer for many years -- and I'm sure there's similar scale operations on east coast if that's really what you need) who have experience running rock solid hosting for ~100s of dedicated servers per customer.
I suspect you may just be able to get your 5-10X cost/month improvement (and bare metal performance gains) without having to take on the financing and hardware bits yourself.
To be sure, I've seen benchmarks (can look up later) that found a 10x+ performance improvement over and above the cost savings, leading to I guess a 100x price/performance improvement.
Right now that makes sense yes.
What about 5 years from now when it doesn't?
This way they get to start building a team and culture to support that kind of infrastructure and wean the baby teeth on nice soft furniture.
I wonder if they could go with a hosting provider, hire a sysadmin, then second them to the hosting provider to learn how it's done, in return for a credit against their costs.
In fact, i wonder if that's a business model for hosting providers. Kind of a sysadmin incubator.
When I looked at how expensive a rented dedicated server was compared to AWS I expected a big difference, but I was still suprised how cheap you can get dedicated servers today. Not worrying about actually owning the hardware is nice, and you still get vastly more hardware for the same price as in the cloud.
There are many advantages to AWS and similar services, but if you can't really take advantage of all the goodies because e.g. you also need to provide a local version of your software (which is the case for Gitlab, as far as I understand), renting dedicated servers is an order of magnitude cheaper.
We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us.
We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=13153455 that this is the time to build a team that can handle this makes sense to me too.
Been a softlayer customer for 4 years now. Their network is pretty awesome. When hosted in the same datacenter it's sub-ms response time always. If there is an issue they get right on it. You can even ask them to host the stuff in the same rack to get even better response time.
We are also a SL customer -- 4-figures of hosts with them. We have had networking problems in the past (latency and loss far higher than I would expect to see in a well-provisioned DC) and talked to them about it. It ended up being contention with another customer, it got fixed, and our network performance has been great since.
I would encourage you to look at what you can get without trying to do your own colo. You're not at the scale where you should be thinking about that.
Few will offer 40Gbps without charging a pretty penny, generally the jump will go from 10Gbps to 100Gbps but not until it becomes cost-effective (and not anytime soon).
That said, SoftLayer does provide 20Gbps access within a private rack and 20Gbps access to the public network.
Most large scale operations are (or soon will be) deploying 25GE and/or 50GE in place of 10Gbps Ethernet. 100GE to each node is unnecessary for most workloads & more importantly it's obscenely expensive and likely to remain so for at least 3 more years.
Architecturally I would honestly wonder why you need that. Git, with its heavy reliance on immutable objects, should replicate very well. I would expect to be NIC-bound, but I would also expect it to horizontally scale reasonably well. Storage is obviously a concern -- you will have to make sure you don't end up needing a full copy of your data on every node -- but there are well known solutions to this problem that account for hot keys/objects well enough for you to get really really far. Even reaching as far back as the BigTable paper there is valuable stuff to look at and you can obviously look at Cassandra to see how the OSS world has tackled that problem.
in the past they've spread servers into different rooms that had interconnects that challenged a team I was on to spend extra time writing compression code. is this better now?
None of that would convince me to lift a finger. Not unless you put dollar amounts on one vs the other. Version control isn't something I even care about performance.
Version control is the primary tool your developers are using at any org doing software development. It's performance is key to ensure devs aren't wasting their time waiting on version control.
Performance along with reliability are the most important metrics for someone providing VCS as a service.
IDE and local dev environment are primary. I'm currently saddled with a proprietary SVN-based enterprisey monstrosity that costs me 10 minutes a day at the absolute worst. It sucks, but we can easily live with it.
I do. We had horrible download\merge\resolve times at my place because of some poor choices (storing many copies of large binary files across all branches in perforce) and it made a huge improvement in development time when those issues were resolved. Granted, this was an extreme case, but VC performance is not irrelevant to the cost of software development.
For server hardware, the Supermicro 2U Twins are a reasonable choice, but I prefer their 4U FatTwin chassis. The engineering quality is a little better IMO, and the cost increase isn't too big. Absolutely do not buy their 1U Twin systems, they are hot garbage.
The FatTwin chassis has similar density, and can support either 1U half width or 2U half width systems in a particular chassis. Typically I use 1U's for app / web servers and 2U's for lower end database / storage. Separate 2U servers for higher end database and 4U servers for bulk storage.
HPE has the Apollo 2600 / XL170r 2U chassis, which I think is somewhat inferior but still a reasonable choice. Dell sells the same thing as the C6300. I really prefer a 4U chassis though from a cooling and power supply perspective, but Dell or HPE may have a better international story for you.
You absolutely should not buy the 2630v4 CPU. I say that because the lower-end Intel CPUs do not support maximum throughput for memory and QPI. The 2630v4 is 8.0GT/s QPI and 2133MHz DDR4. A better solution low-watt part is 2650Lv4 (9.6GT/s QPI and 2400MHz DDR4). I have a guide that I created (and use myself) to determine comparative $/performance of Intel CPUs based on SPEC numbers [1]. If you can go up to 105W the 2660v4 is probably your best bet. Presuming that you're targeting 12-15kW per rack, a 105W part should allow you to deploy between 60-80 hosts per rack.
Also, don't use a W-series CPU that draws 160W. That's crazy power draw per-socket. If you want a super high-end CPU for your database server, I suggest a 2698v4 -- but normally I would go with 2680v4 or 2683v4 depending on the part cost.
In terms of hard drives, absolutely you should specify HGST over Seagate. At some point you may want to dual source this, but if you're only going with one vendor right now HGST is the best option. He8 or He10 8TB are your best bet in terms of cost and availability right now, although start thinking about He10 10T drives. The newly announced He12 drives shouldn't be on your radar until Q2 2017. Stock spares, maybe 2-3% of your total drives deployed, but at least 5-6 drives per site. You don't want to get caught out if there is a supply shortage when you need it most. Your business depends on ready access to considerable quantities of storage.
The P-series Intel SSDs are probably not going to be cost effective for your use case. But they are considerably better in terms of IOPS and remove the need for an HBA or RAID controller. Consider a Supermicro 2U chassis with 2.5" NVMe support, which will allow you to go considerably denser than the PCIe form factor. However, I think it's too early to go with NVMe unless you truly need beyond-SSD performance.
Don't PXE boot every time you boot. This creates a single point of failure (even if your architecture is redundant), and you will regret this at some point. However, DO PXE boot to install the OS.
Don't use 128GB DIMMs. They are not cost effective today.
There is only one solution for database scaling: shard. You'll either shard it today, or you'll shard it tomorrow when the problem is much harder. Scale up each host to what is easily achievable with today's hardware, and if push comes to shove retrofit to get over a hump that arises in the future, but know that you MUST shard in order to keep up with demand. Scaling up simply does not work.
There's a lot more to say, but without doing my job for free in a HN comment, the best advice I can offer is:
1. Simplify what you hope to accomplish in the first round. This is a lot to achieve at once. I think you'll have a hard time achieving the fanciness you want from a software perspective while also forklifting the entire stack over to physical hardware. It's perfectly fine to have something be good enough for now.
2. Find people who have done this before and get their advice. Find a VAR you can trust.
3. Plan, plan, plan, plan. Don't commit until you ...
It's hard to say with absolute certainty, but I think 2-3 years is a reasonable guess. 64GB DIMMs have only recently become semi-reasonable, and I still use a lot of 16GB or 32GB DIMMs on smaller deployments.
Basically whatever the top-of-the-line DIMM option may be (and this applies for CPUs and HDDs and other stuff too), you want to avoid being in a situation where you HAVE to use it. Vendors price these parts accordingly: you pay a premium for top-of-the-line because you must have it. If you can avoid that, do so.
The 2630v4 row was hidden because of the QPI / memory concerns I mentioned (all such models were hidden on the sheet because I don't consider them for my purchases). I re-exposed those rows just now. FYI, the v4 rows are towards the bottom, under Broadwell. I've been tracking this stuff for a while.
The $/perf of 2630v4 is pretty decent ($2.24), but I would personally be leery for the reasons I mentioned. That said, I have used it for bulk storage servers, where CPU performance was not that important. So it's not like it will blow up your machine or something.
To obtain the perf number, I'm averaging single core and multi core fp and int SPEC numbers. If your workload isn't heavily parallelizable, that might not make sense. I'm not too worried about single core performance myself these days and have been tempted to remove it entirely.
One other thing I forgot to mention: v5 Xeon CPUs will be shipping in quantity early 2017, so you may want to consider holding off and looking for better deals on v4 CPUs then.
Likewise, you might be able to obtain a better deal today on v3 CPUs, particularly if you aren't using a large vendor like Dell or HPE. All of my pricing is list (I don't pay these prices), so the math changes significantly if you can get a disproportionate deal on a particular model. I use it as a place to start the conversation with my VAR, and then go with what makes sense in the market right now.
Ive done MMO setups that are more simple. You do not need this large cluster. Spread it out with nodes plus backup nodes near customers. Separate customers. Scaling will be as easy as adding more servers one case at a time.
Z1: the word "monitoring" does not appear in this document.
You will need to monitor:
- ping
- latency
- temperatures
- cpu utilization
- ram utilization
- disk utilization
- disk health
- context switches
- IP addresses assigned and reaching expected MACs
- appropriate ports open and listening
- appropriate responses
- time to execute queries
- processes running
- process health
- at least something for every bit of infrastructure
once you collect that information, you need to record it,
graph it, display it, recognize non-normal conditions, alert on those, page for appropriate alerts, and figure out who answers the pagers and when.
On January 9 our Prometheus lead will join us (who was very valuable already helping with this research behind this blog post) and we're hiring Prometheus engineers https://about.gitlab.com/jobs/prometheus-engineer/
In the short term we might send our monitoring and logs to our existing cloud based servers. In the long term we'll host them on our own Kubernetes cluster.
Gitlab package includes too many things! I remember Gitlab sponsors a guy to do proper packaging to include Gitlab in debian last year, not sure what is the status.
Since GitLab is heavily betting on Prometheus now, my guess is that it will be used to cover almost all of those points and more. Yep, lots of work though.
Disclaimer: Prometheus co-founder and just started as a Prometheus contractor for GitLab.
EDIT: ah, didn't reload the page to see that sytse had already responded with this :)
Since hosting git repositories is core to your business, you should take the time to do it right (https://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/fi...) instead of using vanilla git and relying on a magic filesystems and vertical scaling to solve your issues.
The presentation you linked is something we've considered. But it is build on top of Google's distributed filesystem. So we consider our move to Ceph a first step in that journey.
A bit confused why you are trying to roll your own storage solution. Ceph is great for a lot of applications, but I am not sure it really fits the bill for what you describe. Especially when you indicate you are going to use spinning disk behind it. Have you looked at any of the storage arrays on the market? Your TCO is likely to be much lower and your performance/resiliency much higher if you buy something with $100's of millions of R&D behind it rather than all the hours and costs of rolling your own. Not saying its impossible to make it work, but it just sounds like something that will be a PITA going forward.
I think a storage appliance (NetApp, etc.) makes a lot of sense in the short term. The TCO is lower since we'll spend a lot of time making Ceph work.
In the longer term the storage appliance will lock us in and will get very expensive. I've heard pretty bad stories of companies betting on it. Especially with many small files like us (IOPS heavy).
And one goal of GitLab.com is to gain experience that we can reuse at our customers. Most of our customers use a storage appliance now but are interested in switching to something open source.
You need at least two nodes that do DNS, DHCP, NTP, and other miscellaneous services that you absolutely want to have but do not seem to have mentioned. You want them to be permanently assigned, so that you never have to search for them, and you want them to fail over each service as needed, preferably both operating at once. Three nodes would be better. Consider doing some basic monitoring on these nodes, too, as a backup to your main monitoring system.
Absolutely agree. Typically I deploy about 4-6 "tools" hosts in a site. From a computational standpoint, you could make do with fewer hosts, but there are some things that I prefer to separate out:
1&2. Authoritative DNS (internal)
1&2. NTP
1&2. Caching DNS resolver
3&4. Outbound HTTP proxy (if necessary)
3&4. PXE / installer / dhcp
3&4. Local software mirror (apt / yum / etc.)
5&6. SSH jump hosts.
Or something like that.
Make sure the second host is not just in a separate chassis from the first host, but also in a separate rack.
For external authoritative DNS, don't do it yourself -- pay for someone to run that for you (Route53, ns1, etc.). For e-mail, if you can possibly not deal with it then don't -- use Mailchimp or something.
Sidenote:
We are running coredns.io in production as authoritative internal DNS and as hidden master with NOTIFY to a secondary DNS provider (currently DNSmadeEasy).
The DNS records for the internal records are done using the kubernetes middleware (basically serving the service records).
The external records are pulled in from a git repository hosting our zones as bind files. If need be zones are split into subzones per team/project. Same permission system as our code via MRs using Gitlab.
Our recommendation is build on open standards (BIND, AXFR) and use services on top of these.
I agree that using an external mail provider is usually a good idea. It mostly is your fallback communication channel and is usually easy to switch (doing replication to an offsite mail storage needs to be done to make switching easy/possible/fast). MX records \o/
326 comments
[ 5.1 ms ] story [ 93.7 ms ] threadImpressive.
I think you could probably get the IO performance you're talking about in your blog post from AWS instances or Google Cloud's local NVMe drives, but if you truly need baremetal, I'd recommend Packet or Softlayer. Don't try to run your own infrastructure or in a year you'll be: https://imgflip.com/i/1fs7it
Lower your SLA requirements and go with multiple providers like OVH. Make your site work at multi datacenters. At the end of the day your users will be much happier.
My 2 cents.
A few things to watch out for, if your team doesn't have a lot of experience in direct management of dedicated servers/datacenters:
- There is an increased risk of larger/longer very rare outages due to crazy circumstances. Make sure you have a plan to deal with that. (I've had servers hosted at a datacenter that flooded in Hurricane Sandy, and another where an electrical explosion caused a >3 day outage..)
- It's easy to think you'll rely on managed services, but that rarely works out well. It also can become very, very expensive -- possibly more so than cloud-based hosting.
- Specifically, regarding H1, H2: Dedicated hardware is substantially cheaper than cloud-based hosting, but if you rely too much on managed services you negate a large portion of the savings. Consider that most service providers will be both more expensive and less competent than doing it yourselves. Also, having your own team have direct knowledge + their own documentation of the setup will be beneficial.
- I'd recommend budgeting for and ordering some extra parts to keep on hand for replacement, if you are having datacenter ops handle hardware or can have an engineer located relatively close to your datacenters. (A few power supplies, some memory, a couple drives - nothing too crazy.)
- Supermicro's twins systems are great. In the past I've gone with their 1U models vs. 2U to slightly reduce the impact of unit downtime. (Having to take one 2U Twin down affects four nodes. It sounds like you'll have to decide on balancing that against the increased drive capacity, in your case.)
In a long run, probably. Immediately after deploying, unless they hire very experienced, I expect quite a few "never seen before" issues (may not result in publicly visible downtime though). Monitoring for "thermal events", weird and hard to debug issues requiring firmware updates, "bad cable" issues, etc. are not what you have to deal with in the cloud.
I'm interested to see what they end up with in the end.
It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.
Naively it seems like you should be able to reduce your peak filesystem iops by sharding the data at the application layer. That does introduce application complexity, but it might shake out as being less work than the operational complexity of running my own metal.
Of course, easier said than done -- I just didn't spot any discussion of this option, and it seemed like the design choice of having one filesystem served by Ceph was taken for granted.
Then we have to think about redundancy. The simple solution is to have an secondary NFS server and use DRBD. For the shortcomings of that read http://githubengineering.com/introducing-dgit/
The next step is introducing more granular redundancy, failover, and rebalancing. For this you have to be good in distributed computing. This is not something we are now so we rather outsource it to the experts that make CephFS.
The problem of CephFS is that each file need to be tracked. If we would do it ourselves we could do it on the repository level. But we rather reuse a project that many people have already made better than go through the pain of making all the mistakes ourselves. It could be that using CephFS will not solve our latency problems and we have to do application sharing anyway.
Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with.
(For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)
Etcd is indeed very interesting. I'm thinking about using it for active active replication in https://gitlab.com/gitlab-org/gitlab-ee/issues/1381
email me at bookman@google.com and I'll be sure to get you in contact with the right people at google.
I wouldn't be so quick to jump to that conclusion. It's not just the cost of owning and renewing the hardware, it's everything else that comes with it. Designing your network, performance tuning and debugging everything. Suddenly you have a capacity issue, now what b/c you're not likely to have a spare 100 servers racked and ready to go, or be able to spin them up in 2m? Autoscaling?
Companies spend enormous amounts of engineering hours to maintain their on-premise solutions. And sometimes that's fine b/c you have requirements that you can't easily do in the cloud (think of high frequency trading for example). However, once you tally all that up, plus all the value added services you can buy in the cloud (just take a look at the AWS portfolio for example) the price might well be worth it. That's not to say you won't need engineers to help you with cloud stuff, but you'll probably need less and they'll be able to focus on solving a different class of problems for you.
> There must be a margin in it since the big players are making money at it.
From what I've seen the players aren't making (lots of) money on providing compute power. They're basically racing against each other to the bottom. What they're making money on is all the value added services, the rest of the portfolio AWS/Google Cloud Platform/Azure offers.
Big companies, most of their servers have a pretty stable load, it's unlikely things like internal email, Sharepoint, ERP/MAP systems will take a spike. It's only things like front end order processing that takes the hit.
There are lots of businesses that make sense and some that don't
I like the concept of "racing to the bottom" but they are still making money. But lets take your comment of the Value Added Services other than the ability to spin up capacity. What's the cost to Gitlab to pull this together and keep it running? There is an overflow every day on HN articles about operations monitors, containers, network monitoring. The tools are there, its an effort to glue them together, but then they are there.
So I'll still posit there is cases that the dollars to own are less than the dollars to rent. And I'll agree with your cases of rent because of capacity blowouts is key. The issue, is your ops team savvy enough to figure out what to keep/own, what to rent?
From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys have experienced, is something you want to get right on the first try.
Edit:
N1: Yes, the database servers should be on physically separate enclosures.
D5: Don't add the complexity of PXE booting.
For Network, you need a network engineer.
This is the kind of thing you need to sitdown with a VAR for. Find a datacenter and see who they recommend.
If the 8TB drives don't offer the IO we need we'll have to leave part of them empty. I assume that if you fill them with 2TB they should perform equally well. Word on the street is that Google offers Nearline to fill all the capacity they can't use because of latency restrictions for their own applications.
It is an interesting idea to do SSD cache + SSD storage + HDD instead of just SSD cache + HDD. I'm not sure if Ceph allows for this.
Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.
If you are talking physically, 2TB/8TB would theoretically be faster than a 2TB/2TB disk if the performance attributes were the same. But a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially filling a drive would probably be offset by the slower overall drive.
I'm skeptical here. So your minimum seek time goes up on the 8TB because it's harder to align. But your maximum seek time should drop tremendously because the drive arm only has to move along the outer fifth of the drive. And your throughput should be great because of the doubled linear density combined with using only the outer edge.
A quick google search shows that there are marginal gains on the outer track vs the inner, but that is only on sequential workloads. For something like GitLab, the workloads would be anything but.
http://superuser.com/questions/643013/are-partitions-to-the-...
https://www.pythian.com/blog/hard-drive-inner-or-outer/
1. If I look at 2TB vs. 8TB HGST drives, their seek times are 8ms and 9ms respectively. But if you're only using a quarter of the 8TB drive, the drive arm needs to move less than a quarter as much. Won't that save at least 1ms?
2. The 8TB drive has a lot more data per inch, and it's spinning at the same speed. Once a read or write begins, it's going to finish a lot faster.
3. Here's a benchmark putting 8TB ahead in every single way http://hdd.userbenchmark.com/Compare/HGST-Ultrastar-He8-Heli...
By "Tiering", is this moving data between different drive types? Or by moving the data to different parts of the platter to optimize for rotational physics?
Ceph can probably explain it better than I can. http://docs.ceph.com/docs/jewel/rados/operations/cache-tieri...
Edit. If Ceph is smart enough, it would be aware of the tiers present on the node, and would tier blocks on that node. So a Block on node A will stay on node A.
I read the entire article, and while you talked about having a backup system (in a single server, no less!) that can restore your dataset in a reasonable amount of time, you have no capability for disaster recovery. What happens when a plumbing leak in the datacenter causes water to dump into your server racks? How long did it take you to acquire servers, build the racks and get them ready for you to host customer data? Can your business withstand that amount of downtime (days, weeks, months?) and still operate?
These questions are the ones you need to be asking.
In other words, double your budget because you'll need a DR site in another datacenter.
I'm sure it has its uses! But I'd emphatically advise anyone to obtakn hands-on experience with Bcache before planning around it! As I'm sure you are already doing or considering of course :)
In my experience, the PCIE drives they mention (the DC P3700) are incredible. They're blazing fast (though the 3D X-Point stuff is obviously faster), have big supercaps (in case of power failure), and really low latency for an SSD (150 microseconds or so). They're a pretty suitable alternative to in-memory logbuffers where latency is less crucial. Much cheaper than the equivalent memory too. Having a few of these in front of a RAID 10 made up of 8TB drives will work just fine.
FWIW your experience with a RAID 6 of big disks is unsurprising. Raid 6 is almost always a bad choice nowadays since disks are so big. Terrible recovery times and middling performance.
I would think that GitLab's workload is mostly random, which would pose a problem for larger drives. The SSDs are a great idea, but I've only seen 8TB drives used when there are 2 to 3 tiers; with 8TB drives being all the way on the bottom. I'm not sure how effective having a single SSD as a cache drive for 24TBs of 8TB disks will be.
https://www.backblaze.com/blog/hard-drive-reliability-stats-...
[EDIT]
Oops that's reliability, I can jump the gun sometimes, sorry.
If you were larger, Open Compute Platform might be a way to go. Maybe next generation.
GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).
Another thing I might recommend is a third network (just a simple 1GB and a quality switch) for consensus. Re-replication can max the network out and further cause consensus fails, causes more re-replication winding down everything ... If that's not possible, add firewall rules to prioritize all consensus related ports high.
Just as a suggestion since you seem to be so certain about the location. I basically know every DC in Frankfurt, so if you need any help or info in that regard feel free to contact me :-)
I would assume that the .de lines are saturated with 5 eyes...
EDIT: Funny, every single documented, proven link for the last several years has proven me and GP correct and all you nay-sayers wrong.
I wouldn't make a business decision around a general feeling of paranoia. German spies aren't an improvement over American or British ones.
This is Gitlab. There should never be any data in or out except HTTPS and SSH.
Some financial institutions require all sensitive* data to be stored/hosted in the same country/state(archaic yes).
It's real hard to actually define sensitive data but the IP* in some code a quant wrote can totally be considered a trade secret by a non technical person Please don't get me started on how stupid I think it is that people consider code to be IP
Definitely not archaic. If the FISA courts taught you anything, it should be that the country your things are living in determines which entities can tap your traffic/hardware.
Frankfurt may be a better option solely on the basis that it is more "environmentally" stable (in terms of events that may disrupt the operation of the facility).
Of course there's the issue of potential legalities for your business, but don't kid yourself that you're safe from prying eyes by deploying in a particular country.
The next step (and particularly important for a business like GitLab) is to land a second site in either western Europe or west coast US. Honestly you should be thinking about this right away, and look to sign leases on both spaces simultaneously with a 3-mo delay built in for the second site. This should help you negotiate price as well, if you're able to go with the same dc provider, but be aware that that itself is a single point of failure as well. Make absolutely sure you negotiate your MSA to the n-th degree, get a good SLA, etc. You can still get burned, but do your legal due diligence now because you won't have a chance to change terms later.
Then continue to optimize by having multiple sites per-region (so that failover doesn't involve a big performance hit), adding APAC / AUNZ regions, and so forth. For a service like GitLab, I wouldn't think that time to sync the repo is hyper important, but responsiveness of the web interface is fairly key. So that may lead to a hub/spoke design where there are a few larger sites storing the bulk of the data and more small sites to handle metadata and such to present the web views.
That's all years down the road though. For a first pass, I can't see a problem with northern Virginia as the first site.
I (and others I bet) would be interested in a quick summary of the options (and your opinions of them) for facilities in Frankfurt.
You basically have the big brands like Equinix (7 facilities in Frankfurt), Interxion (They are currently building their 11th facility in Frankfurt), Telecity (now part of Equinix) but also some local ones like e-Shelter, First Colo or Accelerated. There is also a DC only meters away from the Interxion campus that is called Interwerk and is basically the cheapest DC in Frankfurt, but it has an awesome price/performance ratio and you have access to all peering/transit options of Interxion (via CWDM) and can get a 1/1 rack for under 200 EUR/mo. I have been in that DC for several years and never had a single issue, so if you don't rely on any certificates this is a cheap option.
I was also colocating in the First Colo DC for some time, they are also in the locality around the Interxion campus. It's a pretty small DC but it is more premium than the Interwerk and also is kinda cheap. I personally wouldn't go with Accelerated, since they had multiple issues in the past.
For the big brands I would definitely go with Interxion, they are great, have all the certificates and are premium while not going crazy with their pricing like Equinix does. DigitalOcean, Twitch, etc. they are all in one of the many Interxion facilities in Frankfurt. If Price isn't a concern I would probably go with Equinix FRA5.
DE-CIX is present in around 7 facilities, 3 of them are Interxion and I think 2 Equinix.
(I'm from Germany, but I couldn't care less about the country per se or GitLab being co-located in Frankfurt)
Perhaps look into how to have the nodes split between 2 or 3 different datacenters?
It is pretty easy to build such middleware both for the git over ssh (a simple script performing a lookup of where the shard is and then you connect to shard to operate there) and just a little bit more for the http part. At the webapp level, you will have a kind of RPC to run the git related operations which will connect to the right shard to run the operations.
When you use Ceph you are basically running a huge FS at the full scale of your GitLab installation, but practically, you have many independent datasets within your GitLab installation and you do not need to pay the cost for the global consistency of Ceph. You have many islands of data.
[0]: https://github.com/libgit2/
Edit: Typos/missing words.
I do; that's my default scenario. If you can survive that, you can survive all sorts of smaller issues like network congestion, data center power problems, grid power problems, and zombie plagues (or flu, which is more likely.)
But that's a true emergency situation. Don't go offline for multiple days for something that's reasonably likely to happen.
I've ordered a pair of Intel 750 Series 1.2TB NVMe SSD's for it, can't wait to try that out. Still waiting for the SSD drives to arrive.
SuperMicro has a 4 node system available in which each node gets 6 disks, 2028-TP-xx-yyy. Get two of these, populate 2 nodes on each as your databases, and you can grow into the other spaces later. Run your databases on all-SSD; store backups on spinners elsewhere.
Having two node types is not a calamity compared to having just one.
So the proposal now is to have one chassis (excluding the backup system) and two types of nodes (normal and database).
Regarding the 2028 please see the conversation in https://news.ycombinator.com/item?id=13153853
Ohhh boy. I hope this works, and look forward to hearing updates.
"Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters."
I'm getting the same feeling I get when I'm watching those "Hold my beer and watch this..." videos.
From my vantage point they look to have 0 experience in building and running infrastructure... and asking advice on HN. They might ask well post a Ask Slashdot thread if they want armchair advice. Genuinely, I think they've crunched some numbers and think they can run their stuff cheaper and faster in-house... but probably underestimated the human-experience angle.
For just 10-20 physical servers, this is going to be either extremely expensive (if they hire right) or extremely painful (if they don't).
We're not doing this to save money, we're doing it to increase performance https://about.gitlab.com/2016/11/10/why-choose-bare-metal/
A service providers' biggest responsibilities to its customers are security, durability, availability and performance -- in that order. You guys are vastly underestimating the complexity involved in getting first 3 right.
You'll need to maintain an association between - dns, ip, mac address, ssh keys etc.
Hardware break-fix workflow is usually ignored by most production engineers. You'll be doing that a lot. You want to get your hw back into use as fast as possible.
Have you thought about how many spares (CPU, RAM, disks) you'll have to keep at your datacenters ?
does not really agree with
> Each of the two physical network connections will connect to a different top of rack router.
Sure, you can do it with something like MLAG, but that's really just moving your SPOF to somewhere else (the router software running MLAG). Router software being super buggy, I wouldn't rely on MLAG being up at all times.
> N1 Which router should we purchase?
Pick your favorite. For what you're looking for here, everything is largely using the same silicon (broadcom chipsets).
> N2 How do we interconnect the routers while keeping the network simple and fast?
Don't fall into the trap of extending vlans everywhere. You should definitely be routing (not switching) between different routers. You can read through http://blog.ipspace.net/ for some info on layer 3 only datacenter networks.
You'd want to use something like OSPF or BGP between routers.
> N3 Should we have a separate network for Ceph traffic?
Yes, if you want your Ceph cluster to remain usable during rebuilds. Ceph will peg the internal network during any sort of rebuild event.
> N4 Do we need an SDN compatible router or can we purchase something more affordable?
You probably don't need SDN unless you actually have a SDN use case in mind. I'd bet you can get away with simpler gear.
> N5 What router should we use for the management network?
Doesn't really matter, gigabit routers are pretty robust/cheap/similar. I'd suggest same vendor as you go for whatever your public network routers.
Also, consider another standalone network for IPMI. I can tell you that the Supermicro IPMI controllers are significantly more reliable if you use the dedicated IPMI ports and isolate them. You can use a shitty 100mbit switches for this, the IPMI controllers don't support anything higher.
> D5 Is it a good idea to have a boot drive or should we use PXE boot every time it starts?
PXE booting at every boot is cool, but can end up sucking up a lot of time. If you have not already designed your systems to do this, and have experience with PXE, then don't.
> The default rack height seems to be 45U nowadays (42U used to be the standard).
You may not have accounted for PDUs here. Some racks will support 'zero-U' PDUs, but you'd need to confirm this before moving on.
> H3 How can we minimize installation costs? Should we ask to configure the servers to PXE boot?
Assume remote hands is dumb. Provide stupidly detailed instructions for them. Server hardware will PXE by default, so that's not really a concern. IPMI controllers come up via DHCP too, so once you've got access to those you shouldn't need remote hands anymore.
> D2 Should we use Bcache to improve latency on on the Ceph OSD servers with SSD?
Did you consider just putting your Ceph journals on the SSD? That's a lot more standard config then somehow using bcache with OSD drives.
I would strongly consider doing this via pure L3 routing. This is a scale at which the benefits of L2 fabric switching vs L3 multihomed routing (yes, routing decisions on every node) begin to be interesting decisions.
We're already planning a separate router for the management network ("Apart from those routers we'll have a separate router for a 1Gbps management network.").
All Ceph journals will be on SSD too. I've added a question about combining this with bcache in https://gitlab.com/gitlab-com/www-gitlab-com/commit/a9cc9aad...
For switches, yes. Many of the switches share the same merchant silicon Broadcom Trident-II, Tomahawk, et al., however there are switches like the Juniper EX9200 which isn't baed on merchant silicon. Routers (N1) are also not typically based on merchant silicon (Juniper Trio-3D for example).
Bcache is mentioned in the article under the disk header.
Geo-redundancy seems like a luxury, until your entire site comes down due to a datacenter-level outage. (E.g. the power goes down, or someone cuts the internet lines when doing construction work on the street outside).
(This is one of the things that is much easier to achieve with a cloud-native architecture).
As far as moving from cloud to bare metal, another thing to take into consideration (that this was replied to), if you don't architect your AWS (or other cloud solution) to take advantage of multiple geographic regions, the cloud won't benefit you.
I 100% agree that there should be more than one region deployed for this service. As others have said, all it takes is 1 event and the site will be down for days to weeks to months (It may not happen often, but when it does, you go out of business). The size and complexity of this infrastructure will make it nearly impossible to reproduce on short order in a new facility. If I were the lead on this I would have either active / active sites, or an active / replica site.
I would also have both local (fast restores), and off-site backups of all data. A replica site protects against site failure not data loss and point-in-time recovery.
Yep, this is why scaling starts with scalable distributed design. We were moving a fairly large logging stack from NFS to S3 once, for the same reason Gitlab is trying to move to bare metal now. Moving off cloud was not an option, moving to a TCO efficient service was. NFS did not scale and there was the latency problem. I think moving to bare metal cannot help with scale as much as a good architecture can. We will see how deep the datacenter hole goes. :)
To add to my previous comment though, AWS (and cloud in general) tends to make much more sense if you are utilizing their features and services (Such as Amazon RDS, SQS, etc.), and if you aren't using these services I can absolutely guarantee I can deliver a much lower TCO on bare metal than AWS. (Which is why I offered to consult for them) I see this all the time. Company moves from bare metal to AWS as bare metal is getting expensive, then they quickly find out AWS can't deliver the performance they need without massive scale (because they aren't using a proper salable distributed design and can't afford to re-architect their platform)
I don't know much about the Ruby web server landscape, but might Puma (http://puma.io/) be better?
https://gitlab.com/gitlab-org/gitlab-ce/issues/3592
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1899
Go ahead and apply for your ASN now and an IPv6 allocation. Then start working on the paperwork for an IPv4 allocation. Because there is no more IPv4 to allocate you'll have to go through the auction process and then the subsequent transfer process.
You'll easily be able to find a provider that can give you a /24 if you buy transit from them, but you don't wanna go through the trouble of renumbering into your own IP space if you can avoid it.
You're talking about only 64 nodes right now. Your storage and IOPS requirements are not huge. A lot of mid-size hosting companies will give you fantastic service at the 10-1000 servers range. If I were you I'd talk to someone like https://www.m5hosting.com/ (note: happy dedicated server customer for many years -- and I'm sure there's similar scale operations on east coast if that's really what you need) who have experience running rock solid hosting for ~100s of dedicated servers per customer.
I suspect you may just be able to get your 5-10X cost/month improvement (and bare metal performance gains) without having to take on the financing and hardware bits yourself.
In fact, i wonder if that's a business model for hosting providers. Kind of a sysadmin incubator.
There are many advantages to AWS and similar services, but if you can't really take advantage of all the goodies because e.g. you also need to provide a local version of your software (which is the case for Gitlab, as far as I understand), renting dedicated servers is an order of magnitude cheaper.
We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=13153455 that this is the time to build a team that can handle this makes sense to me too.
I would encourage you to look at what you can get without trying to do your own colo. You're not at the scale where you should be thinking about that.
That said, SoftLayer does provide 20Gbps access within a private rack and 20Gbps access to the public network.
Performance along with reliability are the most important metrics for someone providing VCS as a service.
The FatTwin chassis has similar density, and can support either 1U half width or 2U half width systems in a particular chassis. Typically I use 1U's for app / web servers and 2U's for lower end database / storage. Separate 2U servers for higher end database and 4U servers for bulk storage.
HPE has the Apollo 2600 / XL170r 2U chassis, which I think is somewhat inferior but still a reasonable choice. Dell sells the same thing as the C6300. I really prefer a 4U chassis though from a cooling and power supply perspective, but Dell or HPE may have a better international story for you.
You absolutely should not buy the 2630v4 CPU. I say that because the lower-end Intel CPUs do not support maximum throughput for memory and QPI. The 2630v4 is 8.0GT/s QPI and 2133MHz DDR4. A better solution low-watt part is 2650Lv4 (9.6GT/s QPI and 2400MHz DDR4). I have a guide that I created (and use myself) to determine comparative $/performance of Intel CPUs based on SPEC numbers [1]. If you can go up to 105W the 2660v4 is probably your best bet. Presuming that you're targeting 12-15kW per rack, a 105W part should allow you to deploy between 60-80 hosts per rack.
Also, don't use a W-series CPU that draws 160W. That's crazy power draw per-socket. If you want a super high-end CPU for your database server, I suggest a 2698v4 -- but normally I would go with 2680v4 or 2683v4 depending on the part cost.
In terms of hard drives, absolutely you should specify HGST over Seagate. At some point you may want to dual source this, but if you're only going with one vendor right now HGST is the best option. He8 or He10 8TB are your best bet in terms of cost and availability right now, although start thinking about He10 10T drives. The newly announced He12 drives shouldn't be on your radar until Q2 2017. Stock spares, maybe 2-3% of your total drives deployed, but at least 5-6 drives per site. You don't want to get caught out if there is a supply shortage when you need it most. Your business depends on ready access to considerable quantities of storage.
The P-series Intel SSDs are probably not going to be cost effective for your use case. But they are considerably better in terms of IOPS and remove the need for an HBA or RAID controller. Consider a Supermicro 2U chassis with 2.5" NVMe support, which will allow you to go considerably denser than the PCIe form factor. However, I think it's too early to go with NVMe unless you truly need beyond-SSD performance.
Don't PXE boot every time you boot. This creates a single point of failure (even if your architecture is redundant), and you will regret this at some point. However, DO PXE boot to install the OS.
Don't use 128GB DIMMs. They are not cost effective today.
There is only one solution for database scaling: shard. You'll either shard it today, or you'll shard it tomorrow when the problem is much harder. Scale up each host to what is easily achievable with today's hardware, and if push comes to shove retrofit to get over a hump that arises in the future, but know that you MUST shard in order to keep up with demand. Scaling up simply does not work.
There's a lot more to say, but without doing my job for free in a HN comment, the best advice I can offer is:
1. Simplify what you hope to accomplish in the first round. This is a lot to achieve at once. I think you'll have a hard time achieving the fanciness you want from a software perspective while also forklifting the entire stack over to physical hardware. It's perfectly fine to have something be good enough for now.
2. Find people who have done this before and get their advice. Find a VAR you can trust.
3. Plan, plan, plan, plan. Don't commit until you ...
Basically whatever the top-of-the-line DIMM option may be (and this applies for CPUs and HDDs and other stuff too), you want to avoid being in a situation where you HAVE to use it. Vendors price these parts accordingly: you pay a premium for top-of-the-line because you must have it. If you can avoid that, do so.
The $/perf of 2630v4 is pretty decent ($2.24), but I would personally be leery for the reasons I mentioned. That said, I have used it for bulk storage servers, where CPU performance was not that important. So it's not like it will blow up your machine or something.
To obtain the perf number, I'm averaging single core and multi core fp and int SPEC numbers. If your workload isn't heavily parallelizable, that might not make sense. I'm not too worried about single core performance myself these days and have been tempted to remove it entirely.
One other thing I forgot to mention: v5 Xeon CPUs will be shipping in quantity early 2017, so you may want to consider holding off and looking for better deals on v4 CPUs then.
Likewise, you might be able to obtain a better deal today on v3 CPUs, particularly if you aren't using a large vendor like Dell or HPE. All of my pricing is list (I don't pay these prices), so the math changes significantly if you can get a disproportionate deal on a particular model. I use it as a place to start the conversation with my VAR, and then go with what makes sense in the market right now.
You will need to monitor: - ping - latency - temperatures - cpu utilization - ram utilization - disk utilization - disk health - context switches - IP addresses assigned and reaching expected MACs - appropriate ports open and listening - appropriate responses - time to execute queries - processes running - process health - at least something for every bit of infrastructure
once you collect that information, you need to record it, graph it, display it, recognize non-normal conditions, alert on those, page for appropriate alerts, and figure out who answers the pagers and when.
Our Infrastructure lead Pablo will do a webinar of our Prometheus monitoring soon https://page.gitlab.com/20161207_PrometheusWebcast_LandingPa...
We're bundling Prometheus with GitLab https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1481
Brian Brazil is helping us https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1481#not...
On January 9 our Prometheus lead will join us (who was very valuable already helping with this research behind this blog post) and we're hiring Prometheus engineers https://about.gitlab.com/jobs/prometheus-engineer/
In the short term we might send our monitoring and logs to our existing cloud based servers. In the long term we'll host them on our own Kubernetes cluster.
For our monitoring vision see https://gitlab.com/gitlab-org/gitlab-ce/issues/25387
Disclaimer: Prometheus co-founder and just started as a Prometheus contractor for GitLab.
EDIT: ah, didn't reload the page to see that sytse had already responded with this :)
In the longer term the storage appliance will lock us in and will get very expensive. I've heard pretty bad stories of companies betting on it. Especially with many small files like us (IOPS heavy).
And one goal of GitLab.com is to gain experience that we can reuse at our customers. Most of our customers use a storage appliance now but are interested in switching to something open source.
You need at least two nodes that do DNS, DHCP, NTP, and other miscellaneous services that you absolutely want to have but do not seem to have mentioned. You want them to be permanently assigned, so that you never have to search for them, and you want them to fail over each service as needed, preferably both operating at once. Three nodes would be better. Consider doing some basic monitoring on these nodes, too, as a backup to your main monitoring system.
1&2. Authoritative DNS (internal)
1&2. NTP
1&2. Caching DNS resolver
3&4. Outbound HTTP proxy (if necessary)
3&4. PXE / installer / dhcp
3&4. Local software mirror (apt / yum / etc.)
5&6. SSH jump hosts.
Or something like that.
Make sure the second host is not just in a separate chassis from the first host, but also in a separate rack.
For external authoritative DNS, don't do it yourself -- pay for someone to run that for you (Route53, ns1, etc.). For e-mail, if you can possibly not deal with it then don't -- use Mailchimp or something.
The DNS records for the internal records are done using the kubernetes middleware (basically serving the service records). The external records are pulled in from a git repository hosting our zones as bind files. If need be zones are split into subzones per team/project. Same permission system as our code via MRs using Gitlab.
Our recommendation is build on open standards (BIND, AXFR) and use services on top of these.
I agree that using an external mail provider is usually a good idea. It mostly is your fallback communication channel and is usually easy to switch (doing replication to an offsite mail storage needs to be done to make switching easy/possible/fast). MX records \o/