This is serious Server HW porn. Thanks for sharing! Would love to know more information about your bandwidth usage, limits, etc. Also, do you have any closer idea as to how much this would cost in "cloud" services in % terms? (I'm doing general research in Bare Metal & Colocation vs Cloud pricing comparison). Thank you!
But a better answer is: that price keeps changing. We now have a lot of AWS and on-prem experience in house to do a great post. We'll be doing a lot of research and proper comparisons as a huge part of that upcoming post: https://trello.com/c/4e6TOnA7/87-on-prem-vs-aws-azure-etc-wh...
How are the drives used on their SQL server boxes? They have a massive NVMe drive on raid 0 with 20 sata drives in raid 10? Which drives are used for prod data? Do they run backups on the same box?
In the Stack Overflow cluster, the RAID 0 NVMe contains Stack Overflow database and the SATA SSD RAID 10 holds the others (Mobile, Translations, PRIZM, and Sites).
In the Stack Exchange cluster, the RAID 0 NVMe array contains all databases except for a large log database (which I called Careers.BigStuff, because it seemed like a good idea at the time). This larger log database is much more rarely accessed and is on the 10K HDD RAID10.
We run backups on the primary for several reasons, but they are all sent off-box. We have 2 primary on-site backup servers, then those backups go offsite and to tape. Database backups are every 15 minutes with T-Logs and full backups nightly. We also run copy-only backups in the DR data center nightly as an additional backup measure.
Awesome pics; it'd be cool to see some people in them too. Thank you for all the work you're doing for Stack Overflow/Exchange, it's an amazing resource!!
You know someone is a real hardware guy when he calls backup, monitoring and logging non essential. Also a bit confused about the 2960Ss and the choice of Cisco in general. I'd never thought Stack Overflow ran on Network Equipment, what I'd describe as consumer hardware in networking.
What I sadly don't get is the hierarchy of your routers and switches by the pictures and the lack of QSFP+ utilization as well as the amount of copper. What's the reason behind the copper sfp transceivers in the ASR-1001[0]?
The 2960s seem to be in the background for the management side of the network, doing ikvm and the like, therefore don't need 10 gigabits. Also, the copper in the core routers seems to be more a case of the routers still having enough bandwidth for the site, just needing more bandwidth, thus more ports aggregated. 10gig ethernet's a good amount cheaper than 10 gigs of fiber and works just as well for the short run they're doing..
Would love to see details on energy use / cost. Have you done any profiling there? Is energy cost a big deal? Is that the main reason Providence doesn't refresh continuously?
These plans are in flux. At a core level (heh), we nee to have libraries ported first anyway. So we're doing that now. Jil, StackExchange.Redis, Dapper, and Sigil are up on NuGet and core compatible today (some as pre-release). Others like Exceptional are just pending some RC2 goodness to release, and I'll be starting on MiniProfiler after that. We'll likely post other internal tools to .Net Core and start heavy testing in the RC2 time-frame.
Will we run Stack Overflow on core? Probably not for a while. I was asked exactly this in a recent On.Net interview at the 24:22 mark; you can listen here for reasoning: https://youtu.be/DJn8-Psznsw?t=24m22s
Interestingly, the price of 32GB DDR3 LR-DIMMs has been falling as well. I wonder if this is due to oversupply or something else since 8Gbit DDR3 will never happen on servers.
26 comments
[ 3.1 ms ] story [ 68.0 ms ] threadBut a better answer is: that price keeps changing. We now have a lot of AWS and on-prem experience in house to do a great post. We'll be doing a lot of research and proper comparisons as a huge part of that upcoming post: https://trello.com/c/4e6TOnA7/87-on-prem-vs-aws-azure-etc-wh...
In the Stack Exchange cluster, the RAID 0 NVMe array contains all databases except for a large log database (which I called Careers.BigStuff, because it seemed like a good idea at the time). This larger log database is much more rarely accessed and is on the 10K HDD RAID10.
We run backups on the primary for several reasons, but they are all sent off-box. We have 2 primary on-site backup servers, then those backups go offsite and to tape. Database backups are every 15 minutes with T-Logs and full backups nightly. We also run copy-only backups in the DR data center nightly as an additional backup measure.
What I sadly don't get is the hierarchy of your routers and switches by the pictures and the lack of QSFP+ utilization as well as the amount of copper. What's the reason behind the copper sfp transceivers in the ASR-1001[0]?
[0]https://nickcraver.com/blog/content/SO-Hardware-Network-NewY...
There's no compelling reason to invest in such a move for our situation. We're very interested in how Open Compute progresses, though.
Will we run Stack Overflow on core? Probably not for a while. I was asked exactly this in a recent On.Net interview at the 24:22 mark; you can listen here for reasoning: https://youtu.be/DJn8-Psznsw?t=24m22s