I've recently started using their micro-ORM Dapper, and I like it a lot. I get the performance of hand-coded SQL, but without the tedious mapping from SqlDataReader to my entity.
Dapper is great - I promptly dumped NHibernate and Entity Framework for it. It's the only ORM I've found that makes it trivially easy (as it should be) to use stored procedures and views in queries.
+1 for Dapper - I'm a big fan. It is very fast and we use it to serve our live (mostly read-only) traffic, while using something like Entity Framework for the backend Administrative UI, which has a lot of inserts and updates.
I wrote a Dapper extension for working with SQL Server's geospatial queries and types a couple of years ago - have they added anything like that yet? Otherwise I'd be happy to add it.
I've been using Dapper for around three years, "it just works" and is very fast. I've been around the block a few times with ORM's - LINQ to SQL, NHibernate, Entity Framework, Massive, SubSonic. For small to medium size projects there's no contest, provided you don't mind hand crafting your SQL. Hat's off to Marc, Sam and team for building this.
Dapper is nice, but if you want to have the compiler check your queries, and still run fast, I recommend LINQ to DB: https://github.com/linq2db/linq2db
(not to be confused with LINQ to SQL from Microsoft)
To be fair, the compiler is checking against what you told it the database looks like. This can help with lots of things, but still has the fundamental disconnect problem you'll get at runtime with either approach.
A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example we (and by "we" I mean, "I made Marc Gravell do it") added literal replacement to Dapper to help with query parameterization which allows you to use things like filtered indexes. In dapper we also intercept the SQL calls to dapper and add add exactly where it came from. Here's what that looks like, replicate for the other methods:
And here's the our marking method tailored for our code, but you get the idea:
private static string MarkSqlString(string sql, string path, int lineNumber, string comment)
{
if (path.IsNullOrEmpty() || lineNumber == 0)
{
return sql;
}
var commentWrap = " ";
var i = sql.IndexOf(Environment.NewLine);
// if we didn't find \n, or it was the very end, go to the first space method
if (i < 0 || i == sql.Length - 1)
{
i = sql.IndexOf(' ');
commentWrap = Environment.NewLine;
}
if (i < 0) return sql;
// Grab one directory and the file name worth of the path
// this dodges problems with the build server using temp dirs
// but also gives us enough info to uniquely identify a queries location
var split = path.LastIndexOf('\\') - 1;
if (split < 0) return sql;
split = path.LastIndexOf('\\', split);
if (split < 0) return sql;
split++; // just for Craver
var sqlComment = " /* " + path.Substring(split) + "@" + lineNumber + (comment.HasValue() ? " - " + comment : "") + " */" + commentWrap;
return sql.Substring(0, i) +
sqlComment +
sql.Substring(i);
}
This results in a comment at the top of the query like this:
That's a straight copy/paste from the Stack Overflow codebase (Helpers\SqlMapper.cs:332 at time of writing, if you care). Now the typos and snark are out there for everyone to see.
Another library worth mentioning in this space is Massive, which uses .NET dynamics a lot (focusing slightly less on performance than ease of use). https://github.com/robconery/massive
Joel is very much still around and active. Jeff went off to build Discourse [0] although I think he still has a place at the co-founder table as invisible-benevolent-dictator.
Rigid adherence to certain technologies could be a culprit. "We're a Ruby shop!" is fine for problems where Ruby is great, but the limitations of the technology are bound to creep up somewhere. Not using a polyglot approach would be bad.
It can also be caused by too much embrace of polyglot tools. "We use the best tools for the job" can easily mean an architecture with Redis, Memcached, Mongo, MySQL, Angular, JQuery, Node, and several dozen third-party libraries, all glued together with Thrift or JSON-RPC. If all you're doing is a messaging app, you might be much better off with a single in-process Java app.
Usually you want your solution to be just right for the problem; solve the essential complexity but don't introduce any accidental complexity.
There are hundreds of reasons at nearly every level of a company that can contribute to this situation. Its best summed up by Sturgeons Law, "90% of everything is crap."
From my limited experience, it's a lack of the kind of systems thinking present in companies like Stack Exchange, and present on the High Scalability blog, that results in a poor architecture.
And this is compounded by people who have little ability to troubleshoot performance issues. It's quite easy to hunt down the cruddy SQL queries in a DB, or realize that you spinning rust is too slow. But when it starts to come down to things like a blocking network fabric, which has some big fat buffer between two servers, that is killing your transaction speed - many will just start to blame the devs.
Of course, the second point is compounded by the first - the less systems thinking that went into the design in the first place, the harder it is to produce accurate hypotheses about the system in order to troubleshoot.
Whilst there is a lot to be said for the "right tool for the job", if you watch an artisan crafting something, you'll realize that despite having a huge number of tools, they actually get by with relatively few. This is a generalization, but they only use the full range when doing something new, solving a particularly tricky problem, etc. "All the gear and no idea" is certainly applicable in many start ups.
"One problem is not many tests. Tests aren’t needed because there’s a great community... If users find any problems with it they report the bugs that they’ve found."
I'm often surprised at the paucity of test-coverage in relatively large companies.
That bullet point surprised me. It comes right after "110K lines of code. A small number given what it does"; to me this reads like they didn't write tests because that would add complexity.
Can any SO devs give us more details on "not many tests"? Or how many bug reports get filed vs rate of change of software?
The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange).
You can look at reported bugs here: http://meta.stackexchange.com/questions/tagged/bug -- it's pretty frequent, at least one every couple of hours or so. The last commit in our "Tests" project was 14 days ago, where as the last commit in the code base was 4 minutes ago.
Indeed, they seem to have achieved what many have said is literally not possible.
Also, some might remember when they had Uncle Bob on the podcast, the topic being unit testing, and iirc Bob preaching the gospel (~ you must write plentiful unit tests, or else) and Jeff and Joel (especially) more or less saying they don't quite get why.
(Am I remembering this story correctly? I swear that's how I remember it.)
I think we're going trough a thesis/anti-thesis cycle on tests - in the beginning, there was militant testing, 100% coverage, testing getters and setters etc (as well as more complex stuff, obviously). Then some people started coming around to the idea that there are actually large swathes of code that is simple enough that testing doesn't actually add much value especially compared to the effort of writing them, then that probably got a bit out of hand (to what you're referring to). Maybe the pendulum will swing back and we'll find a good heuristic for just how much testing is the right amount that isn't all or nothing?
Actually, in the beginning there was no testing and even justifying having automated test infrastructure or spending developer resources on unit tests was a hard sell to management (what are they paying that QA dept. for after all).
> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune.
I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.
It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive.
I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system.
$4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160.
Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months.
A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).
Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.
AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high.
10tb = roughly $1,000 per month
They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.
they negotiate, still. I know several startups which went to amazon, said, "hey, your competitor is cheaper, what is your value prop" and suddenly had 50% discounts, etc. It's amazing how much you get simply asking!
Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :)
The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc., but apart from these, mostly everything seems “normal” to me.
> e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc.
this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this
384 gigs is pretty common these days. With Windows, you need at least the server edition to address that much, but if you're running a server, you're not gonna be using the desktop versions of the OS anyways. Pretty much everything else, it's just plug and play. The RAM is quite a bit if you're doing a typical cloud deployment, but SO seems to be more in the model of making sure all your eggs are in a really strong basket. Besides, at the end of the day, RAM tends to be a lot cheaper than lag.
To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not.
This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better SMB, and such...but not enough to make the upgrade worth it.
Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!
I'm not sure a full writeup would have much more detail, but yes. With Windows clustering (on top of which SQL 2012/2014 Availability Groups are based), you can only have a homogenous OS version across the board. Since they all must have the same version, you can't have something like a single 2012 R2 instance in an otherwise 2012 cluster...and since you can't do that, you can't upgrade one at a time. Yay! New cluster time instead!
Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.
I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0].
Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.
Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!
The don't have servers with that much RAM, but a 244GB RAM server with 8x800GB SSDs and 32 cores costs <$5k a month ad-hoc and about $1300/month with a 3 year commitment.
$1300/month w/ a 3 year commitment is $46.8k over the lifespan. Which is more then 4x more then the actual hardware cost. Your ROI on a physical server with the same stats at that price tag would be 8 months.
Not saying its totally worth it, but the upside is that they take care of power costs, physical space costs, networking, and if the hardware fails they'll put your application on a new server. Basically you take all of the risk out the equation.
Sure, but add power and cooling for these things plus physical space? (I remember when these big db's needed the floors in the data centre reinforcing since they were a > metric ton )
Still, I agree re. the difference in price. I'd buy too
Worth noting: AWS also doesn't have Enterprise SQL available, so plan on buying that yourself. That's a not-so-insignificant piece of the overall cost to many people.
How common is it to buy one extremely powerful database server? Wouldn't you need the high availability properties of something like a Galera cluster? Or are most people accepting the single point of failure at huge database server?
While you do want HA you also want to keep things simple. You get great HA with just one backup machine. The times you lose two machines are so few that you might crash the site with software failures far more often anyway.
Databases have inherent locking problems that takes more resources to resolve with more machines, more so in some database systems than others. When scaling you often hit a point where you get down to macro logistics, so imagine a car highway: You can get more throughput by adding more lanes. But what that all cars are going to merge to one lane at some point in the middle of the trip (because of a tunnel, bridge or something: it's impossible to have more than one lane at that point)? Now you won't get any throughput benefits of the multiple lanes after all! Just more latency because you get queues up until the single lane and because of the queues all cars are going slow and need to accelerate on the single lane, so the average speed is low too. You are also having too much resources after the single lane because you can never fill all those lanes. It might be better to have one beefed-up single-lane road all the way that people can go fast on. Basically: Remove locks and you get better overall performance.
Yes, this is on the expense of HA. Yes, the costs of scaling up grows asymptotically faster than the costs of scaling out. So this is definitely a trade-off in some sense.
They just run everything through one live database all the time under normal conditions. Personally, I think this is much simpler to manage than to be regularly spreading out queries over many queries.
I believe the idea is to hold as much of the DB in memory as possible, for speed, so huge RAM totals on DB servers are not unusual. 384 GB should be well within modern OS reach. Once you're computing at 64 bits, the next theoretical limit on addresses would be somewhere around 16 million terabytes.
Windows Server handles that much RAM fine, and, so long as it fits, SQL Server will end up caching the entire database in RAM.
If I remember correctly they're using Dell R720's for their SQL Server machines. At current Dell retail pricing 384GB of RAM would be about $5,900.00.
At that price I'd vote for throwing money at raw hardware and reaping the benefits of caching the database in RAM, versus trying to pay programmers to come up with clever techniques to deal with having less RAM.
384 GB is fairly easy to achieve today with 16 GB DIMMs x 24 slots (many general purpose servers have up to 24 DIMM slots). Beyond that you need to jump to 24 GB DIMMs which are fairly expensive.
Even before OS'es supported that much people were sticking that much RAM on expansion bus cards and plugging them in and treating them like fast disks. So it is going to happen either way. The performance boost for the DB is just too much compared to it thrashing disks constantly and there are quite a few benefits to having fewer machines compared to many DB servers as well.
SQL Server only reads data from memory. When something is requested it is first looked for in RAM (or virtual) and if not found is copied from disk to RAM.
The more RAM you have the better without exception here.
The simplest rule of computing is, if you have a problem that can be solved economically by just throwing more RAM into a single machine at it, you should do it (instead of any other dimension, and assuming you don't have other constraints).
No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.
One smart thing they are doing is putting different sites on different databases. It effectively as a kind of partitioning to allow for horizontal scaling if needed.
If nothing else, it keeps less data in each table, so queries should be faster due to smaller datasets.
I'm curious, why not? If your saas was small enough and had few enough customers with large data needs, separating them each out into a separate database seems like a viable solution.
yeah, if you have an experienced team, you can easily refactor your one-customer-per-db model into something that will handle 1,000 new customers signing up every month. given a good enough team, you can even do this while doing other sort-of important things like customer support, bug fixes, and new development.
however, inexperienced developers and teams are not good enough, or fast enough, to do that, and end up painting themselves into a corner when they have thousands of live customers to support and need to change their entire application architecture + database schema when their backup, replication, and housekeeping tasks choke on 1,000+ databases.
and oftentimes, one-customer-per-db also means one-instance-of-application-per-customer, another anti pattern to avoid.
we see this kind of stuff ALL the time. it happens a lot - people make terrible decisions and then are stuck with them 5 years down the line and are looking at a monumental cost to redo. not everyone is experience enough to work their way out of an awful situation like that.
They discussed it quite a bit on the podcasts. I found those a really great resource since Jeff and Joel were discussing the website as they were building it and "aired" the reasons for doing specific things.
If you have extensive Microsoft experience and take advantage of something like BizSpark* (although one suspects with their connections they wouldn't have needed that) it's hard to see how it wouldn't be plenty pragmatic and inexpensive.
* Currently three years of free licenses, all software you download during that time is free forever, and discounted MSDN subs thereafter.
Programmer time is a factor that comes into play when you build something and re-training your people to use a stack they are not familiar with + the associated mistakes can get expensive in a hurry.
If you happen to be familiar with the MS stack and are able to factor the license fees into your business plan there is no strong reason against and many for using that particular stack.
MS programmers tend to be a bit more expensive than people working with one of the many Linux offerings but that's not such a huge difference that it would become the deciding factor and if anything the MS stack is more performant than the Linux one on the same hardware (and I write that as an anti-MS guy, I strongly believe that politics and tech should be kept separate when it comes to discussing systems relative to each other).
So even if there are lots of reasons why I would not have made that particular choice I can completely understand why the SO people made their choice the other way.
There is a good reason majority of the World-Wide-Web is run on 'Nix stacks -- and there is a good reason majority of servers in general are 'Nix stacks. Also, an overwhelming percentage of super-computers (I know we are talking about webservers, but it illustrates performance capabilities) are running a 'Nix stack.
There is also cost involved, as well as flexibility of the stack. With a 'Nix stack, it's infinitely flexible, not so much the case with a Microsoft stack. Also the default MS Stack comes with a lot of additional OS overhead that is not present in the 'Nix stack, which reduces any said box's scalability.
Without trying to turn this into some sort of flame war - I was merely trying to suggest that the MS stack was not/is not the best choice for a highly scalable website. Take the top 10 websites -- they all run on 'Nix. The top cloud providers (except MS Azure), all 'Nix. These are companies that can easily afford MS licensing, so that's not part of the equation.
A MS stack at this scale is unusual to say the least.
That's not to say it won't scale (as evidenced by the SE team), but it doesn't mean there isn't a better alternative that saves more money and scales better with less hardware, etc.
It will be very hard to get into a flamewar with me supporting the Microsoft side of things. That said, none of the links you posted prove anything regarding performance, they do prove something about total cost of ownership, which once you factor everything in leans towards Linux for most installations, however, just looking at the situation for SO seems to me to suggest that they were more comfortable doing the initial development on the stack they were most familiar with and when the license costs started to count against them they used Linux machines to scale out.
Which is a pretty smart decision. Whether an MS stack at this scale is unusual does not say anything at all about whether or not it performs well.
I think we agree for the SE team, given the founders were MS stack familiar, it makes sense.
However, I have to disagree on your assertion the 'Nix's are less performant than the MS stack. The top tier web companies are not running 'Nix because TCO is lower; for most of these companies licensing costs are negligible and if it helped to scale better, it may even save them money going with MS stack... but they don't go with a MS stack...
Sometimes a RHEL license can even cost more than a MS license. Couple that with an Oracle DB back-end, and you easily have a much more costly setup than the MS equivalent. It's not about the money.
These companies are choosing the 'Nix stack because it is performing in an entirely different level than the MS stack. Everything from tiny embedded systems with 64k ram, up to monster systems with TB's of ram.
> "I Contribute to the Windows Kernel. We Are Slower Than Other Operating Systems. Here Is Why."
http://blog.zorinaq.com/?e=74
Keep in mind that it hardly ever is the webserver that is the bottle neck.
My theory on why this is the case is very simple: MS can afford to throw vast amounts of money at optimizations that are next to impossible in Linux simply because the coupling between the layers in Linux is looser. And that's a good thing, it translates into better security and fewer bugs.
As always, optimization alone is not a reason enough to go down a certain path. But for raw speed on requests it's fairly hard to beat IIS, if that's what you're after (I'm usually not, and even when it matters I can comfortably saturate most outbound links from a single server doing light processing, and as soon as the processing becomes the bottle-neck the CPU cache size, RAM and so on matter more than your OS, but given identical hardware a 'dirty' approach should yield better results at a cost of unreliability/complexity).
Tight coupling and money are a good arguments, thanks.
However, i believe we are comparing apples to oranges here. IIS uses it's own kernel module, HTTP.SYS - http://www.microsoft.com/technet/prodtechnol/WindowsServer20...
So, looks like having specific kernel api optimized for your particular usecase is an advantage, which IIS has and others don't.
IMO, it has one of the best tooling sets of any platform out there (Visual Studio) which saves lots of programmer time and increases productivity. With BizSpark the initial cost argument is out...so how's that not pragmatic?
I'm biased of course, but I think it worked out pretty well.
Joel worked at MS on office, Excel if I recall correctly. Jeff's blog is named for a feature from a book published by MS. I would be more surprised if it was MS free.
"I started my career at Microsoft, a little software company outside of Seattle, where I was a program manager on the Excel team. My area was “programmability” and most of what I worked on in those days was replacing the Excel macro language (XLMs) with Excel Basic, and providing an object-oriented interface to Excel. Excel Basic became Visual Basic for Applications and the OO interface is what you know as OLE Automation, a.k.a. IDispatch."
You mean to tell me Joel is partially responsible for the abomination known as Excel 2007 where I can't pull-drag the damn window from screen to screen (but all of the other Office 2007 products do!)
This has nothing to do with Joel. The decision was from Jeff Attwood. Jeff was a c# veteran and MS stack was the easiest decision to come up with an MVP and the journey from there.
One must wonder, how much better this setup would of scaled had it been a more appropriate webserver stack such as Linux/BSD + Apache/Nginx or similar. Perhaps less boxes would be needed, or with the same number of boxes, more concurrent users.
> Microsoft infrastructure works and is cheap enough
not as cheap as just paying your team to maintain the boxes. I wonder how many times in the past few years SE has needed to call Microsoft for support? Would a RHEL license still be cheaper (probably)? If something like CentOS then there would be no support cost unless you need to bring in an outside contractor (for a particularly nasty issue).
> One must wonder, how much better this setup would of scaled had it been a more appropriate webserver stack such as Linux/BSD + Apache/Nginx or similar.
Why is the Microsoft stack not appropriate? I am not aware of any spectacular performance difference between Apache and IIS, or Windows Server 2012 and Linux, and clearly the SO team is not religiously Microsoft, so it's hard to see why they wouldn't have switched if it was a clear win. MSSQL is actually, in my experience coming from MySQL, amazing - I'm back on MySQL and Postgres now and I often wish I wasn't. I'd say it's the most underrated of MS's products, probably because of the expense of the Enterprise/Datacenter editions.
> One must wonder, how much better this setup would of scaled had it been a more appropriate webserver stack such as Linux/BSD + Apache/Nginx or similar
I guess you're gonna have to define "more appropriate" for me, I have a feeling we'll have a fundamental disagreement there. We could (and have) run the entirety of our peak load with 1 SQL server and 2 web servers (and not pegging them). Don't forget we run with a crazy amount of headroom at all times, by design. I'm not sure how much better you picture that scaling on a linux environment.
> not as cheap as just paying your team to maintain the boxes
That makes a lot of assumptions about the team, their expertise, and what support issues would arise.
> I wonder how many times in the past few years SE has needed to call Microsoft for support
In the past 4 years I've been here? Twice. Both to report bugs in CTP versions of SQL server. At no cost, and we improved SQL server as a result of being testers. We have a very good relationship with Microsoft and talk to the developers that make the tools and platform we use in order to make life better for both of us. We do the same thing for redis, Elasticsearch, etc. It's the same reason we open source almost all of the tools we make.
> If something like CentOS then there would be no support cost unless you need to bring in an outside contractor (for a particularly nasty issue)
We use CentOS for all our linux systems and are deploying new servers on CentOS 7 now. We'll me migrating the others in the coming months. That doesn't mean it's free. Developer or sysadmin time to control Puppet deployments and such still eat some factor of time.
Where I come from, it is considered bad engineering to build a product that can only run on a single platform, can only run on one particular OS, and the product is at the mercy of future decisions by a 3rd party. We go out of our way to ensure we are not dependent on any single thing.
Even if it was the best choice at the time to go the Microsoft route, it may not be in the future... however -- SE has zero choice now otherwise they'd have to re-write their entire product... that sucks as a business because you have little choice over your own product now.
>Where I come from, it is considered bad engineering to build a product that can only run on a single platform, can only run on one particular OS, and the product is at the mercy of future decisions by a 3rd party.
This seems like a bit of a waste of resources to try and run an internal bespoke application on every platform imaginable. Now, if you are developing a product that you are then going to sell to other people to run on their own hardware this make more sense.
There are some very big benefits to writing to a specific platform, especially in the performance space.
In the end of the day it is a trade off between trying to eliminate every 3rd party dependency (next to impossible) or picking a solution or company you think will be around for a long time, and forging strong relationships with them.
We regularly talk to people at Microsoft (and all of the people who create and build to tools we use), give them feedback, and get bugs fixed. There is very little that comes down the pipe from them that we are not away of ahead of time and in some cases have helped shape through early access programs.
> SE has zero choice now otherwise they'd have to re-write their entire product...
This is not true at all, we have choices if MS decided to blow everything up. Not great choices, but we have them. Choosing between two or three crap options does not mean that options do not exist.
When you encounter people who don't respect your judgement as much as they "should", I'd suggest it may be because "how well it scales" doesn't guarantee "appropriateness". In most real world situations, the specifics of the chosen stack are rather inconsequential - both MS and *nix stacks perform more than adequately. In my opinion, it is unwise to take the advice of zealots who refuse to acknowledge this reality.
Jeff Atwood hacked prototype and next several versions, and since he's a .net guy (and ruby guy now), that's why .net was picked. Lot about tech stack planning can be heard in first few SO podcasts, don't even know if there're still available online somewhere
I might be a bit late to the party, but if anyone is interested in copying the Stack Overflow setup, we're offering startups free software, 60k in Azure credits for accelerator-backed companies and engineering mentoring, if required. Find me at felixrieseberg.com or felix.rieseberg@microsoft.com and I can set you up.
I could be mistaken, but I think this is in regards to that post Sam Saffron wrote a few years back about getting SO to do Gen.2 garbage collection less often, and using static classes and structs were one of the ways that they fixed it.
I find the opposition to static classes odd. Unless you need the data encapsulation of an object, why force people to create an object to call functions? I find too much C# code loves creating objects for no real purpose other than that's how it works.
For SO, if we guess they max out around 2000 req/sec, then if a bunch of objects are being allocated for each request, there could be added GC pressure. From my own experience developing much higher-scale managed code, allocating objects is a real pain and people will take a lot of effort to avoid it.
I think the real issue (although it doesn't seem to be an issue for SO) is testability. AFAIK, static classes cannot be mocked, and you are pretty much boxing yourself into only doing integration tests for everything. Not necessarily a bad way to go, just lots more setup per test.
Yep, that used to be one of the main reasons to have DI in Java. But now that everything (even static methods) can be mocked, that reason is no longer valid.
Is this not possible in .NET?
public static decimal DiscountValue(decimal orderValue, decimal orderDiscount)
Or even this, which would farm the work out to SQL:
public static decimal DiscountValue(int orderId)
The first would require a mock, the 2nd wouldn't and the third isn't unit testable.
It's a bit of a contrived example, as you're more likely to do the third method if you're trying to get some sort of summary data about an order out which would mean you don't really need/want to load the whole order object.
Isn't that comparing apples to oranges? Taking any application built for large servers and putting the same application onto a cloud-based architecture will be more expensive.
Cloud-based architecture requires ground up differences in how the application is built. Now whether or not it is better to use a cloud-based approach or traditional bare metal is highly subjective and isn't my point.
On another site, in another context, it probably would be, but here it's really presented as a contrast of scale-up versus scale-out - something the regular audience of highscalability will certainly grok.
In context...
'Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t been a problem because they’ve been quite successful at sizing their system correctly.'
It's an acknowledgement of their relatively unique strategy and the short list of caveats that make it possible.
Extremely interesting read, especially because it goes against many fashionable engineering practices. They see (correctly) that abstractions, DI have their own compromises. Also, they're not religious (about technology), that's also quite rare.
> The cost of inefficient code can be higher than you think. Efficient code stretches hardware further, reduces power usage, makes code easier for programmers to understand.
I'm curious what the reasoning is for "Efficient code ... makes code easier for programmers to understand". To my mind, efficient code (in this case, I assume coding to the hardware, as they mention elsewhere), has many benefits, but making it easier to understand is not one of them. A useful comment by the code may help, but that's not a result of efficient coding, that's a result of good commenting practice.
Definitely. Some of the most efficient C code I have seen is incomprehensible to anyone but a C expert and even then it takes several minutes to understand all of the nuances of what might only be a 4-line function.
This is where a good abstraction layer can pay dividends. Of course, a bad abstraction layer (as evidenced by Spolsky's "leaky abstraction" post (http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) on the opposite hand can add immeasurable cost.
But who in their right mind would do the second one?
Obviously you can make code both less efficient and less readable. But starting with code that a competent programmer has written, I find it seldom makes it more readable when you make it more efficient.
It's amazing the sorts of things otherwise sharp programmers will write when they're tired, hurried, and devoting cycles to more interesting/complicated parts of an implementation.
They may just be trying to express the thought that simpler code is both easier to understand and often more efficient. Don't forget Joel Spolsky's contributions to the effort; he's the one who coined the term "architecture astronaut".
Sometimes "efficient" and "easy to understand and be sure is correct" don't have to be mutually exclusive; see this example [1] of Java and Go.
Note that the code is autogenerated, so it should be equally efficient. The Go version also happens to be very simple and no different than most humans would write by hand (without trying very hard to optimize).
I'm not disputing that efficient code can be easier to understand in some instances, just that the assertion that efficient code is easier to understand. Logically, disputing that just requires a single instance of efficient code that is hard to understand. I think that's a trivial enough example to be self evident. In reality, I'm just wondering what they were trying to convey with that statement. I doubt they would have written it without reason, so I'm curious to the reason.
It's possible when they said "efficient", they did not mean "hardware efficient, runs fast, doesn't use a lot of resources" but "developer efficient, faster to write and easier to understand". But that entire phrase as is doesn't make sense, I agree with that.
It's also possible to be a simple mistake. Maybe they went overboard with marketing phrases and claimed something that isn't quite true.
We never said that. The closest I can think of is that simple code (short stack, no DI, layering, etc.) is simpler AND faster at the same time, and in my talk I give specific examples.
To me, that sounds like you are saying simple code is often efficient (or that efficient code can by simple), not that efficient code is simpler (which is how I interpret the original text).
To be very clear, I don't doubt the author (you?) had a good point to make, just that it was unclear what that was from the way it was presented, and to such a degree that it took me out of the flow of reading the post, and perhaps a less ambiguous way of expressing that concept (or omitting it, if it's expressed succinctly elsewhere) would be clearer and more effective.
All, true, which is why I outlined my assumptions and left it open for others to help try to figure out. :)
Unfortunately, if it is just poor communication on their part, we'll likely get no answers that can be considered more likely than others without original author or someone related (work-wise) piping up.
Why would you send a string which is not valid utf8 over the wire in your system? That's the kind of validation that should probably be done prior to that. Or if it hasn't, you can do the validation manually on the receiver. There's no reason to incur the cost on each transmission.
The code samples in your gist aren't doing the same thing. The Java version decodes the UTF-8 stored in the protobuf into Java's native UTF-16 Strings on first access, while Go strings are native UTF-8 and so only a nil check is necessary.
Or are you saying that languages should always use UTF-8 natively? I would agree with you on that, but disagree that this proves your point that "efficient" and "easy to understand and verify correctness" aren't mutually exclusive. pb.getSomeId().charAt(1000) runs in constant time in Java (albeit failing to give correct data if getSomeId() contains codepoints higher than \uFFFF), but pb.GetSomeId()[1000] will give you garbage data in Go if your field contains non-ASCII text. To get a valid codepoint in Go, you'd need to do string([]rune(pb.GetSomeId())[1000]), which runs in linear time and omits the check for valid UTF-8 bytes.
That's a good point, it seems the Java code is checking if the entire string is valid UTF-8 while the Go version isn't. I wonder why the behavior of generated protobufs is different between the two.
In complex systems low-level efficiency is far less beneficial than high-level efficiency. To achieve high-level efficiency you need to have a clear understanding of how the system works as a whole. That is impossible is the code is unintelligible bag of tricks.
I've seen this a lot when working in PHP. I wrote some websites from ground-up (using something similar to my own framework). I optimized int multiple times to great results. The biggest benefits didn't come from making a particular function faster, they came from realizations that large chunks of complexity in templating, routing and permissions checking subsystems simply weren't necessary. It doesn't matter how clever those chunks were written, because I got rid of them completely.
Really hate the separate sites thing. I use a half dozen of them and they are computer related, so it is a pain in the ass always having to register and not being allowed to comment/answer at first, etc.. They have a few improvements now like importing your profile from other sites, but they shouldn't even have so many separate sites in the first place, just a tag or a category or something.
There is a deliberate reason the sites are split. Spolsky talks about in depth here: http://vimeo.com/37309773 Since it's largely sociological, it's also the reason my money is on SE as opposed to Quora in this space.
Essentially they're trying to keep SE from becoming a "chat" community, and keep it focused on being a Q&A community.
With Yahoo Answers, Usenets, even Subreddits, you have people having conversations within posts, which generates noise for people who were looking for answers to questions.
I agree. Diffuses the quality of the related sites. Means you have to deal with twice as many moderators. I dont see why they couldnt have done a "subreddit' approach where you subscribe to the SE category you want and it shows on your homepage.
Hum, each subreddit has its own moderators as well. They also have a front page with questions from the different sites[1], though it's not filtered by the ones you have an account on.
You don't need to create accounts with different passwords on SE sites either. In fact, SE has pushed for single sign-on like probably no other large site did.
But you still have to go to a different site to have any hope of getting the chance to ask your question.
It'd be like going to Home Depot and needing information that is related to building supplies but the employees tell you you must go to a different store down the street to even ask your question.
On Reddit you have to post separately on different subreddits, there's no way to simply "post to Reddit" either.
I don't agree with your analogy; I think it's more akin to going to Home Depot to ask for vegetables. I think it makes perfect sense to have a separation between Cooking.SE, Photography.SE, Christianity.SE, etc.
There are a few edge cases, primarily in the tech sites, but I don't think those disprove the model.
Yeah Im speaking primarily about the technical SE's sites. I agree with the completely isolated SE (Photography, Puzzles, religion).
But when you get the related topics servers, programming, security, tools, web development, web apps... All these items are so closely related it's often disadvantageous to try and ask a vertically silo'd question.
I get they are trying to create a detail 'manual' for all questions, but you get to a point where it's just better to read the manual.
You're off by a factor of 10. I have it at just over 200 page views per second. Also, that's the average load over a month. The same hardware needs to be able to handle peak load.
and they also mention that they could possibly run on a lot less hardware... 7-8 machines + network gear (7 if the load balancer is an appliance, 8 if its a server).
So this is what, a 2000 request/sec peak? Over 11 servers, that's like 200 requests/sec peak per frontend?
The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their transactions in RAM and keep it there (replicated). I know that's what VoltDB is counting on.
Actually most of the websites today are better solved by a battery of cheap boxes.
You simply put a load balancer (Nginx/ELB/HAProxy etc) in front of a fleet of smaller web/application servers that dynamically scale depending on traffic. That way it is cost effective, far more reliable, easier to scale and you can tolerate DC outages better.
As with most systems, the data store is the hard thing to scale out. That requires intelligence at the application level, and that might include CAP issues.
So for us, the SQL data store is the real “up” part of the equation. We have a fair amount of headroom there, so if we can keep sharding (and other data strategies) out of the codebase, so much the better.
(Load-balancing HTTP requests “out” is not a big deal and we are doing that.)
Simply put, this is not true at all. Most of the websites today run happily on a single box and the choice of architecture is frankly irrelevant because of the abysmal amount of traffic they get.
Large, and by large I mean top 100 sites, are large and typically complex systems. Each one is different and bespoke. Making broad generalizations doesn't really work, because typically they have parts that need to be scaled up, parts that need to be scaled out, etc.
Finally, if you read the article you would know that we already use a load balancer with multiple web servers. What we are discussing here is whether it's better to have 100s of cheap/cloud boxes versus a few big ones.
In our use case (we want fast response times) the second solution is demonstrably better.
I took 4 hours at 20 days to get the peak, figuring it probably won't be worse than that. The average over the month isn't a number you'll see in practice. Most perf engineering work seems to go to keep things working nicely while at peak. (And making sure you have good 95th percentile times.)
Peak is more like 2600-3000 requests/sec on most weekdays. Remember that programming, being a profession, means our weekdays are significantly busier than weekends (as you can see here: https://www.quantcast.com/p-c1rF4kxgLUzNc).
It's almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com, meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means we have quite a bit of headroom available. Here's a screenshot of our dashboard taken just now: http://i.stack.imgur.com/HPdtl.png We can currently handle the full load of all sites (including Stack Overflow) on 2 servers...not 1 though, that ends badly with thread exhaustion.
We could add web servers pretty cheaply; these servers are approaching 4 years old and weren't even close to top-of-the-line back them. Even current generation replacements would be several times more powerful, if we needed to go that route.
Honestly the only scale-up problem we have is SSD space on the SQL boxes due to the growth pattern of reliability vs. space in the non-consumer space. By that I mean drives that have capacitors for power loss and such. I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams.
>I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams.
I'm sure some DBAs and devs here would find it interesting.
I'm most impressed that all your traffic is passing through a single active HAproxy which is also terminating SSL?
TLS Session IDs and tickets of course are absolutely essential, but I'd be curious how many peak TPS (number of full handshakes / sec) you are seeing on HAproxy.
The alternative, fanning out your SSL termination to your IIS endpoints, unfortunately means running HAproxy at L2, so you lose all your nice logging.
I'm not sure how many full handshakes/sec we're running - the logging isn't as readily available there because we're explicitly not logging the listeners involved on separate proc binds. We are logging the traffic on the port 80 front end they feed. We have tuned the performance there via best practices and taking into account very recent HAProxy changes for various SSL windows added by Willy, Emeric and help from Ilya Grigorik. They're all awesome and tremendously helpful. We are on a version slightly newer than 1.5.2 (we build straight from the repo).
Even with all that SSL we're only running around 15% CPU at peak so it's not having any trouble. Most of that CPU usage does come from the SSL termination though - it ran around 3-5% CPU before. We're also working on much larger infrastructure changes that mean SSL termination much more local to our users, which means the HAProxy load will drop to just about nothing again. I'm working on a followup to my previous SSL post: http://nickcraver.com/blog/2013/04/23/stackoverflow-com-the-... that will explain why it's not already on for all hits...I think hacker news will have fun with hat one.
All that being said, there's no reason we can't forward that syslog traffic from the listeners to our logging infrastructure to get at least a counter out of it. If you're curious how we log, I'll explain a bit.
This lets us forward on custom headers to logging that the web servers are sending back. It tells us how much time we spent in SQL, redis, etc. We can graph it (bottom of the first link) and identify the root cause of any issues faster. It also handles parsing of that traffic from the syslog entry already, so we use it as a single logging point from HAProxy and it handles the JSON generation to forward that traffic data into logstash (a 300 TB cluster we're working on setting up right now).
As soon as we get the logstash piece working well and dogfooded thoroughly, I'll poke Kyle to open source Realog so hopefully some others can get use from it.
I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams.
Yes! please yes. The occasional real world gem like that is what makes scrolling past a thousand javascript-framework/obscure-language/alternative-datastore posts here worth it (for me).
Relatively new stack dev here. I came in on the other side of the fence of a lot of these technologies (bread and butter is Python in Flask with Mongo on Heroku on a mac) but since I started here, I've been constantly and pleasantly surprised by how performant everything here has been despite my biases. It's mighty fun.
This really needs to stop. Go to stackexchange.com and you'll find that more than half of the HTTP requests are to cdn.sstatic.net.
Looking up the IP addresses for cdn.sstatic.net returned five entries for me, all owned by CloudFlare. None of the CloudFlare servers that they are using seem to be in that 25 count.
Sure, these are all for static assets, that isn't the point. There are way more than 25 servers being used to serve the StackOverflow sites.
Stack Exchange dont own their CDN. If you really start couting external servers, you would be adding DNS, possible client side proxy servers, client side network infrastructure, etc... these are things that OTHER people do, not SO.
I would say though, CDN doesn't fall in the same category as DNS. Serving static assets off other networks does take a huge load off the SE infrastructure and should probably be mentioned in the article.
It's how many servers you would need to provision to do the same, with the same code base. It's basically the number of directly active servers our code runs on.
Sure, we use a CDN, but not for CPU or I/O load, but to make you find your answers faster.
Sort of true, but static assets are also trivial to serve. What they get by using CloudFlare is getting their assets as close to end users as possible to reduce latency. This isn't a complex problem to solve (for a single site), just a really expensive one that very few single app companies could ever justify solving themselves. I believe Google and Amazon are the only ones with their own significant CDN infrastructure, everyone else outsources it.
"Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there’s no compelling reason to change." <- there's your problem :-)
What I'd like to see is how much cache hit they add on each level of cache. I remember some presentation of the facebook images stack where less than 10% (if I remember well) of the requests actually hit the disks; it would be interesting to see the patterns for the whole SE galaxy.
As someone mentioned above, SQL Server hits memory first, and as they have 384GB of memory, most of the requests would sit there. That's just on the DB server. In the linked article it all authenticated requests hit the DB, with anon users getting a cached copy.
I'm grateful to them them for being so transparent about how their systems work. It's rare to get this level of detail on how a large, successful site operates.
You can email us (as George said) or you can just punch your email address into the account recovery tool and get an automated email telling you what to click within the space of a few minutes: http://stackoverflow.com/users/account-recovery
We... probably need to make that link a bit more obvious.
Stack Overflow is the example I'm forever using when arguing against premature scale-out. For non-trivial applications scale-out has substantial complexity costs attached to it, and the overwhelming majority of applications will never truly need it.
It's frustrating to see time wasted obsessing over trying to maintain eventual consistency (or chasing bugs when you failed to do so) on systems that could quite happily run on a single, not that beefy machine.
>For non-trivial applications scale-out has substantial complexity costs attached to it
Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out.
From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of their forum is not an extraordinarily complex problem.
By nontrivial I mean 'application which, in your document database, requires multi-document updates to perform some individual logical operations'. This is a rather low bar :-).
The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course there are real reasons to use scale out - I simply advocate thinking hard about whether your problem will ever truly need it before taking the substantial complexity hit of coding for it.
Er, I don't seem to be able to edit, but I guess I should specify that I'm really talking about scale-out of writes here. Read scalability is an obviously easier problem..
My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this.
ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.
IME that always happens when you try to perform tdd in combination with Javaesque encapsulation. The good solution to the problem is to not be so afraid of classes seeing each others internals. The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.
> The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.
Not doing this will cause much self-righteous snickering from some.
You do realize that most efficient DI framework only inject the dependency once.
Separating Controller, Repository, and Services are good practices as well and let's be honest, we're looking at 3 methods layer at most.
Here's what happened in Java:
1. When you deploy your WAR, the DI will inject necessary component _once_ (and these components only instantiated _once_ for the whole web-app so there you go, Singleton without hardcoding).
2. A request comes in, gets processed by the pumped-up Servlet (already DI-ed). If the servlet has not been instantiated, it will be instantiated _once_ and the instance of the Servlet is kept in memory.
3. Another request comes in, gets processed by the same pumped-up Servlet that is already instantiated and has already been injected with the component (no more injection, no more instantiation, no more Object to create...)
So I've got to ask this question: what GC problem we're trying to solve here?
Some of the static methods are understandable but if Component A requires Component B and both of them have already been instantiated _once_ and have been properly wired up together, we have 2 Objects for the lifetime of the application.
I'd pay for a wee bit extra hardware for the cost of maintainable code ;)
Discipline (and knowledge of object graph) definitely help to reach to that point.
C# or Java, the whole request pipeline processing should be more or less the same. Unless one platform does things less efficient than the other.
Rails and Django do things differently as to my knowledge they do it by spawning processes instead of threads. There are app-server for Rails or Django that may use threads for efficiency/performance reason but I am under the impression the whole LAMP stack is still 1 request 1 process (even though they re-use those processes from a pool of already allocated processes).
Insofar as this is your guess, it's down to the language. In Ruby or Javascript it's very easy to TDD. In Java, much less so. (I'm not sure where C# lies on the spectrum).
At the risk of exposing my slow transmutation into a hipster programmer, I and a colleague found that mocking in Go was much easier than we anticipated, thanks to the way interfaces work.
Marco from Stack Overflow here. We try to measure everything, including GC operation. We know that layers of indirection increase GC pressure to the point of noticeable slowness :-)
It's not a theoretical risk we're trying to avoid here, it's a real problem that needed solving.
Can someone help me understand what they mean by: "Garbage collection driven programming. SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods. While extreme, the result is highly performing code."
I assume it means using C# more as a functional language than an OO language, so that most garbage collection is of short-lifetime objects and therefore cheaper to collect (generation 0).
I take that statement to mean C# as C. Functional languages are terrible for garbage collection. Static methods are great since they are static, loaded once and probably in-lineable by the compiler/JIT.
The usual wisdom in clr-land is that the generational garbage collector can cope just fine with a lot of small, short lived objects. It absolutely doesn't like big, long lived objects that reach G1, G2 or the large object heap and must be traversed for a full gc.
This means that normally, you want to avoid data structures that introduce a lot of objects trees, linked lists and stuff like that. In some circumstances you might lose some speed, but win big time when the full GC comes. Additionally, It may be worthwhile to consider avoiding heap allocations by organizing your data as value types instead of ordinary objects.
It depends on your definitions of "a lot" is. It can scale pretty high by default, but when you're doing hundreds of millions of objects in a short window, you can actually measure pauses in the app domain while GC runs. These have a pretty decent impact on requests that hit up against that block.
The vast majority of programmers need not worry about this, but if you're putting millions of objects through a single app domain in a short time, then it's a concern you should be aware of. This applies to pretty much any managed language.
Are you sure that this is really because of Gen0 collections all by themselves, and not because the short cadence of collections leads to a lot of premature Gen1 promotions? I could imagine something like
var a = InitGiantArray<Foo>();
var b = InitGiantArray<Bar>();
could leave you with a lot of foo garbage in Gen1 or even Gen2, right were it really hurts. On the other hand I'd be surprised if something akin to
do {
Tuple.Create(new object(), new object());
} while (true);
but with something useful thrown in would suffer much under GC pauses. Looks like I have to test it :I)
This is the fascinating part to me, their SSD have not failed:
Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.
I would agree that hundreds of drives is a large enough sample that you would expect to see some failures.
That said, all the SSDs I've had have taken repeated pounding with no complaints. I accidentally swapped a few terabytes to swap on an SSD, and was simply surprised that the job I was doing finished faster.
Yep, still true. We lost one Intel 910 drive (PCIe SSD), and that was very abnormal - died so soon it was almost DOA. We hooked up directly with Intel for them to The replacement is still going strong as is another 910 we have.
All of those 2.5" Intels though, still trucking along! We're looking at some P3700s PCIe NVMe drives now, blog post coming about that.
So doing it right works. Nice :)
I like these articles as it gives good examples/help supporting decisions when you dont want the crappy hack, but the do it right instead.
I love that all the traffic is running through a single HAProxy box - a software load balancer (vs a hardware LB like F5).
And they've moved SSL termination to it.
That's a great quality product, and easy to setup.
Edit: I work I'm software that supports MySQL,PG and SqlServer. SqlServerr seems to be the most stable and consistent in performance - they're hard to kill! One of my few liked MS products :D
277 comments
[ 2.7 ms ] story [ 256 ms ] threadhttps://github.com/tmsmith/Dapper-Extensions
Look towards the end at the tests for example of how to hook up a custom type (it's pretty simple).
(not to be confused with LINQ to SQL from Microsoft)
A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example we (and by "we" I mean, "I made Marc Gravell do it") added literal replacement to Dapper to help with query parameterization which allows you to use things like filtered indexes. In dapper we also intercept the SQL calls to dapper and add add exactly where it came from. Here's what that looks like, replicate for the other methods:
And here's the our marking method tailored for our code, but you get the idea: This results in a comment at the top of the query like this: It's such a simple approach using the caller member attributes, but it saves so much time tracking things down.Also StackOverflow often practices "scale-up" instead of "scale-out".
http://blog.codinghorror.com/farewell-stack-exchange/
[0]: http://www.discourse.org/
Usually you want your solution to be just right for the problem; solve the essential complexity but don't introduce any accidental complexity.
And this is compounded by people who have little ability to troubleshoot performance issues. It's quite easy to hunt down the cruddy SQL queries in a DB, or realize that you spinning rust is too slow. But when it starts to come down to things like a blocking network fabric, which has some big fat buffer between two servers, that is killing your transaction speed - many will just start to blame the devs.
Of course, the second point is compounded by the first - the less systems thinking that went into the design in the first place, the harder it is to produce accurate hypotheses about the system in order to troubleshoot.
Whilst there is a lot to be said for the "right tool for the job", if you watch an artisan crafting something, you'll realize that despite having a huge number of tools, they actually get by with relatively few. This is a generalization, but they only use the full range when doing something new, solving a particularly tricky problem, etc. "All the gear and no idea" is certainly applicable in many start ups.
I'm often surprised at the paucity of test-coverage in relatively large companies.
http://nerds.airbnb.com/testing-at-airbnb/
Can any SO devs give us more details on "not many tests"? Or how many bug reports get filed vs rate of change of software?
You can look at reported bugs here: http://meta.stackexchange.com/questions/tagged/bug -- it's pretty frequent, at least one every couple of hours or so. The last commit in our "Tests" project was 14 days ago, where as the last commit in the code base was 4 minutes ago.
If only for the wailing, gnashing of teeth and rending of garments.
Also, some might remember when they had Uncle Bob on the podcast, the topic being unit testing, and iirc Bob preaching the gospel (~ you must write plentiful unit tests, or else) and Jeff and Joel (especially) more or less saying they don't quite get why.
(Am I remembering this story correctly? I swear that's how I remember it.)
Then after a while it seems like an obvious way to work.
I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.
I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system.
1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...
Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months.
A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).
10tb = roughly $1,000 per month
They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.
The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc., but apart from these, mostly everything seems “normal” to me.
this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this
Says in the article:
Windows 2012 is used in New York but are upgrading to 2012 R2 (Oregon is already on it).
This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better SMB, and such...but not enough to make the upgrade worth it.
Nick Craver - Stack Exchange Sysadmin & Developer
Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.
[0] http://www.dell.com/us/business/p/poweredge-r920/pd
Still, I agree re. the difference in price. I'd buy too
(I honestly don't know, it's just surprising.)
Databases have inherent locking problems that takes more resources to resolve with more machines, more so in some database systems than others. When scaling you often hit a point where you get down to macro logistics, so imagine a car highway: You can get more throughput by adding more lanes. But what that all cars are going to merge to one lane at some point in the middle of the trip (because of a tunnel, bridge or something: it's impossible to have more than one lane at that point)? Now you won't get any throughput benefits of the multiple lanes after all! Just more latency because you get queues up until the single lane and because of the queues all cars are going slow and need to accelerate on the single lane, so the average speed is low too. You are also having too much resources after the single lane because you can never fill all those lanes. It might be better to have one beefed-up single-lane road all the way that people can go fast on. Basically: Remove locks and you get better overall performance.
Yes, this is on the expense of HA. Yes, the costs of scaling up grows asymptotically faster than the costs of scaling out. So this is definitely a trade-off in some sense.
They just run everything through one live database all the time under normal conditions. Personally, I think this is much simpler to manage than to be regularly spreading out queries over many queries.
edit: wrong number
If I remember correctly they're using Dell R720's for their SQL Server machines. At current Dell retail pricing 384GB of RAM would be about $5,900.00.
At that price I'd vote for throwing money at raw hardware and reaping the benefits of caching the database in RAM, versus trying to pay programmers to come up with clever techniques to deal with having less RAM.
64 bit computers has a memmory limit so large that it is not understandeble.
The ram is still lightning fast compared to other stuff.
Memory support for versions of Windows:
The more RAM you have the better without exception here.
No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.
If nothing else, it keeps less data in each table, so queries should be faster due to smaller datasets.
Sounds like they know what they are doing.
for example, do not put every single customer of an saas solution into a separate database.
however, inexperienced developers and teams are not good enough, or fast enough, to do that, and end up painting themselves into a corner when they have thousands of live customers to support and need to change their entire application architecture + database schema when their backup, replication, and housekeeping tasks choke on 1,000+ databases.
and oftentimes, one-customer-per-db also means one-instance-of-application-per-customer, another anti pattern to avoid.
we see this kind of stuff ALL the time. it happens a lot - people make terrible decisions and then are stuck with them 5 years down the line and are looking at a monumental cost to redo. not everyone is experience enough to work their way out of an awful situation like that.
* Currently three years of free licenses, all software you download during that time is free forever, and discounted MSDN subs thereafter.
If you happen to be familiar with the MS stack and are able to factor the license fees into your business plan there is no strong reason against and many for using that particular stack.
MS programmers tend to be a bit more expensive than people working with one of the many Linux offerings but that's not such a huge difference that it would become the deciding factor and if anything the MS stack is more performant than the Linux one on the same hardware (and I write that as an anti-MS guy, I strongly believe that politics and tech should be kept separate when it comes to discussing systems relative to each other).
So even if there are lots of reasons why I would not have made that particular choice I can completely understand why the SO people made their choice the other way.
This is provably not true.
http://w3techs.com/technologies/overview/web_server/all
http://commons.wikimedia.org/wiki/File:Operating_systems_use...
http://w3techs.com/technologies/overview/operating_system/al...
There is a good reason majority of the World-Wide-Web is run on 'Nix stacks -- and there is a good reason majority of servers in general are 'Nix stacks. Also, an overwhelming percentage of super-computers (I know we are talking about webservers, but it illustrates performance capabilities) are running a 'Nix stack.
There is also cost involved, as well as flexibility of the stack. With a 'Nix stack, it's infinitely flexible, not so much the case with a Microsoft stack. Also the default MS Stack comes with a lot of additional OS overhead that is not present in the 'Nix stack, which reduces any said box's scalability.
Without trying to turn this into some sort of flame war - I was merely trying to suggest that the MS stack was not/is not the best choice for a highly scalable website. Take the top 10 websites -- they all run on 'Nix. The top cloud providers (except MS Azure), all 'Nix. These are companies that can easily afford MS licensing, so that's not part of the equation.
A MS stack at this scale is unusual to say the least.
That's not to say it won't scale (as evidenced by the SE team), but it doesn't mean there isn't a better alternative that saves more money and scales better with less hardware, etc.
Which is a pretty smart decision. Whether an MS stack at this scale is unusual does not say anything at all about whether or not it performs well.
However, I have to disagree on your assertion the 'Nix's are less performant than the MS stack. The top tier web companies are not running 'Nix because TCO is lower; for most of these companies licensing costs are negligible and if it helped to scale better, it may even save them money going with MS stack... but they don't go with a MS stack...
Sometimes a RHEL license can even cost more than a MS license. Couple that with an Oracle DB back-end, and you easily have a much more costly setup than the MS equivalent. It's not about the money.
These companies are choosing the 'Nix stack because it is performing in an entirely different level than the MS stack. Everything from tiny embedded systems with 64k ram, up to monster systems with TB's of ram.
> "I Contribute to the Windows Kernel. We Are Slower Than Other Operating Systems. Here Is Why." http://blog.zorinaq.com/?e=74
Could you please provide some evidence to support that?
http://www.webperformance.com/load-testing/blog/2011/11/what...
Keep in mind that it hardly ever is the webserver that is the bottle neck.
My theory on why this is the case is very simple: MS can afford to throw vast amounts of money at optimizations that are next to impossible in Linux simply because the coupling between the layers in Linux is looser. And that's a good thing, it translates into better security and fewer bugs.
As always, optimization alone is not a reason enough to go down a certain path. But for raw speed on requests it's fairly hard to beat IIS, if that's what you're after (I'm usually not, and even when it matters I can comfortably saturate most outbound links from a single server doing light processing, and as soon as the processing becomes the bottle-neck the CPU cache size, RAM and so on matter more than your OS, but given identical hardware a 'dirty' approach should yield better results at a cost of unreliability/complexity).
It turns out there are kernel-mode webservers (or there were, at least), which seriously outperform user-mode ones. https://www.usenix.org/legacy/events/usenix01/full_papers/jo... (page 11; article is old, but anyway)
IMO, it has one of the best tooling sets of any platform out there (Visual Studio) which saves lots of programmer time and increases productivity. With BizSpark the initial cost argument is out...so how's that not pragmatic?
I'm biased of course, but I think it worked out pretty well.
Nick Craver - Stack Exchange Sysadmin & Developer
prag·mat·ic adjective \prag-ˈma-tik\
: dealing with the problems that exist in a specific situation in a reasonable and logical way instead of depending on ideas and theories
-- http://www.joelonsoftware.com/AboutMe.html
> Microsoft infrastructure works and is cheap enough
not as cheap as just paying your team to maintain the boxes. I wonder how many times in the past few years SE has needed to call Microsoft for support? Would a RHEL license still be cheaper (probably)? If something like CentOS then there would be no support cost unless you need to bring in an outside contractor (for a particularly nasty issue).
Why is the Microsoft stack not appropriate? I am not aware of any spectacular performance difference between Apache and IIS, or Windows Server 2012 and Linux, and clearly the SO team is not religiously Microsoft, so it's hard to see why they wouldn't have switched if it was a clear win. MSSQL is actually, in my experience coming from MySQL, amazing - I'm back on MySQL and Postgres now and I often wish I wasn't. I'd say it's the most underrated of MS's products, probably because of the expense of the Enterprise/Datacenter editions.
I guess you're gonna have to define "more appropriate" for me, I have a feeling we'll have a fundamental disagreement there. We could (and have) run the entirety of our peak load with 1 SQL server and 2 web servers (and not pegging them). Don't forget we run with a crazy amount of headroom at all times, by design. I'm not sure how much better you picture that scaling on a linux environment.
> not as cheap as just paying your team to maintain the boxes
That makes a lot of assumptions about the team, their expertise, and what support issues would arise.
> I wonder how many times in the past few years SE has needed to call Microsoft for support
In the past 4 years I've been here? Twice. Both to report bugs in CTP versions of SQL server. At no cost, and we improved SQL server as a result of being testers. We have a very good relationship with Microsoft and talk to the developers that make the tools and platform we use in order to make life better for both of us. We do the same thing for redis, Elasticsearch, etc. It's the same reason we open source almost all of the tools we make.
> If something like CentOS then there would be no support cost unless you need to bring in an outside contractor (for a particularly nasty issue)
We use CentOS for all our linux systems and are deploying new servers on CentOS 7 now. We'll me migrating the others in the coming months. That doesn't mean it's free. Developer or sysadmin time to control Puppet deployments and such still eat some factor of time.
Nick Craver - Stack Exchange Sysadmin & Developer
Even if it was the best choice at the time to go the Microsoft route, it may not be in the future... however -- SE has zero choice now otherwise they'd have to re-write their entire product... that sucks as a business because you have little choice over your own product now.
This seems like a bit of a waste of resources to try and run an internal bespoke application on every platform imaginable. Now, if you are developing a product that you are then going to sell to other people to run on their own hardware this make more sense.
There are some very big benefits to writing to a specific platform, especially in the performance space.
In the end of the day it is a trade off between trying to eliminate every 3rd party dependency (next to impossible) or picking a solution or company you think will be around for a long time, and forging strong relationships with them.
We regularly talk to people at Microsoft (and all of the people who create and build to tools we use), give them feedback, and get bugs fixed. There is very little that comes down the pipe from them that we are not away of ahead of time and in some cases have helped shape through early access programs.
> SE has zero choice now otherwise they'd have to re-write their entire product...
This is not true at all, we have choices if MS decided to blow everything up. Not great choices, but we have them. Choosing between two or three crap options does not mean that options do not exist.
-George (SE Sysadmin)
http://blog.stackoverflow.com/2008/04/podcast-1/
Is performance _really_ an issue here?
For SO, if we guess they max out around 2000 req/sec, then if a bunch of objects are being allocated for each request, there could be added GC pressure. From my own experience developing much higher-scale managed code, allocating objects is a real pain and people will take a lot of effort to avoid it.
It's a bit of a contrived example, as you're more likely to do the third method if you're trying to get some sort of summary data about an order out which would mean you don't really need/want to load the whole order object.
Isn't that comparing apples to oranges? Taking any application built for large servers and putting the same application onto a cloud-based architecture will be more expensive.
Cloud-based architecture requires ground up differences in how the application is built. Now whether or not it is better to use a cloud-based approach or traditional bare metal is highly subjective and isn't my point.
On another site, in another context, it probably would be, but here it's really presented as a contrast of scale-up versus scale-out - something the regular audience of highscalability will certainly grok.
In context...
'Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t been a problem because they’ve been quite successful at sizing their system correctly.'
It's an acknowledgement of their relatively unique strategy and the short list of caveats that make it possible.
I'm curious what the reasoning is for "Efficient code ... makes code easier for programmers to understand". To my mind, efficient code (in this case, I assume coding to the hardware, as they mention elsewhere), has many benefits, but making it easier to understand is not one of them. A useful comment by the code may help, but that's not a result of efficient coding, that's a result of good commenting practice.
Obviously you can make code both less efficient and less readable. But starting with code that a competent programmer has written, I find it seldom makes it more readable when you make it more efficient.
for i = 1 to 100 if x = this do that and something with i else something else with i
easy to read, less efficient than: if x = this for i = 1 to 100 do that and something with i else for i = 1 to 100 do something else with i
from my experience, compilers usually dont hoist very well
edit: sorry, spaces didnt come out well
Surely you jest. I don't know who, but their handiwork is everywhere.
[0] http://www.joelonsoftware.com/articles/fog0000000018.html
Note that the code is autogenerated, so it should be equally efficient. The Go version also happens to be very simple and no different than most humans would write by hand (without trying very hard to optimize).
[1] https://gist.github.com/shurcooL/9f94bbd021b4693cf584
It's possible when they said "efficient", they did not mean "hardware efficient, runs fast, doesn't use a lot of resources" but "developer efficient, faster to write and easier to understand". But that entire phrase as is doesn't make sense, I agree with that.
It's also possible to be a simple mistake. Maybe they went overboard with marketing phrases and claimed something that isn't quite true.
To be very clear, I don't doubt the author (you?) had a good point to make, just that it was unclear what that was from the way it was presented, and to such a degree that it took me out of the flow of reading the post, and perhaps a less ambiguous way of expressing that concept (or omitting it, if it's expressed succinctly elsewhere) would be clearer and more effective.
Unfortunately, if it is just poor communication on their part, we'll likely get no answers that can be considered more likely than others without original author or someone related (work-wise) piping up.
The Go code is very concise and the Java code is very verbose.
The Go code does look more efficient but what is ImpressionData? Where is it checking the string is valid utf8?
Or are you saying that languages should always use UTF-8 natively? I would agree with you on that, but disagree that this proves your point that "efficient" and "easy to understand and verify correctness" aren't mutually exclusive. pb.getSomeId().charAt(1000) runs in constant time in Java (albeit failing to give correct data if getSomeId() contains codepoints higher than \uFFFF), but pb.GetSomeId()[1000] will give you garbage data in Go if your field contains non-ASCII text. To get a valid codepoint in Go, you'd need to do string([]rune(pb.GetSomeId())[1000]), which runs in linear time and omits the check for valid UTF-8 bytes.
Of course, as others have mentioned, it is validating the UTF-8 but the Go version is not.
I've seen this a lot when working in PHP. I wrote some websites from ground-up (using something similar to my own framework). I optimized int multiple times to great results. The biggest benefits didn't come from making a particular function faster, they came from realizations that large chunks of complexity in templating, routing and permissions checking subsystems simply weren't necessary. It doesn't matter how clever those chunks were written, because I got rid of them completely.
I think this is what the author speaks about.
Essentially they're trying to keep SE from becoming a "chat" community, and keep it focused on being a Q&A community.
With Yahoo Answers, Usenets, even Subreddits, you have people having conversations within posts, which generates noise for people who were looking for answers to questions.
[1] http://stackexchange.com/
I don't agree with your analogy; I think it's more akin to going to Home Depot to ask for vegetables. I think it makes perfect sense to have a separation between Cooking.SE, Photography.SE, Christianity.SE, etc.
There are a few edge cases, primarily in the tech sites, but I don't think those disprove the model.
But when you get the related topics servers, programming, security, tools, web development, web apps... All these items are so closely related it's often disadvantageous to try and ask a vertically silo'd question.
I get they are trying to create a detail 'manual' for all questions, but you get to a point where it's just better to read the manual.
Or, to put it another way; 1 server per 8 pageviews per second
Does not seem that efficient when you put it like that. Even a resource hog like a wordpress install can eclipse that level.
*made correction- still not impressed
The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their transactions in RAM and keep it there (replicated). I know that's what VoltDB is counting on.
Not everyone has google-like problems that are betters solved by a battery of cheap boxes.
You simply put a load balancer (Nginx/ELB/HAProxy etc) in front of a fleet of smaller web/application servers that dynamically scale depending on traffic. That way it is cost effective, far more reliable, easier to scale and you can tolerate DC outages better.
So for us, the SQL data store is the real “up” part of the equation. We have a fair amount of headroom there, so if we can keep sharding (and other data strategies) out of the codebase, so much the better.
(Load-balancing HTTP requests “out” is not a big deal and we are doing that.)
Simply put, this is not true at all. Most of the websites today run happily on a single box and the choice of architecture is frankly irrelevant because of the abysmal amount of traffic they get.
Large, and by large I mean top 100 sites, are large and typically complex systems. Each one is different and bespoke. Making broad generalizations doesn't really work, because typically they have parts that need to be scaled up, parts that need to be scaled out, etc.
Finally, if you read the article you would know that we already use a load balancer with multiple web servers. What we are discussing here is whether it's better to have 100s of cheap/cloud boxes versus a few big ones.
In our use case (we want fast response times) the second solution is demonstrably better.
It's almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com, meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means we have quite a bit of headroom available. Here's a screenshot of our dashboard taken just now: http://i.stack.imgur.com/HPdtl.png We can currently handle the full load of all sites (including Stack Overflow) on 2 servers...not 1 though, that ends badly with thread exhaustion.
We could add web servers pretty cheaply; these servers are approaching 4 years old and weren't even close to top-of-the-line back them. Even current generation replacements would be several times more powerful, if we needed to go that route.
Honestly the only scale-up problem we have is SSD space on the SQL boxes due to the growth pattern of reliability vs. space in the non-consumer space. By that I mean drives that have capacitors for power loss and such. I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams.
Nick Craver - Stack Exchange Sysadmin & Developer
I'm sure some DBAs and devs here would find it interesting.
TLS Session IDs and tickets of course are absolutely essential, but I'd be curious how many peak TPS (number of full handshakes / sec) you are seeing on HAproxy.
The alternative, fanning out your SSL termination to your IIS endpoints, unfortunately means running HAproxy at L2, so you lose all your nice logging.
Even with all that SSL we're only running around 15% CPU at peak so it's not having any trouble. Most of that CPU usage does come from the SSL termination though - it ran around 3-5% CPU before. We're also working on much larger infrastructure changes that mean SSL termination much more local to our users, which means the HAProxy load will drop to just about nothing again. I'm working on a followup to my previous SSL post: http://nickcraver.com/blog/2013/04/23/stackoverflow-com-the-... that will explain why it's not already on for all hits...I think hacker news will have fun with hat one.
All that being said, there's no reason we can't forward that syslog traffic from the listeners to our logging infrastructure to get at least a counter out of it. If you're curious how we log, I'll explain a bit.
We feed it into a logging display called Realog built by Kyle Brandt and Matt Jibson in Go. Here are a few snapshots of the dashboard: http://i.stack.imgur.com/OIqhm.png http://i.stack.imgur.com/iQOP9.png http://i.stack.imgur.com/bfpUb.png http://i.stack.imgur.com/JZNy6.png
This lets us forward on custom headers to logging that the web servers are sending back. It tells us how much time we spent in SQL, redis, etc. We can graph it (bottom of the first link) and identify the root cause of any issues faster. It also handles parsing of that traffic from the syslog entry already, so we use it as a single logging point from HAProxy and it handles the JSON generation to forward that traffic data into logstash (a 300 TB cluster we're working on setting up right now).
As soon as we get the logstash piece working well and dogfooded thoroughly, I'll poke Kyle to open source Realog so hopefully some others can get use from it.
Nick Craver - Stack Exchange Sysadmin & Developer
This really needs to stop. Go to stackexchange.com and you'll find that more than half of the HTTP requests are to cdn.sstatic.net.
Looking up the IP addresses for cdn.sstatic.net returned five entries for me, all owned by CloudFlare. None of the CloudFlare servers that they are using seem to be in that 25 count.
Sure, these are all for static assets, that isn't the point. There are way more than 25 servers being used to serve the StackOverflow sites.
Sure, we use a CDN, but not for CPU or I/O load, but to make you find your answers faster.
I would love that problem.
(I'm an SE SysAdmin )
We... probably need to make that link a bit more obvious.
It's frustrating to see time wasted obsessing over trying to maintain eventual consistency (or chasing bugs when you failed to do so) on systems that could quite happily run on a single, not that beefy machine.
Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out.
From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of their forum is not an extraordinarily complex problem.
The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course there are real reasons to use scale out - I simply advocate thinking hard about whether your problem will ever truly need it before taking the substantial complexity hit of coding for it.
I don't understand this at all. What does TDD have to do with reducing garbage collection?
ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.
Not doing this will cause much self-righteous snickering from some.
Separating Controller, Repository, and Services are good practices as well and let's be honest, we're looking at 3 methods layer at most.
Here's what happened in Java:
1. When you deploy your WAR, the DI will inject necessary component _once_ (and these components only instantiated _once_ for the whole web-app so there you go, Singleton without hardcoding).
2. A request comes in, gets processed by the pumped-up Servlet (already DI-ed). If the servlet has not been instantiated, it will be instantiated _once_ and the instance of the Servlet is kept in memory.
3. Another request comes in, gets processed by the same pumped-up Servlet that is already instantiated and has already been injected with the component (no more injection, no more instantiation, no more Object to create...)
So I've got to ask this question: what GC problem we're trying to solve here?
Some of the static methods are understandable but if Component A requires Component B and both of them have already been instantiated _once_ and have been properly wired up together, we have 2 Objects for the lifetime of the application.
I'd pay for a wee bit extra hardware for the cost of maintainable code ;)
Discipline (and knowledge of object graph) definitely help to reach to that point.
Rails and Django do things differently as to my knowledge they do it by spawning processes instead of threads. There are app-server for Rails or Django that may use threads for efficiency/performance reason but I am under the impression the whole LAMP stack is still 1 request 1 process (even though they re-use those processes from a pool of already allocated processes).
At the risk of exposing my slow transmutation into a hipster programmer, I and a colleague found that mocking in Go was much easier than we anticipated, thanks to the way interfaces work.
It's not a theoretical risk we're trying to avoid here, it's a real problem that needed solving.
This means that normally, you want to avoid data structures that introduce a lot of objects trees, linked lists and stuff like that. In some circumstances you might lose some speed, but win big time when the full GC comes. Additionally, It may be worthwhile to consider avoiding heap allocations by organizing your data as value types instead of ordinary objects.
The vast majority of programmers need not worry about this, but if you're putting millions of objects through a single app domain in a short time, then it's a concern you should be aware of. This applies to pretty much any managed language.
Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.
That said, all the SSDs I've had have taken repeated pounding with no complaints. I accidentally swapped a few terabytes to swap on an SSD, and was simply surprised that the job I was doing finished faster.
All of those 2.5" Intels though, still trucking along! We're looking at some P3700s PCIe NVMe drives now, blog post coming about that.
For comparison, for a while we nearly had a 1 to 1 dev to killed (consumer) SSD in work machines (see: http://blog.codinghorror.com/the-hot-crazy-solid-state-drive...). Granted, we did adopt them (see: http://blog.codinghorror.com/the-state-of-solid-state-hard-d...) for local dev use much earlier than we started putting them in servers.
I've killed 2 in my time here, though I do horrible big-ish data things that are either not SSD friendly or actively hostile.
And they've moved SSL termination to it.
That's a great quality product, and easy to setup.
Edit: I work I'm software that supports MySQL,PG and SqlServer. SqlServerr seems to be the most stable and consistent in performance - they're hard to kill! One of my few liked MS products :D