Ask HN: How did you become a hardcore back-end developer?

The best news is that these days, you can build up these skills using a $1k box with Linux or BSD. Years ago, you needed to get a job first because systems were in the order of $millions and they wouldn't fit in your average spare room.

You'll also need to demonstrate so CS/SE chops, because mucking up a big back-end system is not like a web page that occasionally crashes, it can cost $10k's per hour while it's down.

But, nothing beats working directly with geniuses. Earlier this year I made a change (at my last company) that increased the number of simultaneous users by well over an order of magnitude. The change was known and had been tried by others in the group, but was deemed infeasible. I didn't come up with the magic change needed, I found how to apply it. And what I learned in the process is applicable outside of that. Without working directly solving the problems, it's hard to learn how.

2. Remove bottleneck.

3. Repeat.

4. Every once in a while, make a bold move to throw something out that can no longer work that way and replace it with something more scalable. But while this is important, it comes up less often than you might think.

The difference is that you spend a lot more time in that loop than a desktop dev, but if you understand programming it isn't a special black art until the very, very top end.

The other thing to get is that it's always about buying time rather than solving the problem forever. The goal is to have bought enough time that you don't have to be stuck in a local optima or make panicked decisions.

You are fortunate that you live in the age of cloud computing. For instance, you can spend $10 for a day and get access to more compute resources than most people could hope for after months of budget proposals.

Find a problem, solve it, launch it, test it, find bottleneck, kill it. Repeat this enough times and you can start to a feel for where bottlenecks will happen and how fail happens.

For distributed systems there are two main things to learn from: good papers and good deployed systems. A researcher named Leslie Lamport invented a number of key ideas such as Lamport timestamps and Byzantine failure models. Some other basic ideas include quorums for replicated data storage and the linearizability consistency model. Google has published some good papers about their systems like MapReduce, BigTable, Dapper, and Percolator. Amazon's Dynamo paper was very influential. The Facebook engineering "notes" blog also has good content. Netflix has been blogging about their move to AWS.

Every software engineer needs to manage complexity, but there are some kinds of complexity that only show up in big systems. First, your system's modules wil be running on many different machines. The most important advice I can give is to have your modules separated by very simple APIs. Joshua Bloch has written a great presentation on how to do that. Think about what happens when you do a rolling upgrade of a 1,000 node system. It might take days to complete. All the systems have to interoperate correctly during the upgrade. The fewer, simpler interactions between components the better.

The best advice I know of about operating a big distributed system is this paper[1] by James Hamilton. I won't repeat its contents, but I can tell you that every time that we didn't follow its guidelines we ended up regretting it. The other important thing is to get really good with the Unix command line. You'll need to run ad-hoc commands on many machines, slice and dice log files, etc.

How did I learn these skills? The usual mix of how people learn anything - independent study, school, and building both experimental and production systems.

1. http://www.usenix.org/event/lisa07/tech/full_papers/hamilton...

In '06 I joined a startup and we needed to scale. I hadn't had experience with this stuff and neither did most people on my team...so here is what we did.

* Try new things, but basically find out what most people are doing that have already gone down this path (stand on shoulders of giants, as someone mentioned)

* Read, read, more reading...talking to other devs...network...DO NOT REINVENT SOMETHING (I also call this the Kiss of Death). Unless you are Google, Amazon or Facebook, use off the shelf if you can.

* Use technologies that will work for your problem. We chose Erlang for ours b/c it of what we were doing. Something like Java would have worked, but would have made the job 10x harder. C would have been ideal, but we would have to reinvent nearly all of Erlang, so just choose Erlang.

* LEARN about things like good architecture design, SOA and failure (when a system goes down, what happens...).

*Invest in a good test suite or test infrastructure, but realize that it will be nearly impossible to test at scale.

During that time I felt like I was constantly reading every paper I could find, blog on scaling and back-end systems and talking to every dev or had ever done it. It was work, but not the type normally associated w/ dev....but was 100% worth it.

When you discount "learn on the job" and "read books", i'm really not sure what's left, or what you expect the people who have achieved success by doing these things to tell you (while omitting those things.)

http://dbasch.posterous.com/how-did-you-become-a-hardcore-ba...

Tl;dr: in 1998 I created an mp3 search engine that got significant traffic, had to learn on the fly, ended up going to Inktomi where I joined a team tackling much bigger problems. We all learned a lot over the next four years.

Can you take a moment tomorrow and add an edit to your post giving a summary of whether you felt the comments answered your questions?

I ask simply because my first read of your post focused on "How do I get there?" and not "what was your path?" As such, I was surprised to be reading life stories of fellow HN'ers. Since we all absorb info differently, I'm curious to know if the stories helped and what you gleaned from them.

All the best in your endeavor. -- A fellow large-scale enthusiast.

Don't get worried that you won't be able to go in and run the show on the first day. There isn't any secret sauce, and sites that scale to this level are so rare that they probably each have their own arcane and complex way of doing it that has evolved over years of people trying different approaches and failing.

Anywhere that is worth working isn't looking for someone who knows how to scale a website to millions of users, they are looking for smart people who can contribute. Their development budget is probably in the millions of dollars per year, they will be more than happy if you can help.

TLDR; Nobody is going to write a book on this, since only 500 people in the world would benefit from reading it. There is no single answer.

To address the specifics of what you are asking, there is basically a balancing act of consistency vs performance. You need to find the exact balance that is 'good enough' for every problem. The oft quoted 'there are two hard problems in CS, cache invalidation and naming things' pretty much sums it up.

You can run 1000 servers for an hour on Amazon for fairly cheap. If you use that to do some testing/benchmarks etc. of popular nosql systems, for example, and then write about that, you can create some notoriety in the big-systems world fairly fast.

Good luck!