Ask HN: Is this true?
We were in the process of hiring a CTO. One of our programmers suggested that server guys are not normally good coders; and programmers are not normally good server guys. He suggested that, in general, the fields have bifurcated and it is uncommon to find people who excel in both domains. Would you say this is true? mostly true? or BS?
42 comments
[ 2.9 ms ] story [ 101 ms ] threadThen again, there are jaded "specialists" who don't dein to dabble in the "other" field, I'm sure.
Not all programmers are very familiar with hardware issues, or dependent on their area of programming expertise on network design.
I also found that many programmers are oriented towards creating new features, while administrators focus on running established tools at maximum efficiency.
While drawing on the same set of qualities, the interpretation of them can diverge.
Some people posses both sensibilities and the mind space to treat both disciplines well in environments and projects that scale. They are not common.
Those of us who get paged when the shit hits the fan tend to be pretty careful of breaking things; and quite often, we work on systems where relatively short downtimes can lead to losses greater than our yearly salaries.
That said, all the best SysAdmins have at least some background in C and NIX systems programming, and nearly all NIX systems programmers are also pretty good SysAdmins.
In general, an expert program will not be a truly expert sysops because each of them is a deep field and developing truly deep expertise in either of them can be an almost all consuming task.
But, you can easily find people that are pretty good at both, and I would go so far as to say anyone who is good at one also already knows at least the basics of the other, they just might not be a true expert. Certainly anyone who is good at one has shown they have the right mindset and capabilities to become good at the other if they decided to.
All coders do some system administration as a matter of course, and the reverse is true as well. But to specialize in both is probably rare.
Respectfully, I think that once you start dealing with some more interesting problems, and doing so professionally, you might begin to realize just how much you don't know in one field or another.
http://jedi.be/blog/2010/02/12/what-is-this-devops-thing-any...
Just curios on the choice of title.
do you really think a non-sysadmin can effectively choose good SysAdmins?
I mean, there is enough overlap between the fields that a good sysadmin can usually tell if a programmer is completely BSing, and a good progrogrammer (well, a C systems programmer) can usually tell if a SysAdmin is completely worthless, but I wouldn't trust anyone who was 'pure management' to detect either.
An analogy might be basketball. Growing up you learn to just get the ball in the hoop, but those that turn professional specialize in a certain position. Very few can play all five positions quite well (like Magic Johnson).
That said, the value of some level of specialization clear. We have to commit a lot of energy to the things we will become really good at. However that doesn't come without problems. In the scientific community there is concern that it's becoming harder to do cross-discipline research.
While we must accept specialization for practical reasons, and realize that there are also social and biological factors, there is value in encouraging cross-disciplinary experience, especially in leadership positions.
The programmers may like to critique the potential CTO as not as good a programmer as them, and the sysadmins may do the same. But I sorta think you want a CTO who can see the forest for the trees.
(Cue criticisms of dumb CTOs....Now! ;-)
Splitting the roles is also practical. At some point you may have a project deadline coming soon, when suddenly (for example) network fails - do you really want to send experienced programmers on a trip to the datacenter in that case?
Really, it's a question of how big you are and how much time does your infrastructure need...
That being said, I know plenty sysadmins that have gone on to become great programmers. And truly great programmers tend to love computers, both software and hardware.
Larry and Sergey were apparently also server guys: http://www.flickr.com/photos/bsmif/3465464623/
Those corkboard servers you show, BTW, were removed from service in a hurry once the fire marshal found out they were building servers out of corkboard.
Above a certain point, there are so many technologies you need to know that people need to specialize.
The third aspect of this is management. Someone who has been in management for 10 years is not going to be a spectacular programmer or sysadmin because they simply wont be up to date, but they've got a solid tech background and great managerial skills. These tend to be better later hires, but could be right as well depending on your situation.
The best programmers know all the system settings, services, and behaviors that can affect their apps and thus have to know the underlying system.
I have yet to meet a truly great programmer who didn't thoroughly understand the system he was running, or a fantastic sys-admin who wasn't a pretty good programmer.
However: unless the guy you're hiring happens to be an active developer for the operating system that you're expecting to him to admin, then there's a pretty low chance that he could be considered an expert in both fields. Each field is just too deep and broad, and requires too much time to maintain current knowledge in.
A programmer can probably figure out a particular sysadmin problem, and a good sysadmin can probably figure out a decent approach to a programming problem, but while they're figuring it out, you're paying them and potentially also paying for downtime or a slower development process.
I spent the next part of my career as a developer, first at a security company, then at a streaming multicast company I founded, then at a network managment company.
I spent the next part of my career as an almost full-time security researcher for hire, with a pronounced focus on high-end enterprise technology (SAN, WAN compression, replication, database, storage).
Looking at it from all three angles, I'd say: there's definitely something to this. Being a programmer gave me insight into how systems worked. Being a security researcher gave me even more insight (I had projects that were literally months spent reverse engineering equipment firmware and building protocol test suites).
I'm confident that after all this "insight", I knew more about how, say, iSCSI worked than any SAN administrator in Chicago.
So, how would I have done as a manager of a SAN array? Terribly!
The things that actually matter in administration --- understanding the processes that need to be in place to make changes, understanding what kinds of changes will occur, understanding what's a typical kind of failure that will just take a couple hours to diagnose versus what kinds of problems mean that your vendor is putting engineers on an airplane --- those things you learn by being in the operational/engineering role.
To excel in the operational/engineering role, you need to dedicate yourself to the things that matter for that role, and not allow yourself to get distracted by shiny things. Developers and researchers: prone to distraction by shiny things!
What I don't think is true is that strong tech people are predisposed to one of these roles or the other. Could I be a strong ops/engineering person again? I think so. But I know I can't do it while I'm doing this job.
Programmers tend to focus on the very small details and seem to only look at the short term question of "if something is possible to accomplish". A server guy needs to think about the big-picture, with a focus on not only if something is possible, but the impact of a particular approach on the rest of the infrastructure and the long-term maintenance cost of any given action.
The programmer you are describing are a kind of programmer who makes lousy programs. I've read enough of those programs to say: they are not pretty -- they do not work well in large scale -- and they cost a LOT.
I can administrate servers (about 9 years of commercial experience, about 20 years of total experience with UNIX administration, starting from school), but I hate that. My choice is to create copy of the server, then make changes and write them down, then test them on the copy, then apply verified solution to production server (see http://vlisivka.pp.ua/en/modern_administration , sorry for my English).
I even implemented my own run-book automation tool, which reads documentation and then executes steps in automatic or semi-automatic mode (I prefer Documentative Programming style).
I often prefer to develop script or RPM package to make change to the system instead of making changes by hands.
While a CTO is solving today's problems with today's shiny and with shiny duct tape, you also need a CTO that can identify and can hold both your short term and your broader and longer-term organizational goals in sharp focus. You need a CTO that establishes clear goals and measurements and trends, and a CTO that can then push, pull, poke, suggest, prod, cattle-prod or ego-stroke toward those organizational goals. And you need a CTO that adapts.
If a server admin or a programmer has the requisite adaptability and people skills and technology skills and presentation skills and has an indefatigable focus on making your market, hire him. Or her. But you're not selecting for and not hiring for server admin or programming skills here.
You want somebody with the technological skills of and the focus of the Borg.
I think the point is they are distinctly different skillsets. Programming is very different from setting up and maintaining systems. There's a tiny bit of crossover, but not all that much.
Sysadmins do need a basic sense for coding, however they tend use different technologies than the one people generally use to write web applications (though maybe they'll release Bash on Rails sometime soon)
I've worked with some amazing python programmers and without using it day in and out there's no way I could have the in depth knowledge of all the python libraries like they do, so compared to them I'm pretty useless.
One guy who was an early mentor of mine in systems work actually hates doing it, and prefers to program exclusively. I didn't know this when I was under him, I credited him with helping me get to where I am today but he'll hear none of it.
I do think it's true that is is rare to find people who do work effectively in both domains, or even like to. There is a sort of closed mindedness on each side toward the other -- come on guys, we're all engineers! I've had technical and non-technical people alike be surprised at where I can contribute (a memorable one is someone in the accounting department at an old job who was forced to admin a machine back in the day being surprised that I could code, and more recent one is someone being surprised that I was able to successfully contribute to dealing with IE layout issues -- in both cases it's been like "but you're a systems guy, you can't do X", because they've only ever known me to be in that role "officially").