Ask HN: How do I learn how to become a good sysadmin?
Earlier today I read an article on the decline of proper sysadmins in the age of containers. I thought I was doing a decent job of being a sysadmin+developer at the place I work but looking at the problems containers can cause I couldn't help wonder how one should go about becoming a decent sysadmin.
Main scenario is the one where you as an individual might on your own (or with a couple of friends) start some tiny sass product, or build an open source product that you want to keep secure.
At the most basic level, when firing up a new server I follow the how to harden your server guides and install fail2ban, disable root login, enable ssh only login etc.
Things I don't fully understand, user permissions or rather, how to use permissions to ensure good security. I'm also unsure about logging. I'm essentially clueless about where to look to understand what goes into being a good sysadmin.
Any advice would be awesome. Thanks.
123 comments
[ 2.9 ms ] story [ 205 ms ] threadFor example, for a web stack which communicates exclusively over the network stack, run your entire stack as individual non-root user. Nginx can run as the nginx user, django/rails/node as a separate user which has no access to the nginx configs. MySQL/PostgreSQL/Mongo as yet another user, which can't access the previous two.
By default, config files should not be owned by the process reading them, they should be part of the group (if they owned them, they could be re-written by that process).
Logging, you have a few options. Syslog is simple, and re-directable, which can help increase security. Writing to normal log files is perfectly acceptable as well, just be sure to write to a program specific directory and set up a logrotate config.
Also, set up monitoring and notification on everything you possibly can. First, know when something has gone wrong before you're notified by your customers. Then figure out how to know something will go wrong before it actually goes wrong (like running out of disk space - this happens more often than you might imagine).
There are plenty of articles on hardening linux, and sysadmin best practices on the web - I've only outlined a few here. When acting in the sysadmin role, be skeptical, paranoid, and value service uptime above everything. With this attitude, you will be more prone to automate server setup, limit user access, and less inclined to just throwing things out there because they're shiny.
The real fun is when you design and develop a way to safely and securely let your fellow developers just throw code over the fence, because then they can do their job in a way that suits them, without compromising your servers.
Notifying about everything is a very, very bad idea. You'll drown under notifications that do not matter and you will certainly miss those that do matter. There should be as little notifications as possible.
The third time this behavior caused downtime for my clients, I wised up and took my own advice of "uptime is king". I took the "every alert is actionable" to heart, and took a few moments to realize that the action can also be against the monitoring.
Alert for disk at 80% on a 2TB volume? Action: Verify growth in Graphite isn't out of the ordinary, and increase the warning threshold to give you about 3-4 weeks of notice that you might need to get bigger disks.
Alert for an excessive number of 404 responses? Browse the nginx logs and identify someone trying to hack your corporate-mandated WordPress install. Verify they aren't making any traction, and add exceptions to your 404's so you don't alert on known (and non-whitelisted) endpoints they're hitting.
Alert for memory at 80%? What's consuming it; do I have a memory leak and need to restart something? If all is well, and MySQL is just being greedy, up the alert to 90%.
Disk capacity warning at 3am? Put some hours around the warning notification, and add a separate "things are growing out of control" alert which doesn't have hours.
API endpoint is not responding again, but it's not the system at fault? Add the API developer to the notifications for that alert and remove yourself for a week or two.
After an admittedly harrying week of this, you gain two things. One: an operational understanding of what is going on in your system. Two: an alerting system which is tuned to your use case, and which lets you know when you have real problems. Remember - uptime is king.
We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/
Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.
How to use logging (shipping, processing), how to install software (packaging systems), how to one can separate services in different ways, how to communicate with systems, how to work with resources, how to tell what's going on in OS (what's running, what uses what, what subsystems are under how much load) are all derivatives of "know your system and work with it instead of around/against it".
When you don't understand things like user permissions, best is to make up some scenario's and write them down on paper. It makes these things easier to grasp, when you can strike through impossible options or for instance visualize routes or outcomes.
Visualizing on paper also works very good for debugging networking. For instance when a packet coming from the internet arrives at your gateway, how does it travel from there to it's final destination and how does it travel back? Write down at every hop the port/subnet/netmask/gateway for both the incoming and (if applicable) the outgoing interface. Works great for finding connection issues.
Document your infrastructure. Documenting in detail can show design flaws that weren't visible before. It also helps a lot to gain more insight in how everything is connected and how it works together.
These days things are getting more complicated, to be a good sysadmin you also have to be a good developer. Usually as good and sometimes more than the people that write the apps that you will run, maintain and love long after they have decided they have something more shiny they could be working on.
You need to be able to dissassemble and fix programs built by other developers, diagnose issues in many runtimes, understand kernel subsystems and the various issues you can run into in kernel land.
You need indepth knowledge of C, system calls and the behaviour of hypervisors and hardware. Don't believe that running on EC2 and not programming in C gives you the luxury of glossing over these fundamentals.
You will also need networking knowledge, even if you don't intend to run your own networking equipment you will still need to understand things like the TCP handshake, what and skb is and why that's important, understand the differences between select() and epoll().
Learning to be a good sysadmin is relatively easy, learning how to be a great sysadmin takes a good 10,000 hours.
I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never heard of skb or select() and epoll().
I don't even like developing software, which is the reason I once joined the sysadmin team at the company I worked for and fell in love with this job.
I see how these skills are a nice extra, but I understand that knowing too much also creates the problem of having to work harder and stay more often after hours. And saying no isn't an option because nobody else knows how to handle the problems that may arise.
To me, a good sysadmin knows when to say no and spends the least effort in fixing errors in code. The developers should provide good programs or it doesn't get installed in production. Keep strict boundaries in what you do and don't to prevent being abused by your employer. If you are often the last one to leave because you're doing work that someone else should be doing, or asked to be stand by after hours while being the only sysadmin, step on the brakes before this will eventually burn you out or makes you hate your job (or even hobby).
Also, knowing all of what parent describes + all the 'normal' sysadmin knowledge is (eventually) impossible to keep up and will also take too much of your free time. You will regret this later in life.
Those end up creating environments where everything just sorta-works enough to run, for as long as you put a constant stream of sysadmin time into it.
And then as see they doubting people that claim that a person can administer hundreds of servers, or that an admin can go away for a while and nothing will happen.
(Also, exactly what do you look for in SecOps? Most people that use that term are selecting for exactly the wrong things.)
It's good for devs to know some ops, and ops to know some dev from a familiarity standpoint, particularly because mutual understanding fosters a good working relationship, but they're two very different roles.
The companies that do this will have a velocity that outstrips the companies that do what you describe and will naturally have an advantage in the marketplace.
There will still be room for security compliance and someone in the company to be responsible for that, but they'll be writing tests that check software for compliance automatically, they won't be some drone in an office looking at a form and stamping their approval on a change request.
Your industry is changing. You may retire before it really hits you in the ass and kicks you out, but your advice for people getting into the industry is terrible.
You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively.
You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same skill set, just embedded inside a dev team.
And yes, I've had to painfully explain why Linux servers report nearly all their memory as being fully used and what the VM free is was to a Principle Architect at a major internet firm with a string of PhDs and who had built the software architecture of the company from its founding. Controlling a huge chunk of the technical direction of the company and doesn't know the first thing about the VM or how to tell if a server is really out of RAM or not. There's still going to be a skillset which is closer to the hardware, but people need to really have that skillset. Most people who call themselves SAs do not actually have that skillset, they're just semi-technical people who wield root power and throw their weight around the company.
Architecture is a "10,000 foot view" type job. I do a good bit of architecture work, and I don't give a flying fuck about how much RAM is in a server or how many CPUs it has. I care about the function the server performs, whether that performance is adequate relative to current demand, and whether the architecture can scale to handle 10-100x that demand. When I was a sysadmin, I used to care about RAM/CPU tuning, but it's not relevant to my current work so I have forgotten a lot of technical details. I do know that if I think a system isn't adequately tuned to its use case, I can go talk to my performance testing team and they will investigate, generate a hypothesis and test that hypothesis.
Work with your developers to make the application more supportable. Show them the problems you are having and ask for their help in fixing them. Too many sysadmins just throw it over the fence and say "not my problem, you fix it" and that's honestly not acceptable.
This is the entire reason "DevOps" is a thing. They are not fundamentally different roles; you're both involved in building a system that does things for customers. You bring different skills to the table, but that's often the case with every developer: you probably have a dev who is a whiz with databases, another who knows some other library really well... a sysadmin skill set is no different.
I am aware there is plenty of room for the traditional sysadmin in enterprise still but I am not quite sure how long those days will be around.
Google and facebook where first to discover the "Site Reliability Engineer" but they were far from the last.
SRE is becoming the new sysadmin and you can bet your life on it they expect the skills I outlined.
Also when I grew up as a junior system engineer in the mid-90s I looked up to "system administrators" like Wietse Venema who coded in C and wrote tcp_wrappers and postfix, and Larry Wall who wrote his own programming language.
The System Administration / Software Developer is not a new invention of Facebook/Google. What is "new" is the "System Administrator" that started getting hired in the late 90s who couldn't do anything other than install programs and configure them and maybe knew how to 'ps' and 'kill' and that's about it. Even the old school Enterprise-class system admins knew more about debugging crash dumps and using their tools even if they did call up IBM when they needed a patch.
I worked with some old and crusty Perl and C hackers that called themselves sysadmins that definitely would have qualified as "SRE"s during the 90s, it's just they were taken for granted.
All the term SRE has done is create a role that is more appreciated and better compensated with a better understood set of responsibilities and required skills.
The attitude that you don't need to know how to program or understand the architecture of the systems that you run on is a highly privileged attitude that you are practitioner who can remain ignorant of their tools.
The good news is that kids these days are being raised on SREs and there's a billion people in India who will take your job and won't have an attitude about learning how to program. Ultimately, you are going to be a curious dinosaur. You are a unique product of the late 90s Internet bubble where the need for system administrators expanded so much faster than the available supply and anyone halfway technically minded got hired for really good salaries and put in charge of servers.
That is going to change and evolve.
And I spend 5-10 years learning all that knowledge back in the 90s and its 15 years later and I can state with confidence that there's no regrets. If anything my only regret is that I didn't learn how to really practice "software development" as opposed to "programming" even earlier. Going forwards, I expect that more an more the lines between systems and software is going to blur and MY advice to kids these days would be that they will only be able to be ignorant of software development practices at their own peril.
And I stated doing PC Tech work in the late 80s, became a Unix admin in the mid-90s, managed the configuration management system and base configuration of a site that grew to 30,000 linux servers in the 200Xs, and then after 15 years switched to being a Software Developer. I'm fully confident based on experience that you are offering absolutely terrible advice to someone who is just learning, and you are out of touch with the direction of your own field.
No personal attacks on HN, please.
The sysadmin practice is not only installing software and OS, doing basic troubleshooting. It's more than that. It's knowing systems inside out, from kernel internals to application internals. It's knowing and handling more area than anyone else in the field.
If you do not know about these, you do not automate, yo do not code, you do not debug, you become an IT helpdesk employee.
A good sys admin just knows all of the working parts of a server and knows about the latest tools. The only way to get better is hands on practice. Try setting up your own Hadoop cluster from source and run a job through it, then put Hive on it, then attempt to containerize a node ... which not even work but learning why that won't work becomes useful knowledge. Just stumble through it and learn.
But, from the point of view of a developer, the thing that I appreciate the most on a fellow sysadmins is to be calm and methodic at all times. Organised. Having a plan and know what to do. Being ready for disasters.
Do we need to update a security patch on every single server? Ok, list of servers, start with server one, finish with the last one, don't let any one fall behind.
A server suddenly catches fire? No problem, remove it from the load balancer, get a fire extinguisher, remove it, order another one, recover from backup.
Is there a problem on production? What could be wrong? Check logs, think a little, then try to fix it. Do postmortem and come with improvements. Try not to be bitten again for the same thing.
In mi opinion, the core of good sysadmin is to minimise risks and errors in the stability of the system. Mistakes can (and will) happen, but the aim is to make them only once. I think it has a big component of learn from battle stories.
Processes and servers can fail, but the whole system should hold up.
As you said, if a server catches fire, the newbie sysadmin will freak out and start running around in circles screaming. The experienced sysadmin has seen it 10x before and knows its not a big issue - removes, extinguishes and replaces it.
A good sysadmin automates everything he can. It saves time, makes everything uniform, and reduces errors (it can multiply errors but at least they're all the same error).
A good sysadmin documents everything he does. Code, configurations, everything. You want it to be easy for not just yourself to figure out what you did, but your colleagues, customers, or your replacement.
A good sysadmin uninstalls programs that are no longer needed. He doesn't leave 50 old or unused versions of scripts laying around. Not just to save disk space or reduce system resources, but for security and to avoid confusion.
I myself are also a "pretend to be" sysadmin. Not hardcore in any way. However I use Docker for encapsulating almost all the software components, basic security (no-password ssh logins etc.), and most importantly: logging.
>At the most basic level, when firing up a new server I follow the how to harden your server guides and install fail2ban, disable root login, enable ssh only login etc.
This is simply wrong. There are two things wrong with this approach:
1. If I understood correctly, you are essentially following random guides on the internet about setting up your security rig. Not too far from downloading untrusted binaries from the Internet.
2. It shows that you do not understand the core issue and what the main purpose of such tools is, how they function , etc.
This is a bad approach because it does not scale well. Security should be built from ground up, not as icing on top. You have to interact with developers during the build cycle and be the paranoid one. So, for example, when somebody mentions FTP transfers you yell at them, instead of finding a workaround (eg, setting up a box with FTP but filtering IP addresses with iptables or whatever).
Security is an architectural issue, not the issue of which tools to install.
That being said about security, being a good sysadmin for me also means striving for simplicity, documenting and standardizing everything and being meticulous.
How do you learn it? From experience. Listen to your senior colleagues and learn by doing. Also, RTFM (but seriously though, those tech notes are important).
Disclaimer: Used to be a team leader of a large UNIX/Linux team at the big blue. Now I do DevOps.
If you want to learn something, go read (text)books. Not blog posts. You need core understanding. Nobody can give you that in a blog post. Even if the author actually knows what he is talking about, he is translating his knowledge into practical tips for a particular situation. He is applying his knowledge. You get applied knowledge, not real knowledge. When it comes time for you to do your job, the only things you know how to do are bits and pieces from situations which might or might not be applicable to your situation.
Think of it like raw vs. aggregate data. You can easily memorize by heart what the aggregations of the data are. You can learn mean, quartiles, max, min, sum. But what do you do when it comes time to filter? The job of knowledge workers is to filter data, not to memorize aggregations.
http://everythingsysadmin.com/books.html
Learn how to program in C at least well enough to write simple clients and servers, write a trivial threaded application using something like pthreads (just a couple ideas). Then start reading Advanced Programming in the Unix Environment. Really learn how to use Strace -- you should know what the difference is between a system call and a library call and be familiar with many of the common system calls you see in the output of that. You should understand the sockets API and file IO APIs. Should be able to understand socket and file descriptors and understand the output of lsof and be able to use it to debug problems.
Pick up the latest version of the Daemon book and learn how a Unix system is architected and what the different kernel subsystems are. Start poking around in the linux kernel sources. Use your C knowledge to make some toy modifications to the kernel (a /proc entry that echos back 'hello world' when you cat it for example). Read Unix Systems for Modern Architecture by Schimmel as a start if you want to go further here.
Take the same approach to the higher level aspects of systems. Your package manger is important so if you are on a RedHat system you should be able to take a trivial piece of C code that you wrote and properly package it so that you can build it, package it, and install it.
If you've built by hand or seen C packages built by hand you should have encountered 'configure; make; make install'. You should write a toy C program with a portability issue (find a Linux/FreeBSD portability issue and write a toy C program with that problem in it), and write an autoconf script so that you can run your code on either O/S.
Pick up either CFEngine, Puppet or Chef and learn how to use it (you can also add Salt and/or Ansible, but please don't consider yourself an expert if you don't understand the limitations of those two). Study Mark Burgesses Promise Theory.
Learn at least one high level scripting language that is in common use: Ruby, Python or Perl. You should also learn bash (and ideally learn old school /bin/sh and its differences with bash as well), but you cannot stop at just shell scripts.
Ideally you really learn how to program and write 10,000+ line object oriented programs, and write code that is tested. Learn what inheritance and composition are and why some software developers argue you should favor composition over inheritance. Learn the Law of Demeter and other principles.
Install and use something other than Linux to broaden your horizons a bit. FreeBSD at least. Solaris or one of the Solaris derived distros would be even better. Having a Mac as a laptop is also useful but I'm not sure it replaces playing with different server O/Sen.
Then, you definitely do also want Limoncelli's book and Nemeth's Unix and Linux System Administration Handbook and be able to configure a wide range of different systems apps (sshd, ntpd, SMTP mailers, apache, nginx, etc). If all you can do is install and configure apps, though, then you're no better than a scriptkiddie running scripts they download off the internet if you can't write them yourself. You won't understand what you're actually doing, and won't have a prayer of debugging some of the harder problems you can run into in architecting and debugging issue that come up in running those apps in a large and successful environment.
My advice is to really learn how to write some toy C programs at minimum an learn the sockets API and really have a copy of APUE and use it occasionally. I highly recommend a project where you do some kind of trivial 'hello world' patch to the actual kernel sources and things of that nature.
I have actually, on my job, seen issues that require this kind of knowledge -- like one incident with massive amounts of TCP RSTs and I've cracked open the kernel sources and tracked down where the TCP stack can issue RSTs, and found a problem with TCP Timestamps and the PAWS algorithm and the way that our layer 4 hardware loadbalancers at the time were doing packet rewriting and preserving all the random TCP timestamps from the clients which was hitting very high velocity servers and when TIME_WAIT state sockets were hit with a 'backwards in time' TCP timestamp from a different client it would issue a RST and tear down the connection because of PAWS. You might be able to find that kind of problem by studying the TCP RFCs and books like TCP/IP Illustrated instead of opening the kernel sources, but its the kind of problem that you aren't going to be able to Google. And I don't really have the ability to write a [modern, multithreaded] TCP/IP stack (although I could certainly acquire it), but I have demonstrated the ability to read the sources and get real-life work done by doing so.
For all the candidates that I interview I want to see some indication that they can work without the Google safety net and show some ability to track down problems on their own.
That doesn't mean you are going to learn everything there is to server administration by following a random guide on the internet, but for a small startup it's not necesarily a bad way to get started since at the end of the day you are going to learn more from experience than by reading 50 books.
The best argument I could find for "disable root login" was "the attacker has to guess the username, too", which doesn't align with Kerckhoffs's principle and isn't the way security should be done, imo.
Also, fail2ban is a protection against bruteforce. If bruteforce is an issue for you, you're doing something wrong.
Please correct me if you know more.
There are good reasons to use Fail2Ban, and the counterarguments that it doesn't actually improve security miss all the other benefits it brings.
And, I've read all of Theo de Raadt's arguments against these approaches. I understand and mostly agree with them. I get that with ssh key only authentication and sane services configuration and so on that people can hammer away at your server all day and never accomplish anything. But that still doesn't mean I want to provide a test bed for every dumb script kiddie on the internet (and there are many).
The thing is that you can reach pretty much the same effect with a smaller attack service and better efficiency using rate limiting in your packet filter.
E.g. in iptables the 'recent' module can do this, see man man iptables-extensions and search for 'recent'. E.g. you can set up a rule that any IP address making more than 5 connection attempts to port 22 in one minute gets put on a list that is DROPped.
Edit: BTW, if you think the fail2ban attack vector is purely theoretical, you might want to check the CVEs:
http://www.cvedetails.com/vulnerability-list/vendor_id-5567/...
fail2ban helps with a lot of things. It keeps spam out of the logs. Some systems have high CPU cost per login (bcrypt), so similar systems can help prevent brute force attempts turning into (or being actually intended as) DoS.
The mentorship part is a lot more difficult. Good sysadmins are really difficult to find in Sri Lanka. I've worked with a few companies so far. Some examples.
1) One company I worked for didn't even have a policy of hashed passwords and protection against SQL injection. They were developing major enterprise software.
2) I worked as an internal systems developer for a non IT team within another company. This was the one place I could have learnt the most at but the IT team was this very opaque "don't tell people what exactly we do" kind of team.
3) One last example. This other company that I worked at, the sysadmin was pretty good in keeping stuff up and running, but a lot of it was copy paste scripts. I got what I could out of the person but I couldn't pull out much.
Where I'm from, the main cyber security body of the country gave blank looks when asked about heartbleed at a conference held recently after the whole thing exploded.
All that to just sum up why I turned to HN to seek out advice as to what resources I should look at. A lot of threads on the net seem to veer more towards "be a good communicator" and "know your system". While necessary, it's a little too abstract for someone trying to find out what gaps exist and which ones need filling ASAP.
Thanks a lot for the advice. I'll probably start reading up on all those files I had to edit when hardening the server. That should provide a good starting point.
Congratulations, it looks like you are one of the top sysadmins at your country.
Maybe you should try to look in other places, or maybe you really should really congratulate yourself, with no sarcasm at all, and start selling yourself like an expert.
The people who write those guides don't either. I almost never see mention of things like OSSEC or running a WAF or even a basic IDS/IPS. Just lots of naive "hackers get in through passwords, dont use passwords in ssh and enable fail2ban." That's on top of security via obscurity like changing port numbers. Uh, what?
Security needs to have a layered onion approach:
Network> IDS/IPS, Firewall
Server> OSSEC, SELinux, smart permissions, WAF, AV
Storage/Filesystem> Backups, tripwire
DB> Hashed passwords, etc
Software> Patching schedule, writing secure software from day 1, not giving into to team/client demands for insecurity
Monitoring> OSSEC alerts, nagios/zabbix alerts, tripwire alerts, etc
Once you get used to a security stack, this stuff comes easy. The problem is cheap VPS's have turned devs into sysadmins who just plant a naked linux box on the web, turn on fail2ban, and call it a day.
1. Never enter a command if you don't understand what it does.
2. Never make a change on a production system if you don't know how to undo it.
3. Get everything else wrong if you have to, but get the backups right.
I realize this paints with a broad brush but it's a good baseline.
1. I will know exactly what every command or script I run on a system I control is supposed to do - no exceptions. If I don't and are just following instructions, I really need to learn what it means and why. If you need to setup a test system and snapshot before and after to see how things work.
2. I will document a lot. Imagine some poor person showing up after you have won the lottery (think happy thoughts, but watch out for buses just the same). Don't just blindly put down step, put the why down. If I cannot write why I am doing something then I need to think about it more.
The rest just flow from those. Learn to program, be a tool builder, find the best way to learn and dive in, solve problems, and insist on consistent, repeatable, backedup, secure systems.
Do remember though: all your successes will be hidden in the darkness and all your failures will be shown in the full light of day. Its not a fun gig at times.
If you don't have a good boss who can see that you are good at what you do, you will have to be able to speak up if you want to be paid what you are worth.
You want to be a step up from a computer janitor who needs to be told what to do, to being someone who delivers value to the business and helps people get their jobs done, and can anticipate problems before they occur.
And on the software engineer/developer side of things the same applies. This is why whenever I am given a self-assessment or asked to help with a review of myself, I go back through my git log, email, etc. looking for what I've done instead of just attempting to summarize based on memory. Then I keep a personal copy of my self-assessment. That way, I have a record of what I did, and so does my company. Wikis, file servers, and other document repositories change, and when you switch jobs, you have that available to look at to update your resume. If your company doesn't make you do at least annual and hopefully quarterly self-assessments, you should do it on your own.
1) I have more years as a developer than system admin (11 vs 7 and 5 as something I'm still not sure).
I've been in my present position five years. Knowing where the bodies are buried (because I buried half of them) turns out to be one of my most useful functions, and whenever I am asked such a question I add the answer to the internal wiki ...
This. And you learn by not doing this and still having to maintain an infrastructure. After a while you will either start, hang yourself or get fired.
“An expert is a person who has made all the mistakes that can be made in a very narrow field.”
― Niels Bohr
Learn how FreeBSD's ports system (building packages from sources) work and why it is so.
Then learn what Redhat Package Manager (RPM) is and how to make a source rpm.
Get enlightened.)
Some questions and projects to get you started:
Do you know how to start and stop services, how to tell what's running, how to see what network activity is happening?
Try installing Apache or Nginx on an EC2 instance, then see if you can change how it works. Change where it logs to, or change what port it listens on, or change what directory it pulls files from. Can you change where it reads its configuration from? What happens if you give it a bad configuration? Where does it log errors? Can you break it and then fix it?
Spin up a new EC2 instance and try to see if you can lock yourself out of it or otherwise disable it. What happens if you kill the SSH daemon? Or change the port? Or delete your user's entry in /etc/shadow? What happens if you run "rm -rf /"? What do all the fields in /etc/passwd do? What if you delete your home directory? Or "export PATH="? What happens when you delete files that a running program is using? What happens when you fill up the disk? When you run too many processes? Can you make the thing crash? When something gets screwed up, do you understand why?
Run "ls /bin" and see how many of those commands you know what they do. Pick one you don't know how to use but which you've heard of and try to figure out how to use it. Look at the man page, run the command with "-?" or "--help". Play around with it till you feel comfortable. Then pick another one tomorrow and do the same.
Run "ls /etc" and pick a file and try to find out what it's for. See if you can do something interesting by changing the contents of the file. You might need to reboot or restart a service. Tomorrow pick something else from /etc.
Do the same with /usr/bin, /usr/lib, /var/lib. Figure out where the files in /var/log come from and how you can write to them and how you can change their names and how many there are.
Set up two instances and see if you can get them to talk to each other. On EC2 each host has two network interfaces. Do you know how to find both IPs? Can you set up MySQL on one, and connect to it from the other? Once you get that to work, can you block it? Spin up a third instance and see if you can figure out how to make MySQL accessible to one and not the other. Once you figure out one way to do it, figure out another way. Can you get a service to listen on one IP and not the other? Both?
What's the difference between UDP and TCP? Set up NFS. Set up a RAID. Break a RAID, rebuild it. Do you understand crontabs? syslog? Can you send mail to yourself?
Once you're feeling more comfortable with the environment, and you start actually fulfilling a sysadmin role, your basic philosophy should be:
* Expect anything and everything to fail.
* Trust nothing, even your own software and machines.
* Grant the least possible access to make things work. Developers and vendors will always ask for more, and if you are not pushing back against their requests, you are probably giving them too much.
* Always have a plan for rolling back a change.
* When something breaks and you fix it, don't stop working until you understand what went wrong and you take some steps to avoid it in the future.
* When you have a task that takes more than one step to complete that you do more than once, write a script to do it for you.
That should get you started...
Ultimately, good luck and have fun!