Ask HN: Tips on Sysadmin Job
Recently I got a promotion at my job. I was working with another guy in a small company with an even smaller IT department. That guy was fired and I got the position to be the only sysadmin at the moment. I never had any experience working and managing the whole IT infrastructure, because the company has 4 campuses around the city. I'm kinda confused on how to manage everything, my background is with information security and it's going to be a big challenge for me because it's my first job.
64 comments
[ 1.6 ms ] story [ 295 ms ] threadSo, with a security background you have the most important of these already. The next most important is handled with backups. Then the rest is less critical.
Next, you need to decide whether the system as it currently stands is basically working but needs little adjustments, or needs redesigning from scratch. Take a few months of observations before you decide this.
Document the issues that you deal with. Make notes on the nature of the problem, the priority, the expected effort required to deal with it, what happened, how much effort it actually took. An issue tracker is a good way to do this, and I'd recommend using one even if it's a small company.
If that’s how the company handles this, I would start to look for a new job immediately.
- no indication in the post that they going to hire a new person
- If your company IT infra is down, you are probably dead in the water, leaving it to be managed by a single person who does not have any IT infra management experience is a pretty bad business decision
- No indication in the post that the person offered any training or anything
- my experience with 1 person responsible for IT support/management/sysadmin jobs are that they are 24/7, which leads to burnout and mental health issues
I enjoy sysadmin, partly because of the variety of problems you are called-on to solve. But the fact is, if you do the job well, and everything works well, nobody gives you any credit. When things go wrong everybody will be looking at you, and if you haven't fixed it within 20 minutes, then you must be incompetent.
So you're the only sysadmin in a multi-campus company? As someone noted up-thread, you can't do it alone. At least, you need someone on each site that can act as remote hands for you, even if that's not their main job. And you need people to delegate grunt work to - repetitive stuff, like building workstations for new hires. That's IT, but it's not sysadmin.
Get some interesting hardware to take home and play with. Stuff like smart switches and firewall boxes (the same gear your employer uses).
GNU/Linux has much better networking tools than Windows; if you're in a Windows shop, get a laptop with Linux on it.
Whenever you do a manual job you've never done before, let part of the job be automating it, so you never have to do it by hand again. Let the automation scripts be the documentation.
Learn to be pessimistic (a common sysadmin trait). If you have bad news, don't ever sugar-coat it. Things might not be as bad as you thought, in which case your reputation is improved. But what people don't want to hear is that the job takes two hours; and then find after two hours that it's a two-day job.
If you can get yourself a full-time assistant/colleague, work as a pair when you can. It reduces stress a lot if you have someone checking your work as you do it (and your decisions). And you end up with two people knowing the systems, not just one.
Get a hardback A4 notebook, and record everything you do.
In theory, you have a lot of power (c.f. BOFH).
http://bofharchive.com/
You know stuff others don't, and you have superuser powers everywhere. Don't abuse it. Be as helpful as you can. People will be nice to you if they know they might need your help one day. The BOFH was a tyrant, and I don't think he'd get far in a modern business (the BOFH worked in a university IT department, and was God; a sysadmin in a modern business can't behave like the BOFH).*
And seconding the physical logbook.
And it's easier to scribble a line in a physical logbook than to crank-up some editor, open the logfile, append an entry, save the file etc. If you're going to log absolutely everything, then adding an entry needs to be as easy as signing your name.
You could rely on your laptop; but laptops break and crash. a physical notebook is crashproof, drop-proof, and can even survive being dropped in a puddle. And short of pencils and erasers, it's intrinsically an append-only medium (except that you can still annotate old entries).
Maybe an electronic logbook works for you; I found that my logs were more comprehensive and generally easier to use if I kept them on paper. I got through about two notebooks a year; I used a stack of old ones to elevate my monitor (phone directories are hard to come by these days).
Also, opening a big notebook before describing your activities confers an aura of reliability on you; there's a reason why policemen whip out notebooks when testifying in court.
1. Require not 1, but 2 people, for hire, right now. It will take a long time until it gets approved, HR does their thing, and you (or someone else) hire people to help you.
2. Document, as fast as you can, all of the assets in your environment. Don't waste time on looking for software. Plain text or spreadsheet will do for start.
3. Identify the most important assets (servers, network devices, etc) - and try to understand what are your priorities for each. For example, availability for your frontend servers. Confidentiality and integrity for your server storing PII. Based on this, you'll understand what are your biggest risks. Would it be a business disaster if your website is down for few hours? Likely not. What about if the PII is compromised and you get huge fines and be in the news? More likely.
4. Find out who has the access to the most important assets, and cut off anyone who doesn't really need it.
5. Establish some sort of monitoring for the most important assets.
7.5 Test your backup strategy every now and then (nothing is more painful than data loss and dysfunctional backups)
8. "All users lie, there is no exception". Sad but true, get used to it.
9. Automate your infrastructure. If something goes down, just recreate it with Chef/Ansible/Terraform/etc.
[edit: added no.9]
11. Understand the reason the other guy was fired make sure you are not the one holding the bag.
12. Have a issue tracking system in place, you will thank yourself later.
13. Have a change management policy in place.
14. Before you make any change full understand the consequences and have a tested plan to get back to a known good state.
Dependant on what you have to look after you might want more than one book.
You need this to know if your web sites, servers and services are running, and to be emailed or contacted when not. I am used to Nagios. I understand there is a Nagios fork, Icinga, which I know little about. Some people use Graphite, which I also know little about.
You can also have it check for disks running out, high CPU loads, inodes running out, heavy I/O etc.
Also, you can email etc. alerts, or pay for a company like Amelia to monitor a dashboard and page/call people at night if a server is down, or to even do some simple scripted remediation attempts.
Keep a text file, or OneNote or whatever, open while working an incident. Don't overthink it, don't over-complicate it.
Maybe person X built system A, and is now working on system B, but remembers a lot of details - keeping their access to system A can help you recover in an emergency. Maybe someone wrote a monitoring job (or worse, a deployment job) that runs as themselves. Maybe someone has read-only access to a system and is using it to ask occasional good questions to the team that actually runs the system. Maybe team C has a formal API but it doesn't work well so person Y informally got access to C's database and it's making system D able to run at all, and it would take a few months of effort to fix the API that's supposed to be there between C and D.
And, more frustratingly but no less impactful - maybe there's a senior person who likes still having access to system E, and making them mad will cause political problems for you and impose burdens when you you try to make any other changes, which gets in the way of your ability to solve all the other problems that you need to solve.
Your priority in the short term is to keep the business running, not to improve things. Insider threats are real, but they are much rarer than all the other reasons your business could get in trouble. If you don't have a good understanding of your tech debt and why the tech debt exists and how quickly it can be paid off, focus on that first.
7. Eventually, in 5-7 years, become a consultant, either in a consultancy, or freelance.
1. When everything works everyone is happy and nobody pays you any attention.
2. When something is not working you will get shit from everyone for 'not doing your job'.
You will not be able to completely avoid 2. So your only chance to balance the scales is by showing your hard work when in case 1.
That means auto generating graphs of network resources, disk space, creating trend reports with actionable recommendations to our manager/boss.
This way they have visibility into IT world instead of it being magic that just works. It also involves them when making decisions no matter how small.
This of course is job security measure and you need to do your job first. Goodluck.
- The most important rule is that your recovery from backup procedure is working properly. - The second most important rule is that your backups are working properly. - The (distant) third most important rule is to always individually backup any text file you edit before you edit it.
Seriously. These are the most important rules.
Other pro-tips are to test any changes to remote access (SSH most importantly) before you close the session that you made the changes in. Also, schedule restore from system backup and restart before you make any potentially crippling changes to the system configuration. If the changes work then you can cancel the restore / restart. If the changes cripple the system it will (if you've followed rules one and two above) recover itself. Finally, if you can afford it, try to run at least one dev / test instance of each critical system. You can make changes, point a few test clients at the new instance, make sure it's working, then either make the changes to the prod system, or do the dev / prod swap-over.
The actual ways to achieve these rules varies depending on OS mix, hardware, cloud providers (if any), use of certain tech (are you using virtualization? Are you using containers? Are you using some form of orchestration tooling? etc.) and many other things.
The first two rules mentioned above, and the order in which they're mentioned, are by far the most important rules in sysadmin. Ignore them at your own peril ;-)
Good luck. Keep a level head, don't panic, and test your backups!
To add to this.... Don't change things like routing tables or server network device / firewall configurations without physically being on site with the thing you're changing.
- Create a single-source-of-truth you can automate against. Do that now. Use ansible to fetch and create an overview of what your landscape actually is running: which software, which versions. Find out what is going EOL, as this will bite you soon enough.
- don't automate anything else until you understand the full context of what you are automating. Automation is abstraction, and you don't know yet what to abstract.
- When you do automate, start with writing tests. https://testinfra.readthedocs.io/en/latest/ is quick win to ensure all your systems have certain configuration, can reach certain ports, have closed other ports.
- Once you have tests in place you can start creating ansible playbooks for changes, as changes _will_ happen.
- I know with your security background when Bob in accounts demands a domain admin password you won't just give it to him. But who has your back on that? Will he cry to the CEO who in turn will ask you to please do your job and give it to him? Get some relevant policies reviewed and signed off.
- desktop (human!) support
- networking inside buildings
- networking between buildings
- server maintenance
- OS administration
- authentication systems
- local services (DNS and NTP are the most important, but your end users will think of file sharing and printing as most important)
- external services
- security (information, yes, but possibly physical systems, too)
- backups (nobody cares about backups, they only care about restores)
Figure out which of these are currently running, then which are your responsibility. Document everything, and save copies as automatically as possible - you have four sites? You want five copies. One in each location and one in a secure off-site location.
Make lists. Review them in the morning and before you go home.
Get a copy of Limoncelli and Hogan's book, The Practice of Systems Administration, and the follow-up Time Managment for Systems Administrators. Expense them.
Later, figure out costs.
Good luck.
Seconding this. Here's a link to the latest edition.
https://www.amazon.com/gp/aw/d/0321919165/ref=dp_ob_neva_mob...
After he's got his sea legs, I would also get books on how to implement ITSM. IT Service Management is the most straightforward codified way to run an IT department or set of IT services. Learn about Change Management, Incident Management, Disaster Recovery, Availability/Reliability/Redundancy. Learning about any legal regulations/laws that relate to IT. Then eventually dive into SRE, Lean, DevOps, Agile (but far down the road, not immediately).
Always do smoke testing. After setting up a system, and it starts working, reboot once. No preparation, nothing. Just reboot out of the blue.
If the system boots and works as normal, you're done. If not, fix the problems and retry.
Will save a lot of headaches down the line.
Also, document all your procedures. Everything. To something tidy & local. Something like TiddlyWiki or a tool of your choice.
If something bad happens, add a test for it to that email. It’s not an end all, but helps be proactive and keep pulse of things you are on the hook for. Real-time alerts etc can be useful, but also a lot of noise.
Also a kanban board like Trello or Notion has been useful
I have never met very good sysadmin which wasn't programmer at one point.
I am not talking about hard core programming involving serious patterns, DI, migrations etc. But most of the time, you should be writing good, resilient, scripts in more serious shell language such as PowerShell. If you don't do that, your actions are not reproducible and thus your output is prone to interpretations (i.e. it sucks). Plus, there is no such thing in enterprise as one-time-tasks - if you don't script your task you will have to manually repeat it sooner or later and that is extremely inefficient and error prone.
I'm not saying this because I think you need more money and staff, you haven't mentioned either.
The reason you need to do this is to check what kind of organization you're working for. Do they value your presence? If they don't make a move on either of these issues, you know the answer and you can leave immediately. The job market is white hot right now, you don't need them. Either they're dumb and they don't realize they need you, or they're cheap and they think they can get you cheaply. They could also be reasonable and act like it.
Basically it's poker and you've got a decent hand.
Oh yeah, and don't let them do the "we're working on it" thing. Immediate raise, and some kind of job ad in a public place, now.
The most technical aspect: don't lose or destroy the data! Downtime is fine, losing data is catastrophic.
Grow a thick skin or move on. You will be the target of complaints. A lot.
https://www.opsschool.org/
https://www.reddit.com/r/sysadmin/wiki/index/
Also recommend the hangops slack team: https://signup.hangops.com/
You said: > my background is with information security and it's going to be a big challenge for me because it's my first job.
I see a lot of good advice for people who kinda-sorta know what they are doing. You did not say if this was a linux-based job or a windows-based job. This makes a small difference. If this was a windows-only shop, my advice would change slightly (e.g. Do you understand windows automated deployment with a system like SCCM or whatever you have in-house? Do you know powershell? From your comment, I think the answer is "no".) If this was a linux-only shop, again my advice would change (Do you know how to write any code in any language? From your comment, again, I think the answer is "no".)
You say you are confused about how to manage everything, I am assuming you do not mean "I have a huge pile of IT inventory" and what you mean is "I have 200+ different pieces of software/applications running, how to keep track of all of them, their status, upgrades, needs, etc."
Every sysadmin I ever worked with that was worth their weight was at one time a programmer, CS major or CS graduate. I do not know if you have the time to learn programming in this position, so I would ask the following question:
Is it your desire to be a sysadmin as a career or do you want to do Info Sec as a career?
If you want to be a sysadmin, you need to learn some programming. Pick something that fits in with what kind of shop you are in (win/linux/mixed). If you do not want to be a sysadmin, figure out only how to keep the ship afloat, and start looking for an InfoSec job. You will eventually dislike the sysadmin position. It takes a special kind of person to automate everything they do, and build things only to tear them apart later (https://www.sciencedirect.com/science/article/abs/pii/S01672...).
Most people in here have given great advice, but IMHO I am not sure if it fits in with your skill level. I do not mean any insult, but I knew InfoSec people who were expert sysadmins, and others who knew nothing about the job.
Programming background in C and python, also bash and powershell programming.
Knowledge of networking and computers in general( I know how to fix hardware in general and know how to deal with Linux and Windows)
My background is in InfoSec, so I Know how to create scripts in general, analyse and mitigate vulnerabilities.
It's my first job, so I'm really anxious on how to do stuff, because I have some theoretical knowledge but never did it in a company, I always did it at home working on some personal labs.
Thank you again, I really appreciate your comments. Love you all
Something breaks in christmas eve, you have to fix it. Something breaks in the friday evening you have to fix it. This is a job that needs multiple people in long run usually.
You worked together, surely there was some rapport there?
Or at least go to your manager and make very clear that it is simply not possible to provide anything like 24/7 incident response when you are alone. That is the kind of conversation, where it is just better to get fired rather than not getting your point across.
The basic structure of the admin job is, that there is always more to do and it is very easy to burn out. There is always more to do, it is always important, but trying to chase some ideal of a perfect system will just exhaust you. Furthermore, it is not a big problem to check your monitoring before going to bed or a few times on the weekend, but that limits your rest periods to a few hours at best, you no longer get a full weekend, and that will grind you down over time. So, manage your time and sanity, and importantly also manage the expectations of your boss and co workers, there is only so much you can do when you're alone.
Document the dependancies of apps on the above.
Verify backups and that they are backing up the right information and that they are immutable until their lifecycle is complete.
Verify your critical infrastructure has two power sources. Most equipment have two power supplies and I make sure they are independently powered. (UPS and line/generator). Do not rely on just UPS because it will fail you.
Make sure your datacenter has redundant A/C. Cooling failure will kill you as certainly as power failure just more slowly.
Know your facilities contacts especially the after hours emergency numbers and make sure they are current.
Know your IT support contacts. Sysadmins are somewhat jack of all trades master of none. Call the masters and use them when needed. No shame at all in that.
Don’t be a dick. Relationships are important.
Above that go out of your way help people even if it’s not in your wheelhouse.
Be aware of your long term stress level and well being. Your responsibility is to your people and you can’t do that effectively if your are not mentally able.
Communicate with your boss and make sure you are getting the support and tools you need to do your job. If you need it ask for it.
KISS - Don’t be sucked into complexity where it’s not warranted. Simpler is more reliable than complex if you don’t understand the complex.
Don’t automate unless you completely understand the process you are automating and all the dependancies.
Don’t offhand just trust vendors or their promises. Verify.
Pay extra for good hardware.
Read HN
In no particular order.