I think the author meant for this to be a simple set of security guidelines for beginners. But yeah, any reasonably large prod stack should probably be deployed and setup with puppet and what not. It's DRY and less error prone than a human low on sleep.
Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.
You don't have to re-learn puppet/chef/cfengine/bash script every time. You learn it once. It requires much less effort to "maintain" than manually updating more than 1 box.
I was in the same position - too many distinct environments for bash/Fabric, too little time to learn Chef/Puppet/CFEngine. Ansible [1] seems like a good compromise: you get the simplicity (runs over SSH) and host targeting of Fabric with the declarative nature and idempotency of the more complex tools. You can start with all-in-one "playbooks" [2], then split out tasks, handlers, Jinja2 templates, files, and variables [3].
As an aside, I think the default fail2ban config is too loose and quiet. Here's [4] an Ansible task file that configures it to be more aggressive and send notification emails.
I'm using fabric quite frequently, and am trying to understand what makes a configuration management tools a much better choice. I'm currently using fabric for anything from bootstrap a new environment from scratch, via restoring a snapshot from backups, to pushing code updates stored on git. Perhaps I'm being really daft, but it always evades me why something as simple as
I do use a pretty homogeneous environment, which makes things simpler, but this is a deliberate choice to avoid complexity. If I know all my hosts are e.g. debian 6, then what makes ansible/chef/puppet so much better than fabric?
I'm not trying to be provocative or negative. I'm really trying to understand the supposedly big difference between what's labelled a deployment tool, and configuration management tools.
Deployment tools like fabric are imperative, and configuration management tools are declarative. With configuration management, you define the final state you want the server to be in, and it will do whatever is needed to get it into that state. Some or all of the parts might already be done, and it won't change the parts that are already correct (the declarative configuration is idempotent).
Deployment tools just execute whatever script you hand them, and so the scripts are either more brittle (server must be in a precise state beforehand or it doesn't work right), or require more effort to duplicate the work that the configuration management software does to only make the needed changes.
If your deployment scripts get complex, it's more difficult to see at a glance what the end configuration is supposed to be.
Thanks for the clarification. I think I get the theory a bit better now.
I'm still struggling with seeing some practical examples of those differences.
for example, `apt-get` is pretty much idempotent already, no? if I run `apt-get install -y <package>` x 1000 times in a loop it won't install it 1000 times...
Fabric does give you the building blocks for those kind of checks elsewhere, such as `exists`, `contains`, `append` (which is supposed to also be idempotent) etc... I've designed my deployment / bootstrap scripts with fabric to take this into account. It does add a little overhead, but nothing that makes me feel I need a better tool... Maybe my deployment base is still relatively small and homogeneous.
So it's true that I have to put in those checks myself, and that I don't have a very easy way to discover what state a server is at.
I'll try to take another stab at one of those tools. Maybe things will sink in when I actually use them. Thanks again for explaining.
If you're only doing one or two things, the value is a bit more vague. But consider even the simplest interaction: the config file for a service should look like X, and if it has to be changed, the service needs to be restarted afterward. Oh and there are 10 config files, but you only want to restart the service once if any of them changes, after they've all been changed. That's not hard, but it's already starting to look non-trivial.
And what if you want to have the same logic for several services? I guess you abstract it out to a function. But then it turns out one of those services doesn't have a restart command, and you have to do stop+start. And another service won't start if you use restart while it's not running, so you have to check if it's stopped and use start, otherwise use restart.
It's much more than just whether the code is declarative or imperative, or whether it's idempotent or not. An imperative tool can change your system from known initial state A to desired state B. A declarative system can change it from whatever initial state it's in to desired state B, even if you never considered it might be in that state.
When you want to make a configuration change to your servers, and they aren't already in a "known state", why would you think any tool could put them into a known state? When a computer has a virus, say, you don't think it's been put into a "known state" after the anti-virus program gets done with it; the virus may have done any number of things you might be unaware of, altered any number of data or configuration files in subtle ways that the tool doesn't look for, but implicitly relies upon.
Example: what if, say, one of the provisioning requirements is "make sure this gem is installed", but on one of the servers /etc/gemrc has "install: --no-rdoc --no-ri" in it? Now on one server, the docs are missing, while everywhere else they're available. That sort of thing.
If you're already running on an IaaS like EC2, I think there's a simpler, better way: rather than trying to get "unknown state" to "known state", why not use the simplest known state of all--unprovisioned? Write an imperative script that reinitializes a freshly-provisioned IaaS node to your known state, and then do a rolling reprovision, terminating old nodes and provisioning new ones.
An equivalent comparison[1]: would you feel safe running an automated script that would SSH into a production machine and "git pull" a checked-out repo sitting on it, from whatever state it happens to be sitting at, up to refs/heads/master, so as to deploy code from it? Could you guarantee that that repo hadn't been moved to some state where refs/heads/master isn't a fast-forward commit? Or would you rather do a fresh "git clone"?
---
[1] A contrived comparison, though; although the former option is unreliable, the latter is terribly inefficient, and they're both horrible for keeping a .git directory inside of a directory that might very well be web-accessible.
When you install the package you do it as a on-off. When a configuration system does it then it will keep re-happening if anything changes.
Say you install "less", and I remove it. With Chef this will be fixed the next run - with fabric? You'll need to re-run your magic script.
That's the main difference. Automation like Chef/CFEngine/Slaughter will keep fixing breakages - as defined by differences in state from their rules - whereas you'll be playing a game of catchup.
apt-get is most definitively not idempotent without a lot of care. It will upgrade packages if a new version is available, and such upgrades can have widespread implications.
Otherwise I agree with you - Puppet/ Chef etc. are overly big on ceremony for me. We "only" use about 100 VM's on about 20 physical servers at the moment, so it's a "simple" setup.
I think where people get the most value out of systems like Puppet and Chef is if a lot of their servers or VM's have simple enough requirements that they can largely use "off the shelf" community provided configurations. The more customisation you need, the more the leve of ceremony starts biting you. The more you can rely on ready-made scripts, the more those apps help you because of the size of their communities.
I'm not that familiar with Fabric, so maybe it can do some of the things I mention here out of the box without a lot of custom code. I'm only familiar with Capistrano and I associate them in my head, perhaps naively.
Your example is great to illustrate why configuration management is so powerful.
Most importantly, your example will upgrade the package to the latest version every time you run it (which is probably not what you intended, but maybe it is). I'm sure there is an idempotent command for each of apt, yum, ports, homebrew, etc, but I have better things to do with my time than figure that all out.
In puppet your command would be:
package { 'foo': ensure => installed } # or 'latest'
What if you also have RedHat servers? Now you need to figure out the right yum command to use to perform the equivalent. Or your developers Macs? Homebrew. Chef/Puppet are aware of the environment they run in and install appropriate packages using the appropriate package management script with minimal alteration to the underlying recipe.
For RedHat systems, it would be:
package {'foo' : ensure => installed }
On a Mac, it would be:
package {'foo' : ensure => installed }
On that Arch Linux machine that your devs are running, it would be:
package {'foo' : ensure => installed }
Now, let's throw some dynamism into the mix...
What if you want to build an haproxy configuration file that pulls the list of hosts from an authoritative source? Do you "just" build it up from scratch every time and replace the file on every run of your script? With chef/puppet, you can pull that information in from your node classifier using the same query for dev, staging, and production and have the same recipe work for multiple environments. In some environments, the haproxy config has now changed, so I want to tell haproxy to reload. But in other environments, the config stayed the same, so I don't want it to reload. The file and service don't get touched if they didn't change.
What if a service has multiple files associated with it that when one or multiple of them change, you have to restart or reload the service? You don't want to restart each service every time one of the multiple files changes, so you end up having to build a queue to handle the service notifications.
Say I've got a few dozen users on my hosts and someone leaves. Which is easier - writing a purge script which knows how to purge the user from all environments that you support, or this...
user { 'joe' :
...
ensure => absent
...
}
I'm certain that that you can do these things with fabric scripts, but by the time you have, you've either written your own configuration management tool (which is unlikely to be as robust as chef or puppet) or you have a hodgepodge of unmaintainable scripts.
Eh, I didn't like Ansible. Every command is done over SSH, so you don't need an agent/ssh to box and run locally, but... scripts would take much longer to execute than in puppet.
We're currently using local runs of puppet, and it's easy to do dry-runs and see what fails before applying when testing. I'm not sure how Ansible gives feedback on that (I assume it must) but I find puppet pretty good about it.
Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.
Speaking from experience as a part time admin, it is definitely worth it to automate the process, even with one or two servers. Any build system will repay in spades the first time you have to set up a new server or rebuild an old one, esp. if you're in a hurry after a server failure.
That said Linode offers other easier options (Linode specific of course). You can take the commands usually executed, and put them in a shell script in the language of your choice which is uploaded and executed on first run (they call this StackScripts - there are lots of examples and helper scripts on their site, but the scripts are really very basic). It's perhaps easier than learning a DSL as it's just codifying the same commands you would run via ssh by hand. You can also clone nodes so you could set one up in a known good state and clone that each time.
This does tie you to linode, so long term one of the other options available would be worth looking in to.
I think being a part-time admin with other responsibilities is a perfect reason to learn something like Chef or Puppet. It'll end up saving you a lot of time that you can devote to other things.
Without Chef there would be no way for me to rollout a new server in our cluster. Investing time into Chef was one of the greatest things I ever did. Chef is the best documentation of our infrastructure. The second best tool I'm using is fpm[1] to make custom debian packages.
While not directly related to this article, last I checked Puppet/Chef don't have particularly useful windows support. Is there an chef like solution for windows?
I dunno about this guy but apt-get update/apt-get upgrade takes upwards of 15 minutes on new installs (as does yum upgrade on rhel-compatible systems). Kinda eats up into my first 5 minutes on a server.
Look into apt-cacher-ng. I have a slow internet connection at home and it made updates go a lot faster for multiple machines. Newer versions (in wheezy, not squeeze) even work with yum.
While I agree these practices seem pretty safe/standard, it's still enough manual work that many people will just skip half of it out of laziness for small projects.
Does anyone have good reusable Puppet (or bash scripts) published that they use on all new servers?
Here's something I wrote very quickly (bash script): https://github.com/vahek/vSetup
It does most of the stuff mentioned in the post, however doesn't setup automatic updates or Logwatch.
(to both corin and vahe): That kind of shell scripting is the horrorshow that motivated configuration management tools* in the first place. Shell scripts require a lot of added complexity to manage multiple heterogeneous servers, or to be idempotent.
Beginner or not, you should probably use visudo [1] instead of
vim /etc/sudoers
for the sanity checks that it provides, if nothing else. A botched edit of /etc/sudoers that locks you (along with every other user) out of administrative access is an unpleasant way to learn this.
Similarly "ufw allow from {ipaddress-you-will-access-from} to any port 22" sounds like a good way to accidentally lock yourself out unless you have an out of band backup
I've not tried anything complex with UFW so I still use iptables on my bastion host that handles my vpn tap. It's not terribly complex to make rules come back on startup (but probably more involved than one would hope).
This is how I run iptables on a sufficiently large network of machines.
The advice is not complete. IPv6 is real and really works most of the time these days. Back up your ip6tables to a file too. I like /etc/firewall-4.conf and /etc/firewall-6.conf but it's down to preference.
Know about iptables-apply too, lest you be caught unaware.
I've never understood the point of fail2ban if you disable ssh password authentication. Yeah, it might eliminate some spam in the logs, but if you only allow key-based authentication that doesn't really matter.
My only thought is even with password auth. turned off hostile persons can still connect to the ssh daemon and might discover and unknown vulnerability or some type of dos. If you know someone is trouble why let them in even if you are wearing armor?
fail2ban can deal with much more than SSH login attempts. I believe you can configure it to do arbitrary actions on arbitrary log events. Perhaps it comes with a block for HTTP, too?
Seeing as he blocks SSH and only allows it from his IP, there won't be any SSH attempts reaching sshd at all, so fail2ban wouldn't be doing anything if it was looking at SSH only.
I actually do this on my personal server because more than one public wifi (and one of my previous jobs, at a public high school) disallowed outbound 22 connections.
Sure, it'll not stop dedicated manual intrusion attempts, but it will actually prevent a ton of automated bots from even just trying to connect with common passwords through SSH.
2222 is a dumb choice as an alternative port, it's both obvious and quite commonly used. I'm using a port on 4XXXX-range that's normally not used for anything and therefore not scanned by the bots unless all the 65536 ports are. The automated login attempts disappeared almost immediately, except for a few that were quickly blocked.
Now the logs are clean from automated login bots, the only thing left are real dedicated hacking attempts that is worth pursuing further.
And, when some dev screws up by pushing a poisonous build, there is no way to figure out who to contact to fix. I generally have an extensive pam+LDAP setup to control Logins. Having said that, if I am trying to use keys, puppet filebucket works for me.
To prevent having to log into root via remote console, set up a second backup account with an ssh key and sudo access, and then put that key somewhere safe.
The chances of both that and the deploy user getting corrupted at the same time is unlikely.
My big thing is making sure 1) the machine comes back up cleanly after a reboot and 2) is current on all patches 3) is running as little as possible.
Also a big fan of externally verifying what ports are open, and making sure the system is in monitoring, backup, config management systems. Config management is kind of optional if you have a small number of servers which don't duplicate configurations, though, and there's often no need to back up the OS, but any data should be backed up automatically.
Is there a web app that I can just have control my servers from? I'd like to do puppet/chef, but I read that its not 100% working for Windows yet (I need to manage both) and it sounds overly complicated.
I've had df -h write out to the motd before for clients who insisted on tiny slices of disk, no automated monitoring and no manual checking of disk space even after it chewed itself up.
I'm not sure it helped, but it made me feel better.
I just use this bash script when setting up a new VPS. It pretty much takes care of it all for you, plus it sets up a ngnix/mysql stack already optimised for law RAM machines.
Also sets up 3proxy which is handy for viewing Hulu/Netflix :)
1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one.
2) Everything else listed in this document should be configured by a configuration management system.
3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure.
4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does.
For #4, wouldn't the only change when someone leaves the organization be to remove their key from authorized_keys for the shared account? Why would anyone else have to be updated?
But they still would know the 'deploy' password needed for sudo access. And while you could be relatively sure that they couldn't get access, you still couldn't be completely sure since they did have sudo access to begin with. So, the best thing would be to change the shared password. That could be avoided with non-shared accounts.
Are you insinuating that the user could have used sudo access to install a backdoor of some sort? If so, changing the password won't stop them either. Am I missing something?
It's a basic principle of security. Each account represents one person so that you have a full audit of who did what by watching the activity of a given user account. If everything is run as "devops" user for example, you have no idea who actually performed a given task. Was it Bill, or was it an automated job? PCI-DSS requirements also affect your model for user accounts (hint: shared users are often not compliant).
From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
When someone advocates using a VPN, that doesn't mean not using SSH too. VPN + firewall just restricts who has the potential to try to SSH to you, and provides additional protection and central access control/management.
I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits.
Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least).
Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
Yes, I agree, I was just giving an example of how an additional layer can help protect against automated attacks, even for highly-secure services like SSH.
I also agree that SSH is less likely to have flaws than most VPN software. But on a properly configured bastion host, by-passing the VPN would just put you in a position where you can attack SSH. You would still need to by-pass SSH to access production servers.
I am, just from my experience in real-life companies. It takes an awful lot of discipline to treat servers as if they were exposed to the public internet when you know full well that they're not.
anyone who compromises [the VPN server] is in a pretty good position.
Sure. But without a VPN, anyone who compromises even one of your other hosts is in the same position. It's a lot easier to audit a single-purpose VPN server for possible security issues than it is to audit all the application code running on the rest of your production systems.
And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
I wouldn't. And even if you're right, getting a VPN login still doesn't get you anywhere. You still have to be able to ssh to the rest of the hosts. That's why we do security in layers.
I have experience with many open source VPN servers. The purpose is a bit different -- we provide a VPN service to home users to encrypt their internet traffic. But the same problems should apply.
OpenVPN is the most compatible with a variety of clients. OpenVPN runs in userspace, so the clients for each OS and mobile platform interoperate well. The downside is, it does require a client program to be installed and configured. It's considered very secure, using SSL. Since it's userspace, moving large amounts of traffic means more context switching and higher cpu usage. Despite that, I've found it to be faster and more stable than the alternatives.
L2TP/IPSec is built in to most clients -- Windows, OS X, mobile. But every implementation is different and it's hard to configure a server to work with all of them. There are also more moving parts -- an IPSec server (openswan, strongswan, or racoon), and L2TP server (openl2tpd, xl2tpd) and a PPP server (pppd). IPSec seems to be a secure protocol but it's very complicated. I tend to distrust complicated security.
Pure IPSec has many of the problems of L2TP/IPSec with the added problem of difficult to configure in Windows and OS X.
PPTP is not performant or very secure. Other than the fact that almost every client supports it, I see no reason to use it for a new VPN.
Ok, just in case it wasn't clear: I am asking about the advantages of having a VPN server as a bastion host instead of another SSH server.
You would ssh to the bastion host, and from there to internal hosts.
With the appropriate ssh config at the client end, the tunneling through the bastion can be scripted away (using the ProxyCommand directive and RSA keys).
Routing transparency. Sure, you can script a bunch of tunnels, but it's nice to handle routing at a lower layer. Having worked with both setups, I vastly prefer the VPN solution for ease of setup, use, and maintenance.
Also, different attack surfaces. Two layers of the same security measure (ssh) is, all else equal, not quite as good as two layers involving two different measures (VPN, ssh).
openssh is one of the most secure projects. It's developed by the security obsessed (and I mean it in a kind way) folks at OpenBSD.
I, for one, am ready to place for more trust in openssh than in any VPN daemon. The most commonly used ones are propitiatory.
What if there is a 0-day vuln (not exploit) for these VPN daemons? That far more likely. "Securing" ssh with a VPN is just one step beyond of security by obscurity.
If you are afraid of script-kiddies and scanners, let your sshd listen on a non-standard port.
Port scanners do generally scan non-standard ports too, you know...
I don't think that exposing SSH to the internet is that bad, but your argument is not sound - requiring a VPN does add security, because if there happened to be a vulnerability in it that allowed access, all it would do is expose SSH on the machines (I'm assuming you have proper firewalls set up), which you are advocating making public in the first place.
Saying it adds no security is false, because you'd require an unpatched vulnerability both in your VPN server and in the SSH server simultaneously. A zero-day in one is possible, but in both at the same time is far, far less likely.
> "Securing" ssh with a VPN is just one step beyond of security by obscurity.
You're not securing ssh with a vpn. You're adding another layer. ssh is still secured by all of ssh's existing protection.
ssh behind a vpn requires that someone both compromise the vpn _and_ compromise the ssh service to gain that access that, without a vpn, would require them to only compromise the ssh service.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited
So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.
Agreed, anyone who has touched PCI DSS would agree you need to associate access with a human user. This would not work. If you look at the security logs it won't differentiate between which keys were used for that generic account.
If they have had access to a shared account, how much work are you willing to put in to verify that there's nothing in that shared account that will get executed by another user later that will quietly reinstate a key?
A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater)
After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.
But if you give people any access to any system surely this is a concern. As a software dev, maybe I've inserted something into one of the build scripts that quietly re-opens my backdoor to the source control server...
I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?
Quite the contrary: You're probably going to end up with documented procedures for deploying software that are simplified, follow existing standards/best practices, and don't rely on complex stone soup build/init scripts concocted by inexperienced developers (and I've seen some doozies).
A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.
Not quite sure I understand what you're saying here, I'm not talking about the software being produced, but the systems used to produce it.
What I was trying to say was that there's not really any way for you (server admin guy) to know if I (software dev guy) have inserted something malicious into a script that all the other software folks run constantly (software build system, NOT server build/init script, NOT deployment script).
This is not about the end-user's privileges, or server set up, just how in a team-base software dev environment you're probably going to have to have a measure of trust for your employees.
I see, but I think the same principle applies, even in this narrow case. As a server admin guy or fellow software dev guy, I have to trust that any code you've written has been properly reviewed before checking it into a repository that I pull from. Fortunately, version control tools make this trivial, but you're right, the policy and infrastructure supporting it has to be in place, otherwise you're depending only on voluntary peer review.
Note that as an attacker, there's a high risk of exposure and identification in the scenario you describe, and that's a good thing. A well secured system shouldn't merely prevent attacks, it should also protect innocent users from suspicion (another reason why shared accounts are discouraged).
It's commonly stated that 9 out of 10 security threats come from employees or other insiders. You should assume malicious employees. Sooner or later you will hire the wrong person.
Now, you must also have a functioning system, and so you may take risks by leaving things more open than you would like if you don't have the resources to thoroughly lock everything down.
But wherever locking things down further costs you very little, you should take the opportunity. And elsewhere you should asses what level of protection you can afford. Ultimately it is a cost-benefit analysis. Many risks are not worth spending money protecting against. Others are vital.
But even disregarding malicious users: Individual user accounts is not just a protection against malicious users, but against careless users. When someone sets a password that gets guessed, you want to be in a position where exploiting that persons credentials is as hard as possible, and tracking down actions taken via the account is as easy as possible.
And yes, you could insert something into a build script. But if the build script is committed, and the commit was pushed from a named, individual account, you're now at the risk of going to jail. Creating deterrents is often a sufficient risk mitigation strategy to be acceptable.
Chef or Puppet are common choices. There are many others, but those two are modern, have large communities, and decent documentation. There's a decent chance someone has already open sourced a cookbook/module for each of many of these items!
On linode, which he is using, there is one built in - they allow shell scripts to run on first boot which could replay his commands as they are without many changes:
No, chef/puppet/etc are configuration management tools. They automate the manual typing of server setup commands, and then verify that the system stays that way.
They are not security tools. So you're on your own on what to actually tell the tools to do. "Install chef" is not a security tip. It's a repeatability tip, so you can get your system up to a known state repeatedly.
For the security side of things, you're back to figuring out what the right steps are, no matter how they're installed.
But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool.
Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.
Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.
It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet.
Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.
You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)
The problem is that most of the recommendations are bad.
Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior).
The rest is just bad advice.
Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?).
User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server.
Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers.
Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses.
You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.
You are assuming that they have several developers. I bet it's one or two
The article presents this as general advice and doesn't make any mention of a team size that's appropriate for the recommendations. Maybe this guy has only 1 or 2 developers working on boxes, but IMO for any number of people >1 you don't want shared accounts.
Also you are assuming this servers are staying with the company, which may not be the case
In many cases they will be, and, again, the article presents this as general advice. If a server is being configured for a client and sent out, then a general process of setting up local user accounts probably doesn't make sense anyway (unless you only have one client).
Agreed that LDAP isn't the most friendly of things to set up, but there are how-tos for common use-cases, and if you have any skill as a sysadmin, you can do it. As soon as it's set up, it's simple to maintain. Manually keeping accounts in sync across multiple machines is rarely simple. Maybe LDAP is overkill for a handful of hosts and user accounts, but if you expect to grow even a little, expect to need some kind of centralized user account system.
If you only have one or two developers, then you have all the more reason to automate the boring and mundane tasks, since the time savings are even greater as a percentage of total time available!
I agree on a lot of things, except LDAP. Of course, it sounds like the ideal way, but in practice, there's a matter of scale to take into account.
Frankly, you need a lot of developers and servers before investing the time to setup a ldap deployment, integrating it with logins, and spending the inevitable hours debugging why nobody can access anything anymore, becomes more worth it than "just rsync/pssh into all servers and edit /etc/passwd".
Actually, if I had to do it, I'd use chef to automate creation (and destruction) of user accounts over an LDAP any day. Chef can be a pain to learn and use, but any sort of LDAP is even worse.
I would probably stand up an LDAP server after I had 4 or 5 user accounts to deal with, or, more importantly, more than, say, 3 servers to deal with.
I think there's a lot of fear and hate surrounding LDAP, but that's mostly for historical reasons. LDAP has gotten a lot easier to set up. Even in 2009 a colleague and I set one up (using openldap) and had other machines authenticating off of it in an afternoon. It's gotten even easier than that since then.
And hell, you should be using Chef to set up your LDAP master and slave. So once you have the config correct once, you can bring up another machine without trouble when needed.
My friend and I are rebuilding an old, text-based browser RPG, occasionally with another friend helps out with art and game content.
We're on LDAP, which we use to SSH into our ec2 servers, and which we use for authentication when we deploy using `git push production master` to a GlusterFS cluster. We're running our LDAP, application, and file servers on Gentoo. We can easily add new accounts, and we have it set up with group permissions (so the friend can deploy game content to test but not prod, for example).
I refuse to believe that LDAP is "too complicated" or "has to scale before it's useful", when a couple of guys can, in their free time, set it up for themselves. It's saved us a load of time in managing servers that would otherwise take away from the limited time we have to actually write code.
It's also a whole lot cleaner than a bunch of Chef scripts running a script across a quantity of servers; using Chef can too often be a crutch to fixing the actual issue.
I just wrote an ansible script to do these sorts of things yesterday. Now, whenever I get a new server, I just run the script with its ip, and boom, provisioned.
I guess it all depends on how often you bootstrap a new server. If you do it infrequently, keeping a configuration management tool updated is probably more work. I do it perhaps every two years and keep a similar list (I try to keep up to date if I change something in the server configuration) for that purpose.
>Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.
You're right -- many used cfengine. Still others used a custom 'config' rpm / deb that deployed all of these files everywhere.
Automated configuration makes sense not just for repeatability, but for auditability and documentation. Especially when you are going to 'hand the server over', the next admin should be able to know what you've changed.
Also, disallow password-based access to everything (use the keys, Luke.)
I'm a good developer, but setting up systems is something I do so infrequently that I always have to relearn it, and I am positive I could be doing things better. Do you have a link to an article that explains how to go about performing some of these ideas you mention?
Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.
This is an excellent request. Reading through all the comments here, it seems like a lot of people are feeling frustrated with the variety of information available and no clear way to discern what is "good".
I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-)
Thanks! I really appreciate it. All to often we are quick to criticize someone for missing information without any willingness to back it up with better, correct information. The jokes goes that the best way to get an answer is to publish the wrong one. =)
That site is useless, sample pages I looked contained brief overview paragraphs and no real content. Many contained "todo" items.
So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.
If Bastille was still in working order, I'd recommend it as a very good starting point for locking down a configuration. Actual configuration deployment setups I don't put much stock in because I'm not managing enough machines for it to be worthwhile. Would love to hear from someone who manages a large farm/cluster/VM hosting on a very clean and straightforward way to manage configurations. Last I looked, most setups were custom, or some hobbled together packages that didn't "hang" together very well.
This depends entirely upon your level of masochism and the kind of language youlike scripting in, as literally anything will do. Even shell scripts. Even Makefiles. Whatever you're most comfortable with, just start writing out configs. (you will eventually come to hate your job if you allow non-programmers to script/program in your CM, but blah blah keep shipping blah blah) Break it all out intoa wide hirearchy so you can reuse some bits in other bits.
Aaaaand you're done. Of course I haven't tested these configs and something will probably break (most likely my crappy code) but you'll get the idea from looking at the examples.
I wouldn't bother with fail2ban considering password based SSH logins are disabled (which is good).
Since the author is using ufw to control iptables, better to just use "ufw limit" rules for SSH port 22 to slow down the rate of any automated SSH bots trying to give your server a workout.
Indeed. I have always been a bit worried about such approaches, since they parse log files and attackers have some control over what is written to log files (user names and host names).
1. You should do "apt-get dist-upgrade" to get new kernel packages as well, otherwise you are stuck on an old kernel. (You might want that. I prefer updated kernel for the security, firefoxen, etc.). "apt-get upgrade" will only update existing packages - but the kernel updates require new packages to be installed.
2. If you're on ubuntu, root already has no password, and your initial setup user (whether it is called "deploy" or "kilroy") is in the sudoers file.
3. Other things I install in the "5 minutes with server" are: htop molly-guard screen git-core etckeeper
git-core because I prefer my etckeeper in git, but if you want it in bzr you don't need git-core. INSTALL AND CONFIGURE ETCKEEPER AS SOON AS YOU CAN, seriously. You need it. You'll thank me when you try to figure out when and how something in /etc got borked. (you need to edit /etc/etckeeper/etckeeper.conf if you use gif. You need to do "etckeeper init" and then "etckeeper commit" to establish the baseline)
molly-guard stops you from rebooting the wrong server
screen (or alternatively tmux) lets you keep your session open through ssh session disconnects (e.g. when moving from wifi to 3G, or between 3G towers that give you different external IP). The most useful way to use screen is "screen -xR" which also lets you share your session with someone else should you need to.
I'm not sure I get your "on the contrary". Any configuration change on a managed configuration file is transient and bound to overwritten next time state is restored. This makes etckeeper absolutely less useful compared to a system where there is no way to restore state and an unfortunate "rm /etc/passwd" has little remedy. It does not make it "useless", but it's definitely less needed.
Possibly my misunderstanding, but I thought chef and puppet only push changes on the files that they are configured to control - There are tens of files in /etc/, some possibly modified by an "apt-get upgrade", then reverted by chef, some that chef/puppet do not try to modify.
etckeeper is a net that gives you an idea of how /etc looked on a given date or apt-run, not just how it was supposed to look (Which is what you get from the chef/puppet logs).
Question to chef/puppet users: do you restore the state continuously? or only when you've made a change?
Yes, it's true that chef/puppet do not overwrite your entire etc, just a subset of it, but presumably, you'll have your critical components backed by it (otherwise, there is not much point in using a configuration management system). So it is still useful on such a system, but not to the same extent of a system without configuration management.
> but presumably, you'll have your critical components backed by it
Often, the way you discover something is critical is after the first time it breaks. Backups may help, but proper version history is probably going to give better insight into what happened.
It is the point that you "never do that". However, it's been my experience that e.g. when troubleshooting, people always make local modifications, and don't always remember to revert them / restore from configuration management.
I have no experience with chef/puppet, so I might be mistaken, but I'm under the impression that they only push changes when asked to - which means a local change may survive for several weeks before it is overwritten. But I could be wrong about that.
"screen -x" requires a session to already exist, whereas "screen -xR" will join one if it already exists, but will create one if it does not. At least it does in v4.00 which I use. If you have a "screen" call in your @reboot, you already have a session, so they will work the same.
Also check to see what's enabled to run at boot time via whatever your flavor uses.
Check for unusual daemons, ssh running on other ports (yes, the provider pre-loaded systems with a back-door ssh without disclosing it to us).
This is especially important when you are taking over admin on a server you didn't setup yourself. Other folks have weird ideas on how to admin things. Like webmin for example...
I also like epylog for finding unexpected stuff in the logs.
We do something similar, though we have a script that sets up the box. We use linode as well, so the script deploys a new box with a complex root password and my id_dsa.pub file. It also sets up /etc/skel and the profile files so that useradd uses them later.
We don't use a single deploy user though, instead, each user with deploy perms is in the sudo group.
354 comments
[ 3.0 ms ] story [ 290 ms ] threadOne correction for you: the sshd_config lines should have no "=" symbol.
You seriously do this by hand for every server? That seems error prone and a huge waste of time when tools like puppet and chef exist.
As an aside, I think the default fail2ban config is too loose and quiet. Here's [4] an Ansible task file that configures it to be more aggressive and send notification emails.
[1] http://ansible.cc/
[2] https://gist.github.com/dbarlett/5079802
[3] https://github.com/fdavis/ansible-best-practices
[4] https://gist.github.com/dbarlett/5079715
I'm not trying to be provocative or negative. I'm really trying to understand the supposedly big difference between what's labelled a deployment tool, and configuration management tools.
multiple packages can be done this way if you have a lot:
$prep_packages = ['package1', 'package2', 'blahblah']
package { $prep_packages: ensure => installed }
You can of course use extra arguments if you need them.
Deployment tools just execute whatever script you hand them, and so the scripts are either more brittle (server must be in a precise state beforehand or it doesn't work right), or require more effort to duplicate the work that the configuration management software does to only make the needed changes.
If your deployment scripts get complex, it's more difficult to see at a glance what the end configuration is supposed to be.
I'm still struggling with seeing some practical examples of those differences.
for example, `apt-get` is pretty much idempotent already, no? if I run `apt-get install -y <package>` x 1000 times in a loop it won't install it 1000 times...
Fabric does give you the building blocks for those kind of checks elsewhere, such as `exists`, `contains`, `append` (which is supposed to also be idempotent) etc... I've designed my deployment / bootstrap scripts with fabric to take this into account. It does add a little overhead, but nothing that makes me feel I need a better tool... Maybe my deployment base is still relatively small and homogeneous.
So it's true that I have to put in those checks myself, and that I don't have a very easy way to discover what state a server is at.
I'll try to take another stab at one of those tools. Maybe things will sink in when I actually use them. Thanks again for explaining.
And what if you want to have the same logic for several services? I guess you abstract it out to a function. But then it turns out one of those services doesn't have a restart command, and you have to do stop+start. And another service won't start if you use restart while it's not running, so you have to check if it's stopped and use start, otherwise use restart.
It's much more than just whether the code is declarative or imperative, or whether it's idempotent or not. An imperative tool can change your system from known initial state A to desired state B. A declarative system can change it from whatever initial state it's in to desired state B, even if you never considered it might be in that state.
Example: what if, say, one of the provisioning requirements is "make sure this gem is installed", but on one of the servers /etc/gemrc has "install: --no-rdoc --no-ri" in it? Now on one server, the docs are missing, while everywhere else they're available. That sort of thing.
If you're already running on an IaaS like EC2, I think there's a simpler, better way: rather than trying to get "unknown state" to "known state", why not use the simplest known state of all--unprovisioned? Write an imperative script that reinitializes a freshly-provisioned IaaS node to your known state, and then do a rolling reprovision, terminating old nodes and provisioning new ones.
An equivalent comparison[1]: would you feel safe running an automated script that would SSH into a production machine and "git pull" a checked-out repo sitting on it, from whatever state it happens to be sitting at, up to refs/heads/master, so as to deploy code from it? Could you guarantee that that repo hadn't been moved to some state where refs/heads/master isn't a fast-forward commit? Or would you rather do a fresh "git clone"?
---
[1] A contrived comparison, though; although the former option is unreliable, the latter is terribly inefficient, and they're both horrible for keeping a .git directory inside of a directory that might very well be web-accessible.
Say you install "less", and I remove it. With Chef this will be fixed the next run - with fabric? You'll need to re-run your magic script.
That's the main difference. Automation like Chef/CFEngine/Slaughter will keep fixing breakages - as defined by differences in state from their rules - whereas you'll be playing a game of catchup.
Otherwise I agree with you - Puppet/ Chef etc. are overly big on ceremony for me. We "only" use about 100 VM's on about 20 physical servers at the moment, so it's a "simple" setup.
I think where people get the most value out of systems like Puppet and Chef is if a lot of their servers or VM's have simple enough requirements that they can largely use "off the shelf" community provided configurations. The more customisation you need, the more the leve of ceremony starts biting you. The more you can rely on ready-made scripts, the more those apps help you because of the size of their communities.
If you use a single distribution and can get all your stuff packaged using the distribution's packaging system, you're OK.
It can be more difficult when you get to system configuration (editing files), but you can do it manually like you do with fabric if you're careful.
Your example is great to illustrate why configuration management is so powerful.
Most importantly, your example will upgrade the package to the latest version every time you run it (which is probably not what you intended, but maybe it is). I'm sure there is an idempotent command for each of apt, yum, ports, homebrew, etc, but I have better things to do with my time than figure that all out.
In puppet your command would be:
What if you also have RedHat servers? Now you need to figure out the right yum command to use to perform the equivalent. Or your developers Macs? Homebrew. Chef/Puppet are aware of the environment they run in and install appropriate packages using the appropriate package management script with minimal alteration to the underlying recipe.For RedHat systems, it would be:
On a Mac, it would be: On that Arch Linux machine that your devs are running, it would be: Now, let's throw some dynamism into the mix...What if you want to build an haproxy configuration file that pulls the list of hosts from an authoritative source? Do you "just" build it up from scratch every time and replace the file on every run of your script? With chef/puppet, you can pull that information in from your node classifier using the same query for dev, staging, and production and have the same recipe work for multiple environments. In some environments, the haproxy config has now changed, so I want to tell haproxy to reload. But in other environments, the config stayed the same, so I don't want it to reload. The file and service don't get touched if they didn't change.
What if a service has multiple files associated with it that when one or multiple of them change, you have to restart or reload the service? You don't want to restart each service every time one of the multiple files changes, so you end up having to build a queue to handle the service notifications.
Say I've got a few dozen users on my hosts and someone leaves. Which is easier - writing a purge script which knows how to purge the user from all environments that you support, or this...
I'm certain that that you can do these things with fabric scripts, but by the time you have, you've either written your own configuration management tool (which is unlikely to be as robust as chef or puppet) or you have a hodgepodge of unmaintainable scripts.We're currently using local runs of puppet, and it's easy to do dry-runs and see what fails before applying when testing. I'm not sure how Ansible gives feedback on that (I assume it must) but I find puppet pretty good about it.
You can bootstrap 0mq and run them over that instead. That's much faster.
Speaking from experience as a part time admin, it is definitely worth it to automate the process, even with one or two servers. Any build system will repay in spades the first time you have to set up a new server or rebuild an old one, esp. if you're in a hurry after a server failure.
That said Linode offers other easier options (Linode specific of course). You can take the commands usually executed, and put them in a shell script in the language of your choice which is uploaded and executed on first run (they call this StackScripts - there are lots of examples and helper scripts on their site, but the scripts are really very basic). It's perhaps easier than learning a DSL as it's just codifying the same commands you would run via ssh by hand. You can also clone nodes so you could set one up in a known good state and clone that each time.
This does tie you to linode, so long term one of the other options available would be worth looking in to.
[1] https://github.com/jordansissel/fpm
Does anyone have good reusable Puppet (or bash scripts) published that they use on all new servers?
* Puppet, Chef, Ansible, Salt, CFEngine, etc.
Also, since this post is for beginners, you should mention about restarting the sshd over the lish tty and not over the ssh ptty.
[1] http://linux.die.net/man/8/visudo
The ufw man page is pretty decent.
For anyone else that followed the thread to this point- this advice on bringing iptables back up on reboot worked for me http://rackerhacker.com/2009/11/16/automatically-loading-ipt... YMMV
The advice is not complete. IPv6 is real and really works most of the time these days. Back up your ip6tables to a file too. I like /etc/firewall-4.conf and /etc/firewall-6.conf but it's down to preference.
Know about iptables-apply too, lest you be caught unaware.
%admin ALL=(ALL) ALL
So you just need to make sure your 'deploy' user account belongs to the admin group and you're good to go.
$ sudo adduser username sudo
Seeing as he blocks SSH and only allows it from his IP, there won't be any SSH attempts reaching sshd at all, so fail2ban wouldn't be doing anything if it was looking at SSH only.
IMO best practice is to firewall off everything except some bastion hosts or VPN gateway.
http://bsdly.blogspot.com/2013/02/theres-no-protection-in-hi...
Now the logs are clean from automated login bots, the only thing left are real dedicated hacking attempts that is worth pursuing further.
Doesn't take long to port scan a server.
flag arguments should go before any other arguments to be compatible with the most Unix systems.
The chances of both that and the deploy user getting corrupted at the same time is unlikely.
Also a big fan of externally verifying what ports are open, and making sure the system is in monitoring, backup, config management systems. Config management is kind of optional if you have a small number of servers which don't duplicate configurations, though, and there's often no need to back up the OS, but any data should be backed up automatically.
I'm not sure it helped, but it made me feel better.
Looks like an interesting piece of software nonetheless.
I just use this bash script when setting up a new VPS. It pretty much takes care of it all for you, plus it sets up a ngnix/mysql stack already optimised for law RAM machines.
Also sets up 3proxy which is handy for viewing Hulu/Netflix :)
1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one.
2) Everything else listed in this document should be configured by a configuration management system.
3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure.
4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does.
This stuff isn't hard. It's worth doing right.
From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.
Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack.
Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least).
Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
I also agree that SSH is less likely to have flaws than most VPN software. But on a properly configured bastion host, by-passing the VPN would just put you in a position where you can attack SSH. You would still need to by-pass SSH to access production servers.
You are assuming the VPN host is trusted any more than most people trusts random servers on the internet.
Sure. But without a VPN, anyone who compromises even one of your other hosts is in the same position. It's a lot easier to audit a single-purpose VPN server for possible security issues than it is to audit all the application code running on the rest of your production systems.
And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
I wouldn't. And even if you're right, getting a VPN login still doesn't get you anywhere. You still have to be able to ssh to the rest of the hosts. That's why we do security in layers.
I liked the idea of adding VPN layer to SSH, so would like to get as much advice as possible :-)
OpenVPN is the most compatible with a variety of clients. OpenVPN runs in userspace, so the clients for each OS and mobile platform interoperate well. The downside is, it does require a client program to be installed and configured. It's considered very secure, using SSL. Since it's userspace, moving large amounts of traffic means more context switching and higher cpu usage. Despite that, I've found it to be faster and more stable than the alternatives.
L2TP/IPSec is built in to most clients -- Windows, OS X, mobile. But every implementation is different and it's hard to configure a server to work with all of them. There are also more moving parts -- an IPSec server (openswan, strongswan, or racoon), and L2TP server (openl2tpd, xl2tpd) and a PPP server (pppd). IPSec seems to be a secure protocol but it's very complicated. I tend to distrust complicated security.
Pure IPSec has many of the problems of L2TP/IPSec with the added problem of difficult to configure in Windows and OS X.
PPTP is not performant or very secure. Other than the fact that almost every client supports it, I see no reason to use it for a new VPN.
You would ssh to the bastion host, and from there to internal hosts.
With the appropriate ssh config at the client end, the tunneling through the bastion can be scripted away (using the ProxyCommand directive and RSA keys).
Also, different attack surfaces. Two layers of the same security measure (ssh) is, all else equal, not quite as good as two layers involving two different measures (VPN, ssh).
openssh is one of the most secure projects. It's developed by the security obsessed (and I mean it in a kind way) folks at OpenBSD.
I, for one, am ready to place for more trust in openssh than in any VPN daemon. The most commonly used ones are propitiatory.
What if there is a 0-day vuln (not exploit) for these VPN daemons? That far more likely. "Securing" ssh with a VPN is just one step beyond of security by obscurity.
If you are afraid of script-kiddies and scanners, let your sshd listen on a non-standard port.
I don't think that exposing SSH to the internet is that bad, but your argument is not sound - requiring a VPN does add security, because if there happened to be a vulnerability in it that allowed access, all it would do is expose SSH on the machines (I'm assuming you have proper firewalls set up), which you are advocating making public in the first place.
Saying it adds no security is false, because you'd require an unpatched vulnerability both in your VPN server and in the SSH server simultaneously. A zero-day in one is possible, but in both at the same time is far, far less likely.
You're not securing ssh with a vpn. You're adding another layer. ssh is still secured by all of ssh's existing protection.
ssh behind a vpn requires that someone both compromise the vpn _and_ compromise the ssh service to gain that access that, without a vpn, would require them to only compromise the ssh service.
So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.
A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater)
After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.
I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?
A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.
What I was trying to say was that there's not really any way for you (server admin guy) to know if I (software dev guy) have inserted something malicious into a script that all the other software folks run constantly (software build system, NOT server build/init script, NOT deployment script).
This is not about the end-user's privileges, or server set up, just how in a team-base software dev environment you're probably going to have to have a measure of trust for your employees.
Note that as an attacker, there's a high risk of exposure and identification in the scenario you describe, and that's a good thing. A well secured system shouldn't merely prevent attacks, it should also protect innocent users from suspicion (another reason why shared accounts are discouraged).
Now, you must also have a functioning system, and so you may take risks by leaving things more open than you would like if you don't have the resources to thoroughly lock everything down.
But wherever locking things down further costs you very little, you should take the opportunity. And elsewhere you should asses what level of protection you can afford. Ultimately it is a cost-benefit analysis. Many risks are not worth spending money protecting against. Others are vital.
But even disregarding malicious users: Individual user accounts is not just a protection against malicious users, but against careless users. When someone sets a password that gets guessed, you want to be in a position where exploiting that persons credentials is as hard as possible, and tracking down actions taken via the account is as easy as possible.
And yes, you could insert something into a build script. But if the build script is committed, and the commit was pushed from a named, individual account, you're now at the risk of going to jail. Creating deterrents is often a sufficient risk mitigation strategy to be acceptable.
http://www.opscode.com/chef/
https://puppetlabs.com/
http://www.linode.com/stackscripts/
http://ansible.cc/
https://puppetlabs.com/
They are not security tools. So you're on your own on what to actually tell the tools to do. "Install chef" is not a security tip. It's a repeatability tip, so you can get your system up to a known state repeatedly.
For the security side of things, you're back to figuring out what the right steps are, no matter how they're installed.
But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool.
Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.
Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.
Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.
You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)
Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior).
The rest is just bad advice.
Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?).
User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server.
Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers.
Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses.
You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.
If you're dealing with more, let's say, 5, your suggestions became relevant
Also you are assuming this servers are staying with the company, which may not be the case
"User account management is a pain, so that's why we have things like LDAP"
Which is a bag of hurt in itself.
The article presents this as general advice and doesn't make any mention of a team size that's appropriate for the recommendations. Maybe this guy has only 1 or 2 developers working on boxes, but IMO for any number of people >1 you don't want shared accounts.
Also you are assuming this servers are staying with the company, which may not be the case
In many cases they will be, and, again, the article presents this as general advice. If a server is being configured for a client and sent out, then a general process of setting up local user accounts probably doesn't make sense anyway (unless you only have one client).
Agreed that LDAP isn't the most friendly of things to set up, but there are how-tos for common use-cases, and if you have any skill as a sysadmin, you can do it. As soon as it's set up, it's simple to maintain. Manually keeping accounts in sync across multiple machines is rarely simple. Maybe LDAP is overkill for a handful of hosts and user accounts, but if you expect to grow even a little, expect to need some kind of centralized user account system.
Frankly, you need a lot of developers and servers before investing the time to setup a ldap deployment, integrating it with logins, and spending the inevitable hours debugging why nobody can access anything anymore, becomes more worth it than "just rsync/pssh into all servers and edit /etc/passwd".
Actually, if I had to do it, I'd use chef to automate creation (and destruction) of user accounts over an LDAP any day. Chef can be a pain to learn and use, but any sort of LDAP is even worse.
I think there's a lot of fear and hate surrounding LDAP, but that's mostly for historical reasons. LDAP has gotten a lot easier to set up. Even in 2009 a colleague and I set one up (using openldap) and had other machines authenticating off of it in an afternoon. It's gotten even easier than that since then.
And hell, you should be using Chef to set up your LDAP master and slave. So once you have the config correct once, you can bring up another machine without trouble when needed.
We're on LDAP, which we use to SSH into our ec2 servers, and which we use for authentication when we deploy using `git push production master` to a GlusterFS cluster. We're running our LDAP, application, and file servers on Gentoo. We can easily add new accounts, and we have it set up with group permissions (so the friend can deploy game content to test but not prod, for example).
I refuse to believe that LDAP is "too complicated" or "has to scale before it's useful", when a couple of guys can, in their free time, set it up for themselves. It's saved us a load of time in managing servers that would otherwise take away from the limited time we have to actually write code.
It's also a whole lot cleaner than a bunch of Chef scripts running a script across a quantity of servers; using Chef can too often be a crutch to fixing the actual issue.
You're right -- many used cfengine. Still others used a custom 'config' rpm / deb that deployed all of these files everywhere.
Automated configuration makes sense not just for repeatability, but for auditability and documentation. Especially when you are going to 'hand the server over', the next admin should be able to know what you've changed.
Also, disallow password-based access to everything (use the keys, Luke.)
This line of thinking represents a logical fallacy - no one claimed that anything other than Chef or Puppet is "doing it wrong".
Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.
I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-)
In the meantime, a decent source of generalized (not succinct) Ops-type knowledge can be found here: http://www.opsschool.org/en/latest/
So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.
This depends entirely upon your level of masochism and the kind of language youlike scripting in, as literally anything will do. Even shell scripts. Even Makefiles. Whatever you're most comfortable with, just start writing out configs. (you will eventually come to hate your job if you allow non-programmers to script/program in your CM, but blah blah keep shipping blah blah) Break it all out intoa wide hirearchy so you can reuse some bits in other bits.
Hey look, I wrote a crappy one! https://github.com/psypete/public-bin/tree/public-bin/src/si...
Next we implement the OP's comments.
Hey look, I already implemented it in my crappy CM tool! https://github.com/psypete/public-bin/tree/public-bin/src/si...
First push your CM tool and configs to the host:
Then run the main config file which calls the others: Aaaaand you're done. Of course I haven't tested these configs and something will probably break (most likely my crappy code) but you'll get the idea from looking at the examples.Since the author is using ufw to control iptables, better to just use "ufw limit" rules for SSH port 22 to slow down the rate of any automated SSH bots trying to give your server a workout.
2. If you're on ubuntu, root already has no password, and your initial setup user (whether it is called "deploy" or "kilroy") is in the sudoers file.
3. Other things I install in the "5 minutes with server" are: htop molly-guard screen git-core etckeeper
git-core because I prefer my etckeeper in git, but if you want it in bzr you don't need git-core. INSTALL AND CONFIGURE ETCKEEPER AS SOON AS YOU CAN, seriously. You need it. You'll thank me when you try to figure out when and how something in /etc got borked. (you need to edit /etc/etckeeper/etckeeper.conf if you use gif. You need to do "etckeeper init" and then "etckeeper commit" to establish the baseline)
molly-guard stops you from rebooting the wrong server
screen (or alternatively tmux) lets you keep your session open through ssh session disconnects (e.g. when moving from wifi to 3G, or between 3G towers that give you different external IP). The most useful way to use screen is "screen -xR" which also lets you share your session with someone else should you need to.
etckeeper is a net that gives you an idea of how /etc looked on a given date or apt-run, not just how it was supposed to look (Which is what you get from the chef/puppet logs).
Question to chef/puppet users: do you restore the state continuously? or only when you've made a change?
Often, the way you discover something is critical is after the first time it breaks. Backups may help, but proper version history is probably going to give better insight into what happened.
Definitely less important, but not useless, IMO.
I have no experience with chef/puppet, so I might be mistaken, but I'm under the impression that they only push changes when asked to - which means a local change may survive for several weeks before it is overwritten. But I could be wrong about that.
The server should also be rebooted. Applying kernel updates makes no good if you never apply them!
screen -x is equivalent to screen -xR afaict:)
I'd also add @reboot screen to crontab, which will recreate a session on startup - in my bashrc I have :
if [ "a$STY" == "a" ]; then screen -x fi
Other useful things include actually setting up backups (duplicity is a useful first step here), installing munin/nagios to monitor the new box.
Realistically if you are doing this more than once per blue moon, then you should be using something like puppet to do this automagically.
"screen -x" requires a session to already exist, whereas "screen -xR" will join one if it already exists, but will create one if it does not. At least it does in v4.00 which I use. If you have a "screen" call in your @reboot, you already have a session, so they will work the same.
Check for unusual daemons, ssh running on other ports (yes, the provider pre-loaded systems with a back-door ssh without disclosing it to us).
This is especially important when you are taking over admin on a server you didn't setup yourself. Other folks have weird ideas on how to admin things. Like webmin for example...
I also like epylog for finding unexpected stuff in the logs.
what's good beyond this:
Generically (for Linux), take a look at /etc/init.d , /etc/inittab, or /etc/rc?.d.
We don't use a single deploy user though, instead, each user with deploy perms is in the sudo group.