If you start overriding DND, now the user is going to want super-DND. Which somebody will then want to override, and so on. The correct solution is that your users need to not set DND when they in fact need to be disturbed, and your systems shouldn't be disturbing unnecessarily, and to the extent that's a really hard problem, well, yes, it very much is, but an unboundedly-large hierarchy of "bother that person, no don't bother me, SUPER bother that person, no SUPER don't bother me, SUPER MEGA bother that person" isn't part of the solution set.
Excellent. I've been thinking about having a SSHD keylogger post to slack (or some other log). It's crazy that sshd doesn't have this functionality built-in. It's so important to know what your admins are executing on your machines. Aside from the fact that they might have been compromised, it's just good to know what sort of general administration is being done.
What about logwatch? That can do the same and a lot more. You can set it to mail you daily, which gives you an overview who logged in and how often. With a weekly mail you don't get these details, but it might be good enough.
If you don't want to install a "real" solution like Snoopy Logger that works for all users/shells/edge cases, you could always adapt the bash prompt to write the command out. E.g. I personally use something like this in my .bashrc which logs everything per user, but you could easily adapt this to post to Slack instead:
# Adapted from https://unix.stackexchange.com/questions/207813/how-to-log-every-command-typed-into-bash-and-every-file-operation
export ETERNAL_AUDIT_LOGFILE=~/.bash_eternal_auditlog
PROMPT_COMMAND='RET_VAL=$?; history -a; echo "$(who am i | sed -e "s/[[:space:]]\+/ /g") [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//" ) [$RET_VAL]" >> $ETERNAL_AUDIT_LOGFILE'
I've looked into both Snoopy and a bash script as you said. They both have merits, but I think for it to be really reliable there's no better way than to just log keyboard input. Assuming there are malicious users for me is a big part of the motivation.
If you are interested in a commercial solution in this space, check out ScaleFT. Besides the dynamic SSH certificates, we also track both SSH keys used and access events, which you can then pull via an API across all your machines. We also have a Slack notifier, though It needs a little love & cleanup.
Eh, the sysadmins would be monitoring eachother. I'm not saying they should have a manager that keeps their commands in check.
And yes a change management process is very nice and all, and I suppose that at Amazon no line is entered into a root sshd shell without each character being vetted thrice, but at your regular shop you can bet that there's loads of admins that type "ps aux" three times before getting it right. Not that that's terrible, but if you want to look at system administration as an engineering problem you have to know what's going on.
That's some very sensitive information to put in Slack. If Slack has a security breach or if any of your slack accounts are hacked then an outside party could see this channel.
Ever accidentally typed your sudo password at the wrong prompt?
How can we forget that? Literally any time Slack is mentioned on HN, people come out of the woodwork to let everyone know they don't use it cause it's hosted by someone else.
People are _way_ too trusting of Slack and similar services.
You could run your own IRC or Mattermost server accessible only in your LAN or over VPN and this would be fairly safe. Heck, even SSL-only with cranked up SSL options would be fairly safe.
Instead, people ship all their data off to a 3rd party company, often letting that 3rd party have control over their servers and such. Their customer data, their payment data, even code exec on all their servers is exposed and out of their control.
It's insanity, I can't believe people have so little regard for security. Especially with a 3rd party company with a less-than-great security record.
I mean the next step is to have an automated phone call go out to people (which is what we do for critical alerts).
Short of that, slack is on my desktop, laptop, and phone. If i don't have one of those around me at the time, you aren't getting ahold of me for any reason.
So yeah i think it's perfectly valid for security-critical notifications.
Plus this isn't as security critical as you'd think. I don't want klaxons going off every time someone sshs into a server... This can just be an additional layer of security.
I use IRC, but come on. It offers a lot more than IRC right out of the box.
* File uploads
* Embedding portions of links (tweets, images)
* A very good search
* Multi-line posts
* Code-formatting, including multi-line posts, and also snippets.
* A mobile client that alerts you when someone mentions you.
* Scroll-back history when you sign on at any time.
* Syncing between multiple clients.
Yes, you could create a bot or modify an irc server to do this, and then find or write a client that will do all that stuff, and an irc bouncer can fill in for a lot of this.
But Slack does it out of the box. Zero extra work needed.
I like IRC, but if you claim that Slack doesn't offer anything more than IRC, you're either delusional or using an incredibly broad definition of IRC.
I did a similar thing a couple of days ago.
I just added this (with the correct values) in the `sshrc` file inside `/etc/ssh` and enabled a webhook. that's it.
ip=`echo $SSH_CONNECTION | cut -d " " -f 1`
curl -X POST --data-urlencode 'payload={"channel": "#<your channel>", "username": "SSH Login watcher", "text": "User '${USER}' just logged in from '${ip}'", "icon_emoji": ":robot_face:"}' https://hooks.slack.com/services/<rest of the webhook>
70 comments
[ 3.0 ms ] story [ 143 ms ] threadIf you start overriding DND, now the user is going to want super-DND. Which somebody will then want to override, and so on. The correct solution is that your users need to not set DND when they in fact need to be disturbed, and your systems shouldn't be disturbing unnecessarily, and to the extent that's a really hard problem, well, yes, it very much is, but an unboundedly-large hierarchy of "bother that person, no don't bother me, SUPER bother that person, no SUPER don't bother me, SUPER MEGA bother that person" isn't part of the solution set.
This caused more than a few missed announcements and made escalation hard for a bit.
If you are interested in a commercial solution in this space, check out ScaleFT. Besides the dynamic SSH certificates, we also track both SSH keys used and access events, which you can then pull via an API across all your machines. We also have a Slack notifier, though It needs a little love & cleanup.
We're currently in what I could best call a beta: https://www.scaleft.com/
Micromanagement at its finest!
> it's just good to know what sort of general administration is being done.
Your change management process will give you an overview of what your admins are doing.
And yes a change management process is very nice and all, and I suppose that at Amazon no line is entered into a root sshd shell without each character being vetted thrice, but at your regular shop you can bet that there's loads of admins that type "ps aux" three times before getting it right. Not that that's terrible, but if you want to look at system administration as an engineering problem you have to know what's going on.
But that's getting into no-man's land I guess
Ever accidentally typed your sudo password at the wrong prompt?
You could run your own IRC or Mattermost server accessible only in your LAN or over VPN and this would be fairly safe. Heck, even SSL-only with cranked up SSL options would be fairly safe.
Instead, people ship all their data off to a 3rd party company, often letting that 3rd party have control over their servers and such. Their customer data, their payment data, even code exec on all their servers is exposed and out of their control.
It's insanity, I can't believe people have so little regard for security. Especially with a 3rd party company with a less-than-great security record.
https://blog.sucuri.net/2016/01/server-security-integrating-...
I mean the next step is to have an automated phone call go out to people (which is what we do for critical alerts).
Short of that, slack is on my desktop, laptop, and phone. If i don't have one of those around me at the time, you aren't getting ahold of me for any reason.
So yeah i think it's perfectly valid for security-critical notifications. Plus this isn't as security critical as you'd think. I don't want klaxons going off every time someone sshs into a server... This can just be an additional layer of security.
Nobody is selling IRC.
>We're sorry but Grove isn't currently accepting new customers right now.
I guess with that pricing structure they don't need to.
* File uploads
* Embedding portions of links (tweets, images)
* A very good search
* Multi-line posts
* Code-formatting, including multi-line posts, and also snippets.
* A mobile client that alerts you when someone mentions you.
* Scroll-back history when you sign on at any time.
* Syncing between multiple clients.
Yes, you could create a bot or modify an irc server to do this, and then find or write a client that will do all that stuff, and an irc bouncer can fill in for a lot of this.
But Slack does it out of the box. Zero extra work needed.
I like IRC, but if you claim that Slack doesn't offer anything more than IRC, you're either delusional or using an incredibly broad definition of IRC.
Yes.
ip=`echo $SSH_CONNECTION | cut -d " " -f 1`
curl -X POST --data-urlencode 'payload={"channel": "#<your channel>", "username": "SSH Login watcher", "text": "User '${USER}' just logged in from '${ip}'", "icon_emoji": ":robot_face:"}' https://hooks.slack.com/services/<rest of the webhook>
echo "`whoami` logged in at `date` from `echo $SSH_CLIENT`" | mail -s "`hostname` login" youremail@example.com
Note that people can still ssh execute remotely etc.
This post explains how to set it up:
https://blog.sucuri.net/2016/01/server-security-integrating-...
https://github.com/duggan/ansible-slack-notify-ssh