I am not quite sure what "it uses binary SSH" is meant to say. It sounds as if you were refering to the SSH protocol (which of course is "binary"), though my guess is that you mean to say that it relies on executing a separate "ssh binary" (note in particular "ssh" vs. "SSH"). If the latter interpretation is right, perhaps rephrase what you wrote in your README?
Well, having the list is certainly nice, though I don't really see a fair comparison there, more a list of "reasons why I think those other tools suck" ... :-). Some links to those external projects would also be nice. Just saying -- of course the author has probably better things to do than provide a link list to alternative software, but if you go to the trouble of making such a list, that might be handy.
And of course there are more alternatives out there, e.g. it perhaps Fabric <http://www.fabfile.org/> should also be in the list?
Though the reasons listed for what Orgalorg can do but Ansible can't are incorrect. Ansible can accept hosts lists from STDIN (-i '192.168.50.1,192.168.50.2').
By implementing your own ssh, you appear to be ignoring lots of useful default ssh utilities, like ssh-agent and ~/.ssh/config eg.
[derf@pan][09:06:20]-[~] $ orgalorg -o pandora -C uptime
2016-07-25 09:06:28 [FATAL] can't create runner factory
└─ can't read SSH key: '/Users/derf/.ssh/id_rsa'
└─ can't read SSH key from file
└─ open /Users/derf/.ssh/id_rsa: no such file or directory
[derf@pan][09:06:28]-[~] $ orgalorg -p -o pandora -C uptime
Password:
2016-07-25 09:06:36 [WARN] still connecting to address after 2s: [derf@pandora:22]
2016-07-25 09:06:44 [FATAL] acquiring global cluster lock failed
└─ connection to 1 of 1 nodes failed
└─ can't connect to nodes
└─ can't connect to address: [derf@pandora:22]
└─ dial tcp 18.60.0.124:22: i/o timeout
[derf@pan][09:06:45]-[~] $ ssh pandora uptime
09:06:49 up 37 days, 23:45, 0 users, load average: 1.18, 0.70, 0.33
Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS? What about ssh proxy hosts? The fact that you require a single ssh key to live in .ssh/id_rsa is going to make this useless for anyone that does any advanced usage of SSH.
That's a shame. That does immediately rule out my use case of using it at work for working with multiple servers. I have all of our servers in ~/.ssh/config (populated from a script), which use a number of different ssh keys. I assumed the host names in the orgalorg examples would work with the ones defined in ~/.ssh/config
Parsing `ssh_config` is not implemented right now as well as multiple SSH keys for auth. However, it can be implemented easy.
BTW, I have not implemented it, because I think it's very bad approach in maintaining clusters, e.g. storing host names inside local ~/.ssh/config file. Hosts should have DNS records.
yeap, but I have a nice .ssh/config that does wildcard masking , and a few other things to make sure I can log into servers without remembering which SSH Key I need to use, and what username.
e.g. I use the following snippet for my dev build boxes - they are rebuilt all the time, but I need to login and check system state from time to time.
Host 10.250.60.*
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
IdentityFile ~/.ssh/local-dev-key
User build
If you are implementing multiple SSH Keys, remember that a -lot- few people will have keys on GPG Cards, YubiKeys and other devices, not just on disk.
> By implementing your own ssh, you appear to be ignoring lots of useful default ssh utilities, like ssh-agent
What other utilities I'm ignoring besides of `ssh-agent`? `ssh-agent` support is a pending feature and implementing it is quite simple. I will be glad to see PR.
> Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS?
Yes, you should to add your hosts to DNS. Otherwise, it's not production grade (we're talking about production clusters, aren't we?).
SSH config supports lots of esoteric options, like setting non-standards ports, usernames and specialized routing (using Proxycommand options).
Three issues, for me, with adding hosts to DNS:
1) I work from several networks - work, home, tethered. I only control the DNS zone that is set up as my default DNS search domain on one of those networks.
2) I work on hosts in several external networks - they shouldn't all be in the same DNS search domain.
3) I work primarily with cloud hosted, ephemeral servers. Cattle, not pets. Right now I have a tool that adds hosts from my EC2 accounts to my ssh config file (https://github.com/fredsmith/aws-ssh-config.)
I also have solved this problem using 21 lines of bash in my bashrc. (https://github.com/fredsmith/dotfiles/blob/master/bash/dsh) Mine doesn't have quite the level of polish of yours, but since it uses the system ssh binary, it handles special ssh configurations and seamlessly uses ssh-agent. I've used this on hundreds of hosts simultaneously, and it executes in parallel very quickly.
Have to agree with parent, supporting ~/.ssh/config is an absolute requirement for any tool that wishes to "reimplement" SSH. I routinely use custom user names, ports and proxycommands, via wildcard host names.
We built something similar to this a while back: https://github.com/Mojang/Sift, it's great to see more alternatives popping up, seems like a great tool :)
It's always good to see people challenging existing solutions .. this is how innovation happens. But you need to mention other good alternatives like fabric, paramiko, etc to list of alternatives... and plain old shell scripting.
Some of the other options in orgalog seem like a combination of rsync, autossh (which has a heartbeat protocol built in), xargs, and plain old bash might be a better option.
For ad hoc jobs like this, this seems like how I'd go about synchronizing local and remote directories on multiple servers in parallel:
for host in host1 host2 host3
do
rsync -rpae ssh /my/directory/ $host:/remote/directory/ &
done
wait
That will synchronize a local directory with a remote directory, in parallel, and it's readable and uses software you probably already have installed.
Or, if you want to limit parallelization to no more than 16 boxes simultaneously, use in combination with xargs (ie something like xargs --max-procs 16 -n 1 and call a short bash function with the given hostname). Orgalog doesn't seem to support limiting to n hosts at a time?
Here's how to run the example uptime on hosts in parallel, but with batch limits:
orgalorg -o <host>... -p -i ~/.ssh/id_rsa.pub -C tee -a ~/.ssh/authorized_keys
bash:
echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{} scp ~/.ssh/authorized_keys {}
# or, Userify cloud or On Site for >1 users
Of course, I'd just use Userify[1] in a local installation for that (disclaimer: CTO) but you could just modify the rsync example above or just use my old cloudadmins script[2] if your needs are small.
This was obviously a big effort just looking at the dozens of go files! I don't want to disparage what was obviously a huge amount of work from someone who spent a lot of time. A cautionary note for anyone thinking about this: always focus on automating non-security-sensitive tasks first and try to use the smallest amount of magic necessary to do a trick. Rolling your own crypto is never a good idea.
Clearly the OP is trying a lot of new ideas and we should expect good things in the future.
But note, that your solution lacks: valid termination of commands by `CTRL-C`, aggregated progress indication, password authentication, various fail modes (like try to connect to all servers first), elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root), no integration into other unix-way tools.
Also, you will little bit stuck trying to run `tail -f` via xargs ssh.
So, that's the reason for writing production-grade tool.
It's clear that you've put a lot of thought into this!
If I was going for production grade as opposed to a quick ad hoc job, I'd probably automate with a script to handle failures and send logs somewhere (thus wouldn't need or want control-C or progress indicators)
----
* valid termination of commands by `CTRL-C`
Good point. I'd probably use pkill for that, if my ad hoc job went haywire :)
----
* aggregated progress indication
You are correct, this is missing in my solution (nicely aggregated, anyway) -- but again if I was going to do this in production (i.e., automated), I'd probably not be doing it like this anyway.. I'd write a bash script, check status codes, timeouts, etc.
For example, delivering sorted output via log files:
for host in host1 host2 host3
do
rsync -rpae ssh /my/directory/ $host:/remote/directory/ > rsync-$host.log 2>rsync-host.err &
done
and to syslog (hopefully a remote loghost)
for host in host1 host2 host3
do
rsync -rpae ssh /my/directory/ $host:/remote/directory/ | logger -t "myscript-$host" 2>&1 &
done
(note: --tag parameters for logger vary between RHEL and Debian-based distributions.. see the man page for specifics.)
----
* password authentication
Please never do that. We don't even provide that as an option at Userify (https://userify.com, SSH key management). SSH is one of the few places where passwords are completely obsolete.
----
* various fail modes (like try to connect to all servers first)
That seems like a simple thing to accomplish; are you saying orgalog connects to all servers first before connecting to all servers again?
----
* elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root)
If that doesn't work for you, you're probably using a version of red hat with a broken sudoers file (which goes back nearly ten years!) but they're fixing it in the next release:
I don't quite understand what you are saying.. I'd suggest that it's completely integrated into other unix tools, because it uses them instead of inventing a monolithic framework.
> That seems like a simple thing to accomplish; are you saying orgalog connects to all servers first before connecting to all servers again?
orgalorg has two modes: either it will fail and do nothing if even one server fails; or it will skip all errors and continue to do whatever it can with the rest. It allows you to run orgalorg on all set of nodes, but specify different keys and access only subset of servers.
> Please never do that [...] SSH is one of the few places where passwords are completely obsolete.
You simply making declarative statement there without any underlying arguments. Please never do that.
We have passwords as part of our computing infrastructures right now and there are lot of cases where you need to use passwords (e.g., legacy infrastructure). So, supporting valid way of vastly used authentication doesn't sound 'bad' for me.
>> * elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root)
> Sure, that's easy..
I'm talking about rsync. Try to use non-root login whily syncing root-owned files. Orgalorg solves that trivially (just add `-x` flag and it's done).
Of course, I'm aware that key-based authentication is way superior than password based. I'm trying using SSH keys whenever possible and have distributed solution for syncing them across all cluster (https://github.com/reconquest/shadowd). I do not find GUI-based solutions to be productive in work, so I prefer to use dead simple tools which are fail-safe because of simplicity.
Finally,
> but again if I was going to do this in production (i.e., automated), I'd probably not be doing it like this anyway.. I'd write a bash script, check status codes, timeouts, etc.
>>>> * elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root) > Sure, that's easy..
>> I'm talking about rsync. Try to use non-root login whily syncing root-owned files. Orgalorg solves that trivially (just add `-x` flag and it's done).
Fair enough! FWIW, that's actually trivial with native rsync also:
rsync --rsync-path="sudo rsync" (...)
That calls rsync as 'sudo rsync' on the remote system.
* GNU Parallel normally uses 'ssh' and recommends using 'ssh-agent'.
* With 'sshpass' you only have to type the password once.
* GNU Parallel does not have progress indication for the file transfer.
* GNU Parallel uses /./ to indicate which part of the path should be copied to the remote side: foo/bar/./sub/dir/file will append sub/dir/file to --workdir on the remote side.
First of all, congratulations on getting this to a point where you can show it off. A lot of hard work clearly went into this, and it's can't be an easy decision to unleash the hordes of HN programmers knowing the inevitable "constructive criticism" that results. I think a lot of the people who ask "why not just use X" might seem like they miss the point of writing a project to do something you need done just for the fun of it, but I'm an optimist so I assume they're just interested.
I'm an HPC sysadmin, so I use tools like this (specifically pdsh+pdcp) every day. Automation is great and we use it for just about everything, but the nature of the beast is that there's a lot of ad-hoc work and testing that needs to be done.
Specifically with respect to pdsh, just because it appears to have been abandoned, that doesn't necessarily make it the wrong tool for the job. It's pretty-well feature-complete for everything I use it for (that said, it was designed precisely for my needs, which likely aren't yours).
Some minor practical features that would make it a worthy replacement for the venerable pdsh:
* range selection in the hostname selection going beyond bash expansion, like
-o compute[0012-0034,1123,1156-1199]
* taking hostnames from a file is a must
* supporting libgenders would be amazing
Nice work! I'll try it out and file github issues.
56 comments
[ 2.8 ms ] story [ 120 ms ] threadhttps://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
Can you give me an example of such invokation? Layman's `parallel -S host1,host2 hosname` didn't work.
P.S: not a big fan of the name :p
For example:
rex -H "frontend[01..15]" -e "say run 'uptime'"
Also +1 for listing some alternatives and comparisons in the README, so that we don't have to bitch "but you should use $X instead" here :-)
And of course there are more alternatives out there, e.g. it perhaps Fabric <http://www.fabfile.org/> should also be in the list?
BTW, I have not implemented it, because I think it's very bad approach in maintaining clusters, e.g. storing host names inside local ~/.ssh/config file. Hosts should have DNS records.
e.g. I use the following snippet for my dev build boxes - they are rebuilt all the time, but I need to login and check system state from time to time.
If you are implementing multiple SSH Keys, remember that a -lot- few people will have keys on GPG Cards, YubiKeys and other devices, not just on disk.What other utilities I'm ignoring besides of `ssh-agent`? `ssh-agent` support is a pending feature and implementing it is quite simple. I will be glad to see PR.
> Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS?
Yes, you should to add your hosts to DNS. Otherwise, it's not production grade (we're talking about production clusters, aren't we?).
Three issues, for me, with adding hosts to DNS:
1) I work from several networks - work, home, tethered. I only control the DNS zone that is set up as my default DNS search domain on one of those networks.
2) I work on hosts in several external networks - they shouldn't all be in the same DNS search domain.
3) I work primarily with cloud hosted, ephemeral servers. Cattle, not pets. Right now I have a tool that adds hosts from my EC2 accounts to my ssh config file (https://github.com/fredsmith/aws-ssh-config.)
I also have solved this problem using 21 lines of bash in my bashrc. (https://github.com/fredsmith/dotfiles/blob/master/bash/dsh) Mine doesn't have quite the level of polish of yours, but since it uses the system ssh binary, it handles special ssh configurations and seamlessly uses ssh-agent. I've used this on hundreds of hosts simultaneously, and it executes in parallel very quickly.
https://pypi.python.org/pypi/sshpt
"SSH Power Tool, Run commands and copy files to multiple servers simultaneously WITHOUT requiring pre-shared authentication keys"
These days I figured everyone that wanted to do such things would use a tool like Ansible.
DisclaimerAsImAmericanAndIDisclaimEverything: I'm not author. Not even user. I just know it exists.
https://github.com/square/mssh
This tool could be a nice thing to tinker and experiment with a cluster setup, though.
Some of the other options in orgalog seem like a combination of rsync, autossh (which has a heartbeat protocol built in), xargs, and plain old bash might be a better option.
For ad hoc jobs like this, this seems like how I'd go about synchronizing local and remote directories on multiple servers in parallel:
That will synchronize a local directory with a remote directory, in parallel, and it's readable and uses software you probably already have installed.Or, if you want to limit parallelization to no more than 16 boxes simultaneously, use in combination with xargs (ie something like xargs --max-procs 16 -n 1 and call a short bash function with the given hostname). Orgalog doesn't seem to support limiting to n hosts at a time?
Here's how to run the example uptime on hosts in parallel, but with batch limits:
orgalog:
bash: Another orgalog example: bash: Of course, I'd just use Userify[1] in a local installation for that (disclaimer: CTO) but you could just modify the rsync example above or just use my old cloudadmins script[2] if your needs are small.This was obviously a big effort just looking at the dozens of go files! I don't want to disparage what was obviously a huge amount of work from someone who spent a lot of time. A cautionary note for anyone thinking about this: always focus on automating non-security-sensitive tasks first and try to use the smallest amount of magic necessary to do a trick. Rolling your own crypto is never a good idea.
Clearly the OP is trying a lot of new ideas and we should expect good things in the future.
1. https://userify.com
2. https://github.com/jamiesonbecker/cloudadmins
But note, that your solution lacks: valid termination of commands by `CTRL-C`, aggregated progress indication, password authentication, various fail modes (like try to connect to all servers first), elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root), no integration into other unix-way tools.
Also, you will little bit stuck trying to run `tail -f` via xargs ssh.
So, that's the reason for writing production-grade tool.
If I was going for production grade as opposed to a quick ad hoc job, I'd probably automate with a script to handle failures and send logs somewhere (thus wouldn't need or want control-C or progress indicators)
----
* valid termination of commands by `CTRL-C`
Good point. I'd probably use pkill for that, if my ad hoc job went haywire :)
----
* aggregated progress indication
You are correct, this is missing in my solution (nicely aggregated, anyway) -- but again if I was going to do this in production (i.e., automated), I'd probably not be doing it like this anyway.. I'd write a bash script, check status codes, timeouts, etc.
For example, delivering sorted output via log files:
and to syslog (hopefully a remote loghost) (note: --tag parameters for logger vary between RHEL and Debian-based distributions.. see the man page for specifics.)----
* password authentication
Please never do that. We don't even provide that as an option at Userify (https://userify.com, SSH key management). SSH is one of the few places where passwords are completely obsolete.
----
* various fail modes (like try to connect to all servers first)
That seems like a simple thing to accomplish; are you saying orgalog connects to all servers first before connecting to all servers again?
----
* elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root)
Sure, that's easy..
echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{} ssh {} sudo uptime
If that doesn't work for you, you're probably using a version of red hat with a broken sudoers file (which goes back nearly ten years!) but they're fixing it in the next release:
https://bugzilla.redhat.com/show_bug.cgi?id=1020147
In the meantime, just comment out the Defaults requiretty line to get the same fix.
Also, for the tty-requiring solutions that you mention, SSH has a command-line switch already: ssh -t and/or ssh -tt. `man 1 ssh` for details.
Userify configures sudo on remote systems, btw. (SSH key management, https://userify.com, blatant plug: CTO)
----
* no integration into other unix-way tools
I don't quite understand what you are saying.. I'd suggest that it's completely integrated into other unix tools, because it uses them instead of inventing a monolithic framework.
orgalorg has two modes: either it will fail and do nothing if even one server fails; or it will skip all errors and continue to do whatever it can with the rest. It allows you to run orgalorg on all set of nodes, but specify different keys and access only subset of servers.
> Please never do that [...] SSH is one of the few places where passwords are completely obsolete.
You simply making declarative statement there without any underlying arguments. Please never do that.
We have passwords as part of our computing infrastructures right now and there are lot of cases where you need to use passwords (e.g., legacy infrastructure). So, supporting valid way of vastly used authentication doesn't sound 'bad' for me.
>> * elevating privileges from user account to sudo (which is must have when you're logging in using your username, not root) > Sure, that's easy..
I'm talking about rsync. Try to use non-root login whily syncing root-owned files. Orgalorg solves that trivially (just add `-x` flag and it's done).
Of course, I'm aware that key-based authentication is way superior than password based. I'm trying using SSH keys whenever possible and have distributed solution for syncing them across all cluster (https://github.com/reconquest/shadowd). I do not find GUI-based solutions to be productive in work, so I prefer to use dead simple tools which are fail-safe because of simplicity.
Finally,
> but again if I was going to do this in production (i.e., automated), I'd probably not be doing it like this anyway.. I'd write a bash script, check status codes, timeouts, etc.
... and you will write orgalorg in bash.
>> I'm talking about rsync. Try to use non-root login whily syncing root-owned files. Orgalorg solves that trivially (just add `-x` flag and it's done).
Fair enough! FWIW, that's actually trivial with native rsync also:
That calls rsync as 'sudo rsync' on the remote system.To make the emulation easier, make a simple alias:
If you want to supply a password run: or set the password directly: If the above is set up you can then do: There are a few differences:* GNU Parallel normally uses 'ssh' and recommends using 'ssh-agent'.
* With 'sshpass' you only have to type the password once.
* GNU Parallel does not have progress indication for the file transfer.
* GNU Parallel uses /./ to indicate which part of the path should be copied to the remote side: foo/bar/./sub/dir/file will append sub/dir/file to --workdir on the remote side.
I'm an HPC sysadmin, so I use tools like this (specifically pdsh+pdcp) every day. Automation is great and we use it for just about everything, but the nature of the beast is that there's a lot of ad-hoc work and testing that needs to be done.
Specifically with respect to pdsh, just because it appears to have been abandoned, that doesn't necessarily make it the wrong tool for the job. It's pretty-well feature-complete for everything I use it for (that said, it was designed precisely for my needs, which likely aren't yours).
Some minor practical features that would make it a worthy replacement for the venerable pdsh:
Nice work! I'll try it out and file github issues.[1] I'll guess that the current home is hard to find, as a google search finds the old sourceforge.net or googlecode pages.