So happy to see this on HN! Actually I took most of the tips from unix threads here and r/commandline, I highly recommend that subreddit if you like this kind of tricks!
NFS basically breaks the client computer until the host responds, while SMB detects the I/O error earlier and, at least, doesn't hang the terminal.
Other than that, SMB allows for permissions and a bunch of other features. It is, basically, a more modern and robust protocol. Does NFS shine for some use cases? Indeed. Would it be the first choice for most ones? Nope.
Disclaimer: my sysadmin skills are not so great, I'm talking as a user
Soft mounts report errors immediately, hard mounts hang.
And for permissions, NFS provides everything under the sun that you could possibly need via ACLs. NFSv4 is a very modern protocol, much as SMBv2 is (not SMB though, it's awful).
Linux has had NFSv4 for what, 6 years now, at least, and even v2 and v3 had some limited ACL support?
That's nice to know, it seems that we have the "hard" configuration at the lab and it's a real pain. Backwards compatibility, you know. For my mini-cluster we use SMB and couldn't be happier.
If you use any NAS device with a weak CPU and you have a choice of CIFS or NFS, the NFS transfer rates are normally 25% faster and the load is much less on the server.
I third this. I was attempting to set up my Raspberry Pi for sharing a big HDD that was plugged into it on my home network, and the NFS mounts appear to weigh lighter on my vanilla Raspbian setup.
And fwiw, if there's some sort of error with the mount, the terminal lets me know right away.
I believe NFS is a less chatty protocol with fewer roundtrips. SMB is not a particularly well-designed protocol overall.
My only evidence is anecdotal. Back in ~1998 we had Samba running on our small Linux file server using Windows NT desktops via 10mbit Ethernet. It was dog slow, not just for browsing but on sequential things like file transfer. We installed NFS mounts on the same box, and it turned out to be lightning fast. I don't remember just how fast, but it made everyone go "wow".
Nowadays, with NFS over fast 100mbps or gigabit Ethernet the latency difference probably is not significant enough to make a difference. I prefer NFS just because its more Unixy.
1) The caveats of NFS can be crippling if you are a rubbish sysadmin. NFS requires a more thorough understanding than what you can get from a tip sheet.
2) Samba is single threaded, performance will suffer serving SMB from a Linux machine. For this reason it would be better to serve SMB from a Windows machine.
3) Using a foreign protocol between homogenous computers when the native protocol will do is non-ideal. NFS is the right thing when sharing filesystems from Linux to Linux.
If you put that in, for example, `~/.vim/ftplugin/markdown.vim`, it will be used on all markdown files. (`../ftdetect/markdown.vim` controls how Vim determines that a file is markdown.)
Using ssh-agent also means you can practically put a very large pass phrase on your SSH key, because you'll only type it infrequently. Good luck brute forcing my passphrase.
I'm pretty sure ssh-agent doesn't store the password, but the private key. Also, the fact that it supports timed expire (and can be setup to drop keys upon events such as screen lock) make it a wiser choice than passwordless keys.
That's correct. And ssh-agent doesn't give access to the private key either, only to perform operations like signing. The only way to extract the key is to search it in the process memory, which I believe would requires root level access.
re: 4: not only is an ssh agent by far safer, but most agents now allow you to set a timeout on a key, so it's not indefinitely saved in memory.
A passwordless key gives anyone with acces to that file, access to the login associated with it. If that file is inadvertently exposed (oops, checked it into github...), any machine you have a login on must be considered compromised.
ssh-agent and 'ssh -A' is also useful if you have to login to one machine to access another, without having to copy your private key to the first machine.
For example if you login remotely to a machine, and want to access a git repository on another:
I use agent forwarding often, but you still need to be careful, especially if you forward your agent to a machine not under your control. From the ssh man page:
Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.
Consider using a dedicated key for each of those circumstances, set sane defaults in your ~/.ssh/config on all machines, and be very careful about what ends up in any of your ~/.ssh/known_hosts files, as they provide a road map to other destinations.
The ssh_config HashKnownHosts option hashes the contents of the known_hosts file, making it intractable to get a list of hosts. But of course your shell history will still provide it.
3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?
The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff.
I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled it easily
I'm not sure. I haven't closely studied the difference between each algorithm. My guess would be that sort -u would perform better as the data set gets larger with a good block size setting because it does do an external sort. Cardinality would also affect the performance. If the unique set handily fits in memory, an external sort on a large data set wouldn't be very efficient.
Yes it will, and a bit (read: a lot) more than 10GB as it needs to store the contents of variable x in a hash table (with the corresponding hash key and value of the counter). There's no other magic way it can 'know' whether a particular line has been seen before. You can't rely on hash keys alone as the hashes aren't guaranteed to be unique.
For files with relatively few duplicates it's going to be a lot slower than sort | uniq.
Trying it on a 128MB file (nowhere near enough time to test a 10GB file) filled with lines of 7 random upper case characters[1] (so hardly any duplicates):-
$ wc -l x.out
16777216 x.out
$ time ( sort x.out | uniq ) | wc -l
16759719
real 0m17.982s
user 0m42.575s
sys 0m0.876s
$ time ( sort -u x.out ) | wc -l
16759719
real 0m20.582s
user 0m43.775s
sys 0m0.688s
Not much difference between "sort | uniq" and "sort -u".
As for the awk method:-
$ time awk '!x[$0]++' x.out | wc -l
has been running for more than 20 minutes and still hasn't returned. For that 128MB file the awk process is also using 650MB of memory (according to ps). Will check up on it later (have to go out now).
This Linux machine has ~16GB of memory so the file was going to be completed cached in memory before the first test. All things considered equal the awk method will be roughly O(n) (e.g. linear against file size) and sort/uniq will be O(n log n). So, theoretically, the awk method will eventually surpass the sort method because it's having to do less work (it's only checking for a previously seen key rather than sorting the entire file) but I'm not sure the crossover will be anywhere useful if the file doesn't contain many duplicates.
Repeating it for a file containing lots of duplicates (same 128MB file size but contents are only the 7 letter words consisting of A or B, so only 128 possible entries):-
$ time awk '!x[$0]++' y.out | wc -l
128
real 0m1.207s
user 0m1.192s
sys 0m0.016s
$ time ( sort y.out | uniq ) | wc -l
128
real 0m14.320s
user 0m31.414s
sys 0m0.428s
$ time ( sort -u y.out | uniq ) | wc -l
128
real 0m12.638s
user 0m30.366s
sys 0m0.188s
Notice that "sort -u" doesn't do anything clever for files with lots of duplicates.
So awk is much faster for files with lots of duplicates. No great surprises. When I get a chance I'll repeat it for a 1GB file and a 10GB file (with lots of duplicates otherwise the awk version will take far too long).
Yes it will ... [need] to store the contents of variable x in a hash table [...] There's no other magic way it can 'know' whether a particular line has been seen before. You can't rely on hash keys alone as the hashes aren't guaranteed to be unique.
Technically that's true, but the result of a cryptographic hash like SHA-256 is (practically) guaranteed to be unique. Depending on the average length of an input line and how many of the lines are unique, storing only the SHA-256 hash value could take far less memory than storing the input lines along with a non-cryptographic 32-bit hash value.
You are describing a bad implementation of a bloom filter [1]. Anyway, people expect "uniq" to be correct in all cases (i.e., to never filter a unique line). A default implementation where it would possible (even with a minuscule chance) that this doesn't happen would be a recipe for disaster. It may be a cool option though ;)
No, this is not a bad implementation of a bloom filter, and the size of the minuscule chance matters; unless SHA-256 has a flaw in it that we don't know about, SHA-256 collisions are far less likely than undetected hardware errors in your computer. The universe contains roughly 2²⁶⁵ protons, 500 protons per distinct SHA-256 value, and has existed for roughly 2⁵⁸ seconds, which means there are roughly 2¹⁹⁸ SHA-256 values per second of the age of the universe.
Typical undetected bit error rates on hard disks are one error per 10¹⁴ bits, which is about 2⁴⁷. If your lines are about 64 characters long, you'll have an undetected bit error roughly every 2^(47-6-3) = 2³⁸ lines. SHA-256 will give you an undetected hash collision roughly every 2²⁵⁵ lines. That is, for every 2²¹⁷ disk errors, SHA-256 will introduce an additional error. If you're hashing a billion lines a second (2³⁰) then that will be 2^(217-30) = 2¹⁸⁷ seconds, while the disk is giving you an undetected bit error every minute or so. A year is about 2²⁵ seconds, so that's about 2¹⁶² years, about 10⁴⁹. By comparison, stars will cease to form in about 10¹⁴ years, all planets will be flung from their orbits around the burned-out remnants of stars by random gravitational perturbations in about 10¹⁵ years, the stellar remnants will cease to form cold, dark galaxies in about 10²⁰ years, and all protons will have decayed in about 10⁴⁰ years.
And if you somehow manage to keep running uniq on your very large file at a billion lines a second, in a mere 500 times the amount of time from the universe's birth to the time when nothing is left of matter but black holes, SHA-256 will have produced your first random false collision.
Another possibly relevant note: there are about 2¹⁴⁹ Planck times per second. None of the above takes into account the Landauer and related quantum-mechanical limits to computation, which may be more stringent.
Thanks for the suggestion. I used to have a very customized .bashrc with nice little things like that, but have decided to stick with standard stuff for things that work off of muscle memory.
I got tired of sshing into a new box and half the things I'd type wouldn't work properly until I remembered to copy my settings files over, which seemed more trouble than it was worth for a short-lived s3 instance.
That's why I was excited to discover ctrl-r. It's a built-in method of searching history that I can remember and it'll work everywhere.
I've got a public repo of my dotfiles, so the first thing I typically do is "git clone git@github.com:pavellishin/dotfiles.git && cd dotfiles && ./install.sh"
After that, I launch tmux, and it's all hunky dory.
Heh.. right after I posted that comment, my thought was: 'you know - the correct answer here would have been to make an Uber command that would suck all the configs in and install them'.
Thanks for giving me the push to do so. I think I'll take your suggestion but put the command on a site somewhere so I can simply 'curl https://foo.bar/configs | bash'
If you are using 'set -o vi', then you can just use 'ESC' -> '/<somePartOfCommand>' then use 'n' to iterate in reverse order through the commands that match the vi regex. really powerful and useful.
Just to note, splits in tmux and screen behave differently. iirc, in screen you have a set of splits and fill them in with different ports (kind of how vim thinks of viewports). so technically you can have the same viewport open twice on your monitor and the other will mirror the workings of the one that your are working in right now. in tmux each 'tab'/window is a set of splits, which act more like sub-windows, and don't really share across multiple spaces.
An arguably better (and slightly more portable) way to accomplish this is the special variable $_ , which expands to the last argument of the previous command. Since $_ is a variable rather than a history substitution, it still works when there is no command history (e.g. in scripts) and allows all the usual variable expansion forms, for example ${_##*/} to extract the last path component.
That's good to know, although !$ is easier to type since you only need to depress the right shift key, whereas yours requires a quick shift on the opposite side. Makes a big difference when you're trying to quickly type "rm -rf !$" as root. :)
If you are on OS X and use Terminal.app “Alt-.” won’t work because Alt is used for alternate characters. You have two options: enable “use option as meta” in the app settings (but you lose the extra characters) or use “Esc-.” instead.
I use variations on "find" so often that I've created several little commands "fij" (find in any java source), "fit" (find in any text / org file), "fix" (find in any XML file) etc. which I use all the time
Bash and zsh support a surprising amount of emacs editing functionality. Cursor navigation (M-b, M-f, C-a, C-e), text selection (C-space), copy/paste (C-w, M-w, C-y, M-y), and undo (C-_).
If you happen to have a malicious directory named (without double quotes):
".;sudo rm -rf /"
You'd be stuffed.
It's better practice to use the '-print0' flag of find(1) and
pipe the result into xargs(1) with the '-0' flag set. For safety,
it's best to also use '-r #' for expected number of arguments,
'-n' to stop xargs from running without arguments, and "-J %" so
you can use quoting.
EDIT: The reason being the shell is never involved in this process, and the shell is what is responsible for splitting commands on semicolons/newlines.
You are assuming the exact versions of the shell and
find programs that you use are the only ones that exist. It
may not be a problem on your exact system, but it can
be a problem elsewhere.
Sorry, I didn't see your 'EDIT' caveat when I responded --now that will
teach me to reply too soon. ;-)
Also, it seems I failed to be clear; I'm probably too tired I suppose.
My point was there is plenty of ancient and buggy code out there.
It could be "most", or even "many" modern unix variants have fixed a
lot of the old bugs in find(1), but if you don't have the luxury of
working on a current system, and your not allowed to upgrade it, then
plenty bad things can happen due to invoking a shell, handling space,
quote, and delimiter characters, and so forth.
reference:
$ uname -a
OpenBSD alien.foo.test 5.1 GENERIC.MP#207 amd64
setup:
$ mkdir test
$ cd test
$ touch file1
$ touch file2
$ touch file3
$ mkdir ';ls'
bad:
$ find . -type d -exec sh -c {} \;
sh: ./: cannot execute - Is a directory
;ls file1 file2 file3 test.sh
better:
$ find . -type d -exec sh -ec {} \;
sh: ./: cannot execute - Is a directory
also bad:
$ find . -type d -print0 | xargs -0 -r -n 1 -J % sh -c "%"
sh: ./: cannot execute - Is a directory
;ls file1 file2 file3 test.sh
better:
$ find . -type d -print0 | xargs -0 -r -n 1 -x -J % sh -ec "%"
sh: ./: cannot execute - Is a directory
better;
$ find . -type d -print0 | xargs -0 -r -J % sh -c "%"
best:
$ find . -type d -print0 | xargs -0 -r -J % sh -ec "%"
POSIX is all great and wonderful in theory, but in practice it's no
different than the bogus Java "write once, run anywhere" claim. If a
system or utility claims to be POSIX compliant, then you're probably
close, but you'll still need to do testing and debugging.
At least some of the issues with find/xargs are mentioned in the
following wikipedia article. It's probably more clear than I am right
now.
The contrived examples you've shown aren't examples of POSIX-incompatibility, or bugs in `find` at all. You've explicitly involved the shell. Of course trying to run every directory name as a shell command string is going to result in executed code!
Your original argument was that given:
find . -type d -exec chmod g+x {} ';'
It is possible to force code execution of arbitrary commands given a carefully crafted directory name. The key difference in this case is that the shell is not involved _at all_. I challenge you to find an implementation of `find` that is broken in this way.
As a side note, it is even possible to involve the shell in the picture in a safe way with `find`, without the use of `xargs` (and thus avoid the overhead of setting up a pipeline):
find . -type d -exec sh -c 'chmod g+x "$1"' _ {} ';'
(my contrived example is quite poor, though, since it does nothing but introduce unnecessary shell overhead)
Modern (POSIX > 2001?) `find`'s support `-exec {} +` which further reduces the number of reasons to invoke `xargs`:
find . -type d -exec sh -c '
for x; do
do_foo "$x"
do_bar "$x"
do_baz "$x"
done
' _ {} +
(example above shows how to make proper use of this feature with an explicit shell invocation)
It's not a problem with GNU or OpenBSD `find`, and I'm pretty sure that's the case for FreeBSD too. What version of find uses system(3) instead of execv(3)?
Unfortunately OSX uses a badly outdated openssh version just like many other unix tools and it doesn't support the -W option. You could try upgrading openssh using homebrew if you want.
Also, I find that if I'm going to copy the data once, I'm often going to copy it twice, or which to get a more up to date version of it at a later time. Rsync clearly wins in these cases.
Finally, from the compress flag on rsync:
Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection.
Tar can transfer more filetypes and attributes than scp can (even using -p option). `scp -p` only transfers mode, mtime and atime; you lose ownership, extended atributes, symlinks, and hardlinks.
You will also get better compression with tar (or rsync), as it is compressing the files directly, and not just the ssh stream (-C is just passed on to ssh).
In particular, scp is mindblowingly slow on lots of small files. I independently rediscovered the tar-pipe trick while sitting there watching scp laboriously copy thousands of 100-bytes so slowly I could count the files as they went by. That should not be possible, even at modem speeds. Fine for moving one file, OK for directories of very large files, not suitable for general usage where you might encounter a significant number of smaller files.
Absolutely. Connection latency hits you the hardest, since each file is sent serially, and requires 2 (or 3 with -p) round trips in the protocol, and this is on top of an ssh tunnel with it's own overhead. I can't remember what my tests showed, but I have this inkling feeling that tar over ssh was far faster than rsync for an initial load, since there's no round trips required, but you lose some of possible rsync benefits, like resume-ability and checksums.
If my first tar attempt fails for some reason, but it made a lot of progress, I switch to rsync. Best of both worlds. This hasn't come up often enough for me to script it.
Add pv (available in many standard repos these days, from http://www.ivarch.com/programs/pv.shtml if not) into the mix and you get a handy progress bar too.
and so forth will result in a throbber as it can't query the pipe for a length.
If you demoggify the first example to:
pv file | nc ...
you get a progress bar on the sending end without manually specifying a size.
Even without a proper % progress bar, the display can be useful as you can at least see the total sent so far (so if you know the approximate final size you can judge completeness in your head) and the current rate (so you can see it is progressing as expected (so you get some indication of a problem such as an unexpectedly slow network connection, other than it taking too long)).
231 comments
[ 3.0 ms ] story [ 240 ms ] threadEdit: also, @climagic: https://twitter.com/climagic
Second edit: I'm enhancing the file with these comments, so if there are any inconsistencies between the txt and a commenter, it's my fault.
More details would be nice.
Other than that, SMB allows for permissions and a bunch of other features. It is, basically, a more modern and robust protocol. Does NFS shine for some use cases? Indeed. Would it be the first choice for most ones? Nope.
Disclaimer: my sysadmin skills are not so great, I'm talking as a user
http://tldp.org/HOWTO/NFS-HOWTO/client.html
Soft mounts report errors immediately, hard mounts hang.
And for permissions, NFS provides everything under the sun that you could possibly need via ACLs. NFSv4 is a very modern protocol, much as SMBv2 is (not SMB though, it's awful).
Linux has had NFSv4 for what, 6 years now, at least, and even v2 and v3 had some limited ACL support?
http://wiki.linux-nfs.org/wiki/index.php/ACLs
It's a mount option on the client, not the server, so maybe you can change it yourself.
http://www.ietf.org/rfc/rfc1094.txt
If you use any NAS device with a weak CPU and you have a choice of CIFS or NFS, the NFS transfer rates are normally 25% faster and the load is much less on the server.
And fwiw, if there's some sort of error with the mount, the terminal lets me know right away.
My only evidence is anecdotal. Back in ~1998 we had Samba running on our small Linux file server using Windows NT desktops via 10mbit Ethernet. It was dog slow, not just for browsing but on sequential things like file transfer. We installed NFS mounts on the same box, and it turned out to be lightning fast. I don't remember just how fast, but it made everyone go "wow".
Nowadays, with NFS over fast 100mbps or gigabit Ethernet the latency difference probably is not significant enough to make a difference. I prefer NFS just because its more Unixy.
2) Samba is single threaded, performance will suffer serving SMB from a Linux machine. For this reason it would be better to serve SMB from a Windows machine.
3) Using a foreign protocol between homogenous computers when the native protocol will do is non-ideal. NFS is the right thing when sharing filesystems from Linux to Linux.
2) htop is a cpu and memory hog -- every time I've used it I noticed it takes 6+% CPU time
3) there's an awk trick to do the `sort | uniq` recommendation that works on 10+GB files (single pass):
4) Passwordless keys are dangerous -- use ssh-agent to save the password of the keys1. I prefer 'psgrep' because it covers 99% of my use cases for pgrep (ps axuf | grep $NAME)
2. htop is very nice, come on! Would not let it running on the background for hours, but it's nicer than top
3. Note taken, thanks!
4. Is ssh-agent really safer than using passwordless keys? Just asking, I'm curious
2. In my experience on Debian (granted this was in 2010), there is a noticeable performance difference between `htop` and `top`.
4. ssh-agent stores the password in memory and is erased on reboot. OTOH If you use a passwordless key file, anyone can use it if they have the key.
A passwordless key gives anyone with acces to that file, access to the login associated with it. If that file is inadvertently exposed (oops, checked it into github...), any machine you have a login on must be considered compromised.
For example if you login remotely to a machine, and want to access a git repository on another:
Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.
Consider using a dedicated key for each of those circumstances, set sane defaults in your ~/.ssh/config on all machines, and be very careful about what ends up in any of your ~/.ssh/known_hosts files, as they provide a road map to other destinations.
I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled it easily
Awk may still be better for uniquification.
For files with relatively few duplicates it's going to be a lot slower than sort | uniq.
Trying it on a 128MB file (nowhere near enough time to test a 10GB file) filled with lines of 7 random upper case characters[1] (so hardly any duplicates):-
Not much difference between "sort | uniq" and "sort -u".As for the awk method:-
has been running for more than 20 minutes and still hasn't returned. For that 128MB file the awk process is also using 650MB of memory (according to ps). Will check up on it later (have to go out now).This Linux machine has ~16GB of memory so the file was going to be completed cached in memory before the first test. All things considered equal the awk method will be roughly O(n) (e.g. linear against file size) and sort/uniq will be O(n log n). So, theoretically, the awk method will eventually surpass the sort method because it's having to do less work (it's only checking for a previously seen key rather than sorting the entire file) but I'm not sure the crossover will be anywhere useful if the file doesn't contain many duplicates.
Repeating it for a file containing lots of duplicates (same 128MB file size but contents are only the 7 letter words consisting of A or B, so only 128 possible entries):-
Notice that "sort -u" doesn't do anything clever for files with lots of duplicates.So awk is much faster for files with lots of duplicates. No great surprises. When I get a chance I'll repeat it for a 1GB file and a 10GB file (with lots of duplicates otherwise the awk version will take far too long).
1. Example contents:-
EPQKHPH DLJCROB WICVGQY MHWTPSR HMPNECN
Technically that's true, but the result of a cryptographic hash like SHA-256 is (practically) guaranteed to be unique. Depending on the average length of an input line and how many of the lines are unique, storing only the SHA-256 hash value could take far less memory than storing the input lines along with a non-cryptographic 32-bit hash value.
http://en.wikipedia.org/wiki/Bloom_filter
Typical undetected bit error rates on hard disks are one error per 10¹⁴ bits, which is about 2⁴⁷. If your lines are about 64 characters long, you'll have an undetected bit error roughly every 2^(47-6-3) = 2³⁸ lines. SHA-256 will give you an undetected hash collision roughly every 2²⁵⁵ lines. That is, for every 2²¹⁷ disk errors, SHA-256 will introduce an additional error. If you're hashing a billion lines a second (2³⁰) then that will be 2^(217-30) = 2¹⁸⁷ seconds, while the disk is giving you an undetected bit error every minute or so. A year is about 2²⁵ seconds, so that's about 2¹⁶² years, about 10⁴⁹. By comparison, stars will cease to form in about 10¹⁴ years, all planets will be flung from their orbits around the burned-out remnants of stars by random gravitational perturbations in about 10¹⁵ years, the stellar remnants will cease to form cold, dark galaxies in about 10²⁰ years, and all protons will have decayed in about 10⁴⁰ years.
And if you somehow manage to keep running uniq on your very large file at a billion lines a second, in a mere 500 times the amount of time from the universe's birth to the time when nothing is left of matter but black holes, SHA-256 will have produced your first random false collision.
Check out this resource!
http://www.rayninfo.co.uk/vimtips.html
Most of my vim tips are on my .vimrc (my dotfiles here https://github.com/carlesfe/dotfiles/blob/master/.vimrc) and on the links on my homepage: http://mmb.pcb.ub.es/~carlesfe/#programming_h3. The .txt that op linked feels a bit out of context :)
To pipe all the contents of your current directory (including dotfiles) to the destination machine.
Greetings from LSI-UPC!
PS: hey, we're neighbors! :)
tar czf - . | ssh destination "cd /remote/dir; tar xz"
Then I learned to love rsync -av. The benefits are many-fold.
I got tired of sshing into a new box and half the things I'd type wouldn't work properly until I remembered to copy my settings files over, which seemed more trouble than it was worth for a short-lived s3 instance.
That's why I was excited to discover ctrl-r. It's a built-in method of searching history that I can remember and it'll work everywhere.
I've got a public repo of my dotfiles, so the first thing I typically do is "git clone git@github.com:pavellishin/dotfiles.git && cd dotfiles && ./install.sh"
After that, I launch tmux, and it's all hunky dory.
Thanks for giving me the push to do so. I think I'll take your suggestion but put the command on a site somewhere so I can simply 'curl https://foo.bar/configs | bash'
or just use tmux
It substitutes the last argument in the previous command into the current one.
For example:
Also, check it:
$ cd /happily/tabbing/out/some/really/deep/path/ooh/dear/maybe/its/java/WAIT_A.file
bash:> 'WAIT-A.file' is not a directory
$ nano $_ (opens file)
Yes, I know about iTerm. I don’t want it.
is something I use a lot - plus xargs sometimes
On Ubuntu/Debian systems, it is packaged as `ack-grep`.
Bash and zsh support a surprising amount of emacs editing functionality. Cursor navigation (M-b, M-f, C-a, C-e), text selection (C-space), copy/paste (C-w, M-w, C-y, M-y), and undo (C-_).
It's better practice to use the '-print0' flag of find(1) and pipe the result into xargs(1) with the '-0' flag set. For safety, it's best to also use '-r #' for expected number of arguments, '-n' to stop xargs from running without arguments, and "-J %" so you can use quoting.
I believe some on implementations '-n' is unnecessary when '-r #' is specified, but on other implementations it's is the maximum.EDIT: thanks for the vim tip on for finding spelling mistakes!
EDIT: The reason being the shell is never involved in this process, and the shell is what is responsible for splitting commands on semicolons/newlines.
FWIW, your `find -print0`/`xargs -0` is not POSIX.
Also, it seems I failed to be clear; I'm probably too tired I suppose.
My point was there is plenty of ancient and buggy code out there. It could be "most", or even "many" modern unix variants have fixed a lot of the old bugs in find(1), but if you don't have the luxury of working on a current system, and your not allowed to upgrade it, then plenty bad things can happen due to invoking a shell, handling space, quote, and delimiter characters, and so forth.
reference:
setup: bad: better: also bad: better: better; best: POSIX is all great and wonderful in theory, but in practice it's no different than the bogus Java "write once, run anywhere" claim. If a system or utility claims to be POSIX compliant, then you're probably close, but you'll still need to do testing and debugging.At least some of the issues with find/xargs are mentioned in the following wikipedia article. It's probably more clear than I am right now.
http://en.wikipedia.org/wiki/Xargs
Your original argument was that given:
It is possible to force code execution of arbitrary commands given a carefully crafted directory name. The key difference in this case is that the shell is not involved _at all_. I challenge you to find an implementation of `find` that is broken in this way.As a side note, it is even possible to involve the shell in the picture in a safe way with `find`, without the use of `xargs` (and thus avoid the overhead of setting up a pipeline):
(my contrived example is quite poor, though, since it does nothing but introduce unnecessary shell overhead)Modern (POSIX > 2001?) `find`'s support `-exec {} +` which further reduces the number of reasons to invoke `xargs`:
(example above shows how to make proper use of this feature with an explicit shell invocation)https://github.com/ggreer/the_silver_searcher
I think you might like it.
To my surprise, this also works with git:
scp won't copy certain file types correctly. Rsync really is better here, and not just because of that.
I'm not aware of any attributes that tar doesn't preserve.
Also, I find that if I'm going to copy the data once, I'm often going to copy it twice, or which to get a more up to date version of it at a later time. Rsync clearly wins in these cases.
Finally, from the compress flag on rsync: Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection.
Remember: Rsync trades CPU and disk i/o (lots of disk i/o) for network bandwidth.
In the pathological case "thousands of tiny files over a fast network" it can easily be orders of magnitude slower than a straight tar.
You will also get better compression with tar (or rsync), as it is compressing the files directly, and not just the ssh stream (-C is just passed on to ssh).
I did the tests years ago, but a quick google found someone who tried to test the various combinations: http://www.spikelab.org/transfer-largedata-scp-tarssh-tarnc-...
On the receiving machine, in its destination directory:
And on the sending machine, from its source directory: netcat varies a bit from distro to distro, so you may need to adjust these command lines a bit to get it to work.If it doesn't know how many bytes there will be, it just gives you a "throbber" (which is better than nothing, though).
If you demoggify the first example to:
you get a progress bar on the sending end without manually specifying a size.Even without a proper % progress bar, the display can be useful as you can at least see the total sent so far (so if you know the approximate final size you can judge completeness in your head) and the current rate (so you can see it is progressing as expected (so you get some indication of a problem such as an unexpectedly slow network connection, other than it taking too long)).