> Suppose you have a file with a name of "aaa\nbbb.c".
(from the article)
To be honest, I wouldn't care about my shell not handling newlines in files, I'd care about my users creating files with newlines in them. Space is natural language, newline is not.
Apparently you have not been bitten by a filename expansion bug when shell scripting. This kind of thing is fairly common with bash and some of the other ones when dealing with files that have spaces/newlines/whatever. It boils down to learning and knowing your tools and how to work around their limitations.
In this case, if you're not an expert with bash, a good rule of thumb would be to pipe your glob-expanded list of junk into perl -nle 'print' (or echo, or... etc) before running something destructive.
On a sidenote, creating a -i file in important directories can help against accidental "rm -rf"s. The -i will be given to rm as switch and rm will now prompt for each file. Of course backups are better but this might just safe your ass that certain time.
Traditionally Unix filenames can contain everything except for / and null (aka \0). This is why the find utility provides a facility for printing null terminated filenames (-print0), when used in conjunction with xargs -0 this is a fool proof way of iterating over files in the shell.
For example, to delete all .o files in the current directory and its subdirectories:
That's fine for /this/ example because find provides that as an option. But you might want to grep for something in many files of varying depths, or run some home-brewed script on them for which find does not provide a built-in. -exec works of course, but spawns grep (or whatever) once per file; using xargs saves considerable time since it passes lots of files to 1 command.
Or, you can use zsh.
rm **/*.o
grep foo **/*.{java,sql,xml}
# Where is that blasted IBM utility?
ls **/*pmt*(x.) # list e(x)ecutable files (.) that have "pmt" in their name.
# Show me all the "Setup" dirs in my project
ls -d **/Setup(/)
Yep. Though I don't think I've ever seen an actual legitimate use of this capability. I mean really, what's the point of an embedded newline...
Actually the Mac uses (or at least "used") a hidden files named "Icon\r" (yes, an embedded carriage return) for its custom folder icons. I'm still not sure that counts as "legitimate".
I had to write a one-liner to deal with that on Mac OS 10.4 -- had to detect if a user had placed anything on their Desktop, which had a custom folder icon because it was actually a cross-filesystem symlink:
As noted above, I've seen some concepts thrown around about a (Linux) kernel patch to disallow filenames which contain unprintable characters, and I approve of them.
I completely agree that the system shouldn't allow him to do this either. Both the graphical and command shells for Linux (GNOME and bash respectively) do not allow files to be created with newlines characters as far as I know.
Not sure why you're being downmodded. On the simplest of engineering levels, shells don't adequately separate data from execution. They also lack data structures beyond strings and arrays.
ipython / iruby can and do make much nicer shells out of the box.
1) Is he implying that launching a perl interpreter is somehow a more lightweight operation than launching ls and having it stat() half a dozen files? Having ls check if a file exists 99% of the time won't even touch the disk. If you're working in that directory, those inodes will probably be cached. Running stat() on a cached inode is hardly more expensive than a system call. Read: cheap. Now perl launching will probably load a few libs and check a few files before even getting to his one-liner.
2) Am I the only one, who in my ~10 years of running unix-like operating systems have never created a filename with a newline char in it? Like even when opening obscure filesystems, never.
3) Does he think unlink() skips the check for the file's existence?
4) If he wanted to overcomplicate things and make sure everything's kosher, why not:
find . -maxdepth 1 -type f -name '*.c' -delete
Whenever I see anything piped to a perl one liner, 4 times out of 5 it's some sort of hack that can be done with standard utils.
It also grinds my gears when people take longer to think of making something more efficient than the amount of CPU time they'll save in their lives of using that 'more efficient' solution. So he spent a minute coming up with this and ten minutes posting about it. Now, will he save roughly 11 minutes of execution time in his life while running this "superior" command (also count that it takes longer to type)? If not, he wasted his time.
From his sig: "Smalltalk/Perl/Unix consulting". When you have a large-enough hammer....
1) stat() is a system call, so one system call is bound to be faster than two.
2) Sneaky users do this; they create files like /dev/shm/...\n-rf\n.. and you'd better know how to delete it correctly.
3) ???
4) -delete is a GNU extension.
1) Just the system call overhead is not much compared to what the system call actually does, so it depends on what you're calling. Two system calls versus one with identically-cached information, sure it's slower. I'm just saying perl will also probably stat() and if not perl, the unlink() system call would perform the same function. When you're modifying files and flushing it to a drive, especially while using the shell interactively, the sys call overhead is completely negligible. All I'm trying to say is claiming the first command as inefficient is groundless; you execute it as a human and once -- it won't feel any slower.
2) If cleaning up after malicious users, sure why not. Doing this in your source tree or during normal usage? I don't know.
3) He mentioned ls stat()ing files as an inefficiency since the shell does it and ls does as well.
4)
FIND(1) BSD General Commands Manual FIND(1)
NAME
find -- walk a file hierarchy
...
-delete
Delete found files and/or directories. Always returns true.
This executes from the current working directory as find recurses
Randal is pointing out that piping the output of ls to the perl interpreter is dangerous and shows the safer way to do what the original poster wanted.
You aren't going to create a file with a newline in the name. The person who wants to totally screw you over is going to create the file with the newline in the name.
I realize that. The example he used was a collection of c files, leading me to believe he was showing a personal collection of source as an example, wrongly or rightly. And I realize you aren't going to be creating files with newlines in them (or leading dashes, or many dots, etc), but once you get into a territory of someone being able to do that, and you're not competing in a CTF, I'd say there are larger issues to deal with.
I can do no better than quote Randal himself again, from the same thread:
"Well, there was a very embarassing error for Apple a few years ago for the 10.3 to 10.4 upgrade as I recall. Apparently, if you had a space in your home disk name, all sorts of files got deleted.
So, the moral is, no matter how rare you think something is, that's NO EXCUSE for not doing the right thing for all possible characters."
I wonder how many files with a name containing \n are in the wild. And if you encounter such a file, does it really matter if you accidentally delete it?
Waisting brain cycles to save CPU cycles (in the wrong places) is shameful, insulting and boring.
It would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)
I've seen some concepts thrown around about a (Linux) kernel patch to disallow filenames which contain unprintable characters. I approve of them.
With regard to why this matters, it's not uncommon for badly written scripts to interpolate filenames directly into commands. Consider this Perl fragment:
And if you encounter such a file, does it really matter if you accidentally delete it?
It matters. Leaving a bug around is disgusting and wasteful especially when you can fix it.
* would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)*
Fixing bugs and writing programs that have as few bugs as possible is very important.
When you find a bug, by all means, go ahead and fix it. (Well, not always. You need to make sure your fix is not likely to introduce a more severe bug. Not all bugs made equal and some guts-feeling required).
If your goal is to make your system as robust as engineerically possible, you should most likely pay more attention in some directions and less in other. It's not only about quantity - you better have no critical bugs then have only-a-very-few critical ones.
The problem is the file you accidentally delete (or whatever) is probably not the one with the weird name. Most bogus filenames I've seen were created by shell scripts that had a set of commands that was accidentally quoted instead of executed, so they tend to contain the names of files which were important to the author.
This article does point out one thing that I've always found really strange/dumb about Unix. Why does the shell expand the wildcards? That's not how it works in Windows, for example.
I periodically have to cleanup abandoned and overflowing unix mailboxes where every message is it's own file. You'd think a simple "rm *" would work, but it doesn't, because you get an error message saying that there are "too many arguments". From my perspective there's one argument, the asterisk. Very confusing. I end up having to use xargs to get it work. Seems like a poor design.
It's so that every script in the world doesn't have to duplicate the logic for how to interpret globs. They inevitably won't end up doing it exactly the same way.
Shouldn't that be part of the OS API? Why call me with 1,000 arguments, when I could call an API with the wildcard and get the 1,000 files back? My poor argv can't handle it the other way.
41 comments
[ 2.6 ms ] story [ 106 ms ] threadTo be honest, I wouldn't care about my shell not handling newlines in files, I'd care about my users creating files with newlines in them. Space is natural language, newline is not.
Every time. Until you're an expert.
You learn something new (and vaguley horrifying) every day.
$> touch -- --
$> rm --
usage: rm [-f | -i] [-dPRrvW] file ... unlink file
$> rm -- --
$> touch \
$> touch \\ \ \ . \-
favorites of malware authors everywhere.
For example, to delete all .o files in the current directory and its subdirectories:
Or, you can use zsh.
Actually the Mac uses (or at least "used") a hidden files named "Icon\r" (yes, an embedded carriage return) for its custom folder icons. I'm still not sure that counts as "legitimate".
I remember choosing to use ls intentionally, but not what sucked about the alternatives.
You guys are insane.
I have not written nor seen any bash script that is neat, clean and fairly clear. It all seems like a very shady hack that someone think is smart.
ipython / iruby can and do make much nicer shells out of the box.
1) Is he implying that launching a perl interpreter is somehow a more lightweight operation than launching ls and having it stat() half a dozen files? Having ls check if a file exists 99% of the time won't even touch the disk. If you're working in that directory, those inodes will probably be cached. Running stat() on a cached inode is hardly more expensive than a system call. Read: cheap. Now perl launching will probably load a few libs and check a few files before even getting to his one-liner.
2) Am I the only one, who in my ~10 years of running unix-like operating systems have never created a filename with a newline char in it? Like even when opening obscure filesystems, never.
3) Does he think unlink() skips the check for the file's existence?
4) If he wanted to overcomplicate things and make sure everything's kosher, why not:
Whenever I see anything piped to a perl one liner, 4 times out of 5 it's some sort of hack that can be done with standard utils.It also grinds my gears when people take longer to think of making something more efficient than the amount of CPU time they'll save in their lives of using that 'more efficient' solution. So he spent a minute coming up with this and ten minutes posting about it. Now, will he save roughly 11 minutes of execution time in his life while running this "superior" command (also count that it takes longer to type)? If not, he wasted his time.
From his sig: "Smalltalk/Perl/Unix consulting". When you have a large-enough hammer....
2) If cleaning up after malicious users, sure why not. Doing this in your source tree or during normal usage? I don't know.
3) He mentioned ls stat()ing files as an inefficiency since the shell does it and ls does as well.
4)
...Read the context of the thread. It is a reply to this:
http://groups.google.com/group/comp.unix.shell/msg/30e16d21d...
Randal is pointing out that piping the output of ls to the perl interpreter is dangerous and shows the safer way to do what the original poster wanted.
"Well, there was a very embarassing error for Apple a few years ago for the 10.3 to 10.4 upgrade as I recall. Apparently, if you had a space in your home disk name, all sorts of files got deleted.
So, the moral is, no matter how rare you think something is, that's NO EXCUSE for not doing the right thing for all possible characters."
http://groups.google.com/group/comp.unix.shell/msg/504a75cc7... http://groups.google.com/group/comp.unix.shell/msg/0385fc2d1...
Waisting brain cycles to save CPU cycles (in the wrong places) is shameful, insulting and boring.
It would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)
With regard to why this matters, it's not uncommon for badly written scripts to interpolate filenames directly into commands. Consider this Perl fragment:
and what it would do with a file called:It matters. Leaving a bug around is disgusting and wasteful especially when you can fix it.
* would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)*
Fixing bugs and writing programs that have as few bugs as possible is very important.
If your goal is to make your system as robust as engineerically possible, you should most likely pay more attention in some directions and less in other. It's not only about quantity - you better have no critical bugs then have only-a-very-few critical ones.
I periodically have to cleanup abandoned and overflowing unix mailboxes where every message is it's own file. You'd think a simple "rm *" would work, but it doesn't, because you get an error message saying that there are "too many arguments". From my perspective there's one argument, the asterisk. Very confusing. I end up having to use xargs to get it work. Seems like a poor design.