I got into a 20 minutes argument with a coworker once about why it isn't a good idea to put spaces in filenames (my argument being that it can sometimes break handcoded argument parsing in homegrown unix scripts when the escaping of the spaces aren't passed down properly).
Yes, because the potential to break administrative scripts is a far more important consideration than the usability of a system for the average person.
I agree with the most of the points in the article, but the one regarding spaces is, IMHO, ridiculous.
EDIT: Just to be clear, I don't use (or care that much about) spaces in file names. I am talking about the vast majority of non-technical users.
Just a small correction. While nobody uses spaces in file names in Plan 9, and at least in Inferno #U changes spaces to something else, spaces are not forbidden in the kernel since March 23th, 1999: http://swtch.com/cgi-bin/plan9history.cgi?f=1999/0323/port/c...
It has nothing to do with liking or not liking spaces per se, but simply that you should support the features that the operating system allows. Look I hate spaces (and non-ASCII characters for that matter) in files names as much as the next programmer who's ever had his code fail due to them, but if I'm writing for a platform that supports them, then I try my best to deal with them, and tread any failure in handling them as a bug that should be fixed.
Assuming the coworker was a technical user I think it is correct to tell him to stop using spaces. No reason to make life harder for the team members just because you happen to like spaces.
Personally I think spaces and UNIX shells is a mess, but I am not sure what would have been a better solution.
That's actually vulnerable to certain problems. Using "-n" or "-p" or "while (<>)", the files get opened using the 2-argument version of open, so a file names, e.g. "|rm *" can cause problems!
This rant misplaces its frustration. This is not a problem with unix filesystems, this is a problem with Bourne Shell scripts, and with UNIX argument parsing semantics.
Bourne shell is notorious for its problematic quoting, both of filesystem data and of any data from any other source. Every example in which he described a problem with a filename parameter could just as well be a problem with a non-filename parameter. The correct solution is to not program complicated scripts in Bourne Shell, and instead use a language which does not implement variable access by interpolating strings and then re-tokenizing and re-evaluating them. Examples of satisfactory languages include Perl, Python, Ruby.
Regarding UNIX arguments and the dash, it is an unfortunate aspect to the flag argc/argv/envp calling convention for unix programs. Some other operating systems provide more structure in their calling convention, explicitly separating different types of parameters from one another. This is both a strength and a weakness, as it results in a uniform yet inflexible systems interface. One of the greatest strengths of UNIX is that its calling convention is so flexible. The semantics used today are quite different from the semantics used 40 years ago -- yet execve() remains unchanged. I would encourage anyone interested to do a bit of historical digging here, and see how those more ridged system APIs fared over time.
Anyway, the solution to his initial question of using `ls` is the -- argument, which signifies argument parsing should be disabled for the remainder of argv: ls -- *
The correct answer to his dotfile/glob question is: "glob() and the Bourne shell do not have the semantics you're after. Do not use them, use readdir()."
The correct answer to his find -print question is: Yes, print's use of whitespace was a mistake, and it is a mistake repeated continually throughout the land of shell scripting and accompanying standard UNIX utilities. As he notes, it is why print0 was introduced. Making print0 standard is far easier than reworking filesystem semantics (and, reworking userland in this manner is a more complete solution as it addresses data integrity issues from non-filesystem inputs as well). If you want reliable, correct programs, do not write them in shell.
>The correct solution is to not program complicated scripts in Bourne Shell, and instead use a language which does not implement variable access by interpolating strings and then re-tokenizing and re-evaluating them.
No, even "satisfactory" languages can suffer problems. For example a couple of days ago I discovered a nice exploit in the qemu-img program, and using any language to parse the output wouldn't help you:
Your link above shows an author who claims JSON output, yet the output is clearly non-validating JSON (toplevel is not a [] or {}, improper quoting, etc). It appears that instead of using JSON serialization, the author merely printed key/value pairs separated by the string ": ". The problems with this approach are obvious.
This is why using a proper serialization format is important.
If the author had done this correctly and used a proper JSON library to produce this output, the following, completely safe result would have occurred:
The author probably would have been best served by YAML, which is more easily readable -- and which, like json, provides mechanisms to properly represent arbitrary data.
In any event, the discussion is severely confused. Ad-hoc buggy formats cannot be compared with well-formed JSON or YAML. This has nothing at all to do with the language.
You should probably read the link more closely. I'm advocating using JSON so that programs are able to safely parse the output of 'qemu-img'. At the moment there are many programs that parse the (current text) output, and they almost all have security holes as a result.
Yes it does - qemu-img is written in C. The two programs we found exploitable were written in Python and C. They are written in "satisfactory" languages. Bash is not involved. Yet both suffer exploits because of \n (and other) characters in filenames.
The issue you refer to is in a poorly formed, ad-hoc serialization format. It has nothing to do with representation of variables at runtime. It has nothing to do with the language.
It is a programming error, not an inherent flaw in the language.
That's incorrect. As was already pointed out this issue has nothing to do with reading data from the filesystem or manipulating variables internal to the program and everything to do with poor choices made when using printf.
In other words, those files aren't causing the QEMU program internals to re-interpolate one variable as two values. They're merely messing up a poorly written data exchange format.
Other languages such as I listed above simply do not have the same issue. The C program did not mis-interpret a variable as two separate values because it contained spaces. That is the nature of the danger with shell -- any reference to a variable in Bourne involves string interpolation and tokenization. This simply does not happen in C.
Yes, Bourne Shell's variable access scheme is a bit ghetto, but to me the problem is that the shell is doing globbing at all. Why not have the shell pass "*" through to the program, and have the program itself perform globbing? Then filenames would have no impact on how the command-line is parsed.
Because that's how MS-DOS used to work, and it was dumb. It means every program has to do globbing (or often, didn't do globbing). In any case, bash does get this right: ls * will pass the correct filenames to the ls program no matter what the filenames contain. Also quotes around variable expansions can cope with any characters.
So what? If the primary API used by command-line applications to open files does the globbing, then programs will have to go out of their way to not glob. And you'll get the added benefit that globs will only be applied to arguments that are actually meant to specify filenames. There would be none of this escaping "*" when you pass it to "find."
> In any case, bash does get this right: ls will pass the correct filenames to the ls program no matter what the filenames contain.
That doesn't solve the problem; your filename could be called "--help."
bash isn't interpreting '--help' at all, it is just passed on to the program being executed, and most GNU CLI programs conventionally interpret '--help' as a special option.
If your filename is indeed --help, the convention is to use '--' as the separator between your command line options and filenames. Anything after -- is not interpreted as a command-line option.
Another way would be to use a more qualified filename form ('./--help')
Because, as the author points out, different users may want different globbing behavior. Globbing is not performed identically between shells.
If the author so wished, he might trivially create his own shell and allow * to match dotfiles, with absolutely no disruption to the rest of his system. Or one could write a shell which uses a regex instead of a glob. Or the SQL LIKE query syntax. The possibilities are endless. Anyone is free to do this.
The fact is, the current globbing behavior in unix shells strikes a good balance between pedantic correctness and "what I really want." The author's frustration is due to his attempting to use a command line interface as a structured programming language.
A better, and universal, solution to the problem of filenames starting with - is to prefix all relative paths with ./. A path like ./-blah will never be misinterpreted as a command line option, regardless of the tool, and doesn't depend on the -- convention which is only inconsistently present.
The author also mentions a problem with the filesystem: what is the filesystem encoding? Do you treat filenames as blobs, or encoded strings? What do you do if you think the filesystem stores UTF-8 but there's a filename which has a byte sequence which is invalid UTF-8?
I was thinking about this some more. You are right, but both you and I missed the point.
I think the author is saying that the problem is the "unix filesystem" is actually filesystem that doesn't match the unix, where 'unix' includes sh/csh/bash shell and command-line arguments which start with '-'.
If the filesystem wasn't a broad in what it accepted ... and the author is trying to convince us that POSIX allows that ... then it would a unix filesystem which was a better match to unix.
All of these problems arise because we're passing around text that needs to be parsed. Programmers don't know the parsing rules, And they vary by utility. The end result is that it is very easy to make something work, and very hard to make it work correctly.
As was noted by fffggg, most of these problems disappear as soon as you switch to a language that lets you stick the filenames in strings and never again tries to parse them. All of the major scripting languages will do.
But not all, just most. The exception being the UTF-8 issue. It does not solve printability of strings with unknown (and perhaps no valid) encoding. Furthermore scripting languages often will want to try to interpret external bytes coming back from the filesystem as a string, and may have trouble if file names are not some recognized encoding.
And if djb hasn't managed to convince you to stop parsing, watch this presentation from Meredith Patterson at last year's CCC: http://www.youtube.com/watch?v=v8F8BqSa-XY
I use rename[1] to clean up problematic filenames. Sadly I have to rename it to `rnm` because there is a namespace conflict with rename.pl in util-linux. On OSX it is as easy as `brew install rename.` I do not know why it has never made it into the debian.
Taking care of a directory full of madness is as easy as `rename -z`:
-z, --sanitize
Replaces consecutive blanks, shell meta characters, and control characters
in filenames with underscores.
Somewhat random, but I'm the person who "packaged" Aristotle Pagaltzis's outstanding rename for Homebrew. I'm very glad to hear people other than me are using it too.
Many thanks!!! I cant tell you how often I use that command. Or how many times I had scoured `apt-cache search` results looking for this functionality.
I am so very glad that someone wrote this issue up in detail. The fact that the Unix filesystem makes it extremely difficult to write correct scripts and to issue non-problematic commands at the shell is Unix's biggest flaw.
The whole point of Unix, and the reason that it was so revolutionary, was to empower the easy composition of complex tasks via scripts and command line invocations. The “negative freedom" implemented by the filesystem has undermined the "positive freedom" of Unix's original intent.
The kernel hackers' response to this, back in the day, was typically, "Program in C instead." It was a shame that the kernel hackers didn't understand Unix!
What this does is use 'find' to find the file and directory names (with your supplied CRITERIA), send them over to xargs with no funny filename parsing, run those filenames as arguments to COMMAND, batched up to BATCH_SIZE at a time (I usually use 100-500), and ignoring empty batches.
Usually COMMAND would have '--' at the end to prevent filenames from being parsed as arguments, but not all commands need that treatment. If COMMAND doesn't take the filename list as the last argument(s), it can be fixed with the '-i' option to xargs.
The BATCH_SIZE part here is important in many cases. Since large numbers of filenames will break the command line arguments size limit to COMMAND, a reasonable BATCH_SIZE will prevent thousands or millions of filenames from breaking your script. Also, sometimes you may want to run the files through COMMAND one at a time (BATCH_SIZE of 1).
Finally, the '-r' option, which I don't see mentioned often enough, tells xargs not to be derpy and run COMMAND with an empty argument list. Seriously, '-r' is short for '--no-run-if-empty'.
For anything else, a nice scripting language is the only way to fly.
43 comments
[ 3.6 ms ] story [ 94.8 ms ] threadI agree with the most of the points in the article, but the one regarding spaces is, IMHO, ridiculous.
EDIT: Just to be clear, I don't use (or care that much about) spaces in file names. I am talking about the vast majority of non-technical users.
Personally I think spaces and UNIX shells is a mess, but I am not sure what would have been a better solution.
cat * > ../collection becomes
perl -e 'map{open ($f,"<",$_);print <$f>;close($f)}<*>' > ../collection
Strangely, it is far easier to write a correct script in perl than in shell.
Bourne shell is notorious for its problematic quoting, both of filesystem data and of any data from any other source. Every example in which he described a problem with a filename parameter could just as well be a problem with a non-filename parameter. The correct solution is to not program complicated scripts in Bourne Shell, and instead use a language which does not implement variable access by interpolating strings and then re-tokenizing and re-evaluating them. Examples of satisfactory languages include Perl, Python, Ruby.
Regarding UNIX arguments and the dash, it is an unfortunate aspect to the flag argc/argv/envp calling convention for unix programs. Some other operating systems provide more structure in their calling convention, explicitly separating different types of parameters from one another. This is both a strength and a weakness, as it results in a uniform yet inflexible systems interface. One of the greatest strengths of UNIX is that its calling convention is so flexible. The semantics used today are quite different from the semantics used 40 years ago -- yet execve() remains unchanged. I would encourage anyone interested to do a bit of historical digging here, and see how those more ridged system APIs fared over time.
Anyway, the solution to his initial question of using `ls` is the -- argument, which signifies argument parsing should be disabled for the remainder of argv: ls -- *
The correct answer to his dotfile/glob question is: "glob() and the Bourne shell do not have the semantics you're after. Do not use them, use readdir()."
The correct answer to his find -print question is: Yes, print's use of whitespace was a mistake, and it is a mistake repeated continually throughout the land of shell scripting and accompanying standard UNIX utilities. As he notes, it is why print0 was introduced. Making print0 standard is far easier than reworking filesystem semantics (and, reworking userland in this manner is a more complete solution as it addresses data integrity issues from non-filesystem inputs as well). If you want reliable, correct programs, do not write them in shell.
Very well said.
http://www.mail-archive.com/qemu-devel@nongnu.org/msg128802....
Your link above shows an author who claims JSON output, yet the output is clearly non-validating JSON (toplevel is not a [] or {}, improper quoting, etc). It appears that instead of using JSON serialization, the author merely printed key/value pairs separated by the string ": ". The problems with this approach are obvious.
This is why using a proper serialization format is important.
If the author had done this correctly and used a proper JSON library to produce this output, the following, completely safe result would have occurred:
{ "cluster_size":65536, "disk size":"136K", "file format":"qcow2", "image":"/tmp/foo\ncluster_size: bar", "virtual size":"10M (10485760 bytes)" }
The author probably would have been best served by YAML, which is more easily readable -- and which, like json, provides mechanisms to properly represent arbitrary data.
In any event, the discussion is severely confused. Ad-hoc buggy formats cannot be compared with well-formed JSON or YAML. This has nothing at all to do with the language.
It is a programming error, not an inherent flaw in the language.
In other words, those files aren't causing the QEMU program internals to re-interpolate one variable as two values. They're merely messing up a poorly written data exchange format.
Other languages such as I listed above simply do not have the same issue. The C program did not mis-interpret a variable as two separate values because it contained spaces. That is the nature of the danger with shell -- any reference to a variable in Bourne involves string interpolation and tokenization. This simply does not happen in C.
So what? If the primary API used by command-line applications to open files does the globbing, then programs will have to go out of their way to not glob. And you'll get the added benefit that globs will only be applied to arguments that are actually meant to specify filenames. There would be none of this escaping "*" when you pass it to "find."
> In any case, bash does get this right: ls will pass the correct filenames to the ls program no matter what the filenames contain.
That doesn't solve the problem; your filename could be called "--help."
bash isn't interpreting '--help' at all, it is just passed on to the program being executed, and most GNU CLI programs conventionally interpret '--help' as a special option.
If your filename is indeed --help, the convention is to use '--' as the separator between your command line options and filenames. Anything after -- is not interpreted as a command-line option.
Another way would be to use a more qualified filename form ('./--help')
If the author so wished, he might trivially create his own shell and allow * to match dotfiles, with absolutely no disruption to the rest of his system. Or one could write a shell which uses a regex instead of a glob. Or the SQL LIKE query syntax. The possibilities are endless. Anyone is free to do this.
The fact is, the current globbing behavior in unix shells strikes a good balance between pedantic correctness and "what I really want." The author's frustration is due to his attempting to use a command line interface as a structured programming language.
I think the author is saying that the problem is the "unix filesystem" is actually filesystem that doesn't match the unix, where 'unix' includes sh/csh/bash shell and command-line arguments which start with '-'.
If the filesystem wasn't a broad in what it accepted ... and the author is trying to convince us that POSIX allows that ... then it would a unix filesystem which was a better match to unix.
All of these problems arise because we're passing around text that needs to be parsed. Programmers don't know the parsing rules, And they vary by utility. The end result is that it is very easy to make something work, and very hard to make it work correctly.
As was noted by fffggg, most of these problems disappear as soon as you switch to a language that lets you stick the filenames in strings and never again tries to parse them. All of the major scripting languages will do.
But not all, just most. The exception being the UTF-8 issue. It does not solve printability of strings with unknown (and perhaps no valid) encoding. Furthermore scripting languages often will want to try to interpret external bytes coming back from the filesystem as a string, and may have trouble if file names are not some recognized encoding.
Taking care of a directory full of madness is as easy as `rename -z`:
[1] http://plasmasturm.org/code/renameYou just like to live on the edge, don't you?
vs. $ rnm foo.{bar,bar.bak}The whole point of Unix, and the reason that it was so revolutionary, was to empower the easy composition of complex tasks via scripts and command line invocations. The “negative freedom" implemented by the filesystem has undermined the "positive freedom" of Unix's original intent.
The kernel hackers' response to this, back in the day, was typically, "Program in C instead." It was a shame that the kernel hackers didn't understand Unix!
Usually COMMAND would have '--' at the end to prevent filenames from being parsed as arguments, but not all commands need that treatment. If COMMAND doesn't take the filename list as the last argument(s), it can be fixed with the '-i' option to xargs.
The BATCH_SIZE part here is important in many cases. Since large numbers of filenames will break the command line arguments size limit to COMMAND, a reasonable BATCH_SIZE will prevent thousands or millions of filenames from breaking your script. Also, sometimes you may want to run the files through COMMAND one at a time (BATCH_SIZE of 1).
Finally, the '-r' option, which I don't see mentioned often enough, tells xargs not to be derpy and run COMMAND with an empty argument list. Seriously, '-r' is short for '--no-run-if-empty'.
For anything else, a nice scripting language is the only way to fly.