I've never had this problem, what is the use case? I've only ever used /usr/bin/env or /bin/bash, and can't think of a reason to do anything fancier. You would be hardcoding the full (presumably non standard) path to the interpreter right? Why not just add it to PATH and use env?
In Spack https://github.com/spack/spack), we encounter this when users install packages in deep paths within their home directories, or in deep paths within shared project directories (e.g., in NFS or Lustre on HPC machines). So we patch installed scripts that have long shebangs.
We want installed packages to work exactly as built, with the right versions of dependencies, and we don't want to rely on the user getting their environment right to do that. Spack users may install several versions of python or other interpreters. This ensures that scripts work without a special environment.
There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this:
#!/usr/bin/perl arg1 arg2 arg3
At least on Linux, you'll get one argument: "arg1 arg2 arg3". I think perl gets around this by parsing the shebang line itself, but sbang is more general. You can use it with /usr/bin/env to pass more arguments than would otherwise be allowed.
Thanks for sharing this. Wish I had this years ago at an HPC gig at FNAL (similar use case, with users having deep path home directories on Lustre and pNFS).
> There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this:
Most often when calling programs with a shell script it is recommended to make use of env to find it.
Like...
#!/usr/bin/env python3
As in the limitations link above, env is pretty much everywhere, and when it isn't, you're probably going to struggle anyway to make anything portable to those systems.
env actually has an in-built flag for ensuring you're passing arguments.
> you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter)
env lets you solve that one too. You can control PATH. From the FreeBSD manpages:
#!/usr/bin/env -S -P/usr/local/bin:/usr/bin perl
And if you lack the -P flag... env controls the environment variables... So, you could extend (or replace) it like so:
#!/usr/bin/env PATH=/home/my/loc:${PATH} perl
> The -S argument to env is not POSIX
You're right. POSIX only guarantees the -i flag, and the ability to set variables. It hasn't been modified in the standard since 1991, despite other updates to the standard.
OpenBSD and Busybox only supports the crippled POSIX version. It isn't completely portable.
Versions with -S and are available on Linux, FreeBSD (also has -P), macOS (also has -P).
Yeah it would be nice if POSIX would move a little faster, but I guess the slowness is the point.
This is probably obvious but bears repeating since we're on this tangent -- even -S and -P do not prevent the OS from truncating the shebang if it's too long. So env is great for passing multiple args, but it still doesn't let you use really long shebangs (the original point of the thread :)
I remember running into this a few years ago with a python virtual environment created in a deeply nested directory, where (IIRC) scripts installed by pip would wind up with the virtual environment’s python path as the shebang. These failed on some of but not all of our boxes. I forget how we resolved it, but I do remember being shocked to discover that there was a length limit.
Argument cause must be pretty rare, but I can imagine it happening with long paths to a virtual environment's interpreter nested deep somewhere in a home directory.
I do not think that _this_ was or will be ever an issue to myself or any system that i have to deal with and i do not see any use for it but i guess somebody else did. Anyway I believe this page is a very nice read on the background and a little history https://www.in-ulm.de/~mascheck/various/shebang/
This reminds me of one of the _very few_ bits of freebsd userspace that are better than gnu and was only adopted by gnu much later (8.30, the version first in debian buster, I think): `env -S`.
`env -S` introduces an option parser/splitter for #!-lines which can do just about whatever you need (however, it will not work around over-long lines like sbang proposes to do)
Why is the `/usr/bin/env -S` even in there? Why not rather
#!perl -w -T
I know that not all environments support non-absolute paths to interpreters like this, but it feels to me that adding -S to env is a worse hack than allowing non-absolute paths to interpreters. I’m not sure which platforms support resolving the interpreter by PATH (actually, is supporting it a shell thing?), but it may even be more portable than -S at present, GNU coreutils env only having had it for two years. And even /usr/bin/env itself isn’t fully portable, as there are (old) platforms that have env at /bin/env and not /usr/bin/env.
It's there because on some systems all the args are passed as one. -S splits them out. You can try this on Linux and env even warns you about it:
# cat test.py
#!/usr/bin/python3
import sys
print(sys.argv)
$ cat test
#!/usr/bin/env test.py foo bar baz
$ PATH=. ./test
/usr/bin/env: 'test.py foo bar baz': No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
with -S, env does the right thing, and you get this:
$ cat test
#!/usr/bin/env -S test.py foo bar baz
$ PATH=. ./test
['./test.py', 'foo', 'bar', 'baz', './test']
Seems to work (i.e., search $PATH) on macos but AFAIK nowhere else. It fails on both ubuntu20 and centos8. Per this: https://www.in-ulm.de/~mascheck/various/shebang/, relative paths are supposed to be interpreted as relative to ., so I'm guessing that is why everyone uses /usr/bin/env.
Given that shebang is implemented in the kernel (and not in user-space like coreutils) I think env is more likely to change than the shebang mechanism, though who knows -- there have been recent changes to shebang so perhaps further work on this will help: https://lwn.net/Articles/779997/.
The shell you’re using matters, too; I find that PATH search happens when starting things from zsh, but not bash or sh. That might be why you observe it working on macOS, I dunno—they switched to zsh a while back, didn’t they? I’m not sure how much is in the kernel and how much is elsewhere, but I do observe that the shell is at least partly responsible for the error message like `bash: /path/to/script-with-shebang: python3: bad interpreter: No such file or directory`.
The problem with shebangs that I really want to solve is that I often need to edit files on a Windows machine, and then try to run the resulting CR/LF scripts on a Linux machine using a shared filesystem. (Docker, WSL, Vagrant etc...).
I'd like to invoke them with just (eg.) ./dostuff.py - Python, Ruby etc... have absolutely no problem running files containing Win-style line-endings. The only issue is that /usr/bin/env complains that it can't find (eg.) "python3\n".
Yes, I know I could convert these files with dos2unix, and I also know that I can just invoke the interpreter explicitly - but I'm lazy, and this seems like such a trivial thing to solve that I can't believe it's not been done already.
I've taken to actually recompiling /usr/bin/env to strip trailing whitespace from the executable name - but there must be a better solution.
There are editors that allow preserving the line endings (be they LF, CRLF or CR) from the source file. That could be a solution since you already seem to have control over your environments.
Wouldn’t it be better to standardize format and baseline argument spec, and let the evaluating environment figure out the rest? Formalizing the entry path is something that is now becoming two lines of indirection just to lie to whatever is actually interpreting it. It’s another line of implicit code that’s up to interpretation and can easily diverge.
> Wouldn’t it be better to standardize format and baseline argument spec
Better, yes. Easy, no. Impossible is a sort timeframe, probably (given how many systems are out there not confirming to this as yet undecided standard).
Maybe that will actually happen long-term, with this being essentially a polyfil.
I'm not sure how this solves the problem. The goal is to execute an interpreter in a long path via the shebang line, but this is a regular script that executes a program in a long path. You can't use this to make a file invoke /some/long/path when executed. Or am I missing something?
I think the easier method would just be defining an alternative character sequence to shebang lines which binfmt_ misc is happy to do for you. You already have to change the shebang line to support long lines, what’s switching the symbol to ‘#%’ or something?
34 comments
[ 2.6 ms ] story [ 71.0 ms ] threadWe want installed packages to work exactly as built, with the right versions of dependencies, and we don't want to rely on the user getting their environment right to do that. Spack users may install several versions of python or other interpreters. This ensures that scripts work without a special environment.
There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this:
At least on Linux, you'll get one argument: "arg1 arg2 arg3". I think perl gets around this by parsing the shebang line itself, but sbang is more general. You can use it with /usr/bin/env to pass more arguments than would otherwise be allowed.See https://www.in-ulm.de/~mascheck/various/shebang/ for an extremely comprehensive list of limitations on shebangs.
Most often when calling programs with a shell script it is recommended to make use of env to find it.
Like...
As in the limitations link above, env is pretty much everywhere, and when it isn't, you're probably going to struggle anyway to make anything portable to those systems.env actually has an in-built flag for ensuring you're passing arguments.
- you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter)
- The -S argument to env is not POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...
env lets you solve that one too. You can control PATH. From the FreeBSD manpages:
And if you lack the -P flag... env controls the environment variables... So, you could extend (or replace) it like so: > The -S argument to env is not POSIXYou're right. POSIX only guarantees the -i flag, and the ability to set variables. It hasn't been modified in the standard since 1991, despite other updates to the standard.
OpenBSD and Busybox only supports the crippled POSIX version. It isn't completely portable.
Versions with -S and are available on Linux, FreeBSD (also has -P), macOS (also has -P).
This is probably obvious but bears repeating since we're on this tangent -- even -S and -P do not prevent the OS from truncating the shebang if it's too long. So env is great for passing multiple args, but it still doesn't let you use really long shebangs (the original point of the thread :)
`env -S` introduces an option parser/splitter for #!-lines which can do just about whatever you need (however, it will not work around over-long lines like sbang proposes to do)
This:
Seems to work (i.e., search $PATH) on macos but AFAIK nowhere else. It fails on both ubuntu20 and centos8. Per this: https://www.in-ulm.de/~mascheck/various/shebang/, relative paths are supposed to be interpreted as relative to ., so I'm guessing that is why everyone uses /usr/bin/env.Given that shebang is implemented in the kernel (and not in user-space like coreutils) I think env is more likely to change than the shebang mechanism, though who knows -- there have been recent changes to shebang so perhaps further work on this will help: https://lwn.net/Articles/779997/.
I'd like to invoke them with just (eg.) ./dostuff.py - Python, Ruby etc... have absolutely no problem running files containing Win-style line-endings. The only issue is that /usr/bin/env complains that it can't find (eg.) "python3\n".
Yes, I know I could convert these files with dos2unix, and I also know that I can just invoke the interpreter explicitly - but I'm lazy, and this seems like such a trivial thing to solve that I can't believe it's not been done already.
I've taken to actually recompiling /usr/bin/env to strip trailing whitespace from the executable name - but there must be a better solution.
Better, yes. Easy, no. Impossible is a sort timeframe, probably (given how many systems are out there not confirming to this as yet undecided standard).
Maybe that will actually happen long-term, with this being essentially a polyfil.