35 comments

[ 4.2 ms ] story [ 74.8 ms ] thread
It looks like it invokes the compiler every time the script is run? That's nice if you've changed the file, but takes extra time if you haven't. It seems neat at first, but I guess I don't see the advantage over just compiling to an executable and then running that.
I don't want to spam the link, but I actually just posted a top-level comment to a version of this I published back in 2020 that does caching and cache invalidation, using the hash of the script as the key for the compiled output path.

EDIT:

Actually, that was how the original version worked, but it had the limitation that once the script changed you could no longer determine the path to the previous executable output to delete, so you could end up accumulating a lot of defunct binaries. Now the hash is stored alongside the target.

There was some mention in the wiki about caching the result and rebuilding it on change, so it seems well thought out from that perspective.
it does some dependency checking, so no compiler invocation if the source isn't changed, the binary is cached
For the rustaceans in the crowd:

It's not a universal "for all compiled langs," but I published a two-liner [0] with zero (non-standard) dependencies or pre-requisites that can be copy-and-pasted to the top of any rust code to turn it into an executable script (cached! also correctly cache-invalidating!) back in 2020.

It even lets you optionally omit the `fn main() { ... }` surrounding your code as well, if you really want to feel like you're just shell scripting <insert appropriate emoji here>.

It's an example of "polyglot source code" that is valid under two languages at once (in this case, standard sh and rust), (ab)using the rust `#[allow(...)]` "macro" to do double-duty as a shell script comment opener.

[0] https://neosmart.net/blog/self-compiling-rust-code/

Nice hack! Would it have been possible back then to use cargo to pull in some dependencies?

The clean solution of cargo script is here: https://github.com/rust-lang/cargo/issues/12207

Following the links, the RFC contains a nice list of similar tools: https://rust-lang.github.io/rfcs/3424-cargo-script.html#prio...

Thanks!

> Would it have been possible back then to use cargo to pull in some dependencies?

Not at the time, at least not that I know of.

I wonder if it's possible to get a link to this solution added to that "prior art" section.

Thats awesome, now I'm going to have to find a reason to use this for something ;)

Tiny nitpick: the #![allow()] generates a 'warning: unused attribute' warning for me.

If you change it to #![allow(unused_attributes)] it disables the warning for itself.

Nice catch! That warning definitely wasn't emitted the last time I tried this.. I don't know about using something like your suggestion which has actual side effects, but probably something like #![cfg_attr(invalid, allow(unused_attributes))] instead would work (without side effects).
I prefer to take advantage of the fact that without a proper shebang the file will be first executed by the shell. Then it is easy to do almost whatever you want.

For example I put this in front of my test C files:

  #if 0
  set -e; [ "$0" -nt "$0.bin" ] &&
  gcc -Wall -Wextra -pedantic -std=c99 "$0" -o "$0.bin"
  exec "$0.bin" "$@"
  #endif
I call those: polyglot shell scripts. All my examples are valid files for the language while working as shell scripts at the same time.

In the past when Python 3 was still not obviously there and under different names it was once useful for me to have this on top of a python script:

  "set" "-e"
  "exec" "$(for p in python3.7 python3 python; do which p && break; done)" "$0" 
  "$@"
It works, because Python doesn't care about stray strings.

You can also use the trick with Makefiles, although I never needed to use it:

  #\
  set -e
  #\
  exec make -f "$0" "$@"
This is not a useful example, but it works. This works, because make will treat slash followed by a new-line as a comment continuation and bash will ignore it.
I appreciate a good hack more than most, but this is the kind of cleverness which makes life hell for anyone else who needs to understand what the fuck is happening and why.

At least with TFAs shebang at the top of the file, it kind of gives a clue. Upon reflection, I don't feel great about either way, haha. Even if it makes sense to me, it'll be a little too sneaky for most.

A few lines of comment can make the trick
For the sweet and rare soul who bothers to document it, absolutely.
> I prefer to take advantage of the fact that without a proper shebang the file will be first executed by the shell

I know that bash does this, and possibly other shells, but it doesn't work in general when you `execv` the file (or when using the subprocess API on python f.i). But cool trick anyway

Somewhat related: nix-shell supports an additional shebang, allowing summoning of any combination of packages, to be made available inside a script. An example from the docs[1]:

  #!/usr/bin/env nix-shell
  #![allow()] /*
  #!nix-shell -i bash -p rustc
  rsfile="$(readlink -f $0)"
  binfile="/tmp/$(basename "$rsfile").bin"
  rustc "$rsfile" -o "$binfile" --edition=2021 && exec "$binfile" $@ || exit $?
  */
  fn main() {
      for argument in std::env::args().skip(1) {
          println!("{}", argument);
      };
      println!("{}", std::env::var("HOME").expect(""));
  }
and another:

  #! /usr/bin/env nix-shell
  #! nix-shell -p "haskellPackages.ghcWithPackages (p: with p; [turtle])" -i runghc

  {-# LANGUAGE OverloadedStrings #-}

  import Turtle

  main = echo "Hello world!"

1. https://nixos.wiki/wiki/Nix-shell_shebang
In my experience, writing small but critical shell-script-style mini-programs in rust can actually be a really good idea. The biggest drawback is, IMO, all the overhead of setting up an entire crate just to compile a tiny script, so I'm really happy to see projects like this (and other tools [1]) for this workflow.

[1] https://rust-lang.github.io/rfcs/3424-cargo-script.html#prio...

The problem with these systems is third party dependencies and IDE support. You really want to be able to use third party dependencies, and have IDEs know about them.

The only systems I know of that do that properly are Deno and I believe F#, though I haven't tried the latter.

Unfortunately Deno has a stupid issue where `deno run` will check for updates and print a "new version of Deno is available" message which rather sours the otherwise great experience of using it for shell scripting.

This is one of the things that drive me nuts about open source projects. Sane defaults are critical to onboarding. Yet default configs seem to mostly be setup for maximum pedantry.
For python

    #!/usr/bin/env -S PIP_RUN_RETENTION_STRATEGY=persist pip-run

    # Requirements:
    # pendulum>=1.0.0
    # requests

    import requests
    print('hello')
Executable, single files with dependency specification should have first class support in all languages.

But that's not going to happen.

So next best option is probably weirdly a markdown file executor which runs triple quoted blocks.

The upside is that you could mix different languages, which is interesting.

That sounds a lot like a Jupyter notebook
Also Emacs' org-mode.
Literate programming, Donald Knuth
I use org mode (and a custom tangler, because Emacs' tangler is slow and gets slower with every file you tangle) precisely for Knuth-style literate programming.

The only reason I didn't mention it here was that GP was talking about running one SRC block at a time, rather than building the entire project to export and run as an executable.

In both Linux and Macos source files are usually not executable. So what's the point of this? Applicable only in Windows with ex-FAT/FAT32 (probably, also not a thing in NTFS, but not sure whether needs explicitly forbidding 'execute' flag or how it's called there) which is able to run any file.
A utility called 'chmod' can adjust the permissions to allow execution
And utility called 'dd' can overwrite your hard disk with nulls. But should it?

The key word above is "usually". There's no good reason to make source files executable. And also, there's no much sense in compiling any source file individually instead of relying on a proper build system.

This is probably meant for tiny script-like programs rather than multiple dependent source files. E.g., I've seen standalone C programs with `#!/usr/bin/tcc -run` shebangs before.
Script files are typically executable. The whole point here is to make a script using a language that requires compilation. So yeah, mark it as executable, it's a script now.
> The whole point here is to make a script using a language that requires compilation.

Then it's a bit of non-orthodox approach here, if not more. As defined in wiki [0] (and on par with what we all are used to):

> Scripting languages are usually interpreted at runtime rather than compiled.

I would not be much happy if script would do some heqvy lifting and output some binary in the current or tmp folder. Which hash needs to be checked each run (against what?) and would require lots of other security-related protections, in real world. In ideal world - not much, but that's not what we are talking about, right?

0 - https://en.m.wikipedia.org/wiki/Scripting_language

It's interesting being able to execute the source file like this, by passing in build flags and compilation instructions. But in Emacs I can do this by simply writing my code in scratch and then evaluating each piece of it with ctrl-j. I will immediately get output. The LISP read-eval-print loop enables this type of power.
Emacs didn't invent REPL, and it's common everywhere. For Rust: https://github.com/evcxr/evcxr/blob/main/evcxr_repl/README.m.... But heck, the compiler is reasonably fast enough that any IDE can REPL by compiling the code.

The value here is more in being able to read a script before you run it, then have it run fast, maybe tweaking something here and there. And a compiled script will run 10,000 times faster than LISP, which can be important.

> a compiled script will run 10,000 times faster than LISP

Putting aside some hyperbole, this depends how long it takes to compile.