HN Explorer

Tell HN: People forget that you can stick any data at the end of a bash script

321 points by BasedAnon ↗ HN
This is a neat trick I've used to write self-extracting software or scripts that extract files from archives by just using

    tail -c <number of bytes for the binary> $0
All you have to do is make sure you append an explicit 'exit' to the end of your program before your new 'data section', so that bash won't parse any of the 'data section'.

One thing to bear in mind is that if you append binary data, it will be corrupted if you save it in most text editors so when I want to make changes I just delete all the binary and reappend it.

148 comments

[ 3.8 ms ] story [ 209 ms ] thread
jjgreen
Ruby (and earlier, Perl) formalised this with the __END__ section: https://www.honeybadger.io/blog/data-and-end-in-ruby/
x86x87
yup. after that you can use the global var DATA to access the data injected after the __END__
jjgreen
When I was first learning Ruby knowing Perl rather well, it was discovering that it supported DATA/__END__ that made me feel like I was at home ...
brasic
Are you sure that Perl took it from ruby and not the other way around?

(edit: a subsequent correction has obsoleted this comment)

vram22
They said "earlier, Perl".
[deleted]
(comment deleted)
throwawaaarrgh
Back in the day I made a Perl script that would use an inline encryption algorithm to decode a payload and execute it in memory so it would never hit the disk. Data after the __DATA__ line.
_rwo
While applying for new job once I made a self-unpacking CV in ruby using this trick; whole binary content was compressed with zlib, I also added some blank padding to get some 'nice' numbers, so I could stuff like DATA.seek(1337 * 32, IO::SEEK_CUR); needless to say either nobody appreciated the idea, then again, what I was expecting? :D
cocodill
I can vaguely remember that many programs used to install themselves this way under Linux.
teddyh
It was used on Unix systems even before that.
dekhn
definitely used something similar on VAX/VMS called VMS_SHARE (https://www.glaver.org/ftp/multinet-contributed-software/vms...) circa '90-91

in fact I found an old archive of mine floating around on usenet and wrote a python script to unpack it. Looking at the original, it was using a scripting. language bootstrap to make a COM script unpack embedded the original code.

nerdponx
I've seen it recently with the Conda and Mamba package managers.
a2tech
Lots of commercial Linux software use this still for installing their stuff. It’s a neat trick
[deleted]
(comment deleted)
nottorp
Shell archive it was called? There used to be a lot of installers like that.
linsomniac
Minor nit: every "shar" I've seen (from distant memory) used a "here document" rather than appending (possibly binary) data to the end of a shell script.

https://en.wikipedia.org/wiki/Here_document

nottorp
Hmm i have a yocto generated sdk installer handy and it seems to do just this:

payload_offset=$(($(grep -na -m1 "^MARKER:$" "$0"|cut -d':' -f1) + 1))

and at the end of the code it just has:

exit 0

MARKER:

kkfx
Makeself archives are a classic self-extracting tarball who do exactly that...
xg15
If you care less about space efficiency and more about maintainability of the script, you can also encode the binary as base64 and put an

  echo '...base64 data...' | base64 -d > somefile
in your script.

Or add compression to reclaim at least some of the wasted space:

  echo '...base64 gzipped data...' | base64 -d | gunzip > somefile
Also note that bash accepts line breaks in quoted strings and the base64 utility has an "ignore garbage" option that lets it skip over e.g. whitespace in its input. You can use those to break up the base64 over multiple lines:

  echo '
    ...base64 gzipped data...
    ...more data...
    ...even more data...
  ' | base64 -di | gunzip > somefile
dheera
Is there an encoding that is less wasteful that base64 but not vulnerable to text editor corruption issues? I think avoiding 0x0 to 0x20 should be enough to not get corrupted by text editors, though base64 avoids a lot more than that.
ElectricalUnion
If you can count on every printable ascii character being not-mangled, you can use ascii85/base85/Z85 (5 "ascii characters" to 4 bytes) instead of base64.
raverbashing
There's probably a base(bigger number) with Unicode chars today
sudobash1
But you need to make sure to use utf-16 or utf-32 instead of utf-8, or you may be worse off.
dheera
Those get mangled by text editors that don't support them.
Dylan16807
While a couple of people suggested Base65536, that encoding isn't particularly compact, and it can't be as elegant as 65536 would suggest because it has to dodge special cases in unicode.

It's almost always the case that either Base32768 is denser, or encodings with 2^17 or 2^20 characters are denser.

balou23
there's also base91, with an efficiency of 6.5 bits of data per printable character, compared to 6.4 with ascii85, 6.0 with base64
delusional
At that point you're basically doing yEnc.
bazzargh
if you mean the thing you want to encode is mostly-ascii, then https://en.wikipedia.org/wiki/Quoted-printable ... it's a real throwback, I've not seen this in the wild since the 90s, but it's there in the python standard library (quopri), perl (MIME::QuotedPrint) etc
Tepix
Base85, also called Ascii85. Also yEnc.
mike_hock
If you care about maintainability, you keep the binary data out of the source file and have a build process.
dopidopHN
For something small. I would take a data.bin file rather than a build process. But yes.
mike_hock
The "build process" in this case is concatenation.
xg15
I meant "maintainability" in the sense that you can open it in a text editor without either corrupting the file or having the editor crash.

It's also useful if you want to check the file into a git repo and want to keep using the diff and merge tools.

saltcured
You can also use here-documents to avoid hitting any argv length limits:

    { base64 -d | gunzip > output; } <<EOF12345
    ...data...
    EOF12345
AlDante2
Just to be sure I’m following you correctly, what is the advantage of zipping the base64 data vs having the original binary, zipped if you like?
newx
As I understood, you base64 the zipped data on input and the other way around on output.

The reasoning being that the base64'd binary data is safe from being corrupted when the file is edited in text editors, as a response to the warning stated on the last paragraph of the original post.

xg15
The idea is to first zip the binary, then base64 the zipped data. Conversely, the script first decodes the base64 to a zipped binary, then unzips the binary.

It's just to mitigate the wastefulness of base64 encoding. You end up with a file that is text editor friendly and not quite as bloated as directly encoding the binary would be - but of course the file is still larger than simply appending the binary directly like in the OP.

Also, if you don't care about text editor friendlyness, you could indeed just zip the binary and then append it to the script for an even smaller file.

sudobash1
An even simpler way would be to include a marker to denote the end of the shell script, and the start of the data. For example, if you put this in extract.sh

    #!/bin/sh
    sed -E '1,/^START-OF-TAR-DATA$/d' "$0" | tar xvzf -
    exit
    START-OF-TAR-DATA
and then run:

    cat extract.sh ../foobar.tar.gz > foobar.tar.gz.sh
You can then run foobar.tar.gz.sh to self-extract. And you still get the benefit of being able to modify the shell script without needing to count lines or characters without sacrificing any compression.
[deleted]
(comment deleted)
oxygen_crisis
You've already got a marker to denote the end of the shell script baked in there, it's ^exit$
withinboredom
There may be more than one exit.
oxygen_crisis
There will be only one that is neither preceded by indentation nor followed by an exit code, so that it could match ^exit$, unless you contrive some hypothetical nonsense purely for the sake of contrarianism.

Any reasonable person will indent a conditional exit within the block testing its condition, and more than one unconditional exit doesn't make sense.

withinboredom
Ah fair enough. I didn’t think about using the indent, that’s pretty clever.
8n4vidtmkvmk
It would still make me uncomfortable that a 2nd exit or lack of indent would break my script. And it's less clear than an explicit marker
snowstormsun
Beware that often you need to append -n to echo to not include a newline when you base64 encode/decode something.
xg15
I think the -i flag when decoding should handle this and cause the decoder to skip the newlines.
floomk
The build process would produce the concatted file
[deleted]
(comment deleted)
dietrichepp
This trick is used in the demoscene. Instead of using -c, I use -n,

  tail -n +2 $0
The -n +2 option means “starting at line 2”, which is what you want if you cram your script into one line. You can make an executable packed with lzma this way,

  a=`mktemp`;tail -n+2 $0|unxz>$a;chmod +x $a;$a;rm $a;exit
This is the polite way to do it, using mktemp. You can save some bytes if you don’t care about that stuff.
londons_explore
There must be a way to run something without needing a temp file...
BasedAnon
Basically everything in Linux will create a temp file one way or another even pipes. For you to take a binary and run it directly it has to have an inode. At best you can use Python & the ctypes module to write a program into part of Python's memory and trick it into continuing execution from there.
dietrichepp
Pipes are not temp files, they’re more like kernel buffers associated with no file. When I think of temp file, I think of something that is at least associated with a filesystem.

The reason that running a binary needs a file is because execve() takes a path as an argument. But, as you said, there are other ways to load code into memory.

Denvercoder9
> Basically everything in Linux will create a temp file one way or another even pipes.

pipe(2) doesn't create a file, at least not in the sense that I usually think about files (something that's accessible through the filesystem).

dietrichepp
Yes. You can load the decompressed code into memory, mprotect() it to give it execution permissions, then jump in. Hard to do in a shell, though.
10000truths
If you're willing assume a Linux platform and a Python installation, you can do something like:

  echo -n "base64encodedelf" | base64 --decode | python3 -c 'import os;import sys;f=os.memfd_create(os.MFD_CLOEXEC);os.write(f,sys.stdin.buffer.read());os.execve(f,["dummy"],[])'
londons_explore
Surprising that no combination of bash symbols lets you do this.
Tepix
Neat. But is the base64 even necessary?
erik_seaberg
I don’t know that every editor today will gracefully handle a blob that isn’t valid UTF-8. Emacs needed a library (so-long) to disable some features that choke trying to render very long lines.
Tepix
You'll to edit the shell script without the binary at the end and once you're done you append the blob.
INTPenis
That's how I made a bash backdoor once. It was just a script somewhere on the FS, until it unpacked itself and executed the rest of the rootkit.

Long story but trust me that I had good intentions.

JohnFen
This is my default approach to writing installers for the Unices. The program is compressed and added to the end of the script, and the script does the unpacking and any needed setup/configuration for the specific platform it's getting installed on.

I don't append it in binary form, though. I uuencode it. That way, there is no danger in using text editors.

themerone
Why uuencode? Base64 is the defacto standard these days.
vram22
I've used both, but only briefly. I think I used uuencode when using uucp. And Base64 in one of my Python programs.

What are their pros and cons, in your opinion?

themerone
Base 64 is slightly more space efficient. Other than that it's just more popular and better supported.
vram22
Got it, thanks.

Yes, uuencode / uudecode are probably older too.

They are from the uucp dialup comms era of networking.

JohnFen
Sorry, I did mean base64. I have a bad habit of calling all "binary as text" encodings "uuencode". I usually catch myself before I put it in writing, though.
onion2k
This reminds me of ZX Spectrum Basic where all the graphics, sound, and level layouts were defined using DATA lines at the end of the program.
allarm
You could also put the binary data in the first line of the Basic program after the ‘rem’ command, change the line number to 0 using the poke command, so that it’s not possible to edit this line. The second line would run the code using ‘randomize usr’. There were also fun tricks with control sequences, that would hide the ‘rem’ command and the line number, and put something like “Cracked by Bill Gilbert (c) 1982” instead. Gosh, why I still remember all this nonsense after all these years…
antod
Or any machine code routines you wanted to POKE into memory.

A suppressed obscure part of my lizard brain secretly wishes I could just code for 8bit computers from the 80s, just with all the modern niceties like text editors, assemblers and emulators etc.

vram22
BASIC and Perl had or have something like that too.

IIRC, Perl copied it from BASIC, because BASIC came much before Perl.

And, again, IIRC, I've read about the shar (shell archive) method that someone else commented about in this thread (and which even has a Wikipedia entry), in either the classic Kernighan and Pike book, The Unix Programming Environment (which I've recommended here multiple times before), or in some Unix man pages, long ago.

So it's quite an old method.

nurettin
I remember writing DATA lines on commodore basic. But you could put them on any line.
vram22
Yes, the DATA statement was the way of doing it in BASIC.
davidw
I seem to recall that you can do the opposite as well: stash some extra data at the end of a binary file. The 'tclkit' system used this to package up an executable with the scripts you wanted to ship.
giantrobot
The Löve game development framework packages games like this. The games are nominally Alia scripts but if you cat a zip file to the end of the Lua script the framework lets you access assets in that zip as if they were in the game's cwd.

It's a cool abstraction, during development you can have assets just living on the file system and then "deploy" a flat file that accesses those assets the same way.

2OEH8eoCRo0
I think this is how GOG ships the Linux version of Battletech.
aquova
I believe this is how GOG ships all of its Linux titles, all of the installs I've used from them are downloaded as a single *.sh file. I just checked an example game, and it looks to be using this method.
twic
Since zip files use a directory at the end, you can make a kind of mullet file - script at the front, archive at the back. I generated single-file runnable Java binaries like that at once point.
ndsipa_pomu
> mullet file

That's a great expression

BonoboIO
In German that would be VokuHila: Vorne Kurz, Hinten Lang

Front Short, Back Long

https://de.wikipedia.org/wiki/Vokuhila

sumosudo
I use a fun little hack, a la awk:

``` #!/usr/local/bin/bash

echo "HELLO"

TAIL_REMOTE_MARKER=`awk '/^__THE_REMOTE_PART__/{flag=1;next}/^__END_THE_REMOTE_PART__/{flag=0;exit}flag' ${0}`

eval "$TAIL_REMOTE_MARKER"

exit 0

__THE_REMOTE_PART__

echo "WORLD"

__END_THE_REMOTE_PART__ ```

[dead] hey00
[flagged]
norir
This is a great trick, but no one should ever run someone else's script that does this unless they have verified the script line by line beforehand.
ygjb
Sure, but that's turtles all the way down... any time you run untrusted code, you are making a risk based decision, usually based on the provenance of the code.
michaelcampbell
Maybe? People run all manner of binaries/installers without checking them; I'm not sure why these sorts of things require any EXTRA scrutiny.
fargle
EXACTLY! I can't stand the overblown security posturing with regards to things like bash and git, even python.

When everyone in the world INCLUDING cyber-security professionals seems to think that it's perfectly OK to download setup.exe or install.msi from sketchy driver mfgs. and run them. But SHARs and curl | sh is too scary!

It's not that any of these are a good idea, it's the blind-spot to traditional binary installers that's annoying. How do you audit them? Why are they OK (because they have to be)

oefrha
curl | sh is actually bad because a completely innocuous script can become dangerous with a network hiccup, e.g.

  rm -rf /path/to/directory
becomes

  rm -rf /path
if the connection accidentally drops half way. Or something less dramatic, but still leaving you in a broken state.

Some people took the “don’t curl | sh” advice as “you have to inspect every line of a shell script installer you download”, which is of course absurd.

fargle
that's a very interesting point and example. TCP is normally lossless and byte-for-byte before data is forwarded to the application. I think your example may illustrate a loophole - truncation.

It'd only work if, like your example, it was the truncation that caused the flub. You can't have random corruption.

Does curl in a pipeline really behave that way - if the connection is interrupted (TCP RESET), does it just end the pipe? It probably actually does, from what I see.

It's an interesting edge case.

Personally I use, which would seem more immune:

    curl -o /tmp/a
    vi /tmp/a
    {get bored easily, fuckit}
    sh /tmp/a
ElectricalUnion
Ideally, scripts that are in any danger of being truncated are supposed to be wrapped in immediately invoked functions exactly because of this gotcha, and most (but not all) of those that I inspect before running often are.
michaelcampbell
Could be confirmation bias/rage on my part of course, but I never took...

> Some people took the “don’t curl | sh” advice as “you have to inspect every line of a shell script installer you download”, which is of course absurd.

... to ever mean anything BUT "inspect every line". Which is, as you say, completely absurd.

mook
I don't think I've ever read through the Nvidia binary drivers that way. (They're named *.run but are basically shar files)
thinkmusic2000
I used to do something similar for Windows executable files. Append a large file to the end as necessary.
Fudgel
I vaguely remember this is what Ocaml does for one format of its executable.
mbreese
Java JAR files are similar, but reversed. You can add anything you want to the beginning of the JAR file (or is it any ZIP file?) so long as it doesn't include the Zip file header "PK". So, I use this to prepend a bash script that ultimately calls

    java -jar $0
It makes it very easy to setup and use Java based command line programs on a server.
zmmmmm
this sounds incredibly useful but I couldn't get it to work. I just get

    java.util.zip.ZipException: invalid CEN header (bad signature)
        at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1623)
if I try to do anything with a JAR file that has leading text. I'm creating it just using

   echo 'java -jar $0' | cat - test.jar > test.run.jar
Is there more to it?
nneonneo
Technically, you should update the offset to the central directory in the Zip footer, along with the offsets to each file header in each central directory entry. If you don’t, the zip file reader has to apply some heuristics to locate the central directory; not all readers implement these heuristics, and those that do won’t always be robust.

The “unzip” utility can be useful as a sanity check; run “unzip -t” to test the integrity of the file.

karmicthreat
I did a similar thing for a lowish volume embedded product. The update files are just bash scripts with a tar file cat'd on them. The unit just looks for a particular file on an external flash drive to run and the bash script runs, copies off a tar and checks that it has the right hash. Super simple and flexible when customers need me to do something special. Like extract some specific log onto a flash drive.