I have another story un the general spirit of doing crazy pointless things with bash.
A couple of jobs ago, I had to write a job to download a large set of files scattered in a deep hierarchy from the web interface of an internal distributed file system. Because the directory page was non standard and didn't contain any url, I couldn't simply use wget or curl in recursive mode, but had to do some light parsing of each tree level.
My simple bash one liner quickly grew in a larger script. Of course, this being bash, it was trivial to make it do the scraping and downloading in parallel, but I needed a way to limit the maximum number of concurrent downloads.
To make a long story short, I ended up reimplementing semaphores in bash on top of message passing via pipes and 'coproc' (which I had just found out).
In retrospect I could have probably implemented something simpler and more robust on top gmake (which interestingly also uses pipes to limit the number of subprocesses but doesn't need a supervisor process).
This morning I was scraping the images from the 2016 report and given they were like ~230 images I thought it could be cool to download them in parallel, but I quickly realized this would lead to such path and downloaded these in serial.
It's the same number of bits either way, so the relevant info is were the images are stored and what the bottleneck is in accessing them. If they're all on a single disk drive, for example, it would probably make no difference if you did it in parallel or serial.
You can do this with xargs -P (with GNU xargs at least). I often use the pattern:
do_one_thing() {
local arg=$1
... do stuff with $arg
}
do_with_limited_parallelism() {
cat things-to-do.txt | xargs -P 10 -n 1 -- $0 do_one_thing "$@"
}
"$@"
That limits the number of concurrent processes, which is almost always good enough for limiting the number of concurrent downloads. If you have really tiny downloads, then the overhead of starting a connection per process may matter. In that case it's easy to do more than one download per process with say -n 50 instead of -n 1.
xargs -P is great and I have used it often for quick parallelization. The problem I had was that the crawler would recursively invoke itself while crawling the (possibly unbalanced) tree so xargs wouldn't have worked.
OK, I see. Your implementation sounds a little like how "make -j" is actually implemented (in C).
This basically sounds like implementing a semaphore with a pipe.
"""
Rather than using complex schemes such as a central server, shared memory, sockets, etc. for synchronizing the various instances of make, he suggested we use the simplest, most basic UNIX construct: a pipe. Not only is this easy, but it’s also a relatively portable concept. Most operating systems these days implement some sort of pipe feature, because that paradigm is so useful.
The idea is ingeniously simple: the initial, top-level make creates a pipe and writes N one-byte tokens into the pipe. That pipe is shared between that make and all submakes, and any time any make wants to run a job it first has to read a token from the pipe. Once the job is complete, it writes the token back to the pipe. Since there are only N tokens, you know that you will never invoke more than N jobs.
"""
Yes, except that gmake implementation is brilliant as it uses the pipe itself as a semaphore, while my version had an explicit process keeping a count and listening to V and P messages.
Unfortunately I didn't know how make -j was implemented at that time.
it's just a moderately clever way to do a moderately ugly thing. When I saw the title of the post, I thought "uh, how on earth would you do that?"
I didn't stop and try to reason it out on my own, but then I clicked and saw the answer, and said "heh. cute." So there you go- enough people felt the same way I did, that the voting system moved this to the 4th link on the top page. /shrug. That's the internet for you!
How about a while true loop with cases in it on a state variable.
goto=42
while true ; do
case $goto in
1 )
code for label 1
goes here
break # finish
;;
42 )
code for label 42
goes here
goto=1
;;
esac
done
You don't have the syntactic sugar here, and there are some inconveniences, but think of the positives: this does not claim a new stack frame for each jump, let a lone a new stack frame holding a big string variable that holds most of the script source code.
That's OK. It's often regarded as bad practice to use variables and functions before they're declared in scope, and clearly that's what the author was after :)
How so? The jump function has access to the entire script source code via parameter $0. It finds whatever label is referenced, and trims the script from there, executing it. This looks like it will jump backwards and forwards. Unfortunately, it chews up stack and memory with each jump, so a loop with many iterations is a bad idea.
I recently read Knuth's paper "Structured Programming with GO TO Statements", which I highly recommend to anybody like myself who never wrote code using GO TO.
These days, we all understand why GO TO is bad. However, in this paper, Knuth does a good job at explaining ways that GO TO can be good. I found that the paper took time to get through, but was well worth the effort.
Having worked in the JavaScript-github-npm-node-environment-complex the last few years, I feel what Dijkstra had against GOTO has largely been lost on the current crop of CADTs (https://www.jwz.org/doc/cadt.html). The spaghetti is stronger than ever today.
Downvotes if you want. I've actually used GOTO before there was an option not to.
My first programming languages were Apple (and Microsoft's) BASIC, which I didn't do very much with, and the Casio BASIC (with which I did quite a bit).
So there was a time when it was the only type of flow control I understood and going to C with these newfangled looping constructs was new and interesting because I understood just how useful they were compared to the GOTO statements I had needed to structure so carefully before.
If you're implying that callbacks are the GOTO of our day, and if you aren't careful it results in code that devolves into into a horrible mess where it's hard to determine the control flow, you aren't going to hear any dissent from me.
I've had way too many days of debugging webpack/browserify/react/jquery-whatever nonsense that leads to stupid dead-end threads on stackoverflow or the issues tab in Github. When you're debugging an 8th-party connector library that allows the 3rd-7th party piles of github crap to integrate into your app, and you find yourself stuck in the goddamn issues tab on Github to find the solution to your highly specific problem, something is wrong. It's time to throw it all away and start over.
> [...]at $work, I have a particular script which takes
> several days to run, each part of which may take many
> hours [...] If a command fails, the state up to that
> point is preserved, so I just need to continue where
> that left off.
Unfortunately, the stack grows with the length of the goto chain! Each goto starts a new call to the jump function, which never returns: the embedded eval contains another goto which invokes the function again, and so on. The function and eval frames will pile up and eventually blow the shell's stack. Associated with each eval frame is a copy of an entire suffix of the script starting at the target label, too, held in the string variable which captures the output of sed.
I have just coined a name for this crude technique: "trumpolines". Under this nomenclature, the program is said to construct a trumpoline by extracting a suffix of itself beginning with the label into a local variable, and then effectively branches to it via eval.
This is vaguely reminiscent of "trampolines" in compiled languages like C, whereby machine code is constructed in local storage (on the stack) and then branched to (requiring an executable stack). GCC uses trampolines for indirection on pointers to GNU C nested functions.
Trumpolines are so named because they will trump the process's memory and stack resources.
There was a separate goto utility in early UNIX. Pretty sure you can still find the source code if you look. Meanwhile, it's mentioned here: http://www.in-ulm.de/~mascheck/bourne/
solution without using sed, grep and eval (yikes!);
does this suffice? i probably missed something obvious;
run with ./
case ${1-start} in
start)
# your script goes here...
x=100 $0 foo
;;
mid)
echo "This is not printed!"
x=101 $0 foo
;;
foo)
x=${x:-10}
echo x is $x
esac
Don't understand the "...One of the (mostly hated) features of the (mostly hated) language was that any statement required a line number...". Basic was a wonderful entry level language - user friendly and understandable. It introduced countless kids to programming. It never pretended to be some enterprise level, hardcore language and it was certainly more readable than the code accompanying the article:
40 comments
[ 2.9 ms ] story [ 90.2 ms ] threadA couple of jobs ago, I had to write a job to download a large set of files scattered in a deep hierarchy from the web interface of an internal distributed file system. Because the directory page was non standard and didn't contain any url, I couldn't simply use wget or curl in recursive mode, but had to do some light parsing of each tree level.
My simple bash one liner quickly grew in a larger script. Of course, this being bash, it was trivial to make it do the scraping and downloading in parallel, but I needed a way to limit the maximum number of concurrent downloads.
To make a long story short, I ended up reimplementing semaphores in bash on top of message passing via pipes and 'coproc' (which I had just found out).
In retrospect I could have probably implemented something simpler and more robust on top gmake (which interestingly also uses pipes to limit the number of subprocesses but doesn't need a supervisor process).
This morning I was scraping the images from the 2016 report and given they were like ~230 images I thought it could be cool to download them in parallel, but I quickly realized this would lead to such path and downloaded these in serial.
I still wonder what's the _right way_
http://make.mad-scientist.net/papers/jobserver-implementatio...
""" Rather than using complex schemes such as a central server, shared memory, sockets, etc. for synchronizing the various instances of make, he suggested we use the simplest, most basic UNIX construct: a pipe. Not only is this easy, but it’s also a relatively portable concept. Most operating systems these days implement some sort of pipe feature, because that paradigm is so useful.
The idea is ingeniously simple: the initial, top-level make creates a pipe and writes N one-byte tokens into the pipe. That pipe is shared between that make and all submakes, and any time any make wants to run a job it first has to read a token from the pipe. Once the job is complete, it writes the token back to the pipe. Since there are only N tokens, you know that you will never invoke more than N jobs. """
Unfortunately I didn't know how make -j was implemented at that time.
So, is it just me that keep refining bash 'one liners' well past the point they should be implemented on a real language?
I didn't stop and try to reason it out on my own, but then I clicked and saw the answer, and said "heh. cute." So there you go- enough people felt the same way I did, that the voting system moved this to the 4th link on the top page. /shrug. That's the internet for you!
EDIT: /me sighs, this isn't the one I was looking for but it'll do for now. http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html / http://www.chiark.greenend.org.uk/~sgtatham/coroutines.h
"Prepare to cringe."
It's a dirty hack, this is Hacker News.
These days, we all understand why GO TO is bad. However, in this paper, Knuth does a good job at explaining ways that GO TO can be good. I found that the paper took time to get through, but was well worth the effort.
Here is a link to the paper: http://pic.plover.com/knuth-GOTO.pdf
Having worked in the JavaScript-github-npm-node-environment-complex the last few years, I feel what Dijkstra had against GOTO has largely been lost on the current crop of CADTs (https://www.jwz.org/doc/cadt.html). The spaghetti is stronger than ever today.
Downvotes if you want. I've actually used GOTO before there was an option not to.
So there was a time when it was the only type of flow control I understood and going to C with these newfangled looping constructs was new and interesting because I understood just how useful they were compared to the GOTO statements I had needed to structure so carefully before.
I've had way too many days of debugging webpack/browserify/react/jquery-whatever nonsense that leads to stupid dead-end threads on stackoverflow or the issues tab in Github. When you're debugging an 8th-party connector library that allows the 3rd-7th party piles of github crap to integrate into your app, and you find yourself stuck in the goddamn issues tab on Github to find the solution to your highly specific problem, something is wrong. It's time to throw it all away and start over.
I find it pretty ironic he calls HN users "man-children" by redirecting us to this image, but oh well.
I think it would be pretty amusing if HN automatically proxied links to his site through norefer.com or similar.
http://repository.readscheme.org/ftp/papers/ai-lab-pubs/AIM-...
Yikes.
I have just coined a name for this crude technique: "trumpolines". Under this nomenclature, the program is said to construct a trumpoline by extracting a suffix of itself beginning with the label into a local variable, and then effectively branches to it via eval.
This is vaguely reminiscent of "trampolines" in compiled languages like C, whereby machine code is constructed in local storage (on the stack) and then branched to (requiring an executable stack). GCC uses trampolines for indirection on pointers to GNU C nested functions.
Trumpolines are so named because they will trump the process's memory and stack resources.
cmd=$(sed -n "/$label:/{:a;n;p;ba};" $0 | grep -v ':$')
To see for yourself: run a non-trivial batch script in cmd.exe with plenty of gotos and check out all the CreateFile calls in [Process Explorer](https://technet.microsoft.com/en-us/sysinternals/processexpl...).