> How to run shell commands in parallel while limiting their concurrency.
Depending upon the task, generate a Makefile and run it with:
make -j <concurrency>
> My original use case had to do with making a large number of web requests
This is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:
Yeah, and all those concurrency controls of the shell suck. Here's a snippet to run N commands in parallel, wait for all of them, while supporting interrupting them with simple Ctrl+C:
(
trap 'kill 0' SIGINT;
for i in {1..$N}
do
whatever_command $i &
done
wait
)
The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT. No wonder people keep re-inventing PID files, manually tracking $! and all kinds of other crazy bullshit that work 90% of the time, but sometimes fail to properly terminate the runaway children.
> The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT.
This is a gross misunderstanding of what is going on (signals are never "passed through", the process is either in the same group or it isn't), but the real issue here is the confusing semantics of bash's job controls. I personally would have swapped the semantics of { command blocks } and ( subshells ) too, but bash is pushing 40 so maybe it made more sense then.
It will sorta behave properly if you disable job control or put it in a script:
set +m
{
trap 'kill 0' SIGINT;
for i in {1..$N}
do
whatever_command $i &
done
wait
}
it might also kill your current shell too though. This really seems to boil down to not understanding the difference between { } and ( ) in bash.
> signals are never "passed through", the process is either in the same group or it isn't
Yes, I know. It's actually a major nuisance because sometimes you do want to pass the signals through, e.g. to implement termination of all transitive children processes, process groups notwithstanding, so then you have to do it yourself.
> it might also kill your current shell too though.
It absolutely will kill your current shell, but only if it's non-interactive — so that's a funny way to have a script that misbehaves when ran non-interactively, but works as intended when you try to debug it. So, you either need a subshell (and then the subshell having a fresh process group makes it mostly work on its own), or fake it with "trap 'jobs -p | xargs -n 1 kill -SIGINT' SIGINT" bullshit or the like (for all the signals you can remember). The subshell way is somewhat more robust.
> It's actually a major nuisance because sometimes you do want to pass the signals through, e.g. to implement termination of all transitive children processes, process groups notwithstanding, so then you have to do it yourself.
Yes, I would prefer some form of structured concurrency or even just better defaults. Plan9 does it a bit better with rfork and having note groups be its own thing.
> The subshell way is somewhat more robust
Oh it totally is, just trying to illustrate that technically the subshell isn't necessary
I shamelessly use concurrency in Bash shells but typically I do it because I already have a Bash shell and it's now outgrowing its purpose and becoming to slow (say because I begin to use on too much data).
So instead of rewriting everything to another language, I add concurrency to that already existing shell scripts.
xargs / max args / forking to "bash -c" etc.
It's tricky. There are gotchas (with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc.).
But it works and it's a cheap way to greatly speed things up.
Also that's the kind of stuff that became not just a bit but much easier now: just give a shell script to a LLM and ask it to add concurrency where needed.
> just give a shell script to a LLM and ask it to add concurrency where needed.
You know, I personally consider the whole LLM revolution a sort of indictment of our whole industry up to this point. We had more than thirty years to figure out and make robust, composable, reusable building blocks, that would not require fiddling and filing away the sharp edges and writing megabytes of gluing boilerplate code — and we failed. Miserably. So now LLMs come in and finally give us the reusability in a very brute-forced kind of way: by being able to regurgitate those pieces of software, slightly adjusted to the context as needed.
So. In a sane world, the shell would already have a bloody "par" builtin that would deal "with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc." out of the box, without additional fuss — but we don't. Well, we do have GNU parallel, but frankly, almost nobody knows about, or gives a damn, about anything that's beyond the GNU coreutils. Like, have you heard of e.g. GNU stow? Apparently it's been a thing for decades.
Anyway. We're not living in the sane world, so instead of a simple, robust, widely available built-in tool for parallel execution, we have dozens of little bespoke homegrown systems, cobbled together from xargs, jobs, pid- and lock-files lying around, and goodness knows what else, all of them mostly working 95% of the time but subtly broken in different subtle ways 5% of the time... and so the advent of LLMs which you could just ask "hey, here's the shell script, just parallelize it", and then get the working output in a minute, was a godsend. Of course it was! But that's because most of our software is barely fit for human use and consumption.
> How to run shell commands in parallel while limiting their concurrency.
Depending upon the task, generate a Makefile and run it with:
> My original use case had to do with making a large number of web requestsThis is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:
Or, to emulate one of the examples in the blog post: Run with: > You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.
There is also GNU Parallel's sem: https://www.gnu.org/software/parallel/sem.html
I really like pueue for this. Like a light weight slurm or parallel without the nagging citation notice.
https://github.com/Nukesor/pueue
I guess that works.
I've always been fond of using make for things to get free parallelism / ability to interrupt / restart / incremental updates.
> language with poor concurrency controls
Few other languages have as many builtins/keywords for concurrency control: &, wait, jobs ...
Yeah, and all those concurrency controls of the shell suck. Here's a snippet to run N commands in parallel, wait for all of them, while supporting interrupting them with simple Ctrl+C:
The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT. No wonder people keep re-inventing PID files, manually tracking $! and all kinds of other crazy bullshit that work 90% of the time, but sometimes fail to properly terminate the runaway children.> The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT.
This is a gross misunderstanding of what is going on (signals are never "passed through", the process is either in the same group or it isn't), but the real issue here is the confusing semantics of bash's job controls. I personally would have swapped the semantics of { command blocks } and ( subshells ) too, but bash is pushing 40 so maybe it made more sense then.
It will sorta behave properly if you disable job control or put it in a script:
it might also kill your current shell too though. This really seems to boil down to not understanding the difference between { } and ( ) in bash.> signals are never "passed through", the process is either in the same group or it isn't
Yes, I know. It's actually a major nuisance because sometimes you do want to pass the signals through, e.g. to implement termination of all transitive children processes, process groups notwithstanding, so then you have to do it yourself.
> it might also kill your current shell too though.
It absolutely will kill your current shell, but only if it's non-interactive — so that's a funny way to have a script that misbehaves when ran non-interactively, but works as intended when you try to debug it. So, you either need a subshell (and then the subshell having a fresh process group makes it mostly work on its own), or fake it with "trap 'jobs -p | xargs -n 1 kill -SIGINT' SIGINT" bullshit or the like (for all the signals you can remember). The subshell way is somewhat more robust.
> It's actually a major nuisance because sometimes you do want to pass the signals through, e.g. to implement termination of all transitive children processes, process groups notwithstanding, so then you have to do it yourself.
Yes, I would prefer some form of structured concurrency or even just better defaults. Plan9 does it a bit better with rfork and having note groups be its own thing.
> The subshell way is somewhat more robust
Oh it totally is, just trying to illustrate that technically the subshell isn't necessary
perhaps that's more an indictment of most programming languages
What about using GNU Parallel ?
"If you know all commands that you want to execute up-front, then the solution gets far simpler":
https://emmer.dev/blog/concurrent-shell-command-queues/#why-...
What prevents you from feeding your `xargs -P# sh -c` with a FIFO kept open by a dummy reader-writer?
Or plain xargs...
I shamelessly use concurrency in Bash shells but typically I do it because I already have a Bash shell and it's now outgrowing its purpose and becoming to slow (say because I begin to use on too much data).
So instead of rewriting everything to another language, I add concurrency to that already existing shell scripts.
xargs / max args / forking to "bash -c" etc.
It's tricky. There are gotchas (with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc.).
But it works and it's a cheap way to greatly speed things up.
Also that's the kind of stuff that became not just a bit but much easier now: just give a shell script to a LLM and ask it to add concurrency where needed.
> just give a shell script to a LLM and ask it to add concurrency where needed.
You know, I personally consider the whole LLM revolution a sort of indictment of our whole industry up to this point. We had more than thirty years to figure out and make robust, composable, reusable building blocks, that would not require fiddling and filing away the sharp edges and writing megabytes of gluing boilerplate code — and we failed. Miserably. So now LLMs come in and finally give us the reusability in a very brute-forced kind of way: by being able to regurgitate those pieces of software, slightly adjusted to the context as needed.
So. In a sane world, the shell would already have a bloody "par" builtin that would deal "with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc." out of the box, without additional fuss — but we don't. Well, we do have GNU parallel, but frankly, almost nobody knows about, or gives a damn, about anything that's beyond the GNU coreutils. Like, have you heard of e.g. GNU stow? Apparently it's been a thing for decades.
Anyway. We're not living in the sane world, so instead of a simple, robust, widely available built-in tool for parallel execution, we have dozens of little bespoke homegrown systems, cobbled together from xargs, jobs, pid- and lock-files lying around, and goodness knows what else, all of them mostly working 95% of the time but subtly broken in different subtle ways 5% of the time... and so the advent of LLMs which you could just ask "hey, here's the shell script, just parallelize it", and then get the working output in a minute, was a godsend. Of course it was! But that's because most of our software is barely fit for human use and consumption.