Ask HN: Why is concurrent programming so hard ?
Lots harder than the simple relation between the number of threads of execution and a single thread should make you suspect.
Unless your problem is one that is easily parallelized you are left with limited alternatives.
Usually this is put down to either architectural issues (von Neumann vs alternatives), sometimes to the difficulties of locking and such, most times to how easy a program can be parallelized.
But even the latter is an effort on the part of the programmer, rarely of the environment the program is written in (though there are some interesting developments in that area).
I have this weird theory that it has to do with something completely different:
Our brains are not wired to have 'multiple parallel threads' of consciousness.
So we have a hard time envisioning in our sequentially wired brains what is going on in a parallel program, which means we feel much more 'at home' in a sequential environment.
It takes a good bit of training to get around that limitation and even then it is not easy.
So we tend to 'break up' our programs into sequential pieces of code, each running in parallel with the rest, and the spots where they meet, like trains using a railroad track, have to be scheduled to make things execute in the proper sequence.
This makes me wonder if we are truly capable of devising a method of parallel execution that will feel 'natural'.
Even in the real world, when we 'concentrate' we single out a thread from all the possible inputs and go step-by-step to analyze that one single process.
What are your thoughts on this ?
12 comments
[ 4.0 ms ] story [ 104 ms ] threadDead on. We've been working on single threads for SO long and then just adapted that onto multi-threaded architecture now that we have to scale outwards.
I spent the last year of university writing a research project (final year project) trying to come up with a viable form of concurrent programming that didnt just look like a fancy way to multi thread.
But no luck :D
The best I can think of is to create some rock solid standard and an underlying architecture. With the idea being you can write your basic code and the compiler, vm, interpreter does the parallel work for you. But that's basically the same thing and not really solving the problem :)
It's like learning formulas for 2D geometry and then trying to extrapolate to 3D formulas, such as the sum of all the sides of a square vs a cube. If we learned about generalized formulas in n-D, we could easily apply to any dimension at all. But it's harder to think so abstractly in the beginning.
Same thing here, now you're trying to think of algorithms that work with n threads instead of just one.
Whenever you need access to information you don't have available, there's a barrier before you can continue that task: maybe you have to go look for somebody, or send a message and wait for a response, or buy a book.
While these dependencies exist, you have some options. You could decide to work on something else; you could stop everything; or you could speculate on what the answer is likely to be, proceed as if that is the correct course, and expect to backtrack if you're wrong. To learn from the experience and not waste as much time in the future, you can remember key facts.
Like any dependency graph, it can explode as more points are involved. If you have many people (or tasks), there's a greater risk that many of them will essentially be unproductive while waiting directly or indirectly for something else. The challenge is to keep everybody in tune and to give them enough to work on so that they're never really wasting time.
This is one reason that languages good for concurrency tend toward functional programming or other more-declarative paradigms. By eliminating observable state mutation, you can write concurrent programs that are still guaranteed to be deterministic.
Concurrent programming isn't hard, debugging concurrent programming is hard. Eliminate side effects, use Software Transactional Memory, and figure out how easy concurrent programming _can_ be.
obvious Clojure plug
I thank my lucky stars every day that I go to a school where the first thing you learn is Scheme. Thanks, Matthias!
This intrigued me to read the original: First Draft Report on the EDVAC, June 30, 1945. Here are some interesting insights: Pp. 16-19
"5.3 At this point there arises another question of principle. In all existing devices where the element is not a vacuum tube the reaction time of the element is sufficiently long to make a certain telescoping of the steps involved in addition, subtraction, and still more in multiplication and division, desirable.To take a specific case consider binary multiplication. A reasonable precision for many differential equation problems is given by carrying 8 significant decimal digits, that is by keeping the relative rounding-of errors below 10exp(−8). This corresponds to 2exp(−27) in the binary system, that is to carrying 27 significant binary digits. Hence a multiplication consists of pairing each one of 27 multiplicand digits with each one of 27 multiplier digits, and forming product digits 0 and 1 accordingly, and then positioning and combining them. These are essentially 272 = 729 steps, and the operations of collecting and combining may about double their number. So 1000-1500 steps are essentially right."
Point 5.3 is essentially a parallel computing but von Neumann abandoned this idea. This document is worth reading in original.
I posted some information and link to the Report here:
http://blog.thstart.com/2009/07/first-draft-report-on-edvac-...
Consider driving. Your brain is processing millions of things in an out-of-order manner, and producing continuous motions in your hands and feet that guide the car so perfectly that few would be surprised to hear of someone driving for a lifetime without a major accident.
Same deal with many concurrent activities, like having a conversation, or playing sports, or listening to music, writing, thinking about the future, or fixing the fridge. In every case, we set our massively parallel architecture on many problems simultaneously, and then integrate a solution in real time without deadlocks or timeouts.
Humans have a corresponding weakness for sequential thought. We find it very hard to write a piece of sequential software that lasts more than a week without crashing hard or suffering from a major design flaw. We need to be taught for decades in the most advanced institutions to think sequentially and logically. Only the most elite of us ever manage to produce something that others would actually pay for, and even then we may solve a different problem for people than the one we set out to solve. (Think of startups like twitter that find that the market likes them, but for completely different reason than what they thought.)
Then we try to make people trained in sequential thought solve parallel problems, and they come up with steamy piles of tangled mess, full of deadlocks, races, and side-effects.
Our DEFAULT state is concurrent thought. We've all been trained to think otherwise, by our computers, languages, and education. We're hard on the poor folks who were carefully trained as sequentialists when they fail to make a parallel problem easy.
I work on a compiler for sequential source at a major corp by day, but my personal long-term goal is to create a compiler for a language that can form the kernel for a new school of thought separate from the sequentialists and their half-measures towards solving this problem. As hardware evolves, it will only become more like our own brains, and less like calculators and simple programmable machines.
When you learn something it is conscious. Every bit of the activity that you are learning is new and requires lots of hard work. But at some points you have 'mastered' the skill, and you can do it 'without thinking', which is really just another way of saying that your unconscious has enough information to do the thinking without direction.
For instance, most of the people reading this are probably good typists (to me that's a part of 'hacking', the sound a typewriter keyboard would make when you 'hack' out a piece of text). All of us had to learn that skill at some point, one conscious and slow stroke at the time. But right now I'm typing this just a little slower than I can think it, or maybe even faster in spurts, depending on how frequently I've used those patterns.
If I tried to consciously think of every letter that I'm typing immediately I slow down to a crawl.
So we're good at parallel thought, no doubt about that, we multitask like we're made for it. But we have but one state of consciousness and we can not fork that even for the simplest of tasks.
Take chess players that can play on many boards. That's as close to human 'multitasking' as you can get when making a computer analogy. The state of each board is the thread of execution, the consciousness is doing 'multiple-instruction-single-data' across each board. On a 'task switch' (move to another player) the state gets reloaded and another move is produced.
If you could consciously think multiple threads then it should be much easier to play on many boards at the same time.
I wonder how well octopuses would perform at that task...