That quote (and you misquoted, btw) is from an interview discussing his working relationship with Dennis Ritchie. The particular quote is from a section of the article titled "On Collaborating With Dennis Ritchie"
"Dr. Dobb's: Was there any concept of looking at each other's code or doing code reviews?
Thompson: [Shaking head] We were all pretty good coders."
Nothing to do with Rob Pike, who has worked with Ken for decades since.
Ken Thompson was replying to a question about code reviews there. Code reviewing is not quite the same as pair programming (and they probably didn't even call it pair programming then).
Your link is discussing the development of Unix, which was before Rob Pike came on the scene. Rob Pike is discussing an incident that happened a number of years later when Rob would have been significantly junior to Ken.
There is no reason that both versions of history can't be true.
Don't kid yourself. Systems may be larger and seemingly more complex, the individual parts are as simple or complex as they ever were. Anyway, if things are more complex the necessity of thinking it through are even greater.
You forget the fact that Go is a young language while modern C implementation has to support several standards, countless extensions and probably decades of cruft in the codebase. Additionally, if you factor in that go includes a tool that replaces most of what autotools gdoes in the C world, we're talking about a huge blob of accidentally complex code.
It's very reasonable to think that the Go implementation is simpler.
I'm not forgetting. The make tools are not part if the language and irrelevant. Most C developers I know avoid extensions for portability reasons. The cruft part doesn't matter much for this discussion either, you can start from scratch if you want like TCC. The language definition of Go requires some things that C does not for any reasonable implementation. I think it is much more reasonable to think a C implementation will be much simpler.
Simple compared to what? The Go implementation is very immature, I'll grant you that, but it is definitely more complex than an implementation of C at a similar point in its life. At the very least Go supports, fairly straight forwardly, a lot of C and it has a GC and multiplexing user land threads over multiple cores.
Ahh, if we are comparing equivalent implementations, you are definitely correct. GC and goroutines are much more complex than anything in C. In fact, off the top of my head, the only features truly missing in Go from C are unions (unfortunately) and the preprocessor (effectively a part of C).
There is a big difference between implementing them for fun and implementing them for a production system. Regardless, even if they are simple to implement they are more than what C90 has so, by definition, more complex implementation.
I think if you are just pushing through (brute force) to fix a bug "without thinking" then you will never figure it out. Also, "thinking before" as the article claims, without actually looking at the code or stack traces, that's just mental brute force.
Pretty much it's a combination of both: you look at the code, you think about what's happening and what could go wrong, you look at the code again, you think some more, you look at stacks, variables, output.. and then you think again and BOOM: you figure it out.
Actually, I think it's a bit more subtle than that. By using stack traces and debuggers etc, I can come up with a working and correct solution. But, by thinking through the problem at a deeper level I can come up with an improved design which may simplify future programming and avoid similar bugs.
I was reacting to the idea that instead of fixing the immediate bug you just found you're going to redesign the whole thing to prevent that type of bug from occurring ever again in the future.
You are only so smart. Once the complexity of the programming model reaches a certain point a debugger is necessary to validate and discover the true nature of a system. Often that point is quite low.
>>You are only so smart. Once the complexity of the programming model reaches a certain point a debugger is necessary to validate and discover the true nature of a system.
That is true but if you rely only on the tools, you will never gain a smarter understanding of the systems you are working with.
I disagree, you'd be surprised how many people have the mental model of various large code bases (think Linux kernel) in their minds. It's not as if you are some savant memorizing all the lines of code and it helps to differentiate between "complexity" and "a mess." :)
Linus refused to merge the kernel debugger patches for a very long time because he felt that if a bug could not be solved by thinking (and logging), then the code was too complicated.
I've found that liberal logging and careful error management has almost replaced the debugger entirely. In the last three months, I've only fired up the debugger twice, and that was when I was working with untyped memory and peeking at memory through the debugger was the most efficient way to get things done.
I'm also one to use logging and error management to help my way through a program. I don't understand how anyone else does it any other way. But surely the debugger helps a lot when no other way to reason about your code exists.
I have a lot of problems explaining this to the people I work with. Most of the errors we make are in some way related to having imperfect information through the debugger.
At some point complexity gets so high the debugger isn't even capable of handling the system under inspection. What are the tools we use in these cases?
One should always try to write well modularized code with lots of good abstraction barriers so that you only ever have to have a fraction of the complexity of the system in your head at a time. If you had to understand everything down to the transistor level you could never even understand a Hello World program. That's because the language you use gives you good abstractions so you don't have to worry about linking, operating systems, memory allocation, etc. When writing complex systems you should strive to create such good abstractions yourself.
This is my method of debugging. I tend to think through code for a long time away from a keyboard before writing it, trying to understand corner cases, implications, and so on -- developing a mental model. Well, I wouldn't do this for everything, more for core aspects of a complex/somewhat-complex system.
I think this might be an advantage of pair programming that isn't generally talked about.
The time to find a bug with a debugger has a much tighter distribution than the time to find one by thinking about the code. There are some problems that you just can't get to the bottom of by thinking.
By having one person take each path, you get the advantages of both. It might seem that a single programmer should be able to achieve the same by switching between the two approaches, but the cost of context switching is so great that it will be more like starting again each time. So it's really hard to know when to stop thinking about it and to get the debugger out.
The advice that once you find the bug you should work out what the problem really is definitely holds solid, though.
A professor of mine who worked at Bell Labs once made the same point. "In the old days we had to think a lot about how our punch card program worked because we'd only find out if it worked the next day. Nowadays you guys just throw crap at the wall and see what sticks. Find the middle ground."
Same with the former French department store programmer who taught us Pascal in high school - punch cards have shaped that generation. That mindset still exists in industries with physical processes, but "measure twice, cut once" is wisdom lost to the desktop generation - experimenting can replace some thinking... But we certainly overestimate how much.
I agree there's been an overall shift in styles, but instant-feedback aspects of programming also have a pretty long pedigree, in the form of Lisp's REPL.
I don't think in code. My mental model is the process, not the series of functions and objects that make up the process. When the process breaks, I'll dig around to see what code makes up that step of the process.
That's one of the less obvious (to me, at least) benefits of test driven development: When you're writing out your unit test, you're forced to think about how the implementation is going to work.
I had the opposite reaction: I wonder if tests make it easier for you to fix code without forcing you to develop a mental model of it, assuming you're working in an unfamiliar codebase. That seems like something of a hidden drawback.
That may be possible, but it isn't inevitable. I use tests to validate that my mental model is correct. When I'm doing something greenfield, you'll see my tests are full of rather stupid-looking assertions of really basic stuff, and the reason for that is that about 5% of the time, my really basic so-simple-it-couldn't-be-wrong is wrong.
If you're building something that's going to be used as a foundation by lots of other things, those 5% errors add up really fast.
It does help in some sense, but not always. I found that, if i wrote out test cases like i am preparing a test scenario document for someone in plain English it works. If i have to open vim and write test cases, i seem to the hack mode and write out the most trivial cases, causing painfully slow development. Test Document + thinking/visualization works better for me.
I've found this as well. Just blindly writing test cases doesn't work so well unless you've already understood the higher level operation of what you're trying to build, and obviously does tend to slow down development.
How do you find the middle ground? My hypothesis is that one can write the tests first and use them as a compass. If debugging proceeds monotonically, you have thought enough. If you fix the bug revealed by test r, but later when you fix the bug revealed by test s, test r starts failing again, that is a clue that you didn't think enough.
What the clue means will depend on the order of the tests. If the tests were written in order of increasing code coverage it is probably a clue that the algorithm needs more thought, but it could be a clue that one hasn't thought enough about test coverage and has ones tests in an unhelpful order.
That's why it's very hard to juggle multiple projects at the same time. Every time you switch projects, you have to flush the old mental models of the old project out and reload the mental models of the new project.
Same thing with interruption, it takes a while to rebuild the mental model after being interrupted.
Ah, Yes. I use to have a strong mental model of our security infrastructure and could troubleshoot problems by just hearing the description of the issues and a couple of variables. I would literally see visual gaps where my knowledge of the system stopped if I was drawn there. I would then fill in those gaps.
Now as a developer I haven't worked on a model long enough (my excuse) to be able to do that. But now I realize I already have a process for identifying gaps that I don't use.
I work with a guy who can do this and recently he has amazed me with how he can do it. Contrary to this though, sometimes I think he works in outdated models that limit his creativity.
I am beginning to realize this recently at my new job. Due to a couple of people leaving and stuff like that, i got handed a reportedly(half-written) code base with no test cases. And a knowledge transfer document, that was mainly written by a poor guy who had gotten the code(before me) and left the project in 3 weeks. In any case, couple of months into the project i got handed a different requirement. Guess what, a total of 10 months after i took the job the second tool is done and is in testing while i am still working on the first one. But the main problem so far has been, lack of clear visualisation of the requirements beforehand. Documenting a project specifications (functional or technical) is unheard of here it seems. I swear taking the time to gather all the requirements would have halved the time i spent developing,demoing, getting feedback,correcting/fixing suggestions etc.. sigh
Sometimes visual but more often I imagine the code physically. It has weight, or friction, or rigidity, depending on what aspects of the code I'm trying to think about.
Interesting. I should try using that. That is mechanical models. I have come across it(using a mechanical device analogy/metaphor for thinking about something) in other fields(http://www.ribbonfarm.com/2010/06/30/the-philosophers-abacus...).But never really used it in code/design consciously so far.
I am a logical thinker and good at creating sensible (that is high-level or very granular dependent on importance) abstractions in my head. Once this is done, it's easy to think "this abstract data gets computed by this function which passes its output to this function, etc."
The actual solution will then appear from (0) the association of an (erroneous) result with a particular piece of data/behaviour, (1) strictly following through this model in my head, (2) a holistic understanding of the flow of data and the computational roles inside the system (or external to it in the case of end-users), or (3) a whim to subvert and reframe the question/problem in interesting ways. As Carl Jacob's said: "Invert, always invert."
Generally you progress from (0) to (2) before taking out a debugger. Most easy problems are solved at (0). (3) is unusual but intellectually gratifying if you find a way of solving a problem in a way that could not be understood simply through using a debugger and hacking a couple of lines of code.
Very. It's just how my mind puts things together, not something I can try to do or not do. I tend to see programs in blocks and patterns interacting rather than sequences of logical operations, probably why I tend to prefer the object-oriented paradigm over others. I get bogged down when I can't visualize the interactions. It isn't just programming, either -- working with derivative products in finance I formed very detailed visual mental models of how they operated.
This reminds me an episode in "Surely You're Joking, Mr. Feynman!" where he fixes a noisy radio just by trying to understand how can the problem happen and comes to conclusion that the most likely cause is that amp's vacuum tubes heat first and generate a lot of noise before the rest of the circuit is ready.
He swaps tubes, problem is gone and the owner of the radio (which was very sceptical at first) goes around telling about this twelve years old boy "he fixes radios by thinking!".
The power of thought is strange. I happen to have this ability as well. That's mainly why I'm still employed.
A couple of years ago, we originally had a 4 man team. 3 of those guys left, leaving me (the junior dev) to deal with the whole platform.
I didn't have any time for error, but I didn't know a darn thing. I did, however, had the ability of the "hunch". Basically, when a problem happened, instead of opening the text editor and looking for the problem, I just stood back and thought about it. I'm not sure what I'm thinking about. Its strange. Its like my brain just traverses through the infrastructure and comes up with the LOGICAL solution. Once I get that answer, I then go look at the code base. 99% of the time, its my hunch that finds the problem.
I amaze myself all the times with this. To me its just logic, but I'm sure its something more. I'll probably never figure it out, but that's not to my detriment.
This also helps with writing new applications, because you already know what will work and what won't before you get there. Its, craziness!!
Me too. I used to really get frustrated when others couldn't see what I thought were obvious answers looking at a situation. Now I've come to sincerely believe there's something strange in me that allows me to see these things because when I've tried to take the Socratic method to get others to see what I see in various situations, I realize there are a lot of people who actually can't. Some people can get it via experience, but I've been able to do the same thing in areas or situations where I have zero experience (and now I'm talking about life, not just technical stuff).
I don't want to get on a high horse and say I have superpowers or something like that. But for some reason I'm able to do something that others can't, even though I think it's simply simple logic. It's sometimes a scary thing, because when I'm wrong, I have a difficult time figuring out why I'm wrong if someone else isn't able to do the same thing and "see" ahead.
Sometimes it seems to me that "seeing" is a better word than "thinking" here, because for the simpler things, it's not like I try to use brain cycles on the issues at hand.
And I hate saying this stuff, because it makes me sound arrogant when I know that I'm really incompetent at so many things.
It's sad to say, but the majority of people cannot follow a logical progression more than a two or three steps. It's not even an issue of "smart" vs "dumb", as many of my quite clever coworkers have this issue. It seems to me to be predicated on the ability to keep a lot of state in your head. If in order to reach an answer you need to synthesize the results of two logical conclusions and you can't keep the first result in your head while you work on the second, you're going to get lost.
> It seems to me to be predicated on the ability to keep a lot of state in your head.
There's some research and discussion among educators and neuroscientists about how working memory may be more important than IQ. IANAE but my anecdotal experience also supports this.
I amaze myself all the times with this. To me its just logic, but I'm sure its something more. I'll probably never figure it out, but that's not to my detriment.
I vaguely recall a story that my grandfather used to be very good at doing math in his head, and actually lost this skill when he tried to understand his own thought process well enough to explain it to others...
I think a lot of engineers get to this point through different methods.
For me when I was new engineer (no CS background) I'd never spend too much time on a bug before I got frustrated and would head over to a more senior engineer's cube for advice. Usually during the walk over while I tried to properly form the exact question to ask I'd figure out the answer. Dan (my go to guy for this) got used to seeing me head his way then just say hi, he'd ask "Figured it out?" and I'd say "Yep" and head back to my desk.
Thanks for the advice. I've always had a feeling that this was the right approach, yet I was usually too lazy to actually do it. Seeing Rob Pike coming to this conclusion is a good motivation. I've also noticed that the most persistent and hardest to solve bugs came either from logical errors in my mental model or its implementation.
If you're ‘just a’ developer in a larger company, you may rarely get to actually fix these high-level problems you discover. You kind of have to live with it, which is frustrating.
And if you're working alone, too ‘high-level’ thinking may well slow down getting things done. If you don't have around someone with more pragmatic attitude, be sure to have a bit of it yourself. =)
That's why most people 'here' are probably not 'just a dev' at a larger company. There is a lot frustrating about that position.
I agree with your second remark; you need pragmatism, but yes, as I get older I notice that I solve bugs in my head instead of debugging in the usual fashion. I strive to type as little as possible and to do that you need to do a lot of head work; 20 years ago, I was the opposite.
I don't understand the lesson to be about "high-level problems", but about not seeing the forest for all the trees. A debugger can give an invaluably direct view of what is happening, but it can allow be a very narrow view.
But the debugger is not the problem - in fact, when you don't know the code perfectly, a debugger can be very useful in enabling you to form a sufficiently complete model of the code in the first place.
It only becomes a problem if, at that point, you keep looking at details rather than stopping to think about what you've learned.
Can't agree more. Infact at my current company working alone has led me to slow down on getting stuff done. my solution? Write obsessively, i.e to the extent of opening up 750words.com and putting down whatever thoughts interrupt work even if it is at 30 minute intervals.
Didn't realise either of those points. I guess that adequately explains my wondering. What's amusing now are the insecure defensive posts and the downvoting.
Agreed - we'd probably all be most productive working in a language we'd designed ourselves, irrespective of the merits or demerits of that language for other programmers.
Yeah, but not everybody designs a language. Right?
[1]The usage of a language directly depends on comfort, productivity and efficiency.
Maybe ''Pike's language'' was a factor, but all the above relations[1] come true in wider light.
I'm not bashing Go here, just to clarify - I'm just saying that given Rob Pike's deep knowledge of the language and its libraries, and his heavy design input to it, it doesn't mean much that he's productive in it (well, besides the implied statement that he thinks it's not a toy language any more).
Toyota's '5 Whys' root cause analysis, don't just fix the manifestation of the problem, but keep asking why it happened until you get to the real cause.
When they find a bug, they go to great lengths to find how and where in their process of programming this was able to happen, sometimes finding other bugs before they surface.
As he says it's indeed a matter of style/preference.
I've personally spent 10 years in academia deeply understanding many things that more often than not were good for nothing. Now I'm OK with just getting things to work quickly and not looking back.
Good advice - after I noticed that I solved most of my tough debugging problems on the walk home from work, I started going outside to take a walk around the building whenever I got stuck. Removing yourself from immediate access to the code lets you think at a higher level how it's organized and what could go wrong.
i've recently had to deal with similar situation, accept the person that refactored my code didn't make such a big improvement. instead he went on vacation and a lot of things broke when we released the code. even though i've looked at all his check-ins, and they looked harmless, i was not able to foresee the problems we experienced when this code run in prod..
i'm conflicted because on one hand i don't want to be the code police that simply refuses any changes from others, but at the same time its hard to be responsible for a system when so many core updates are done by other team mates w/out proper testing.
i think the happy middle ground is: no such changes in common trunk. Do those in a branch, and only merge them in when benefits are clear, team is on board with all changes, you're ready to release to production, and will be around during the release.
Always something to be said for running the code through the compiler in your head. Nowadays you need a virtual distributed environment in your head to emulate the design.
141 comments
[ 2.9 ms ] story [ 161 ms ] thread"Dr. Dobb's: Was there any concept of looking at each other's code or doing code reviews?
Thompson: [Shaking head] We were all pretty good coders."
Nothing to do with Rob Pike, who has worked with Ken for decades since.
There is no reason that both versions of history can't be true.
Unix was also much simpler than Multics.
Software doesn't need to always grow in complexity, but it takes lots of determination to accomplish this.
It's very reasonable to think that the Go implementation is simpler.
"The make tools are not part if[sic] the language and irrelevant"
No, they're quite a lot more than just make replacement. And definitely not irrelevant!
* Assume C90
Pretty much it's a combination of both: you look at the code, you think about what's happening and what could go wrong, you look at the code again, you think some more, you look at stacks, variables, output.. and then you think again and BOOM: you figure it out.
The best debugging tool is you. Use all you have.
Don't get me wrong, you have a point - there is certainly a balance to be struck.
That is true but if you rely only on the tools, you will never gain a smarter understanding of the systems you are working with.
I've found that liberal logging and careful error management has almost replaced the debugger entirely. In the last three months, I've only fired up the debugger twice, and that was when I was working with untyped memory and peeking at memory through the debugger was the most efficient way to get things done.
I have a lot of problems explaining this to the people I work with. Most of the errors we make are in some way related to having imperfect information through the debugger.
At some point complexity gets so high the debugger isn't even capable of handling the system under inspection. What are the tools we use in these cases?
Does the order of the two terminal conditions matter? / Try it out!
Does the order of the two previous answers matter? / Yes. Think first, then try.
- The Little Schemer
The time to find a bug with a debugger has a much tighter distribution than the time to find one by thinking about the code. There are some problems that you just can't get to the bottom of by thinking.
By having one person take each path, you get the advantages of both. It might seem that a single programmer should be able to achieve the same by switching between the two approaches, but the cost of context switching is so great that it will be more like starting again each time. So it's really hard to know when to stop thinking about it and to get the debugger out.
The advice that once you find the bug you should work out what the problem really is definitely holds solid, though.
If you're building something that's going to be used as a foundation by lots of other things, those 5% errors add up really fast.
What the clue means will depend on the order of the tests. If the tests were written in order of increasing code coverage it is probably a clue that the algorithm needs more thought, but it could be a clue that one hasn't thought enough about test coverage and has ones tests in an unhelpful order.
Same thing with interruption, it takes a while to rebuild the mental model after being interrupted.
Now as a developer I haven't worked on a model long enough (my excuse) to be able to do that. But now I realize I already have a process for identifying gaps that I don't use.
I work with a guy who can do this and recently he has amazed me with how he can do it. Contrary to this though, sometimes I think he works in outdated models that limit his creativity.
Also, I'm curious about something. Those of you who are good at building mentals models: are you also visual thinkers?
I am a logical thinker and good at creating sensible (that is high-level or very granular dependent on importance) abstractions in my head. Once this is done, it's easy to think "this abstract data gets computed by this function which passes its output to this function, etc."
The actual solution will then appear from (0) the association of an (erroneous) result with a particular piece of data/behaviour, (1) strictly following through this model in my head, (2) a holistic understanding of the flow of data and the computational roles inside the system (or external to it in the case of end-users), or (3) a whim to subvert and reframe the question/problem in interesting ways. As Carl Jacob's said: "Invert, always invert."
Generally you progress from (0) to (2) before taking out a debugger. Most easy problems are solved at (0). (3) is unusual but intellectually gratifying if you find a way of solving a problem in a way that could not be understood simply through using a debugger and hacking a couple of lines of code.
link for anyone else who's been meaning to read this: http://amzn.com/0393316041
It has both "Surely You're Joking, Mr. Feynman!" and "What Do You Care What Other People Think?" in a nice hardcover.
A couple of years ago, we originally had a 4 man team. 3 of those guys left, leaving me (the junior dev) to deal with the whole platform.
I didn't have any time for error, but I didn't know a darn thing. I did, however, had the ability of the "hunch". Basically, when a problem happened, instead of opening the text editor and looking for the problem, I just stood back and thought about it. I'm not sure what I'm thinking about. Its strange. Its like my brain just traverses through the infrastructure and comes up with the LOGICAL solution. Once I get that answer, I then go look at the code base. 99% of the time, its my hunch that finds the problem.
I amaze myself all the times with this. To me its just logic, but I'm sure its something more. I'll probably never figure it out, but that's not to my detriment.
This also helps with writing new applications, because you already know what will work and what won't before you get there. Its, craziness!!
I don't want to get on a high horse and say I have superpowers or something like that. But for some reason I'm able to do something that others can't, even though I think it's simply simple logic. It's sometimes a scary thing, because when I'm wrong, I have a difficult time figuring out why I'm wrong if someone else isn't able to do the same thing and "see" ahead.
Sometimes it seems to me that "seeing" is a better word than "thinking" here, because for the simpler things, it's not like I try to use brain cycles on the issues at hand.
And I hate saying this stuff, because it makes me sound arrogant when I know that I'm really incompetent at so many things.
There's some research and discussion among educators and neuroscientists about how working memory may be more important than IQ. IANAE but my anecdotal experience also supports this.
http://duckduckgo.com/?q=working+memory+more+important
I vaguely recall a story that my grandfather used to be very good at doing math in his head, and actually lost this skill when he tried to understand his own thought process well enough to explain it to others...
For me when I was new engineer (no CS background) I'd never spend too much time on a bug before I got frustrated and would head over to a more senior engineer's cube for advice. Usually during the walk over while I tried to properly form the exact question to ask I'd figure out the answer. Dan (my go to guy for this) got used to seeing me head his way then just say hi, he'd ask "Figured it out?" and I'd say "Yep" and head back to my desk.
If you're ‘just a’ developer in a larger company, you may rarely get to actually fix these high-level problems you discover. You kind of have to live with it, which is frustrating.
And if you're working alone, too ‘high-level’ thinking may well slow down getting things done. If you don't have around someone with more pragmatic attitude, be sure to have a bit of it yourself. =)
I agree with your second remark; you need pragmatism, but yes, as I get older I notice that I solve bugs in my head instead of debugging in the usual fashion. I strive to type as little as possible and to do that you need to do a lot of head work; 20 years ago, I was the opposite.
But the debugger is not the problem - in fact, when you don't know the code perfectly, a debugger can be very useful in enabling you to form a sufficiently complete model of the code in the first place.
It only becomes a problem if, at that point, you keep looking at details rather than stopping to think about what you've learned.
http://spinroot.com/pico/index.html
http://cm.bell-labs.com/cm/cs/who/gerard/cnn.mpg
http://cm.bell-labs.com/cm/cs/who/gerard/pixelface.mpg
....Unless maybe you understand implicitly that you're probably not as smart as Rob Pike.
Toyota's '5 Whys' root cause analysis, don't just fix the manifestation of the problem, but keep asking why it happened until you get to the real cause.
The NASA space shuttle programmers
http://www.fastcompany.com/28121/they-write-right-stuff
When they find a bug, they go to great lengths to find how and where in their process of programming this was able to happen, sometimes finding other bugs before they surface.
I've personally spent 10 years in academia deeply understanding many things that more often than not were good for nothing. Now I'm OK with just getting things to work quickly and not looking back.
PS: I'm a really fast typer.
i'm conflicted because on one hand i don't want to be the code police that simply refuses any changes from others, but at the same time its hard to be responsible for a system when so many core updates are done by other team mates w/out proper testing.
i think the happy middle ground is: no such changes in common trunk. Do those in a branch, and only merge them in when benefits are clear, team is on board with all changes, you're ready to release to production, and will be around during the release.