It's easy to swamp useful messages in tons of noise, so just logging whatever is not necessarily an improvement. Logging means code, and more code means more work, less readability, and more bugs. In particular, logging somewhere, sometime means I/O, which opens up a whole new class of bugs you'd rather avoid. For instance, I've seen several systems go down simply because the disk space ran out.
Logging is also slow - his example of finding a cheapest path is a good example of this. These are well-studied algorithms that explore huge state spaces. Adding logging as he suggests can easily mean the application's running time is spent almost entirely logging. This might not be an issue: maybe you don't need this very often, or maybe the path's aren't very complex, so you can get away with huge overhead because systems are so fast anyhow. But often as not, you can't. And even when you can, you might be adding a trivial DoS vector for your app, or you might be needlessly limiting its applicablity.
If you have an error, it's nice to log whatever you can to help understand it. But the benefit of understanding the odd error more easily needs to be weighed against the non-trivial costs in time, readability, maintainability, disk space and runtime, particularly if you're pre-emptively logging before an error ever occors.
Good points. As I added in the comments to the post, you should be able to turn the logging/tracing on or off on a running system. Furthermore, you need to be able to select what what should be logged, for example only messages to a certain mobile phone. Simply logging everything will swamp the system.
My main point was this: if you don't get the results you expected, do you know where to start looking for the problem or not? If you have no idea, then there is not enough logging/tracing present in the program.
"As I added in the comments to the post, you should be able to turn the logging/tracing on or off on a running system. Furthermore, you need to be able to select what what should be logged, for example only messages to a certain mobile phone."
There's little to gain that way. If the logging was off when the problem happened, you have no logs. You can switch it on and try to reproduce. However, if you are able to reproduce, you can as well attach the process to the debugger and check what's happening. No logging needed.
Attaching a debugger is not always an option - if you set break-points things will start timing out. Also, you can ask operations people to enable certain logging on a system, but you can't ask them to attach a debugger.
This depends very much on the type of business you're running. If you're creating software to run yourself, fair enough (as long as your debugger is sufficiently low overhead). If you're selling (and supporting) software for other people, on the other hand, being able to get them to turn on logging on a certain class and try again is a godsend.
One useful thing to do is to actually used log files to debug problems during development and testing so you check whether they are actually useful. You don't want to be looking at the structure of log files for the first time when you encounter a serious issue in production.
Logging means a I/O operation in some way or another to observe the message. If it's not the disk, it's a buffered stream with a syscall and all kinds of slow. Any other magic (using another thread to do the slow stuff, etc) risks the log message getting lost during a crash and the whole thing being useless.
But the point of logs is to put them in a place where they can be observed. That means I/O pretty much by definition. You might put those logs out to the network, where some dedicated logging machine could handle storing them someplace, but that's still I/O.
You assume that logging has happen to disk. It does not. Look at Varnish. For every request it can log as much as 400 lines. Because the logging is done to memory the cost of logging is more or less zero.
And since it is a circular buffer the memory cost is fixed. Since the memory is shared and available through the filesystem the log stays around if the application crashes.
One important aspect of logging is to include enough information when something fails. This sounds like it should be common sense, but I have seen this too many times:
Why not include at least the hostname and port? And what connection is this exactly that failed? You might be able to find out what it was trying to connect to based on the stack trace, but that requires knowledge of the internal structure of the software.
This seems to be a common problem in software written in Java, I don't know why exactly. Maybe something to do with the way exceptions are handled, so that the relevant information is not available when printing the error?
So true! One of my pet peeves! Truly pathetic to not include dynamic info like hostname, port and reason.
Who are these people writing code like that - they never had to debug a problem before? Include as much as you can! A stack trace is only part of the story.
Definitely, I often catch exceptions and wrap them just to add more information from the try-finally scope. It is so frequently useful to get a full story of what the app was attempting to do.
Misleading headline alert. All code is debuggable. Logging, mentioned in the article, is more similar to code tracing. It may or may not be useful, because the log might omit information crucial for debugging. There is no way someone can write logging code for every scenario that might break in production.
To me, easily debuggable code (whatever that means) is code written clearly with good documentation. Often well architected systems are modular (rather than a spaghetti mess) and problems are easy to isolate.
It's very hard to think so,our IDE or gcc has already located where we can see the errors.And to write two versions of code is a very annoying thing.To great programmers,source code is enough.
I'm curious about this whole "have to be contrarian deal." Guy writes blog post which is good advice in 99% of the situations and someone just has to bring up that, yeah, somewhere, sometimes more logging is not necessarily better. Why always nitpick on things?
In my work, if people wrote more and better log messages I would be delighted. They almost never do and instead rely on the Visual Studio debugger to inspect program state. It works great when debugging your own code, but is pretty useless when debugging someone elses code (you dont know where to place the breakpoints) or when analyzing a crash on a production site.
I always figured that this was because a developer ran into a situation where the concept was naively applied and it ended up biting him. So, instead of staying quiet, he let people know the problems he ran into so that others don't get bitten as well and approach the problem more carefully.
The thing is, the article does not say "There are many circumstances where logging is a great tool." Who would object to that?
However, what the article says is "One of the differences between a great programmer and a bad programmer is that a great programmer adds logging and tools that make it easy to debug the program when things fail." In other words, he's more or less explicitly saying people who do not use logging are bad programmers. So it's not at all surprising that there is vocal disagreement.
> I'm curious about this whole "have to be contrarian deal." Guy writes blog post which is good advice in 99% of the situations and someone just has to bring up that, yeah, somewhere, sometimes more logging is not necessarily better. Why always nitpick on things?
It's just the nature of HN. I've given up on bitching about it.
Logging has its place, but I'd argue it's far more important that great programmers write code that fails fast. In his example, he didn't have instrumentation in production to determine why a module was returning null. If null isn't an expected return value, then an exception should be thrown to avoid the system moving forward in a crippled and unpredictable state. A well tested system that stops processing and fails fast with a descriptive exception greatly reduces the need for time-consuming and resource depleting logging systems.
Yes but no, in many cases you do not want to fail fast and loudly. If a corner case in a minor module crashes all the pages of your app, you better fail silently, and log.
Thanks for a good comment. Failing fast is great. In this case, since null was returned, the whole call failed gracefully (returning an error code to the calling party instead of simply throwing an exception).
The key point though: however the problem is detected, you need to have enough information to help you answer the question "Why did it fail?". You can add information in an exception, or in traces/logs, as long as you can tell why it happened, not just that it happened.
Even getting logs is luxury in many cases. I found it better to write 'fail fast' code. Module in that example should fail with reasonable error message, if it was used incorrectly.
If your codebase is a bit more complex than hello world, a single helpful error message is hard to do right, because if you have a crash it should mean something unknown happened that you cannot fix at coding time. So it is better, in error messages, to not try to guess what is the problem and just state the wrongful facts. But then only the log and the traceback will help.
Oh cmon. Why can't programming be just fun. Why this obsession with great programmer and expert coders and the likes. Just pick a problem and hack away. Most importantly, have fun.
Every week I come across at least 10 articles telling people about what makes a great programmer. Sometimes its test code, sometimes its some concoction of agile or xp etc, other times its about functional and oop.
Forget writing "debuggable code", whatever that is. Normally that's called "code that makes sense" or "code that a human can read", which I'll freely admit is a subset of the kind of code you'll encounter in the real world. It's still what you'd expect a competent programmer to produce.
Write testable code. Each method should have a specific mandate, should specialize, and should delegate additional functions to other methods that follow the same pattern. It should be straightforward to verify that a particular method is working by exercising it according to the various branches it might take.
This is very hard to facilitate with huge methods that try to do too much, there are too many permutations.
1. Write code that you think is almost trivial.
That way it is simple enough to debug.
If you think your implementation gives you bragging rights down the pub, its waaaaay too complicated
2. log everything, write tools to parse the logs. All this turning logging off / on / up / down. It's disk space, its cheap and trust me when you wanted -vvv debugging, the live server was set to -q
3. testable code - oh yes, yes, yes. Functions that return common structures that are then sent on not wrapped up for that one specific use you had in mind(my most common crime) . But make it simple. If you cannot throw in a config file and run it there on the command line, then its hopeless. Stubs, dependancy injection - I get it, I just want to avoid it.
"1. Write code that you think is almost trivial. That way it is simple enough to debug. If you think your implementation gives you bragging rights down the pub, its waaaaay too complicated"
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian Kernighan
That said, sometimes cleverness is necessary, or actually serves the goals of making things testable and debuggable.
"2. log everything, write tools to parse the logs. All this turning logging off / on / up / down. It's disk space, its cheap and trust me when you wanted -vvv debugging, the live server was set to -q"
While I tend to agree, it's not (just) disk space - it's IO, which sometimes isn't cheap.
Yeah I could not remember who said the Kernighan quote. Should have remembered.
I am not against cleverness, just appropriate for the system - for example some part of the system needs to be very clever else its a pretty mind-numbing project. However the rest of the project should be less clever.
1. Write code that you think is almost trivial. That way it is simple enough to debug. If you think your implementation gives you bragging rights down the pub, its waaaaay too complicated
One of my colleagues is a fantastic programmer. Like anyone else he'll occasionally suggest stuff that won't work, but you can tell he's come up with something great when he starts out with "I understand this is trivial, and the stupidest implementation possible, but..." Those ideas have led to some of the most maintainable and debuggable code in our repo.
When speaking about reasons not to log, I have one: What about performance?
If your software is a computer game, for example, you might have dozens of items, characters, and particle effects being tracked by the game engine at a rate of ~60 Hz. That's ~2000 log lines per second, or 160 KB / sec. Steam says I've played 200 hours of Civ5. Where are we going to put 115 GB of logs?
Not to mention the string formatting and I/O of that much logging potentially leading to unacceptable lag.
Ok, write lots of logs everywhere, but how and where should those go? Wont all those messages get in the way? Will they obfuscate my code? Most of the codebases I've worked with didn't have judicious logging, so where should I look for examples?
Spring's source code doesn't look too bad. Is that a good standard?
45 comments
[ 3.5 ms ] story [ 108 ms ] threadIt's easy to swamp useful messages in tons of noise, so just logging whatever is not necessarily an improvement. Logging means code, and more code means more work, less readability, and more bugs. In particular, logging somewhere, sometime means I/O, which opens up a whole new class of bugs you'd rather avoid. For instance, I've seen several systems go down simply because the disk space ran out.
Logging is also slow - his example of finding a cheapest path is a good example of this. These are well-studied algorithms that explore huge state spaces. Adding logging as he suggests can easily mean the application's running time is spent almost entirely logging. This might not be an issue: maybe you don't need this very often, or maybe the path's aren't very complex, so you can get away with huge overhead because systems are so fast anyhow. But often as not, you can't. And even when you can, you might be adding a trivial DoS vector for your app, or you might be needlessly limiting its applicablity.
If you have an error, it's nice to log whatever you can to help understand it. But the benefit of understanding the odd error more easily needs to be weighed against the non-trivial costs in time, readability, maintainability, disk space and runtime, particularly if you're pre-emptively logging before an error ever occors.
My main point was this: if you don't get the results you expected, do you know where to start looking for the problem or not? If you have no idea, then there is not enough logging/tracing present in the program.
There's little to gain that way. If the logging was off when the problem happened, you have no logs. You can switch it on and try to reproduce. However, if you are able to reproduce, you can as well attach the process to the debugger and check what's happening. No logging needed.
Logging to disk means I/O.
Also, if you can turn logging/tracing on and off dynamically, the average volume of output can still be quite manageable.
Apparently when saying logging doesn't require I/O at runtime, you gotta talk it out a lot more before people get it.
And since it is a circular buffer the memory cost is fixed. Since the memory is shared and available through the filesystem the log stays around if the application crashes.
java.net.ConnectionException: Connection refused ...
Why not include at least the hostname and port? And what connection is this exactly that failed? You might be able to find out what it was trying to connect to based on the stack trace, but that requires knowledge of the internal structure of the software.
This seems to be a common problem in software written in Java, I don't know why exactly. Maybe something to do with the way exceptions are handled, so that the relevant information is not available when printing the error?
Who are these people writing code like that - they never had to debug a problem before? Include as much as you can! A stack trace is only part of the story.
To me, easily debuggable code (whatever that means) is code written clearly with good documentation. Often well architected systems are modular (rather than a spaghetti mess) and problems are easy to isolate.
In my work, if people wrote more and better log messages I would be delighted. They almost never do and instead rely on the Visual Studio debugger to inspect program state. It works great when debugging your own code, but is pretty useless when debugging someone elses code (you dont know where to place the breakpoints) or when analyzing a crash on a production site.
However, what the article says is "One of the differences between a great programmer and a bad programmer is that a great programmer adds logging and tools that make it easy to debug the program when things fail." In other words, he's more or less explicitly saying people who do not use logging are bad programmers. So it's not at all surprising that there is vocal disagreement.
It's just the nature of HN. I've given up on bitching about it.
The key point though: however the problem is detected, you need to have enough information to help you answer the question "Why did it fail?". You can add information in an exception, or in traces/logs, as long as you can tell why it happened, not just that it happened.
No wonder most people just Google error messages.
Every week I come across at least 10 articles telling people about what makes a great programmer. Sometimes its test code, sometimes its some concoction of agile or xp etc, other times its about functional and oop.
Let it rest guys. Just go and hack.
Write testable code. Each method should have a specific mandate, should specialize, and should delegate additional functions to other methods that follow the same pattern. It should be straightforward to verify that a particular method is working by exercising it according to the various branches it might take.
This is very hard to facilitate with huge methods that try to do too much, there are too many permutations.
2. log everything, write tools to parse the logs. All this turning logging off / on / up / down. It's disk space, its cheap and trust me when you wanted -vvv debugging, the live server was set to -q
3. testable code - oh yes, yes, yes. Functions that return common structures that are then sent on not wrapped up for that one specific use you had in mind(my most common crime) . But make it simple. If you cannot throw in a config file and run it there on the command line, then its hopeless. Stubs, dependancy injection - I get it, I just want to avoid it.
4. Don't do as I do, do as I say :-)
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian Kernighan
That said, sometimes cleverness is necessary, or actually serves the goals of making things testable and debuggable.
"2. log everything, write tools to parse the logs. All this turning logging off / on / up / down. It's disk space, its cheap and trust me when you wanted -vvv debugging, the live server was set to -q"
While I tend to agree, it's not (just) disk space - it's IO, which sometimes isn't cheap.
I am not against cleverness, just appropriate for the system - for example some part of the system needs to be very clever else its a pretty mind-numbing project. However the rest of the project should be less clever.
One of my colleagues is a fantastic programmer. Like anyone else he'll occasionally suggest stuff that won't work, but you can tell he's come up with something great when he starts out with "I understand this is trivial, and the stupidest implementation possible, but..." Those ideas have led to some of the most maintainable and debuggable code in our repo.
If your software is a computer game, for example, you might have dozens of items, characters, and particle effects being tracked by the game engine at a rate of ~60 Hz. That's ~2000 log lines per second, or 160 KB / sec. Steam says I've played 200 hours of Civ5. Where are we going to put 115 GB of logs?
Not to mention the string formatting and I/O of that much logging potentially leading to unacceptable lag.
Ok, write lots of logs everywhere, but how and where should those go? Wont all those messages get in the way? Will they obfuscate my code? Most of the codebases I've worked with didn't have judicious logging, so where should I look for examples?
Spring's source code doesn't look too bad. Is that a good standard?
Maybe other languages?