AFAIK it's mostly intended as a way to generate tickets: hard errors go into sentry, and the logs can be used as breadcrumbs, but it's not intended to help you diagnose e.g. how and why a business process did something wrong (unless it ended up crashing I guess).
Yes, ultimately I think that's the stronger underlying argument that the author fails to make clearly—when you log something, it needs to be in a machine-readable format so that you can aggregate that data and gain value from it, and you need a system set up to deal with it on the other end. For example, on Heroku I have access to the wonderful Librato log connector, which makes metrics as easy as `stdout.write("my-metric-name=#{whatever}")`. I would also call that a form of "fancy logging"—now my logs are automatically connected to a graphing and alerting system, with a very long retention period. Sentry is the same way—instead of logging something into the aether, you have structured data (the error payload) that's connected to a system that can bug you about errors and help you aggregate the common sources of failure. Unstructured logging can occasionally be useful, but most of the time it's just not much more than noise.
Our approach is to accumulate logs on the grain of individual user work items - one SQLite database per.
This has several benefits, the largest of which being that you only look at log files when a thing logically goes wrong, and every entry you see pertains to that exact activity.
The other benefit is that these logs get purged at the same grain as the work items being cleaned up. Successful items are purged quickly, exception items can be kept perpetually.
So, the implication for us is to log with a lot of verbosity with the expectation that we are probably never going to see any of it, but also with the knowledge that a ton of helpful debug lines will be available when it does all go wrong. Stack traces are not enough for us, we need to know exactly what communications occurred between us and external systems over time to figure out many issues.
We do maintain a global log file, but this is mostly just service start/stop entries and other administrative actions.
This basic philosophy looks very similar to Google's Dapper request tracing system. Since it's impossible to log every request at their scale, they have various scaling controls for dropping requests from the trace or summarizing them (which can be overridden when a trace goes exceptional).
I've had this discussion at lot too. My basic argument always ends up boiling down to "Don't make the software do anything you don't have a use for". If you want to log something, you better have some sort of plan for how that line is going to be used, and a line in some stream somewhere has to be the best way to get that outcome.
That's usually enough argument for people that I get to not have a logger statement in the catch. Frankly it's a religious thing. No one really wants to have the argument, and the best you can hope for is a sigh of defeat.
sort of related, but id say limited logging for things like fail2ban. youll need a rolling 24-72 hour window to identify threats at least...unless im missing something?
> I either want a quick notification about some critical error with everything inside, or I want nothing: a peaceful morning with tea and youtube videos. There’s nothing in between for logging.
1. No, there's lots in between.
2. You should be able set the log severity threshold, and hopefully the log verbosity threshold (e.g. with/without stack trace and other information).
“Use better tools instead: like error and business monitorings with alerts”
I don’t quite understand the alternatives.
However, I do agree that logging is a side effect and should be done in specific ways. That does not mean that you should remove all of it. A little goes a long way.
I have no idea what the authors qualifications are.
There have been uncountable cases where a GOOD log trail has been the difference between a 20 hour journey through a code base and a short stop to immediately discovering the problem. Common failure points should be logged, especially when they are deeply nested, or they involve complex hard-to-diagnose processes.
Overlogging is just as bad. But to say "do not log" and then suggest using services like sentry or datadog instead is really reaching around your nose to touch your face. You're doing yourself a disservice by lacking the final bit of correlation - the approximate location and nature of the failure. There have been times when I've been on a service so complex that just building a harness to reproduce the case takes hours or days. Not everything is a nice, clean, small and simple microservice where you only have N guesses at where the problem could originate. On a large enough, or old enough, service the upper bound could approach N! very quickly.
I'm not sure why this was posted to HN as it seems like a very low-quality ranty post by someone whom I'm not sure has ever worked on a large enough project where it becomes non-obvious what the origin of some error alerts are.
It really is not: you can trim overlogging, you can't add missing logs.
Now one thing I am very much in favor of, but which I didn't see anywhere in TFA, is the ability to dynamically instrument instances (add or remove log points).
It's not necessarily easy to know what to log or not to log. With "normal" logging adding new log points is a complicated process (they're static so you have to introduce them, ensure they're correct, and then go through whatever deployment process you have), so there's a natural tendency to overlog because if you underlog you may miss critical information.
In systems where the capability exists, instrumenting log points "from the outside" without having to update the program being observed is much better, not only does it let you refine your logging more easily, it also lets you brutally increase logging density (possibly locally) without downtime.
That's true. Annoying and pointless logs at least tell you where they are :). I suppose what I meant by this is in the heat of the moment having to sift through overlogging probably somewhat justifies the "no log" rant on the surface. But the underlying problem isn't logging but rather the abuse of it so it seems a little premature to blame logging itself for this.
I try to produce many small log files by hour or day. This makes it pretty easy to discard old logs or reduce the amount of data you have to look at if you know the problem was within a certain timeframe.
In some systems / runtimes, yes. For instance every browser supports "logging breakpoints" which can be added and removed on the fly (create a breakpoint then right click, you should get an option or sub-option about logging). I would also expect that there's Java instrumentation things which do it, though I haven't done any java for a very long time.
It's pretty rare though, obviously so for runtime-less programs, somewhat less so for those with a runtime but while "external" profiling is common logging is much less so (e.g. Python supports updating the configuration of the existing static logging system on the fly, but not defining the log points).
> It sounds like an application restart would be warranted at the very least, which can be disruptive.
See above, if you can profile an application on the fly without restarting it you should be able to instrument for logging.
And tools like dtrace/ebpf should allow this, but I've yet to take the plunge and actually try and look at that, it's only been an idle thought so far.
“the difference between a 20 hour journey through a code base and a short stop to immediately discovering the problem.”
I had situations where the difference was between having an intermittent problem for months or years that nobody can figure out and pouring over logs for days and finding the problem.
I do pretty extensive logging in my code and it has saved my ass many times. Before people complain about performance or storage problems I want to see hard profiling numbers. In most cases you it’s only a few places where the logging is causing problems and you can reduce it there.
Overlogging is almost never a real problem in my experience. You can always discard log data you have but you can’t create log data you may need but didn’t produce.
> Overlogging is almost never a real problem in my experience
I would disagree, after managing a 100 node ELK monstrosity to try to process the terabytes of logs generated by our systems.
Overlogging is not a problem if log levels can be easily changed (preferably without an app restart). One app logging like crazy is not a problem. A thousand of microservices, is.
I agree and push this to it's logical conclusion: logging is for development (test) only, on live I have everything hard routed to my mail, if something fails I'll know about it and be encouraged to fix it because I set it on a 24 hour reset. + Instead of wearing my SSDs down, it wears my mail providers.
The real solution is to make both automated and manual feedback extremely easy, for example send crashdumps without asking and have a forum connected to the users account where you redirect the user upon a crash.
Just like downgrading my GPU to a 1030, so I cannot play stupid games; this is a motivational trick to curb me into making things better!
How is using Sentry (or whatever) not adding complexity? How is Sentry not introducing side effects? How is using Sentry magically creating information about why a business operation failed? Does Sentry even know your business operation failed? It doesn’t, unless you tell it. How is Sentry not I/O-bound? (That’s right, they abstracted that away, like a good logging framework does, too.)
In the end, to integrate Sentry and get actually useful information for monitoring, you have to do the same work you’d do for logging. Except now you’re tied to Sentry. Want some other service? Good luck.
Do log. And then send the log messages to Sentry or whatever you’re using, too.
Also, do overlog. It can be hard or impossible to attach a debugger to the deployed application. But you can always turn the log level up to 11 and then reconstruct what happened from the log messages.
That, however, does not at all work when exceptions are part of the "normal" process and should be ignored or cleanly retried (having spent a while interacting with github, if you take the entire thing down every time github replies with an error and log that to sentry, your sentry instance will last all of 5mn before you have to declare bankruptcy or just plonk the alters and ignore everything).
But in that case you probably still want to know about these failures, so you can review them from time to time, or see if there are upticks in failure frequency.
With some systems I have indeed been tempted a few times to only throw an exception if everything went right in one try, but my suggestions have been rejected so far.
> I have a third party which returns errors ~20% of the time outside of working hours
Then presumably those errors shouldn't be exceptions. Of course, there's a lot of a lot of assumptions behind this:
1. You have control over the code that interfaces with the third party. Otherwise presumably you'd have to write a shim that catches the exception and turns it into an error value, and that's probably not worth the effort.
2. You're work in a language that doesn't type check exceptions (i.e. not Java with checked exceptions) and your language also doesn't have a nicer way of handling errors as a (discriminated) union.
3. Your co-workers will go complete bananas if you start using some sort of result type or [success, value|error] tuple instead of an exception because that's how they've done it for 20 years.
The reason for this is that exceptions in most languages escape the type system and are in general more awkward to catch.
Of course, I'd also disagree with "do not log" in this context, though I might prefer a logging mechanism that allowed me to write queries on it (like recording values in a database).
Well, unless it's reasonable for your own software to fail the same 20% of the time (well, more, but knowing how much more requires more information) when outside of working hours, you will always need a shim anyway.
In some languages, the standard way to signal failure is to throw an exception. You can't really define exceptions as abnormal in those cases, because failures are part of normal operation; you can't expect them not to happen if you want robust software.
Because that's highly un-idiomatic in pretty much every language that has exceptions, except G++, which arguably doesn't really have exceptions as they are strengstens verboten.
Even if you really really like error return codes and really really don't like exceptions, you'll likely agree that writing very un-idiomatic code that goes against the grain of the language and all the libraries you might be using is likely a much bigger loss than the value you might have added by avoiding exceptions in your part of the block.
It's really comes down to wether you think exceptions are an everyday thing not. I like going by the dictionary definition of the word "exception" which states that they are not.
Out of curiosity, in what languages are error return types non-idiomatic?
Python and Java are two that come to mind. It is idiomatic in both to return null/None on error, but it is not idiomatic to use a result type. Hell, Python even uses exceptions to terminate iterators.
This isn't to say the idioms are never broken or that they shouldn't be broken. I agree that if you are doing something like parsing a string to an integer, that should probably be a result type and not just throw an exception.
In almost all dynamic languages I know of for example. Way too easy to forget about error handling if you use values. Exceptions at least tend to cause noise.
In languages like Java an Python, you can return error values but because you can also always return nulls, you're just making more trouble for yourself than it's worth avoiding exceptions.
In your code, you don't have to use exceptions to indicate a failure but you still have to catch exceptions thrown by third-party libraries (or their dependencies).
It is not easy to stay away from exceptions
if your language supports them.
I think errors and failures get often conflated. There are truly exceptional situations where your application can do nothing and will just have to abort (passing off the failure to a higher layer in the system) but often failures aren't really exceptional. You can not assume that your database connections live forever, for example, and it's not an error when a commit fails due to an interrupted network connection. Such cases need to be handled.
If you know that an operation is unreliable, you need to add retry / recovery logic and perhaps log or increment a metric that you can track what's going on; a high number of failures often indicates an exceptional situation.
In some cases, your retry logic will be a "Please try again" message to the user. I think that's perfectly acceptable, but you'll still want to know about it.
> Catching just to log something is generally wrong
Yes, but you should catch as soon as possible (or validate stuff as soon ass possible) and re-throw with a better error message if possible. You will thank yourself when some problem appears and instead of a random "errorcode 123" you get a detailed message like "Operation Xyz failed with errorcode 123";
I think what the post is trying to say is that you should let your exception bubble up.
Usually, but not always. That is, for the classic "exceptions should only be used in exceptional cases" scenario, this is usually what you want to do.
A good portion of the time, though, this scenario doesn't apply. You're dealing with a subsystem that is throwing an exception (for good reasons or bad) but no, you don't want to just the application crash. You want to ... catch it, and do something else.
Which is what try-catch was devised for in the first place, after all.
> I think what the post is trying to say is that you should let your exception bubble up.
It's not an either-or situation though. Even if you let the exception bubble up and sentry logs the exception, logging performed prior to the exception and at the time the exception is caught (prior to being bubbled up) as valuable context to how the exception happens, which can lower debugging time.
The exception just tells you that something went done, it doesn't always have the reason why something went wrong. And the context for why is not always available at the time of raising the exception (the context may be from a prior function call, or the consumer of the function that raised the exception).
Eh... a lot of this article can be reworded as "do error handling better" (make error paths unreachable through your type system, crash for unrecoverable errors, if errors are recoverable, they should be Results, etc etc.)
Ok, I did all that. And I still want to know why this bug report happened, and logs help me. Why should I Not Log?
Also... Sentry is logging. It's a very fancy logger that can capture all sorts of data about the exception and what was happening - but at its core, it's still a essentially a logger.
I was thinking the same thing once they got to that point! Our "logger" is literally a library from DataDog, but Sentry has similar (we actually migrate from Sentry to DataDog at my job) and there's tons of other services now that do similar with "loggers" as well. There is good advice here about questioning when to log but Sentry and its peers are loggers.
>Except now you’re tied to Sentry. Want some other service? Good luck.
If this is a concern, just create a class with a method that takes an exception, message, context, etc. as parameters and use that. Now you can change it in one place instead of throughout your codebase.
Agree on logging messages as well; is useful to see live what is happening in the system as well as it is useful to see logging broken up by different errors
> Also, do overlog. It can be hard or impossible to attach a debugger to the deployed application. But you can always turn the log level up to 11 and then reconstruct what happened from the log messages.
Please don't overlog. Only ever do this if you can dynamically change your log level on demand, without impacting users.
If you do have to reconstruct a sequence of events and you do not have distributed tracing, fine. Log the sequence of events _when needed_.
But please don't log messages on a perfectly fine system for no reason. It's noise and health can be better served by looking at metrics.
Because of this overlogging nonsense our systems generate terabytes of logs daily. Most of it is never ever looked at.
Well yes of course? That’s what log levels and log categories are for. Whether the configuration can change at runtime or if your services have to cycle doesn’t really matter IMHO.
Oh i see you are logging exceptions caught in try/catch blocks... easy, remove the logging then just refactor that code to not have any exceptions by accounting for every possible situation. Then in 2 years when you are done, you can proceed to the other try/catch block.
(2020). Not this article again. Observability is crucial in reducing time-to-resolution in incidents. I would much rather have to debug a chatty system than one which says nothing yet fails monitoring checks.
This is inane. Logging has been a thing since the stone ages of software engineering. The first thing you do in any environment is set up a debug log of some kind. You can make as many error states unreachable as you can, but what the hell do you do when there's an error anyway? Psychically debug the process?
edit: how are you going to communicate information as to why something crashed to your colleagues who do not understand certain parts of the system as well as you? A stack trace is not effective at expressing intent. Explaining yourself in logs helps people triage more efficiently.
I think the point is "logging isn't free" (in fact, that might make for a better title for the blog post).
A lot of devs think of logging as free; it's easy, it's almost always something that's either global-scope or standard library, and it's the first obvious solution to inspect internal program state when the program misbehaves.
But logs are a subsystem and adding any subsystem to your code increases your risk surface (for both crashes and privacy violations).
Took an operating systems class in college where the first thing the prof drilled into us was "Don't debug with printf. You're going to want to. It's the thing you're most accustom to having written application code. But you're writing an operating system now. If you have a memory overrun, is the first thing you want to do really allocating a kilobyte of stack to mangle strings and determine if you need localization support to tell yourself something's wrong with memory? By the time the output gets emitted to console, the memory you want to inspect will be a blasted wasteland."
As an example of how logging can break...
> you can buffer in memory, and write to storage later
You may as well not have logged then if the failure issue is a segfault; your program context will collapse before the buffer is output.
Great point about using business metrics to notice real customer problems, but when you have to find out _why_ those metrics are off, logging is helpful.
This reeks of functional programmer brain.
The world is not immutable. the world is weird. Your systems will interact with other systems that don't always quite work. Sometimes, their systems are wrong, or update without your knowledge, or theres other issues in routing, networking and whatever else.
Logging errors like "I can't resolve this domain" is more helpful than "my customers can't use this integration and business is at zero"
Even if logging does not give you a complete picture of the exact run-time state of the system or the context in which the error happened, I’d be damned if I’d willingly deprive myself of a logged error. At least you know where it happened. Invaluable first clue IMHO.
I don't get why people get so theological about logging.
If logs bring you some sort of value, then log away. If they don't, then don't. In some domains you absolutely crucially need logs, in others they're largely a waste of time.
Logging is probably the most foolproof way to extract debugging information from your application. The killer feature of logging is that standard out very rarely breaks, no matter how bad things get it works. It will work from the moment the application starts, it will work in an OOM situation, will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building, it will work no matter how bad things are. If logging does somehow fail, then that is on you, in that scenario you have fucked up your architecture somehow.
Most applications today run as replica sets (k8s pods/VM asgs) so log aggregators are another interesting part of the problem. Most shops are moving to saas logging for log collection which presents another failure mode: your logs are only as reliable as the saas provider.
It is true however that looking at the individual replica, application logs should be very reliable.
Definite up vote here. Theological is the perfect word for this. It would be helpful if the author substituted the word "blog" for the word "log" in the article, and took the new meanings that result seriously.
> I don't get why people get so theological about logging.
Because the reason to log is about people, not about technology.
There's no technical reason to make logs: Just don't make bugs, and you won't need the logs to find out what went wrong because nothing will have gone wrong. Go to the beach!
This is hard though, and bugs happen for all sorts of reason that have nothing to do with the programmer making a mistake-- the customer can be unsatisfied because they didn't know what they wanted. Programmers learn very early in their career to distrust the requirements, because they know that saying but that's what the ticket says gets them nowhere besides an argument: They have to do what the business/customer meant.
Put another way, the user doesn't want logs: The user wants to know if everything is okay or not, and presumably they have some way to figure that out, but if it isn't working correctly (or they merely think it isn't), the programmer will probably want those logs to help them.
In many environments, the loop is larger than just those two people, there's QA support people, and internal experts/operators, and triage, and all of those people are going to have different needs from those logfiles. People might also have a second use for those logs involving anticipation of failures and tracking capacity.
So there are a lot of non-technical, non-functional needs for logs, and once you understand this, it may be a little easier to understand why people would have almost rabid opinions about what those logs should look like (or if they can get the cure they need another way, what they might prefer instead of logs)
> [standard out] will work in an OOM situation
If you use write() which doesn't allocate memory, that's almost true, but if you use printf() it might allocate memory, and it might not work in an OOM anyway depending on what "standard out" actually is.
> will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building
And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway.
I've been programming for over three decades at this point, and I'm convinced there is no "easy one-size-fits-all answer" to whether to log or not, or if to log, how to log, so I find it useful to learn about this bit of theology so that I'm better equipped to talk to others about this, and maybe convince them of my opinions.
> And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway.
Reducing the number of points of failure is almost always a good idea. The network being down is one problem, not being able to ever know what your applications did when the network was down is another problem added to the first problem.
> Reducing the number of points of failure is almost always a good idea.
Reducing the single points of failure is usually a good idea.
You can easily reduce the number of points of failure by reducing redundancy and removing sanity checks from your process. Doing that does normally not improve your reliability.
Sending the logs to your SAN for it to write in a couple of disks arrays is much more reliable than writing to stderr (stdout is not a reasonable place to log failures) and redirecting it to your local disk.
Reducing points of failure in general is a good idea too. Single points of failure are especially bad, but most major errors are the confluence of multiple things going wrong that while alone would be bad, but together make for an absolute disaster.
Reducing points of failure in general is an iffy idea that can be good or bad depending on how you do it.
A lot of complexity is there for good reason, a lot for bad reason, and some for no reason at all. It takes a very nuanced and specific work to determine what is what, and talking in generalities about it is a very bad idea on itself.
> Reducing the number of points of failure is almost always a good idea.
I think it is good to think about what happens when things fail: Things fail all the time and most of those failures are benign -- and even those that aren't can often be remedied by turning it off and on, but some of them aren't, so what happens then?
If you run out of disk space, writing a log line to the disk complaining that you're out of disk space is no help, but what else can you do? You could send a network alert, or print something to hardcopy (if there's a printer nearby), or ring a bell if you have one attached -- all of these things are also potential points of failure, but they can also help prevent a meltdown.
And if that sounds surprising, you might also be shocked the number of problems that can be solved with:
It's for this reason (I think) that people usually qualify your statement as reducing the number of "single" points of failure, instead of the way you put it, because having two single-points of failure isn't exactly like having two weak links in a chain, and in many cases, it's nothing like it at all.
In any event, there's another advantage to thinking about the failure-process instead of trying to merely avoid failures:
The biggest reason reducing points of failure is a bad idea is when it is more expensive to do so than to simply tolerate the failure: One thing that dominates the attention of startups looking to improve their availability beyond 99.999% uptime, is "removing the failure points" to get higher uptime, but if Amazon is down, what are Amazon's customers going to do? Are they going to buy from someplace else? Sometimes it is simply that more-established brands have the benefit of ignoring some problems, but you can be sure a large company like Amazon has someone who knows exactly how much more they can spend per second of uptime they may desire, and I think it is worth it for a small company to get good at that exercise as well, because sometimes the juice just isn't worth the squeeze.
As a specific example: My ad servers (HTTP event collectors) are critical, and I want 100% uptime for them because missing a click means missing money, but the reporting dashboard (so the publishers know how much money they made, and the advertiser knows how much they owe me) doesn't need that, so as long as the user got redirected to the landing page correctly, and I can generate line charts and excel spreadsheets once a day, nobody cares that the reporting UI might be down for a few minutes a day-- they will simply press refresh, not knowing if it is their internet or mine, and combined with aggressive javascript-side caching and automatic retries, I am not quite sure any of my customers even notice my downtime.
If instead, I focused on reducing the things that could go wrong (because as you say, it is almost always a good idea), I might spend time trying to improve the availability of my database simply because that's the thing that fails instead of whacking a setTimeout into the JavaScript since the impact is that the user would just retry on failure. That setTimeout saved me the cost of a whole server (and power and heat and electricity) and in the early days of my ad server, that would've been a lot of beer.
> "When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway."
That's the normal state of life; when a building burns down, forensic investigators pick over the evidence trying to work out what started a fire. They don't ask everyone to record every time two bits of metal touch, every switch switched, every use of electricity, every flammable material entering the building, just in case there's a fire one day. Use a teaspoon in a metal cup? Log it. Post-it note? Flammable, log it. Don't know if your shoes are flammable? Log them anyway just in case.
And yet the cause of the fire is not likely to be any of those things, it's likely to be the normalization-of-deviance fireworks next kept next to the open gas tank where the employees take smoke breaks which people were reporting for ages until they were told to shut up about it.
Beverley Hills Supper Club was known to be overcrowded, badly designed and ignoring fire department reports for years before it burned down. Cocoanut Grove nightclub had flammable decorations and an employee using flames for light. Rana Plaza had big cracks appear in the building before it collapsed, employees were told to go back to work or be sacked. The World Trade Center collapsed because a plane hit it. Titanic sank because it was going too fast and ignoring reports of icebergs and chose nice looking corridors over high bulkheads. Grenfell Tower burned while "everyone" knew the building was coated in flammable coating and nobody in power wanted to pay to do anything about it. Station Nightclub burned because someone accidentally brought outdoor fireworks indoors, and the sound damping foam was flammable not fireproof and it killed people because the emergency exit doors were locked.
We have a culture of log every order the captain makes, every response from the crew, every movement of the rudder, every radio signal sent and received, and at the same time "why did the front fall off?" "a wave hit it" "is that unusual?" "Yeah, at sea? Chance in a million."
> "you can write logs that nobody will ever be able to read, but what's the point of that?"
Cover your ass, cargo culting, habit, defaults are powerful, the cognitive bias effect like psychic powers where
the one time people predict a bad thing happening and it happens is very memorable but the all the times people predict a bad happening and nothing happens are very forgettable, it's easy to remember one time logs helped and ignore all the times they didn't and the costs associated with them. For some possibilities.
> They don't ask everyone to record every time two bits of metal touch, every switch switched, every use of electricity, every flammable material entering the building, just in case there's a fire one day.
It's industry specific. In my line of work, yes, they do ask you to make extensive records like that and it is why we log nearly everything. Though I've mostly been in safety-critical and aerospace work (space now, previously just aero). I imagine medical, pharmaceutical manufacturing, and others have similar requirements. If you're making systems that don't matter (in the sense of, if they fail no one cares except a few possibly angry users), then don't bother logging anything.
My personal reason for having strong opinions on logging is that nearly every performance issue I have come across has had an easy win of 2x-3x improvement (small sample size, to be fair) by simply disabling logs. Not that this tends to stick because people fear the lack of information that comes from not logging, but you need to be aware of exactly how expensive logging to a synchronized output is, and how expensive that serialization is compared to the rest of the critical path.
People treat logging as if it's free, when the reality is that it very often has a significant impact on performance, exacerbated by the fact that it's treated as free.
Add in the fact that today's distributed systems approaches add complexity to the logging paradigm and make it much harder (vs something like Jaeger/OTel) to understand the context of a single log message, and I would rather logging disappear almost entirely in favor of metrics (fast, coarse) and tracing (slower, fully contextualized), with some ability to fall back to just printing metrics and traces to stdout or other local debugging options as a fallback.
Yes there is performance cost, but removing logs removes your ability to solve problems that happened in production. At best without logging you will have a snapshot of your program at the instant the problem was detected such as a process dump or even just a stack trace. Logs tell you what happened before and while things were breaking.
And that's exactly where tracing comes in. Except instead of writing a dozen log queries in your ELK/datadog/splunk/etc system and switching contexts that many times, you have a single view that centralizes who called whom when, and even aggregates text metadata that you may have chosen to include. In other words, logs, but you don't need "started xyz" because that's just the natural beginning of each span.
I worked on a real-time telecom system, and had to debug issues that heavily depend on timing. Regular logging impacted performance and changed timing, but when we implemented logging properly we fixed this. Our solutions was in-memory ring buffer that stores only data and pointers to formatting functions, logs are formatted and printed on-demand.
A kindred soul. This is exactly my preferred approach to logging. A ring buffer gives you enough information to debug an issue when it occurs, and nothing otherwise.
Visibility into programs running in environments you can’t access is unreasonably important, but any code that creates visibility will impact your performance, too. The “right” solution is to balance those two needs for your situation, which means there is no “right” solution for everybody.
There's your performance issue, logging doesn't need to be persisted or even processed by your main program thread or any thread/process actually doing the work. Who's logging straight to stdout or directly ::write'ing a file in a production application?
Right, and depending on the actual buffer implementation when logs are buffered, given the massive volume of logging I've seen, it's likely not all that much better than synchronous because those buffers are going to fill quickly and need flushed frequently.
My comment wasn’t about buffered writing in a main thread, it was about moving the work out of your threads actually executing your program and having them push raw log actions into a queue for another thread to do the formatting and writes to file from.
If you’re logging so much that you’re saturating a dedicated thread that just reads from a queue in shared memory, does log formatting and writes to a file then you have bigger problems than figuring out how to log because there isn’t going to be a solution that lets you log with a deterministic latency without skipping events or wrapping around your queue.
Of course, the approach scales fine if you can have multiple log files per process - e.g. a normal log file and a message log or transaction log, because you can give each file its own queue and thread.
Me, not write(2)’ing though, mostly because you don’t want to syscall-block every few bytes. Modern service managers redirect 1,2 into logging facilities (or just files if configured such way), and 1,2 are no different than 3+. As if there was a regular way for data to escape a program through non-fd (mmap/shm/fork doesn’t count).
That said I’m not sure what context we are discussing. I work on nodejs apps now where blocking is a non-issue. I’d obviously think about it more in C’s fprintf() or a similar block-by-default environment.
Edit: also, my area is fintech with $0.1-$1000 range per request, not telecom. I log everything because my employer can ask about everything, and it’s always money.
I get the idea, but then how would you decide when to enable tracing and in which area?
IMO, the advantage of logging is that it some degree of information about actual failure events - in contrast, monitoring only gives you aggregated information and tracing is too expensive to be applied everywhere at any time.
Tracing should be sampled at a rate that makes sense for the dollar impact of an error in the service, versus the dollar cost of your tracing system.
$thousands per transaction? Sample 100%. Cheaper to refund a customer than debug the problem every time? Then sample. You'll have plenty of sampled traces with the bug/error shown as soon as the problem is expensive enough to matter.
Metrics are great for demonstrating a problem exists (and alerting), and tracing is great for helping you understand how and why it happened once it's at a big enough scale to matter.
Beyond that, with a lot of what's coming out of Open Telemetry, you can get fancier like sampling a higher percent of traces to a localized processor middleware that then filters out any traces you doesn't care about (e.g. dropping non-errors or low-quantile latencies).
Yeah, but once again, this requires you to already know there is an error, have a dollar impact calculated, etc.
There is a lot of talk in this thread about the performance impact of logging. I don't see how sampling 100% wouldn't have a far greater performance impact.
> I don't get why people get so theological about logging.
> If logs bring you some sort of value, then log away.
I have been on the other side of the problem for years now. Logging has a cost. And the cost benefit is also not great. It requires a human to read it(either that or a regepx hellscape awaits you). This can be fine if you are working with a small service, but it quickly gets out of hand once scale increases.
I've had to deal with double digits(approaching triple) terabytes of logs daily. Most of the information logged was, like the author suggests, completely irrelevant. It makes it much harder to see what's actually relevant. And, for whatever was actually relevant, generally not enough context was provided (something the author also argues).
What you really have to have in production systems are metrics. That's what's going to tell you if shit is on fire and which ass is burning.
Has your app has hit a catastrophic failure and cannot continue? OOM? Sure, by all means log away. Did it have an error while trying to talk to a service? Are you retrying? Then _do not log_ until your retries (or whatever recovery mechanism) fail. Networks have become very reliable, but failures are expected and normal. Spit out metrics instead.
If you don't do this, some schmuck will have to spend their days trying to keep the logging infrastructure from falling over (Cthulhu helps you if you are on an ELK stack) while management will be complaining why noone seems to notice failures before customers do.
If you find yourself parsing logs to alert on issues, please revisit your architecture.
METRICS FIRST. Logs are a safety net and a tool for troubleshooters.
What you are describing is pretty clearly within a domain where logging doesn't bring much value. It's either superfluous debug messages from an earlier phase in development, or at the wrong granularity. If you are dealing with thousands of events per second, you can't log at every event. Your logging probably shouldn't produce more than a couple of lines/second on average unless something has gone very wrong.
Logging is something that needs to be maintained (like all code), it can't be just some vestigial by-product of the development that's left to its own devices, at least not if it's to be of any use. The same is true for metrics. If your metrics are just some arbitrary selection of times and counts that seemed good at the time, they're probably not very useful. Which data to collect is something that needs to be deliberately chosen, and re-evaluated over the lifetime of the application. When you are launching a new product, then lots of logging is probably a good idea. When that product is mature, you probably should have dialed it back several notches, as you most likely already know the general areas where things go wrong.
> I don't get why people get so theological about logging.
IMO, it’s because the right logging solution for one situation is not the right solution for another situation. That means that one person is “right” and the other person is “wrong” (because the context required to identify why one person is right is complicated and hard to express, and is thus missing).
Observability (of which logging is a subset) has a performance impact, and software has performance requirements. This means that the right amount of observability is going to depend on the software, not a standard.
> If logs bring you some sort of value, then log away. If they don't, then don't.
The issue is -- some people find value in the logs but some other people don't.
I am in the strong camp that find logging provide negative value, mostly due to the extra complexity it brings and its obstruction on code reading. And I believe the reason the other camp finds value in logging is because they lack insight to pin-point issues and lack effort to gain that insight. But I can't really use that argument because it can easily be deemed offensive and solicit slash backs rather than constructive discussions and debates.
When writing something that feels like it could break, or when dealing with unknown external APIs, I like to add an exception handler to _all_ new methods with a TODO and expiry date 3 months from now. Then on clean up days, if an exception wasn't triggered in that time frame, just remove the catch block.
Perhaps I'm being too cautious (or too naive)? But it's saved my skin a few times.
The only thing that I don't like about this technique is that the 3 month period is arbitrary. But it works well enough for me.
Sentry, in this case, is just a form of logging. So I don't agree with the premise of the article. Sure, prefer to eliminate error scenarios instead of allowing/logging them, but to me that has nothing to do with logging - the approach that doesn't handle errors is just bad code.
Strong disagree on this. Logs should serve a single purpose of allowing developers to introspect issues with their applications. If you want to generate an event stream, a more flexible and stable architecture would be to emit events deliberately.
Imagine you build all sorts of dashboards and Workflows on top of logs or log based metrics and the dev team changes the structure of the logs or just stops logging. Suddenly all your analytics is out of whack; this is because you created a hidden dependency.
I’m not sure about hidden dependency if the metrics break immediately. Important metrics should have alerts when absent or change unexpectedly.
Communication and the decision to depend on the logs as a source of metrics would be key to keeping these systems connected and functioning.
Taking a dependence on logs based metrics without communicating that is the failing here.
The same breakage could happen by depending on the event being emitted and a developer changing the event structure.
Communication and understanding helps both scenarios, not duplicating the effort by emitting logging for debugging, events for auditing and metrics for business.
Logging can be useful during development, but PLEASE get rid of it for deployment. The usefulness of my own logging while working at Apple was severely diminished by the sheer volume of logs from other frameworks where (I guess) removing the log message was considered to be of low priority. Just as an example: on my pure production M1 I see thousands (really) of this completely useless log message:
PPSocialHighlightStorage: Constructed social attribution has missing data: (null)
What is that message doing in shipping software? It cannot possibly help anyone.
> What is that message doing in shipping software? It cannot possibly help anyone.
Correct. Those low level messages should not be present in 'production' environments _unless requested_. The ability to log extra info for debugging is fine. Doing that by default isn't.
(User-visible) Warnings and errors only please. Can always dig more and turn up verbosity if needed.
Good point, but often you can solve this by setting log levels appropriately. Usually something like "DEBUG for my own modules/packages, WARN for 3rd party libs or frameworks, TRACE for that particular class I'm currently debugging" gives you back your sanity.
For anyone else wondering what the heck "Sentry" is, I'm guessing it is this: https://sentry.io/welcome/
If the author is reading this, the Sentry-related link you have in the article [1] is currently giving the readthedocs.io 404 page. Same for the "docs on logging" link about 4 sentences earlier [2].
Okay, if revenues suddenly drop 50% - then what? Where does the problem lie and once it’s discovered how can we alert on a recurrence of the same issue and fix the issue? Presuming the underlying issue cannot be trivially permanently remediated.
The author seems to see logging as essentially making excessive alert noise inevitable when logs are meant to just be collected somewhere and scanned for exceptions known to cause incidents. Then when that exception happens at the same time revenues fall 50%… there you have it?
The authors observations are mostly valid but their conclusions just don’t make sense. The observation you shouldn’t ping an admin every time a log generates an error doesn’t mean “don’t log”
The author's premise seems to boil down to 1) don't have errors and 2) "simply" make sure that your APM system has sufficient context to identify what went wrong. Of course, both of those are pretty impossible and they ignore the similarity between creating a constant flow of information for later use (logging) and creating a constant flow of information for monitoring (not logging). I could go on and on here.
The "real" point seems to be that logging should be useful and consistent, which I agree with.
156 comments
[ 2.6 ms ] story [ 222 ms ] threadThis has several benefits, the largest of which being that you only look at log files when a thing logically goes wrong, and every entry you see pertains to that exact activity.
The other benefit is that these logs get purged at the same grain as the work items being cleaned up. Successful items are purged quickly, exception items can be kept perpetually.
So, the implication for us is to log with a lot of verbosity with the expectation that we are probably never going to see any of it, but also with the knowledge that a ton of helpful debug lines will be available when it does all go wrong. Stack traces are not enough for us, we need to know exactly what communications occurred between us and external systems over time to figure out many issues.
We do maintain a global log file, but this is mostly just service start/stop entries and other administrative actions.
That's usually enough argument for people that I get to not have a logger statement in the catch. Frankly it's a religious thing. No one really wants to have the argument, and the best you can hope for is a sigh of defeat.
-------
> I either want a quick notification about some critical error with everything inside, or I want nothing: a peaceful morning with tea and youtube videos. There’s nothing in between for logging.
1. No, there's lots in between.
2. You should be able set the log severity threshold, and hopefully the log verbosity threshold (e.g. with/without stack trace and other information).
3. We don't just log failures.
Ye just log everything. You mostly don't know in advance where the error will be anyway.
The author seem clueless. Don't log but use a Cloud $$$ SaaS for monitoring? Really?
And then he goes on to argue that the code should just not fail. Like, "write good code".
I don’t quite understand the alternatives.
However, I do agree that logging is a side effect and should be done in specific ways. That does not mean that you should remove all of it. A little goes a long way.
There have been uncountable cases where a GOOD log trail has been the difference between a 20 hour journey through a code base and a short stop to immediately discovering the problem. Common failure points should be logged, especially when they are deeply nested, or they involve complex hard-to-diagnose processes.
Overlogging is just as bad. But to say "do not log" and then suggest using services like sentry or datadog instead is really reaching around your nose to touch your face. You're doing yourself a disservice by lacking the final bit of correlation - the approximate location and nature of the failure. There have been times when I've been on a service so complex that just building a harness to reproduce the case takes hours or days. Not everything is a nice, clean, small and simple microservice where you only have N guesses at where the problem could originate. On a large enough, or old enough, service the upper bound could approach N! very quickly.
I'm not sure why this was posted to HN as it seems like a very low-quality ranty post by someone whom I'm not sure has ever worked on a large enough project where it becomes non-obvious what the origin of some error alerts are.
It really is not: you can trim overlogging, you can't add missing logs.
Now one thing I am very much in favor of, but which I didn't see anywhere in TFA, is the ability to dynamically instrument instances (add or remove log points).
It's not necessarily easy to know what to log or not to log. With "normal" logging adding new log points is a complicated process (they're static so you have to introduce them, ensure they're correct, and then go through whatever deployment process you have), so there's a natural tendency to overlog because if you underlog you may miss critical information.
In systems where the capability exists, instrumenting log points "from the outside" without having to update the program being observed is much better, not only does it let you refine your logging more easily, it also lets you brutally increase logging density (possibly locally) without downtime.
In some systems / runtimes, yes. For instance every browser supports "logging breakpoints" which can be added and removed on the fly (create a breakpoint then right click, you should get an option or sub-option about logging). I would also expect that there's Java instrumentation things which do it, though I haven't done any java for a very long time.
It's pretty rare though, obviously so for runtime-less programs, somewhat less so for those with a runtime but while "external" profiling is common logging is much less so (e.g. Python supports updating the configuration of the existing static logging system on the fly, but not defining the log points).
> It sounds like an application restart would be warranted at the very least, which can be disruptive.
See above, if you can profile an application on the fly without restarting it you should be able to instrument for logging.
And tools like dtrace/ebpf should allow this, but I've yet to take the plunge and actually try and look at that, it's only been an idle thought so far.
I had situations where the difference was between having an intermittent problem for months or years that nobody can figure out and pouring over logs for days and finding the problem.
I do pretty extensive logging in my code and it has saved my ass many times. Before people complain about performance or storage problems I want to see hard profiling numbers. In most cases you it’s only a few places where the logging is causing problems and you can reduce it there.
Overlogging is almost never a real problem in my experience. You can always discard log data you have but you can’t create log data you may need but didn’t produce.
I would disagree, after managing a 100 node ELK monstrosity to try to process the terabytes of logs generated by our systems.
Overlogging is not a problem if log levels can be easily changed (preferably without an app restart). One app logging like crazy is not a problem. A thousand of microservices, is.
The real solution is to make both automated and manual feedback extremely easy, for example send crashdumps without asking and have a forum connected to the users account where you redirect the user upon a crash.
Just like downgrading my GPU to a 1030, so I cannot play stupid games; this is a motivational trick to curb me into making things better!
Apparently I'm also somewhat of a masochist.. :S
In the end, to integrate Sentry and get actually useful information for monitoring, you have to do the same work you’d do for logging. Except now you’re tied to Sentry. Want some other service? Good luck.
Do log. And then send the log messages to Sentry or whatever you’re using, too.
Also, do overlog. It can be hard or impossible to attach a debugger to the deployed application. But you can always turn the log level up to 11 and then reconstruct what happened from the log messages.
And then catch it at the highest level, including all the stack trace. And then maybe send it to sentry, or log it, put it on data dog.
But don't catch exceptions all over your application code.
I have a related mental model for this: When you crash, crash loudly.
But in that case you probably still want to know about these failures, so you can review them from time to time, or see if there are upticks in failure frequency.
I have a third party which returns errors ~20% of the time outside of working hours (yes, really).
1. Exceptions should be sent to sentry. 2. Exceptions should not be normal. 3. Do not log.
Solve for ability to review & debug after the fact.
Then presumably those errors shouldn't be exceptions. Of course, there's a lot of a lot of assumptions behind this:
1. You have control over the code that interfaces with the third party. Otherwise presumably you'd have to write a shim that catches the exception and turns it into an error value, and that's probably not worth the effort.
2. You're work in a language that doesn't type check exceptions (i.e. not Java with checked exceptions) and your language also doesn't have a nicer way of handling errors as a (discriminated) union.
3. Your co-workers will go complete bananas if you start using some sort of result type or [success, value|error] tuple instead of an exception because that's how they've done it for 20 years.
The reason for this is that exceptions in most languages escape the type system and are in general more awkward to catch.
Of course, I'd also disagree with "do not log" in this context, though I might prefer a logging mechanism that allowed me to write queries on it (like recording values in a database).
Even if you really really like error return codes and really really don't like exceptions, you'll likely agree that writing very un-idiomatic code that goes against the grain of the language and all the libraries you might be using is likely a much bigger loss than the value you might have added by avoiding exceptions in your part of the block.
Out of curiosity, in what languages are error return types non-idiomatic?
This isn't to say the idioms are never broken or that they shouldn't be broken. I agree that if you are doing something like parsing a string to an integer, that should probably be a result type and not just throw an exception.
In languages like Java an Python, you can return error values but because you can also always return nulls, you're just making more trouble for yourself than it's worth avoiding exceptions.
It is not easy to stay away from exceptions if your language supports them.
But even for exceptional exceptions, you may still want or need to retry or branch based on what was thrown.
If you know that an operation is unreliable, you need to add retry / recovery logic and perhaps log or increment a metric that you can track what's going on; a high number of failures often indicates an exceptional situation.
In some cases, your retry logic will be a "Please try again" message to the user. I think that's perfectly acceptable, but you'll still want to know about it.
Yes, but you should catch as soon as possible (or validate stuff as soon ass possible) and re-throw with a better error message if possible. You will thank yourself when some problem appears and instead of a random "errorcode 123" you get a detailed message like "Operation Xyz failed with errorcode 123";
Usually, but not always. That is, for the classic "exceptions should only be used in exceptional cases" scenario, this is usually what you want to do.
A good portion of the time, though, this scenario doesn't apply. You're dealing with a subsystem that is throwing an exception (for good reasons or bad) but no, you don't want to just the application crash. You want to ... catch it, and do something else.
Which is what try-catch was devised for in the first place, after all.
It's not an either-or situation though. Even if you let the exception bubble up and sentry logs the exception, logging performed prior to the exception and at the time the exception is caught (prior to being bubbled up) as valuable context to how the exception happens, which can lower debugging time.
The exception just tells you that something went done, it doesn't always have the reason why something went wrong. And the context for why is not always available at the time of raising the exception (the context may be from a prior function call, or the consumer of the function that raised the exception).
Ok, I did all that. And I still want to know why this bug report happened, and logs help me. Why should I Not Log?
If this is a concern, just create a class with a method that takes an exception, message, context, etc. as parameters and use that. Now you can change it in one place instead of throughout your codebase.
Agree on logging messages as well; is useful to see live what is happening in the system as well as it is useful to see logging broken up by different errors
Please don't overlog. Only ever do this if you can dynamically change your log level on demand, without impacting users.
If you do have to reconstruct a sequence of events and you do not have distributed tracing, fine. Log the sequence of events _when needed_.
But please don't log messages on a perfectly fine system for no reason. It's noise and health can be better served by looking at metrics.
Because of this overlogging nonsense our systems generate terabytes of logs daily. Most of it is never ever looked at.
edit: how are you going to communicate information as to why something crashed to your colleagues who do not understand certain parts of the system as well as you? A stack trace is not effective at expressing intent. Explaining yourself in logs helps people triage more efficiently.
Not necessarily, e.g., you can buffer in memory, and write to storage later.
> loggers can fail and break your app
Anything can fail and break your app. Weak point.
A lot of devs think of logging as free; it's easy, it's almost always something that's either global-scope or standard library, and it's the first obvious solution to inspect internal program state when the program misbehaves.
But logs are a subsystem and adding any subsystem to your code increases your risk surface (for both crashes and privacy violations).
Took an operating systems class in college where the first thing the prof drilled into us was "Don't debug with printf. You're going to want to. It's the thing you're most accustom to having written application code. But you're writing an operating system now. If you have a memory overrun, is the first thing you want to do really allocating a kilobyte of stack to mangle strings and determine if you need localization support to tell yourself something's wrong with memory? By the time the output gets emitted to console, the memory you want to inspect will be a blasted wasteland."
As an example of how logging can break...
> you can buffer in memory, and write to storage later
You may as well not have logged then if the failure issue is a segfault; your program context will collapse before the buffer is output.
This reeks of functional programmer brain.
The world is not immutable. the world is weird. Your systems will interact with other systems that don't always quite work. Sometimes, their systems are wrong, or update without your knowledge, or theres other issues in routing, networking and whatever else.
Logging errors like "I can't resolve this domain" is more helpful than "my customers can't use this integration and business is at zero"
Also, this [1] is doing logging right. Completely underutilized.
[1]: https://stripe.com/blog/canonical-log-lines
If logs bring you some sort of value, then log away. If they don't, then don't. In some domains you absolutely crucially need logs, in others they're largely a waste of time.
Logging is probably the most foolproof way to extract debugging information from your application. The killer feature of logging is that standard out very rarely breaks, no matter how bad things get it works. It will work from the moment the application starts, it will work in an OOM situation, will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building, it will work no matter how bad things are. If logging does somehow fail, then that is on you, in that scenario you have fucked up your architecture somehow.
It is true however that looking at the individual replica, application logs should be very reliable.
And then you hope you are not logging to a logging service that'll try to execute something that was logged.
Because the reason to log is about people, not about technology.
There's no technical reason to make logs: Just don't make bugs, and you won't need the logs to find out what went wrong because nothing will have gone wrong. Go to the beach!
This is hard though, and bugs happen for all sorts of reason that have nothing to do with the programmer making a mistake-- the customer can be unsatisfied because they didn't know what they wanted. Programmers learn very early in their career to distrust the requirements, because they know that saying but that's what the ticket says gets them nowhere besides an argument: They have to do what the business/customer meant.
Put another way, the user doesn't want logs: The user wants to know if everything is okay or not, and presumably they have some way to figure that out, but if it isn't working correctly (or they merely think it isn't), the programmer will probably want those logs to help them.
In many environments, the loop is larger than just those two people, there's QA support people, and internal experts/operators, and triage, and all of those people are going to have different needs from those logfiles. People might also have a second use for those logs involving anticipation of failures and tracking capacity.
So there are a lot of non-technical, non-functional needs for logs, and once you understand this, it may be a little easier to understand why people would have almost rabid opinions about what those logs should look like (or if they can get the cure they need another way, what they might prefer instead of logs)
> [standard out] will work in an OOM situation
If you use write() which doesn't allocate memory, that's almost true, but if you use printf() it might allocate memory, and it might not work in an OOM anyway depending on what "standard out" actually is.
> will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building
And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway.
I've been programming for over three decades at this point, and I'm convinced there is no "easy one-size-fits-all answer" to whether to log or not, or if to log, how to log, so I find it useful to learn about this bit of theology so that I'm better equipped to talk to others about this, and maybe convince them of my opinions.
Reducing the number of points of failure is almost always a good idea. The network being down is one problem, not being able to ever know what your applications did when the network was down is another problem added to the first problem.
Reducing the single points of failure is usually a good idea.
You can easily reduce the number of points of failure by reducing redundancy and removing sanity checks from your process. Doing that does normally not improve your reliability.
Sending the logs to your SAN for it to write in a couple of disks arrays is much more reliable than writing to stderr (stdout is not a reasonable place to log failures) and redirecting it to your local disk.
A lot of complexity is there for good reason, a lot for bad reason, and some for no reason at all. It takes a very nuanced and specific work to determine what is what, and talking in generalities about it is a very bad idea on itself.
I think it is good to think about what happens when things fail: Things fail all the time and most of those failures are benign -- and even those that aren't can often be remedied by turning it off and on, but some of them aren't, so what happens then?
If you run out of disk space, writing a log line to the disk complaining that you're out of disk space is no help, but what else can you do? You could send a network alert, or print something to hardcopy (if there's a printer nearby), or ring a bell if you have one attached -- all of these things are also potential points of failure, but they can also help prevent a meltdown.
And if that sounds surprising, you might also be shocked the number of problems that can be solved with:
It's for this reason (I think) that people usually qualify your statement as reducing the number of "single" points of failure, instead of the way you put it, because having two single-points of failure isn't exactly like having two weak links in a chain, and in many cases, it's nothing like it at all.In any event, there's another advantage to thinking about the failure-process instead of trying to merely avoid failures:
The biggest reason reducing points of failure is a bad idea is when it is more expensive to do so than to simply tolerate the failure: One thing that dominates the attention of startups looking to improve their availability beyond 99.999% uptime, is "removing the failure points" to get higher uptime, but if Amazon is down, what are Amazon's customers going to do? Are they going to buy from someplace else? Sometimes it is simply that more-established brands have the benefit of ignoring some problems, but you can be sure a large company like Amazon has someone who knows exactly how much more they can spend per second of uptime they may desire, and I think it is worth it for a small company to get good at that exercise as well, because sometimes the juice just isn't worth the squeeze.
As a specific example: My ad servers (HTTP event collectors) are critical, and I want 100% uptime for them because missing a click means missing money, but the reporting dashboard (so the publishers know how much money they made, and the advertiser knows how much they owe me) doesn't need that, so as long as the user got redirected to the landing page correctly, and I can generate line charts and excel spreadsheets once a day, nobody cares that the reporting UI might be down for a few minutes a day-- they will simply press refresh, not knowing if it is their internet or mine, and combined with aggressive javascript-side caching and automatic retries, I am not quite sure any of my customers even notice my downtime.
If instead, I focused on reducing the things that could go wrong (because as you say, it is almost always a good idea), I might spend time trying to improve the availability of my database simply because that's the thing that fails instead of whacking a setTimeout into the JavaScript since the impact is that the user would just retry on failure. That setTimeout saved me the cost of a whole server (and power and heat and electricity) and in the early days of my ad server, that would've been a lot of beer.
That's the normal state of life; when a building burns down, forensic investigators pick over the evidence trying to work out what started a fire. They don't ask everyone to record every time two bits of metal touch, every switch switched, every use of electricity, every flammable material entering the building, just in case there's a fire one day. Use a teaspoon in a metal cup? Log it. Post-it note? Flammable, log it. Don't know if your shoes are flammable? Log them anyway just in case.
And yet the cause of the fire is not likely to be any of those things, it's likely to be the normalization-of-deviance fireworks next kept next to the open gas tank where the employees take smoke breaks which people were reporting for ages until they were told to shut up about it.
Beverley Hills Supper Club was known to be overcrowded, badly designed and ignoring fire department reports for years before it burned down. Cocoanut Grove nightclub had flammable decorations and an employee using flames for light. Rana Plaza had big cracks appear in the building before it collapsed, employees were told to go back to work or be sacked. The World Trade Center collapsed because a plane hit it. Titanic sank because it was going too fast and ignoring reports of icebergs and chose nice looking corridors over high bulkheads. Grenfell Tower burned while "everyone" knew the building was coated in flammable coating and nobody in power wanted to pay to do anything about it. Station Nightclub burned because someone accidentally brought outdoor fireworks indoors, and the sound damping foam was flammable not fireproof and it killed people because the emergency exit doors were locked.
We have a culture of log every order the captain makes, every response from the crew, every movement of the rudder, every radio signal sent and received, and at the same time "why did the front fall off?" "a wave hit it" "is that unusual?" "Yeah, at sea? Chance in a million."
> "you can write logs that nobody will ever be able to read, but what's the point of that?"
Cover your ass, cargo culting, habit, defaults are powerful, the cognitive bias effect like psychic powers where the one time people predict a bad thing happening and it happens is very memorable but the all the times people predict a bad happening and nothing happens are very forgettable, it's easy to remember one time logs helped and ignore all the times they didn't and the costs associated with them. For some possibilities.
It's industry specific. In my line of work, yes, they do ask you to make extensive records like that and it is why we log nearly everything. Though I've mostly been in safety-critical and aerospace work (space now, previously just aero). I imagine medical, pharmaceutical manufacturing, and others have similar requirements. If you're making systems that don't matter (in the sense of, if they fail no one cares except a few possibly angry users), then don't bother logging anything.
People treat logging as if it's free, when the reality is that it very often has a significant impact on performance, exacerbated by the fact that it's treated as free.
Add in the fact that today's distributed systems approaches add complexity to the logging paradigm and make it much harder (vs something like Jaeger/OTel) to understand the context of a single log message, and I would rather logging disappear almost entirely in favor of metrics (fast, coarse) and tracing (slower, fully contextualized), with some ability to fall back to just printing metrics and traces to stdout or other local debugging options as a fallback.
No it doesn't. Distributed tracing can do a far better job. As do metrics.
Visibility into programs running in environments you can’t access is unreasonably important, but any code that creates visibility will impact your performance, too. The “right” solution is to balance those two needs for your situation, which means there is no “right” solution for everybody.
There's your performance issue, logging doesn't need to be persisted or even processed by your main program thread or any thread/process actually doing the work. Who's logging straight to stdout or directly ::write'ing a file in a production application?
If you’re logging so much that you’re saturating a dedicated thread that just reads from a queue in shared memory, does log formatting and writes to a file then you have bigger problems than figuring out how to log because there isn’t going to be a solution that lets you log with a deterministic latency without skipping events or wrapping around your queue.
Of course, the approach scales fine if you can have multiple log files per process - e.g. a normal log file and a message log or transaction log, because you can give each file its own queue and thread.
That said I’m not sure what context we are discussing. I work on nodejs apps now where blocking is a non-issue. I’d obviously think about it more in C’s fprintf() or a similar block-by-default environment.
Edit: also, my area is fintech with $0.1-$1000 range per request, not telecom. I log everything because my employer can ask about everything, and it’s always money.
IMO, the advantage of logging is that it some degree of information about actual failure events - in contrast, monitoring only gives you aggregated information and tracing is too expensive to be applied everywhere at any time.
$thousands per transaction? Sample 100%. Cheaper to refund a customer than debug the problem every time? Then sample. You'll have plenty of sampled traces with the bug/error shown as soon as the problem is expensive enough to matter.
Metrics are great for demonstrating a problem exists (and alerting), and tracing is great for helping you understand how and why it happened once it's at a big enough scale to matter.
Beyond that, with a lot of what's coming out of Open Telemetry, you can get fancier like sampling a higher percent of traces to a localized processor middleware that then filters out any traces you doesn't care about (e.g. dropping non-errors or low-quantile latencies).
There is a lot of talk in this thread about the performance impact of logging. I don't see how sampling 100% wouldn't have a far greater performance impact.
> If logs bring you some sort of value, then log away.
I have been on the other side of the problem for years now. Logging has a cost. And the cost benefit is also not great. It requires a human to read it(either that or a regepx hellscape awaits you). This can be fine if you are working with a small service, but it quickly gets out of hand once scale increases.
I've had to deal with double digits(approaching triple) terabytes of logs daily. Most of the information logged was, like the author suggests, completely irrelevant. It makes it much harder to see what's actually relevant. And, for whatever was actually relevant, generally not enough context was provided (something the author also argues).
What you really have to have in production systems are metrics. That's what's going to tell you if shit is on fire and which ass is burning.
Has your app has hit a catastrophic failure and cannot continue? OOM? Sure, by all means log away. Did it have an error while trying to talk to a service? Are you retrying? Then _do not log_ until your retries (or whatever recovery mechanism) fail. Networks have become very reliable, but failures are expected and normal. Spit out metrics instead.
If you don't do this, some schmuck will have to spend their days trying to keep the logging infrastructure from falling over (Cthulhu helps you if you are on an ELK stack) while management will be complaining why noone seems to notice failures before customers do.
If you find yourself parsing logs to alert on issues, please revisit your architecture.
METRICS FIRST. Logs are a safety net and a tool for troubleshooters.
Logging is something that needs to be maintained (like all code), it can't be just some vestigial by-product of the development that's left to its own devices, at least not if it's to be of any use. The same is true for metrics. If your metrics are just some arbitrary selection of times and counts that seemed good at the time, they're probably not very useful. Which data to collect is something that needs to be deliberately chosen, and re-evaluated over the lifetime of the application. When you are launching a new product, then lots of logging is probably a good idea. When that product is mature, you probably should have dialed it back several notches, as you most likely already know the general areas where things go wrong.
IMO, it’s because the right logging solution for one situation is not the right solution for another situation. That means that one person is “right” and the other person is “wrong” (because the context required to identify why one person is right is complicated and hard to express, and is thus missing).
Observability (of which logging is a subset) has a performance impact, and software has performance requirements. This means that the right amount of observability is going to depend on the software, not a standard.
The issue is -- some people find value in the logs but some other people don't.
I am in the strong camp that find logging provide negative value, mostly due to the extra complexity it brings and its obstruction on code reading. And I believe the reason the other camp finds value in logging is because they lack insight to pin-point issues and lack effort to gain that insight. But I can't really use that argument because it can easily be deemed offensive and solicit slash backs rather than constructive discussions and debates.
Logging is my most frustrating pet peeves.
Perhaps I'm being too cautious (or too naive)? But it's saved my skin a few times.
The only thing that I don't like about this technique is that the 3 month period is arbitrary. But it works well enough for me.
Most log collecting solutions have ways of producing metrics and alerts from logs.
It takes more discipline to be able to understand why a metric or alert happened without supporting logs.
Sourcing logs from multiple services, including dependencies, allow for creating more intelligent metrics.
Treat logs as a stream of events rather than noise created by developers and logs become more useful.
Imagine you build all sorts of dashboards and Workflows on top of logs or log based metrics and the dev team changes the structure of the logs or just stops logging. Suddenly all your analytics is out of whack; this is because you created a hidden dependency.
Communication and the decision to depend on the logs as a source of metrics would be key to keeping these systems connected and functioning.
Taking a dependence on logs based metrics without communicating that is the failing here.
The same breakage could happen by depending on the event being emitted and a developer changing the event structure.
Communication and understanding helps both scenarios, not duplicating the effort by emitting logging for debugging, events for auditing and metrics for business.
PPSocialHighlightStorage: Constructed social attribution has missing data: (null)
What is that message doing in shipping software? It cannot possibly help anyone.
Correct. Those low level messages should not be present in 'production' environments _unless requested_. The ability to log extra info for debugging is fine. Doing that by default isn't.
(User-visible) Warnings and errors only please. Can always dig more and turn up verbosity if needed.
If the author is reading this, the Sentry-related link you have in the article [1] is currently giving the readthedocs.io 404 page. Same for the "docs on logging" link about 4 sentences earlier [2].
[1] https://stories.readthedocs.io/en/latest/contrib/sentry/
[2] https://stories.readthedocs.io/en/latest/debugging/
>don’t use logging, use business monitoring
Okay, if revenues suddenly drop 50% - then what? Where does the problem lie and once it’s discovered how can we alert on a recurrence of the same issue and fix the issue? Presuming the underlying issue cannot be trivially permanently remediated.
The author seems to see logging as essentially making excessive alert noise inevitable when logs are meant to just be collected somewhere and scanned for exceptions known to cause incidents. Then when that exception happens at the same time revenues fall 50%… there you have it?
The authors observations are mostly valid but their conclusions just don’t make sense. The observation you shouldn’t ping an admin every time a log generates an error doesn’t mean “don’t log”
The "real" point seems to be that logging should be useful and consistent, which I agree with.