“Do not log specific things, just pay for a service that logs everything”
The false dichotomy between monitoring and logging here is so bizarre, they’re not mutually exclusive. When an alert wakes you up on a weekend, having a log statement with useful context is not going to kill you...
Also:
> Isn’t it a bit too hard?! Look, I just want to write strings into stdout. And then store it somewhere.
So do that? Why is the author pretending you need to use ELK stack?
And what about their example is “too hard”, if you’re deploying your own application that’s the least of your complications, and if you aren’t and already paying a premium for it, there’s plenty of managed ELK and ELK-like products (and I’d also be surprised if your managed deploy platform doesn’t already support you just logging to STDOUT)
This article doesn't look at the usecases for logging.
In my mind, there are audit logs, to know what happened during an incident. You don't know what the incident will be, so you should log as much maybe-useful stuff as possible, and you only look at the logs if some incident requires attention.
There also logs to debug an issue that occurs in production but can't be diagnosed any other way. When a customer says "I clicked the log in button, but just saw a blank screen", and you have no other records, the logs will be handy.
I wonder if that person ever tried to debug an obscure support case for a product that is installed on some customer on-prem hardware with ridiculous settings.
This article is pretty confused. For example, the concluding section is titled "What to do instead?" but doesn't actually address that question. (If this were the only problem, I'd think it was an accidental leftover from an edit, but it's all only partially coherent.)
There are some good parts. It's not all wrong or anything. But I don't think it's worth the read.
Maybe a good takeaway is: consider which monitoring systems might accomplish your business purpose better.
Also, I guess: a full-featured logging subsystem in a distributed system can be substantial. True but not too interesting because that's pretty obvious, often not needed, wouldn't normally be implemented from scratch any more than a monitoring system might be, etc.
Articles like this are often written by someone that manages a trivial or non-critical path system. They have strong feelings about logging because they lack other things to have strong feelings about.
This person seems confused in general. They say "don't log, send it to sentry". Sentry is logging. Sentry is usually configured as a logging destination. It just does some preprocessing on those logs to aggregate and enrich them.
I don't know anyone that fishes through text logs over ssh anymore. We use simple automations to roll everything up for convenient access.
Furthermore he overlooks other use cases for logging. Including analytics, fixing concurrency/heisenbugs, and a myriad of other problems that logging addresses.
> I don't know anyone that fishes through text logs over ssh anymore.
Good points, though it's funny: In my day job I only ever fish through logs (not by ssh -- I don't have that much access to the systems).
But that's only because by the time it gets to me multiple levels of people (who really know what they are doing, BTW, so it doesn't happen often) have done everything else.
There is a lot of overlap between logging, monitoring and crash/error reporting. In Sentry with some systems it does hook up to your logging framework to gather extra context through breadcrumbs or adds a hook to send logged error reports (in addition to the error reports from exceptions and crashes). They also follow logging good practices and integrate with them, but if you need logging, you can't use sentry for that. It only keeps the relevant information to a specific crash, and it aggregates data from crashes of a specific type so you won't even get historical data from crashes in some instances. Quoting from one of their blogs:
>>>
Sentry is built on top of best practices like logging. However, this can cause confusion and signal-to-noise issues if you don’t understand the the difference between logging and error reporting.
Throughout your application you are likely logging many kinds of events. While these are commonly errors, you are probably logging a lot of debug information as well. This is the fundamental reason why logging has levels (e.g., debug, warning, error). When we abstract our crash reporting through logs, the challenge is to differentiate between actual errors and debug noise.
>>>
https://blog.sentry.io/2016/02/09/what-is-crash-reporting
I was introduced to the real benefits of logging on an Erlang project. They used a pseudo-csv log format and it was possible to do a lot with these logs. For example you could process them with cut/uniq/grep/etc in order to produce statistics for stuff that you weren't recording statistics for. If there was some kind of dataloss because of a bug it was sometimes possible to reconstruct the corrupted/missing data from the logs. If you had a problem with a particular transaction you could just grep for the transaction_id and see how it moved through the system to narrow down what happened to it.
I'm now working on an Elixir project but the way Elixir does logging is kind of dangerous. So you have a dynamically typed language that relies on unit testing in order ensure stuff works, then you have logging methods where the compiler will remove them depending on what log-level you have configured for your environment. Now run different log-levels for production and testing. What could go wrong? It seems insane that the framework would let you do this and not have a more sensible default like run the log methods but drop the output to the floor and you have to do a bit more work in order to make the log methods be completely removed.
Just flew over the article, but it made me wonder: When I'm developing scripts for our workflow systems I'm often also "logging" successful events. It helps with looking how a process works but in the end I never remove that logging output.
I wonder how you people handle that. Do you also log "good events", for example when nothing happened just to be able to see everything went well?
If you are not using something to aggregate statistics using something like prometheus or graphite then logging good events can be really useful in order to produce statistics by doing text processing on the logs. It can also be a good first step towards introducing something like prometheus or graphite. Once people can see the kind of insights into an app you can get from dumb logging and manual text processing it becomes easy to sell some kind of metrics recorder to the team.
I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems. Relatedly, all they look at is error logging.
Admittedly, there's a blurry line here between logging and tracing, but the latter word generally evokes externally attaching additional tools. But that "far end" of tracing is frequently not what's needed... there's a reason logging systems have "info" and "debug" severities.
Either way good (i.e. configurable, low overhead) logging is worth its weight in gold.
When I noticed that some of the people who complained loudly about certain practices also tended to disappear when a problem was too difficult for them to solve, I stopped letting what they think bother me.
Bosses notice exceptional events. As the team or cluster get bigger, they start happening closer together. Depending on context, any event that happens once a week or once a day will be characterized as happening “all the time” and that creates a new set of problems.
You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.”
> You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.”
That is one of the feelings I dislike the most as someone who takes pride in building quality software. It’s important to be honest when you don’t know why something is wrong, but even more important to minimize the number of times it happens. Logging is incredibly beneficial for the latter, and complements metrics + exception-tracking.
> I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems.
That includes rare conditions where you have to backtrack from error to root cause based on information already in the log. I worked for years on data replication and got into the habit of logging anything that might affect data integrity at INFO level to ensure it would be available to guide debugging.
Simple examples: where did you read config values from on startup? Log the file name including the full path. Rebooting?
Log all steps in the boot sequence in detail. Retries of calls to remote components? Log that too--put in a counter to log every N failures so you don't flood the log. Transaction aborts? Log them as well using the same counter method. Extra credit: Use ring buffers to dump current transaction state on fatal errors.
If things are OK the log is fairly small.
This kind of logging is essential in transaction processing systems. You can't log every transaction because (at least in my case) there were billions of them. But you should log everything that gives insight into whether the system is meeting preconditions for safety and liveness. If you need more info you can turn on traces but this will at least give you an idea where [not] to look.
I think the point here is to not do unstructured, string-based logging. If logs are valuable, then you should treat them the same way as any other valuable data: rather than blatting string soup into some datastore and hoping you can somehow index it in the future, write a proper data schema and represent the trace of your program execution in a structured, machine-readable way.
This ends up overlapping a lot with event sourcing: if you're doing event sourcing right, with every substantial business decision recorded as an event and your logic decomposed into lots of fine-grained transformations between event streams, then the event streams can be your logs: each successive event transformation either succeeds or fails, and if it fails then you have the input event and the small piece of code in which the failure occurred, so you shouldn't need any more. If a given transformation gets too complicated to debug, split it into two smaller transformations!
You still need alerting on failures, but again, that's something that's better done in a structured way, with proper stacktraces and business-relevant information.
I could never get structured logging to work, syslog receiver for structured logs is a nightmare to set up. Also, when I start writing logs, I just want to quickly set up something to check my error state and move on. The onus of setting up structured logging output in the code is too much work in the immediate moment. Additionally, structured logging does not solve the problem we think it solves, what is the log trying to say. What do I use as my key-value naming convention? fault_type = major, component_name = X, subcomponent_name = X.aaa desc1 = "some text" desc2 = "fooblat" -- I just made all those things up, the log consumer is still going to need to parse out the KV pairs and decipher what that means and what to do. Meanwhile, we have RFC5424 which does a fair job of defining and standardizing those made-up things.
I meant store logs the same way as any other data - be that an SQL database, Cassandra, or whatever you're favouring for this system. I'd expect your fields to be somewhat business-specific, and reading them works the same as reading any other business data - you write the data schema in one place and generate read and write code from it. key-value pairs aren't perfect and you certainly can (and should) define a more detailed schema than that, but even just a bag of key-value pairs is a lot more structure (and therefore a lot easier to query) than a raw string.
Well yeah, because when we're storing structured data we don't think of that as "logging". But the point isn't to throw that information away, it's to store it properly.
No I’m probably going to do what he had in the introduction and just dump major events or errors to a text file because it’s cheap, easy, and it’s usually all that’s needed to diagnose end user issues (I do desktop apps and can’t do any real time monitoring but on the other hand logs are perfect chronological records of the actions of one user on one document - not of just one system as is common for web apps).
It’s not that dumb logs are great, but they provide 50% of what you need at almost no cost.
If I were able to dream and could have the amount of thought and design that he describes in any part of my code - logging would be pretty far down the list of subsystems I’d refactor.
As greatly represented in this comment section, this article touches on maybe 1/10 of the usecases of logging...and than pretends that the opinion is applicable everywhere.
> And yes, you got it correctly: it sounds like a job for Sentry, not logging.
okay, I guess I will not write a log file for my Raspi bash script then and instead setup a third party service?
> We don’t really care about exceptions and how to handle them. We care about the business value that we provide with our app.
That's how you get shitty software which can be fixed relatively quickly regarding large scale black swan events (everything broken) and retains their annoying edge case bugs forever, slowly driving users away...
All the while, the arguments against logging seem to be the same against implementing, well anything. More things = more complexity. New things = new problems. By all means be aware of these pitfalls in the context of logging and anything else.
But as a whole, this article awfully smells like an unreasonably strong opinion wrapped in pseudo-intellectualism.
> this article touches on maybe 1/10 of the usecases of logging...and than pretends that the opinion is applicable everywhere.
It's definitely a weird article.
I maintain an open source tool called dbcrossbar that automates data transfer between several kinds of databases. It uses multiple async streams to copy data, and it has a "backpressure" system that keeps every step of the transfer in sync. Usually it's very reliable, but if it goes wrong, then I need debugging tools.
We use the "slog" library for structured logging. This allows us to add extensive structured data to our logs, and to keep track of what's coming from where. We can increase or decrease the log level for each component individually.
And these logs are invaluable for debugging. Like, almost every time we have any problem at all, we all sit around saying, "Wow, we're so glad we have such good logging."
>> And yes, you got it correctly: it sounds like a job for Sentry, not logging.
> okay, I guess I will not write a log file for my Raspi bash script then and instead setup a third party service?
They're not even exclusive. You can and should feed your logs into sentry. It can use low-sev logs as breadcrumbs and the high-sev logs as events: https://docs.sentry.io/platforms/python/logging/
Having extensive logging and sentry makes your sentry reports significantly better, it's not an either/or situation.
> We don’t really care about exceptions and how to handle them. We care about the business value that we provide with our app.
This is famously not true. When the system is working, “we”, as in the business, only care about the business value. The moment things break or get slow it’s all recriminations about “how could you let this happen?”
Then later it’s right back to, “why can’t you go faster?” until it happens again.
>If so, let’s refactor our code to be type-safe. And eliminate all possible exceptions that can happen here with mypy.
All possible exceptions eliminated when calling doSomethingComplex(). Perhaps I'm missing something here and this overly broad language is more targeted than i've realized while reading... But that just sounds silly to my ears. And saying that making something type safe solves "all possible errors"... What am I missing?
I don't want to have to guess why your service is crashing. I'd much rather be able to jump in to the logs and find the exact exception information so I can fix it quickly.
I kinda agree and disagree with article's author. You see, do log, but do so in a compiler switch so that your production code has no logging in final form and is easy to generate.
Get 2 variants, one with logging and debugging and whatnot, which is used for alpha, beta, gamma / whatever version and the clean one that you release officially.
And yes, I'm aware that sometime a bug will creep only in the release version while in the debug version will not happen. What can I say, those are where your experience really is showing, so you can understand the root of the problem.
> I either want a quick notification about some critical error with everything inside, or I want nothing: a peaceful morning with tea and youtube videos. There’s nothing in between for logging.
Until you do. Then, instead of having access to it, you need to hope you can recreate it.
Meanwhile you can filter your logging such that the critical stuff still comes to you and the other stuff is sent into the ether and eventually GC'd
> I am fighting an “overlogging” culture. When logs are used just for no good reason. Because developers just do it without analyzing costs and tradeoffs.
The article would have been better if it started with the conclusion, and could have done with a good bit of editing, but outrageous claims do stimulate discussion.
As a user of many programs I always find it very frustrating when some program
does not work and produce no log information whatsoever.
As an engineer, I always write detailed and human-readable log entries with as
much context as possible which is split into trace, debug, warning and info
levels.
With trace level enabled it's typically possible to reconstruct program
execution step-to-step, which is _invaluable_ when you need to debug some rare
problem.
Even if logs are ephemeral and not persisted it's better to have logs for a run
than not having logs at all.
Logging as a side effect is irrelevant if you look at your code from the
pragmatic standpoint, from which program execution transparency matters more
than some abstract elegancy.
At least in Javaland, the "should I log" condition test for most logging libraries takes nanoseconds. Unless you're in an extremely tight loop, performance is fine.
If it was absolutely necessary you could even have the JIT remove the code and then de-optimize when you enable tracing[1]. That would solve the tight-loop issue..
Correct, in simple cases it can do it, but you need to be very careful with computing the arguments for the logging call. Java has no lazy evaluation of args nor macros, so if you accidentally concat strings there or do something expensive, it is likely JVM won't optimize these out. That's why typically you want to guard your logging calls on the hot path with an if condition.
I code mostly in Go. And yes, I'm writing `log.Tracef` everywhere.
Might seem tedious, but in fact it's easily automated using snippets in the editor.
Also when I'm inserting trace statements during development it provides a great help since it's much easier to debug logic.
Regarding performance: I don't think that we need to worry about log performance. Most applications we have this days are not performance-bound to the logging. And all it takes to disable particular log level is single `if` in the code. Or a build flag in more extreme cases.
> There are a lot of cases when we don’t care. Because of the retries, queues, or it might be an optional step that can be skipped. If this failure is not important - then just forget about it.
No! Log it and still retry.
There are lots of reasons why might want to know later on if something failed the first time and then had to retry, for example when debugging possible timing issues.
> And yes, you got it correctly: it sounds like a job for Sentry, not logging.
Sentry is a form of logging. Also, sometimes you have potentially private data in local variables that you absolutely cannot risk sending to a service over the network; a log message allows you to be very careful with what information you reveal and what you don't reveal.
Which brings me to another use case for logging: sometimes you simply cannot report exact errors to customers (because they might leak the existence or nature of other customer's data), but if the customer contacts support, you absolutely want that information in the logs (even if just to decide if the vague error message was intentional or a bug).
Don't get me wrong, I'd love to live in a world where every action straight out succeeds or results in a bug ticket, but the real world is much more complicated, and logs do help making sense of that.
Both logging and metrics can be useful in such a case.
Metrics are useful to get a general picture, and logging is useful when you try to find out in one specific instance what happened.
If you have a billion requests per day, you likely don't want logging for retries; if you have a small number of requests, many of them important enough to investigate individual errors, you absolutely want logs on what happened to each request.
Imagine the do_something_complex function is executed as an automated nightly job. No humans involved. And let's say it needs to access a remote API or a database. One fine day some network changes happen in the external system without your knowledge and your function simply can't connect to the external API. Next morning how are you gonna know why your job failed without logging the exception?
It is my first time working on production grade code which has SLAs with other parties and well, for me, logging is crucial.
1. Logging the whole end-to-end chain allows us to view "whose fault" it is basically. We work in a microservices landscape and any request coming in the system goes through several teams. Pinpointing the team responsible for fixing their shit is important to get things up again.
2. Sometimes it happens that we (my team) drops the ball, we have some custom logging that traces the whole request through our application and we can quickly pinpoint what function failed or again whether it is a third party API call that failed or something.
For me, logging is very important, but perhaps there are better solutions available that I'm unaware of, always eager to learn and improve.
86 comments
[ 2.8 ms ] story [ 159 ms ] threadThe false dichotomy between monitoring and logging here is so bizarre, they’re not mutually exclusive. When an alert wakes you up on a weekend, having a log statement with useful context is not going to kill you...
Also:
> Isn’t it a bit too hard?! Look, I just want to write strings into stdout. And then store it somewhere.
So do that? Why is the author pretending you need to use ELK stack?
And what about their example is “too hard”, if you’re deploying your own application that’s the least of your complications, and if you aren’t and already paying a premium for it, there’s plenty of managed ELK and ELK-like products (and I’d also be surprised if your managed deploy platform doesn’t already support you just logging to STDOUT)
In my mind, there are audit logs, to know what happened during an incident. You don't know what the incident will be, so you should log as much maybe-useful stuff as possible, and you only look at the logs if some incident requires attention.
There also logs to debug an issue that occurs in production but can't be diagnosed any other way. When a customer says "I clicked the log in button, but just saw a blank screen", and you have no other records, the logs will be handy.
There are some good parts. It's not all wrong or anything. But I don't think it's worth the read.
Maybe a good takeaway is: consider which monitoring systems might accomplish your business purpose better.
Also, I guess: a full-featured logging subsystem in a distributed system can be substantial. True but not too interesting because that's pretty obvious, often not needed, wouldn't normally be implemented from scratch any more than a monitoring system might be, etc.
Articles like this are often written by someone that manages a trivial or non-critical path system. They have strong feelings about logging because they lack other things to have strong feelings about.
This person seems confused in general. They say "don't log, send it to sentry". Sentry is logging. Sentry is usually configured as a logging destination. It just does some preprocessing on those logs to aggregate and enrich them.
I don't know anyone that fishes through text logs over ssh anymore. We use simple automations to roll everything up for convenient access.
Furthermore he overlooks other use cases for logging. Including analytics, fixing concurrency/heisenbugs, and a myriad of other problems that logging addresses.
Good points, though it's funny: In my day job I only ever fish through logs (not by ssh -- I don't have that much access to the systems).
But that's only because by the time it gets to me multiple levels of people (who really know what they are doing, BTW, so it doesn't happen often) have done everything else.
Sentry IS logging.
See: https://news.ycombinator.com/item?id=22574712
>>> Sentry is built on top of best practices like logging. However, this can cause confusion and signal-to-noise issues if you don’t understand the the difference between logging and error reporting.
Throughout your application you are likely logging many kinds of events. While these are commonly errors, you are probably logging a lot of debug information as well. This is the fundamental reason why logging has levels (e.g., debug, warning, error). When we abstract our crash reporting through logs, the challenge is to differentiate between actual errors and debug noise. >>> https://blog.sentry.io/2016/02/09/what-is-crash-reporting
I'm now working on an Elixir project but the way Elixir does logging is kind of dangerous. So you have a dynamically typed language that relies on unit testing in order ensure stuff works, then you have logging methods where the compiler will remove them depending on what log-level you have configured for your environment. Now run different log-levels for production and testing. What could go wrong? It seems insane that the framework would let you do this and not have a more sensible default like run the log methods but drop the output to the floor and you have to do a bit more work in order to make the log methods be completely removed.
I wonder how you people handle that. Do you also log "good events", for example when nothing happened just to be able to see everything went well?
Admittedly, there's a blurry line here between logging and tracing, but the latter word generally evokes externally attaching additional tools. But that "far end" of tracing is frequently not what's needed... there's a reason logging systems have "info" and "debug" severities.
Either way good (i.e. configurable, low overhead) logging is worth its weight in gold.
Bosses notice exceptional events. As the team or cluster get bigger, they start happening closer together. Depending on context, any event that happens once a week or once a day will be characterized as happening “all the time” and that creates a new set of problems.
You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.”
That is one of the feelings I dislike the most as someone who takes pride in building quality software. It’s important to be honest when you don’t know why something is wrong, but even more important to minimize the number of times it happens. Logging is incredibly beneficial for the latter, and complements metrics + exception-tracking.
That includes rare conditions where you have to backtrack from error to root cause based on information already in the log. I worked for years on data replication and got into the habit of logging anything that might affect data integrity at INFO level to ensure it would be available to guide debugging.
Simple examples: where did you read config values from on startup? Log the file name including the full path. Rebooting? Log all steps in the boot sequence in detail. Retries of calls to remote components? Log that too--put in a counter to log every N failures so you don't flood the log. Transaction aborts? Log them as well using the same counter method. Extra credit: Use ring buffers to dump current transaction state on fatal errors.
If things are OK the log is fairly small.
This kind of logging is essential in transaction processing systems. You can't log every transaction because (at least in my case) there were billions of them. But you should log everything that gives insight into whether the system is meeting preconditions for safety and liveness. If you need more info you can turn on traces but this will at least give you an idea where [not] to look.
What is it? Performance?
This ends up overlapping a lot with event sourcing: if you're doing event sourcing right, with every substantial business decision recorded as an event and your logic decomposed into lots of fine-grained transformations between event streams, then the event streams can be your logs: each successive event transformation either succeeds or fails, and if it fails then you have the input event and the small piece of code in which the failure occurred, so you shouldn't need any more. If a given transformation gets too complicated to debug, split it into two smaller transformations!
You still need alerting on failures, but again, that's something that's better done in a structured way, with proper stacktraces and business-relevant information.
* make logging an explicit part of your contracts
* Logging should be done right
No I’m probably going to do what he had in the introduction and just dump major events or errors to a text file because it’s cheap, easy, and it’s usually all that’s needed to diagnose end user issues (I do desktop apps and can’t do any real time monitoring but on the other hand logs are perfect chronological records of the actions of one user on one document - not of just one system as is common for web apps).
It’s not that dumb logs are great, but they provide 50% of what you need at almost no cost.
If I were able to dream and could have the amount of thought and design that he describes in any part of my code - logging would be pretty far down the list of subsystems I’d refactor.
As greatly represented in this comment section, this article touches on maybe 1/10 of the usecases of logging...and than pretends that the opinion is applicable everywhere.
> And yes, you got it correctly: it sounds like a job for Sentry, not logging.
okay, I guess I will not write a log file for my Raspi bash script then and instead setup a third party service?
> We don’t really care about exceptions and how to handle them. We care about the business value that we provide with our app.
That's how you get shitty software which can be fixed relatively quickly regarding large scale black swan events (everything broken) and retains their annoying edge case bugs forever, slowly driving users away...
All the while, the arguments against logging seem to be the same against implementing, well anything. More things = more complexity. New things = new problems. By all means be aware of these pitfalls in the context of logging and anything else.
But as a whole, this article awfully smells like an unreasonably strong opinion wrapped in pseudo-intellectualism.
It's definitely a weird article.
I maintain an open source tool called dbcrossbar that automates data transfer between several kinds of databases. It uses multiple async streams to copy data, and it has a "backpressure" system that keeps every step of the transfer in sync. Usually it's very reliable, but if it goes wrong, then I need debugging tools.
We use the "slog" library for structured logging. This allows us to add extensive structured data to our logs, and to keep track of what's coming from where. We can increase or decrease the log level for each component individually.
And these logs are invaluable for debugging. Like, almost every time we have any problem at all, we all sit around saying, "Wow, we're so glad we have such good logging."
> okay, I guess I will not write a log file for my Raspi bash script then and instead setup a third party service?
They're not even exclusive. You can and should feed your logs into sentry. It can use low-sev logs as breadcrumbs and the high-sev logs as events: https://docs.sentry.io/platforms/python/logging/
Having extensive logging and sentry makes your sentry reports significantly better, it's not an either/or situation.
This is famously not true. When the system is working, “we”, as in the business, only care about the business value. The moment things break or get slow it’s all recriminations about “how could you let this happen?”
Then later it’s right back to, “why can’t you go faster?” until it happens again.
All possible exceptions eliminated when calling doSomethingComplex(). Perhaps I'm missing something here and this overly broad language is more targeted than i've realized while reading... But that just sounds silly to my ears. And saying that making something type safe solves "all possible errors"... What am I missing?
I don't want to have to guess why your service is crashing. I'd much rather be able to jump in to the logs and find the exact exception information so I can fix it quickly.
Get 2 variants, one with logging and debugging and whatnot, which is used for alpha, beta, gamma / whatever version and the clean one that you release officially.
And yes, I'm aware that sometime a bug will creep only in the release version while in the debug version will not happen. What can I say, those are where your experience really is showing, so you can understand the root of the problem.
Until you do. Then, instead of having access to it, you need to hope you can recreate it.
Meanwhile you can filter your logging such that the critical stuff still comes to you and the other stuff is sent into the ether and eventually GC'd
> I am fighting an “overlogging” culture. When logs are used just for no good reason. Because developers just do it without analyzing costs and tradeoffs.
The article would have been better if it started with the conclusion, and could have done with a good bit of editing, but outrageous claims do stimulate discussion.
No.
As a user of many programs I always find it very frustrating when some program does not work and produce no log information whatsoever.
As an engineer, I always write detailed and human-readable log entries with as much context as possible which is split into trace, debug, warning and info levels.
With trace level enabled it's typically possible to reconstruct program execution step-to-step, which is _invaluable_ when you need to debug some rare problem.
Even if logs are ephemeral and not persisted it's better to have logs for a run than not having logs at all.
Logging as a side effect is irrelevant if you look at your code from the pragmatic standpoint, from which program execution transparency matters more than some abstract elegancy.
For context, I mostly program in python. I could imagine compile-time loglevel settings could address the performance, though.
[1]: https://community.oracle.com/blogs/forax/2011/12/17/jsr-292-...
Might seem tedious, but in fact it's easily automated using snippets in the editor.
Also when I'm inserting trace statements during development it provides a great help since it's much easier to debug logic.
Regarding performance: I don't think that we need to worry about log performance. Most applications we have this days are not performance-bound to the logging. And all it takes to disable particular log level is single `if` in the code. Or a build flag in more extreme cases.
No! Log it and still retry.
There are lots of reasons why might want to know later on if something failed the first time and then had to retry, for example when debugging possible timing issues.
> And yes, you got it correctly: it sounds like a job for Sentry, not logging.
Sentry is a form of logging. Also, sometimes you have potentially private data in local variables that you absolutely cannot risk sending to a service over the network; a log message allows you to be very careful with what information you reveal and what you don't reveal.
Which brings me to another use case for logging: sometimes you simply cannot report exact errors to customers (because they might leak the existence or nature of other customer's data), but if the customer contacts support, you absolutely want that information in the logs (even if just to decide if the vague error message was intentional or a bug).
Don't get me wrong, I'd love to live in a world where every action straight out succeeds or results in a bug ticket, but the real world is much more complicated, and logs do help making sense of that.
You will likely say this is a form of logging, and you’re right, but I think this is a better fit for a metric rather than a log.
“Something something retried at…” isn’t nearly as useful as the “the number of retries peaked at…”
Metrics are useful to get a general picture, and logging is useful when you try to find out in one specific instance what happened.
If you have a billion requests per day, you likely don't want logging for retries; if you have a small number of requests, many of them important enough to investigate individual errors, you absolutely want logs on what happened to each request.
Record retries: yes. Log them? Mmm, maybe not.
How to take the information out of information technology.
or
How to have no clue about why an edge case crashed your systems.
1. Logging the whole end-to-end chain allows us to view "whose fault" it is basically. We work in a microservices landscape and any request coming in the system goes through several teams. Pinpointing the team responsible for fixing their shit is important to get things up again.
2. Sometimes it happens that we (my team) drops the ball, we have some custom logging that traces the whole request through our application and we can quickly pinpoint what function failed or again whether it is a third party API call that failed or something.
For me, logging is very important, but perhaps there are better solutions available that I'm unaware of, always eager to learn and improve.