Ask HN: How do you automate logging bugs in your product?
Recently, at my place of work, we've come into a bit of an issue with the way that log our bugs. Currently we automate logging many different types of exceptions and errors to our bug tracking system from our product. We're currently using unfuddle (which is a whole different religious war) to track bugs.
We've noticed that due to the automated manner of tracking bugs we have started to decrease our signal/noise ratio in our bug tracker due to the large number of duplicates that we're seeing. This makes triaging time consuming as we aren't sure which bugs are legitimate issues or possible duplicates of other bugs we've seen before. Our team has been on a recent stent of patching noisy bugs as they come in, but it feels like these are merely bandaids to a larger problem.
How do you automate logging bugs in your product?
Some examples of bugs that we've been having issues with :
- Loops in automated processes can hit states that raise exceptions. These can generate a large number of bugs.
- Database restores that may fail in a staging/production environment
22 comments
[ 2.2 ms ] story [ 55.8 ms ] threadYou can check out their github repo as well https://github.com/getsentry/sentry
For the side business I work (https://styleci.io) we use Bugsnag; https://bugsnag.com and a custom application.
Directly posting every single exception raised to your bug tracker is a sure fire way to flood and overwhelm you. Logging to unfuddle, to me, is the wrong way to do it.
Learn proper Ops. Learn the concepts properly: logging for application monitoring and trace debugging is one thing. Posting a confirm functional bug report is another.
Error detection/escalation is best left to proper tools such as nagios/zabbix where they have grouping of errors etc.
Application level tracing is best left to a logging tools like sentry/logstash/new relic/etc.
Use the d
Wanting to know when runtime exceptions happen (and grouping them) is very helpful a lot of times but it seems like by itself (without context) would only get in the way of trying to parse what is and isn't really happening in some cases.
Especially when trying to sort out what needs immediate attention and what needs attention at some point later.
What does your workflow for logging exceptions and errors look like?
What sort of separation do you have between application exceptions and other types of exceptions such as database connectivity? How do either of the aforementioned exceptions/errors materialize into a bug?
I meant "confirmed functional bug report"; and by that I mean a reproducible scenario that actually is caused by faulty code path leading to wrong result/behaviour. Usually, in my experience, bugs are ideally reported by people after some repeatable / confirmable thought-out process.
If I understood you correctly, you are automatically posting every exception to your bug tracker. My point is that not all exceptions are bugs. And certainly not all bugs are discoverable through exceptions.
> What sort of separation do you have between application exceptions > and other types of exceptions such as database connectivity? How do > either of the aforementioned exceptions/errors materialize into a bug?
These are great questions that you should be asking while architecting / implementing / testing / running your application (i.e. whole lifecycle). Again, not all exceptions are bugs. Many, but not all exceptions are to die / crash your program. Many exceptions are definitely recoverable (unexpected input), while others are not (failed network, unreachable DB). If you are using a decent logging framework, it’s obvious that they have different levels for logging messages. FATAL/ERROR usually mean unrecoverable, while WARNINGs are typically recoverable but may prompt further investigation for the log audience, and INFO/DEBUG are generally harmless, non-error messages. This design has become widespread across many languages/frameworks and makes you ponder why.
> What does your workflow for logging exceptions and errors look like?
My view is that error handling is generally a delicate art and science (see why Golang doesn’t have exceptions, a decision made by veteran language designers!). I follow the principle of Fail Fast, especially for unrecoverable errors; that is, don’t hide fatal errors. Bubble up exceptions. Carefully design your layers to raise and catch the appropriate exceptions, if at all (see also Erlang’s let it crash principle). c2 wiki has some good starting discussions: http://c2.com/cgi/wiki?FailFast http://c2.com/cgi/wiki?FirstRuleOfLogging
Note, these are all abstract and conceptual, and not tool oriented. They go back to the design of your system and your processes, to fit your organization / communication structure.
Also not stated is the great importance of automated tests (unit tests / integration tests / functional tests). Remember I mentioned confirmed bugs? Because at the very least, every bug discovered should have a corresponding test written to prevent regression. A decent engineering team should have good grasp and practice of many levels of automated testing.
P.S. as for our tools, our stack uses logstash/elasticsearch for distributed logging + opsview/nagios NRPE for ops-related detection/escalation of issues that warrant attention.
Disclaimer: I work for Sumo Logic.
Right now, we don't have a ton of traffic so the errors channel receives a very manageable volume. I imagine this sort of approach would get overwhelming for a product with more concurrent users or more runtime-error-generating code paths.
https://developers.google.com/analytics/devguides/collection...
[0] https://rollbar.com/