errorShip is a python library that sends exceptions/errors generated by your application to your datadog account.
Are you tired of looking at metrics in datadog and then switching over to another website to track your applications exceptions?
errorship exists to solve that context-switching problem, among others.
It's a bit like sentry, bugsnag, rollbar etc; except implemented in your own datadog account.
I'm happy to receive any feedback or just chat about it.
Can you add a little more color commentary? I haven't used datadog, but they seem clear that there's both a logging and metrics (apm) side to their product. Is the logging side neglected somehow?
> but they seem clear that there's both a logging and metrics (apm) side to their product.
yes they have those two sides. what they do not have is an error tracking side; something like sentry[1].
errorship exists to bring the error tracking side into datadog.
The logging side of datadog is like splunk or elasticsearch, and their APM side is like newrelic. errorship is bringing the error tracking side.
some of the analogies may not apply 100%, but I hope I'm making sense.
I'm guessing that using the DD event stream rather than purchasing logging support could be a cost saving, although not sure how DD would feel about that.
errorship is not using the datadog eventstream to send all logs to it. Rather, errorship just sends exceptions, their stack traces(and a few other metadata like hostname) to the eventstream.
> although not sure how DD would feel about that
The datadog documentation[1] says; "An event represents any record of activity noteworthy for engineers"[1].
We certainly believe that exceptions that occur in engineer's applications are noteworthy activities and as such belong to the eventstream.
> errorship is not using the datadog eventstream to send all logs to it.
Yep, noted that, and I like how you preserve syntax highlighting in the stack traces :)
Out of curiosity, I've only ever dabbled in Datadog, is there any limit on event stream size or retention? I couldn't find anything mentioned in the docs.
I had also noticed the lack of mention in the docs, so I reached out to datadog sometime last year. Their response was that it doesn't have any limits or quota
Interesting. Is there some disincentive that discourages it? Is it hard (or costly) to query, for example...compared to paying specifically for logging?
The main benefit is that you do not have to context switch from datadog metrics/logs to go look at your exceptions in sentry.
With errorship, all these are made available in one place; in your datadog account.
You also do not need to maintain two services. If you are already using datadog(maybe their APM and their logging and metrics service) then you might as well use them for error tracking instead of maintaining an additional account with sentry.
However, it is not a must that you give up sentry to use errorship. errorship will work just as fine, if you choose to continue with sentry.
If your server is down, my application would crash too. Just cut the license validation out of the library. If I wanted to use the library without an license I could do so anyway.
edit: as noted by the author below, this is not the case :). If the server is not available, it won't raise an exception. I did miss that part somehow.
> If your server is down, my application would crash too.
The errorship library is written in such a way that it fails open. If our servers are down(or any other failure), it does not affect your application and your application continues to work okay.
> If it fails it raises an Exception... If your server is down, my application would crash too
So first off, let me just say that I completely agree. If this were the case, that'd be fucking atrocious and would definitely be a blocker for using it.
But I'm curious how you came to that conclusion.
It took me < 2 minutes of looking at the source code[0] to determine that your claim was incorrect. Not only does it appear to gracefully handle the server being unavailable, the developer literally commented that code explaining that they wanted to ensure users could continue uninterrupted if the errorship server is unavailable.
> We give people the benefit of doubt.
> We only consider people to be not authorized if the backend comes back with an authoritative answer to that effect.
> Else, any errors or any other outcome; we assume authorization is there and also assume they belong to the highest pricing plan: Enterprise
> # failure of errorship should not lead to people been unable to ship exceptions
And it took even less time than that to run a new Python Docker container, install the library, run the sample code, and validate my assumptions[1] (the first attempt fails because the key is invalid, I disabled Internet access for the second attempt and it succeeded).
So I'm legitimately curious - did I miss something? Is there another failure case I didn't catch or test for? Or did you just make an assumption and not bother to verify it? And if it's the latter, why? What was the point? Like, to be frank, if this was a news piece I could understand the (possibly inaccurate) commentary. But why take the time and energy to write your comment and tear down someone's personal project with seemingly inaccurate claims?
(To be clear, no affiliation with errorship, I'm not even a DataDog user. Just a random dev browsing HN).
Can you resolve exceptions like in Sentry? Having to look at the list of all exceptions and figure out which ones are of interest wouldn’t be ideal. (In the demo I did notice the level/priorities filters, but nothing resembling an unresolved filter.)
Also, nitpick: it’s 2020, maybe update the front page screenshot to Python 3? My immediate reaction seeing that py27 screenshot: is this even maintained?
> Having to look at the list of all exceptions and figure out which ones are of interest wouldn’t be ideal.
You can filter for exceptions by tags. You can also use full text search to filter for exceptions. ie all the functionality provided[1] by the datadog eventstream[1] is available for your errors.
> Also, nitpick: it’s 2020, maybe update the front page screenshot to Python 3: is this even maintained?
errorship is compatible with both python2 and python3. Yes it is maintained. We have a testsuite that is ran in CI under both python2 and python3. Some of our customers during early trials still had some python2 applications that they had yet to port over.
For Java I'd written a Logback exception sender[0] that creates metrics tagged by level, class, exception class, cause classes. I'll say that it is a quick and convenient way to see changes in system error characteristics. I had used a similar setup for Graphite but didn't work as well without tags.
Since DataDog charges by number of metrics and each tag combination counts as one, actual exception messages should only be included in tags if they don't have instance-varying text.
IMO only if you are using the Logs functionality. You can log exceptions/errors, and it should be pretty easy to set-up monitoring around these events.
As a Datadog customer, I still use Sentry next to it, mostly for the top notch exception grouping, reporting etc. It’s just their core product, they’re better at it when you need to zoom in and actually debug.
This looks like an admittedly clever hack to avoid the fantastically expensive APM and still get some useful exception logging. I really wanted to use Datadog’s APM but $31 per host was too much. The downsides to logging to the event stream like Errorship is that you can’t associate exceptions with requests and aggregation is limited to tags. I think event tags have a low-ish cardinality limit similar to metrics.
I’m going to make the classic engineer mistake and assume that it’s almost as easy to implement this myself. My understanding is that errorship does something like:
- register a global exception handler, probably tweaked to hook into framework specific exception mechanisms.
- make the exceptions “pretty”
- send the exceptions to the datadog event log.
The benefits of rolling my own is avoiding a soft dependency on errorship and that I can tweak exception aggregation and reporting. Is the primary defense that errorship only costs $10 per month or is there additional complexity I’ve missed?
30 comments
[ 2.2 ms ] story [ 69.1 ms ] threaderrorShip is a python library that sends exceptions/errors generated by your application to your datadog account.
Are you tired of looking at metrics in datadog and then switching over to another website to track your applications exceptions? errorship exists to solve that context-switching problem, among others. It's a bit like sentry, bugsnag, rollbar etc; except implemented in your own datadog account.
I'm happy to receive any feedback or just chat about it.
I see, for example, docs on how to log from Python: https://www.datadoghq.com/blog/python-logging-best-practices...
> but they seem clear that there's both a logging and metrics (apm) side to their product.
yes they have those two sides. what they do not have is an error tracking side; something like sentry[1]. errorship exists to bring the error tracking side into datadog.
The logging side of datadog is like splunk or elasticsearch, and their APM side is like newrelic. errorship is bringing the error tracking side.
some of the analogies may not apply 100%, but I hope I'm making sense.
1. https://sentry.io/
But also, errorship will still capture(and send to datadog) any exceptions that you may have forgot to capture or log.
> although not sure how DD would feel about that
The datadog documentation[1] says; "An event represents any record of activity noteworthy for engineers"[1]. We certainly believe that exceptions that occur in engineer's applications are noteworthy activities and as such belong to the eventstream.
1. https://docs.datadoghq.com/events/
Yep, noted that, and I like how you preserve syntax highlighting in the stack traces :)
Out of curiosity, I've only ever dabbled in Datadog, is there any limit on event stream size or retention? I couldn't find anything mentioned in the docs.
Seems like a big loophole.
With errorship, all these are made available in one place; in your datadog account.
You also do not need to maintain two services. If you are already using datadog(maybe their APM and their logging and metrics service) then you might as well use them for error tracking instead of maintaining an additional account with sentry.
However, it is not a must that you give up sentry to use errorship. errorship will work just as fine, if you choose to continue with sentry.
Sentry lets you send exceptions to Datadog though?
If your server is down, my application would crash too. Just cut the license validation out of the library. If I wanted to use the library without an license I could do so anyway.
edit: as noted by the author below, this is not the case :). If the server is not available, it won't raise an exception. I did miss that part somehow.
> If your server is down, my application would crash too.
The errorship library is written in such a way that it fails open. If our servers are down(or any other failure), it does not affect your application and your application continues to work okay.
So first off, let me just say that I completely agree. If this were the case, that'd be fucking atrocious and would definitely be a blocker for using it.
But I'm curious how you came to that conclusion.
It took me < 2 minutes of looking at the source code[0] to determine that your claim was incorrect. Not only does it appear to gracefully handle the server being unavailable, the developer literally commented that code explaining that they wanted to ensure users could continue uninterrupted if the errorship server is unavailable.
> We give people the benefit of doubt.
> We only consider people to be not authorized if the backend comes back with an authoritative answer to that effect.
> Else, any errors or any other outcome; we assume authorization is there and also assume they belong to the highest pricing plan: Enterprise
> # failure of errorship should not lead to people been unable to ship exceptions
And it took even less time than that to run a new Python Docker container, install the library, run the sample code, and validate my assumptions[1] (the first attempt fails because the key is invalid, I disabled Internet access for the second attempt and it succeeded).
So I'm legitimately curious - did I miss something? Is there another failure case I didn't catch or test for? Or did you just make an assumption and not bother to verify it? And if it's the latter, why? What was the point? Like, to be frank, if this was a news piece I could understand the (possibly inaccurate) commentary. But why take the time and energy to write your comment and tear down someone's personal project with seemingly inaccurate claims?
(To be clear, no affiliation with errorship, I'm not even a DataDog user. Just a random dev browsing HN).
[0] https://gitlab.com/errorship/errorship_python/-/blob/master/...
[1] https://gist.github.com/citruspi/16d359ac2dafef6fc876e2dd101...
Also, nitpick: it’s 2020, maybe update the front page screenshot to Python 3? My immediate reaction seeing that py27 screenshot: is this even maintained?
You can filter for exceptions by tags. You can also use full text search to filter for exceptions. ie all the functionality provided[1] by the datadog eventstream[1] is available for your errors.
> Also, nitpick: it’s 2020, maybe update the front page screenshot to Python 3: is this even maintained?
errorship is compatible with both python2 and python3. Yes it is maintained. We have a testsuite that is ran in CI under both python2 and python3. Some of our customers during early trials still had some python2 applications that they had yet to port over.
1. https://docs.datadoghq.com/events/#event-stream
Since DataDog charges by number of metrics and each tag combination counts as one, actual exception messages should only be included in tags if they don't have instance-varying text.
[0] https://github.com/karmakaze/logback-metrics-datadog
For the bird’s eye view, Datadog is fine.
[0] https://www.datadoghq.com/apm/
- register a global exception handler, probably tweaked to hook into framework specific exception mechanisms.
- make the exceptions “pretty”
- send the exceptions to the datadog event log.
The benefits of rolling my own is avoiding a soft dependency on errorship and that I can tweak exception aggregation and reporting. Is the primary defense that errorship only costs $10 per month or is there additional complexity I’ve missed?
1. https://errorship.com/pricing.html