Show HN: HyperDX – open-source dev-friendly Datadog alternative (github.com)
Github Repo: https://github.com/hyperdxio/hyperdx
Coming from an observability nerd background, with Warren being SRE #1 at his last startup and me previously leading dev experience at LogDNA/Mezmo, we knew there were gaps in the existing tools we were used to using. Our previous stack of tools like Bugsnag, LogRocket, and Cloudwatch required us to switch between different tools, correlate timestamps (UTC? local?), and manually cross-check IDs to piece together what was actually happening. This often made meant small issues required hours of frustration to root cause.
Other tools like Datadog or New Relic come with high price tags - when estimating costs for Datadog in the past, we found that our Datadog bill would exceed our AWS bill! Other teams have had to adjust their infrastructure just to appease the Datadog pricing model.
To build HyperDX, we've centralized all the telemetry in one place by leveraging OpenTelemetry (a CNCF project for standardizing/collecting telemetry) to pull and correlate logs, metrics, traces, and replays. In-app, we can correlate your logs/traces together in one panel by joining everything automatically via trace ids and session ids, so you can go from log <> trace <> replay in the same panel. To keep costs low, we store everything in Clickhouse (w/ S3 backing) to make it extremely affordable to store large amounts of data (compared to Elasticsearch) while still being able to query it efficiently (compared to services like Cloudwatch or Loki), in large part thanks to Clickhouse's bloom filters + columnar layout.
On top of that, we've focused on providing a smooth developer experience (the DX in HyperDX!). This includes features like native parsing of JSON logs, full-text search on any log or trace, 2-click alert creation, and SDKs that help you get started with OpenTelemetry faster than the default OpenTelemetry SDKs.
I'm excited to share what we've been working with you all and would love to hear your feedback and opinions!
Hosted Demo - https://api.hyperdx.io/login/demo
Open Source Repo: https://github.com/hyperdxio/hyperdx
Landing Page: https://hyperdx.io
185 comments
[ 6.3 ms ] story [ 397 ms ] threadSome other landing pages I loved and had along side while designing ours were Vercel, Resend, and WorkOS :)
Definitely striving to be the opposite of that - and would love to hear how it goes and any place we can improve!
Do also check out SigNoz [1] We are working on a similar problem statement ;)
[1] https://github.com/signoz/signoz
Feature-to-feature, I'd say the things we do better is browser-side monitoring (session replay), event patterns/clustering, and we have first-party SDKs built on OpenTelemetry to make the setup a lot easier than vanilla OpenTelemetry.
I think Signoz has built a nice one-stop platform for observability, whereas we go one step further and focus on the developer experience to ensure anyone can fully leverage that observability data!
One other thing I think you'd love if you're coming from Datadog is that you're able to full text search on structured logs as well, so even if the value you're looking for lives in a property, it's still full text searchable (this is a huge pain we hear from other Datadog users)
If there's anything you love/hate about Datadog - would love to learn more!
Btw I haven't checked your product out yet; I was just reminiscing :-) I'll take a look soon.
We dogfood a ton internally and (while obviously biased) we're always surprised how much faster we can pin point issues and connect alarms with bug reports.
Hope you give us a spin and feel free to hop on our discord or open an issue if you run into anything!
[1] https://github.com/metriport/metriport
We include a "query explainer" - which translates the parsed query AST into something more human readable under the search bar, hopefully giving good feedback to the user on whether we're understand their query or not. Though there's lots of room to improve here!
Edit: happy to chat more about it as well if you're looking for more specific feedback - it's an area I've spent a decent amount of time on and would love to improve projects like liqe or others based on our experience if we can.
https://www.antlr.org/
The background of someone with a DX background comes through! I will be looking into this a lot more.
Here are a few comments, notes, and questions:
* I like the focus on DX (especially compared to other OSS solutions) in your messaging here, and I think your hero messaging tells that story, but it isn't reinforced as much through the features/benefits section
* It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Or is the cloud backend a bit different where you can offer that scale without making open source so complex?
* I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features? Or are you considering functional things? Especially with recent HashiCorp "fun" I think more and more it is useful to be open about what you think the split will be. Obviously that will evolve, but I think that sort of transparency is useful if you really want to grow the OSS side
* on OSS, I was surprised to see MIT license. This is full featured enough and stand alone enough that AGPL (for server components) seems like a good middle ground. This also gives some options for potentially a license for an "enterprise" edition, as I am certain there is a market for a modern APM that can run all in a customer environment
* On that note, I am curious what your target persona and GTM plan is looking like? This space is a a bit tricky IMHO, because small teams have so many options at okay price points, but the enterprise is such a difficult beast in switching costs. This looks pretty PLG focused atm, and I think for a first release it is impressive, but I am curious to know if you have more you are thinking to differentiate yourself in a pretty crowded space.
Once again, really impressive what you have here and I will be checking it out more. If you have any more questions, happy to answer in thread or my email is in profile.
> It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source?
Scaling any system can be challenging - our experience so far is that Clickhouse is a fraction of the overhead of systems like Elasticsearch has previously demanded luckily. That being said, I think there's always going to be a combination of learnings we'd love to open source for operators that are self-hosting/managing Clickhouse, and tooling we use internally that is purpose-built for our specific setup and workloads.
> I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features?
Our current release - we've open sourced the vast majority of our feature set, including I think some novel features like event patterns that typically are SaaS-only and that'll definitely be the way we want to continue to operate. Given the nature of observability - we feel comfortable continuing to keep pushing a fully-featured OSS version while having a monetizable SaaS that focuses on the fact that it's completely managed, rather than needing to gate heavily based on features.
> on OSS, I was surprised to see MIT license
We want to make observability accessible and we think AGPL will accomplish the opposite of that. While we need to make money at the end of the day - we believe that a well-positioned enterprise + cloud offering is better suited to pull in those that are willing to pay, rather than forcing it via a license. I also love the MIT license and use it whenever I can :)
> On that note, I am curious what your target persona and GTM plan is looking like?
I think for small teams, imo the options available are largely untantilizing, it ranges from narrow tools like Cloudwatch to enterprise-oriented tools like New Relic or Datadog. We're working hard to make it easier for those kinds of teams to adopt good monitoring and observability from day 1, without the traditional requirement of needing an observability expert or dedicated SRE to get it set up. (Admittedly, we still have a ways to improve today!) On the enterprise side, switching costs are definitely high, but most enterprises are highly decentralized in decision making, where I routinely hear F500s having a handful of observability tools in production at a given time! I'll say it's not as locked-in as it seems :)
One more follow-up on the scale side (which I mentioned with sibling comment), it isn't so much about clickhouse itself, but about scaling up ingest. From my own experience and from talking with quite a few APM players (I previously worked in streaming space), a Kafka / durable log storage kind of becomes a requirement, so I was curious if you think at some point you need a log to further scale ingest.
For enterprise side, I was previously in data streaming space and had quite a few conversations with APM players and companies building their own observability platforms, happy to chat and share more if that would be useful!
Similar to Elastic - I think a lot of architectures are available to choose on that side when users want to scale.
Will reach out to connect!
I have seen orgs remove datadog because of unpredictable pricing. If you do flat price self hosted platform, you will get attention. I dont think orgs would mind hosting clickhouse. You can also bundle it with your helm charts or initial proof of concept might have lower barrier. I know some orgs have million dollar annual contracts with datadog, a cheaper more predictable priced alternative will definitely get attention.
Honest question: What makes you think that you are not turning into a Datadog (price wise) once reach a certain scale?
The problem what I see with software companies that the pricing is dominated by investor requirements and when a company reaches a certain milestone change up the licensing model and the pricing with it.
I'd also add that I don't think all services trend their price upwards. AWS has historically lowered prices on services and continue to offer new service-tiers with lower prices (S3 tiering as an example). As the tech matures and costs fall for our service as well, it'd be surprising if we don't do the same.
This page shows event pattern available for both oss vs. cloud. The blog doesn't mention exactly how this is being which would be an interesting read but I understand if a secret sauce.
I recall quite a few years ago a standalone commercial & hosted tool for doing something like this just on logs for anomaly detection. Anyone has any reference for similar tools for working with direct log data (say from log files) or in a similar capacity like hypderdx (oss or commercial)
The technical details are best explained by the authors of the original paper [1]. We weren't smart enough to come up with it on our own and can't take credit for that haha
[1] http://jiemingzhu.github.io/pub/pjhe_icws2017.pdf
Where do you discover these papers to read?
Generally pretty good way to approach research papers in a new field I recall being taught, there's always some sort of "landscape overview" paper being published that can help distill down the SoTA and you can just follow the references.
So often that means you need something like a Kafka to get the bulk ingest to really perform to get batch sizes large enough.
That kind of gets into one of the challenges of OSS observabilility systems, you don't want to make the dependencies insane for someone who only has a few thousand logs a second, but generally at some point of scale you do need more.
We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type.
[1] https://clickhouse.com/docs/en/optimize/asynchronous-inserts [2] https://github.com/highlight/highlight/tree/main [3] https://github.com/highlight/highlight/blob/4d28451b1935796d...
I wonder why not Apache Druid
No. Clickhouse is opensource with Apache License [0].
[0] - https://github.com/ClickHouse/ClickHouse/blob/master/LICENSE
Especially Ruby, which is the one that I would be most interested in using.
Feel free to pop in on the Discord if you'd like to chat more/share your thoughts!
One thing to consider with your messaging is that when you start speaking to large companies, they won't see you as a datadog alternative. They'll see you as a mix of sentry + fullstory + honeycomb.
Datadog originally found its success with its metrics products, and the larger the buyer of datadog gets, the more metrics-esque use case a company finds. The session replay, logging and other things are simply products that datadog tacks on.
That being said, this is clearly a large market (which is why we're working on it). I particularly like the tracing UI that y'all have and I'd love to chat with your team at some point. Good luck.
We both charge a cloud saas fee as well:
https://www.hyperdx.io/pricing https://www.highlight.io/pricing
Look forward to the syslog integration which says coming soon. I have a hobby project which uses systemd services for each of my Python apps and the path with least resistance is just ingest syslog (aware that I lose stack traces, session reply, etc).
For Syslog - it's something we're actually pretty close to because we already support Heroku's syslog based messages (though it's over HTTP), but largely need to test the otel Syslog receiver + parsing pipeline will translate as well as it should (PRs always welcome of course but it shouldn't be too far out from now ourselves :)). I'm curious are you using TLS/TCP syslog or plain TCP or UDP?
Here's my docker stats on a x64 linux VM where it's doing some minimal self-logging, I suspect the otel collector memory can be tuned down to bring the memory usage closer to 1GB, but this is the default out-of-the-box stats, and the miner can be turned off if log patterns isn't needed:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
439e3f426ca6 hdx-oss-miner 0.89% 167.2MiB / 7.771GiB 2.10% 3.25MB / 6.06MB 8.85MB / 0B 21
7dae9d72913d hdx-oss-task-check-alerts 0.03% 83.65MiB / 7.771GiB 1.05% 6.79MB / 9.54MB 147kB / 0B 11
5abd59211cd7 hdx-oss-app 0.00% 56.32MiB / 7.771GiB 0.71% 467kB / 551kB 6.23MB / 0B 11
90c0ef1634c7 hdx-oss-api 0.02% 93.71MiB / 7.771GiB 1.18% 13.2MB / 7.87MB 57.3kB / 0B 11
39737209c58f hdx-oss-hostmetrics 0.03% 72.27MiB / 7.771GiB 0.91% 3.83GB / 173MB 3.84MB / 0B 11
e13c9416c06e hdx-oss-ingestor 0.04% 23.11MiB / 7.771GiB 0.29% 73.2MB / 89.4MB 77.8kB / 0B 5
36d57eaac8b2 hdx-oss-otel-collector 0.33% 880MiB / 7.771GiB 11.06% 104MB / 68.9MB 1.24MB / 0B 11
78ac89d8e28d hdx-oss-aggregator 0.07% 88.08MiB / 7.771GiB 1.11% 141MB / 223MB 147kB / 0B 11
8a2de809efed hdx-oss-redis 0.19% 3.738MiB / 7.771GiB 0.05% 4.36MB / 76.5MB 8.19kB / 4.1kB 5
2f2eac07bedf hdx-oss-db 1.34% 75.62MiB / 7.771GiB 0.95% 105MB / 3.79GB 1.32MB / 246MB 56
032ae2b50b2f hdx-oss-ch-server 0.54% 128.7MiB / 7.771GiB 1.62% 194MB / 45MB 88.4MB / 65.5kB 316
Actually I am not really using syslog per say, but systemd journalctl which default behaviour on Debian (rsyslog) also duplicates to /var/log/syslog.
Is there a better integration to pull logs from my systemd services and journalctl up to HyperDX?https://github.com/open-telemetry/opentelemetry-collector-co...
Happy to dive more into the discord too if you'd like!
That's why this is MIT right, so folks can contribute stuff like this?
We did explicitly choose MIT for the freedom of end users to deploy and modify the code how they want - and tried to open source pretty much everything that doesn't have a hard 3rd party dependency. We do touch a bit on how we think about the open core model as well in the README, and largely align with Gitlab's stewardship model [1] when it comes to paid vs OSS. In this case, a contribution to add SAML specifically to OSS will likely not be merged. It'd also introduce complexities with maintaining that alongside our cloud version that already includes a specific implementation of SAML.
[1] https://handbook.gitlab.com/handbook/company/stewardship/
There's nothing wrong with asking folks to pay for software instead of giving it away via FOSS, especially if you're honest about your intentions and goals. When you choose FOSS to gain traction and rug pull your users when no one converts later on, you end up reaping what you sow.
My main point was you should get these ducks in order first and be genuine with your intentions. Don't use FOSS as a growth hack, it never ends well for the creator or the user. I don't think HyperDX is genuine with their intentions, as with all open core, it's all kumbaya FOSS until you start encroaching on their enterprise feature set.
I think that HyperDX is a bit different from tools like Mongo, Redis or Hashicorp in that we're a vertically integrated product from SDKs/UIs to ingestion pipeline and DBs, which is opposite kind of offering from done by the above companies (which has made them more vulnerable to the kind of rug pull you mentioned)
We're trying to be permissive with freedoms granted to the user of our code, while still maintaining governance over the project to make it sustainable.
We don't want to be source-available, as that's pretty much the opposite of what we want to accomplish (and is why we consciously did not pick a license such as BSL/SSPL/etc.)
How do you plan on doing that while being VC-backed? Why did you choose to be VC backed in the first place? You can create a sustainable open source project and community without any VC funding.
Interestingly, it seems like "HyperDX" might've been part of their original product offering that they decided to open source--their main website (https://www.deploysentinel.com) doesn't include any references to "HyperDX for CI" in May of 2023: https://web.archive.org/web/20230321102146/https://www.deplo.... Seems like they're pivoting to metrics? Even more of a reason to be weary about this.
The open core model relies on a delicate balance of ensuring that the OSS product is featureful and standalone, while successfully monetizing value added features for advanced users and enterprise customers. Not many companies do this right, but there are those that understand and handle this balance well, and manage to have both a successful OSS and commercial product. Grafana comes to mind, for example.
Just because you think that SSO is a required feature that should be part of the OSS product doesn't mean that HyperDX is using OSS as a growth hack. Nor is it fair to label a young startup that for a product that just launched.
FWIW I agree with their decision to make SSO a paid feature, but we can go over any number of features, and some OSS user is guaranteed to demand a specific feature, yet will not be willing to pay for it. SSO is not special, unless it's a core feature that the product depends on, which doesn't seem to be the case here.
When done right, open core is the best model to monetize OSS projects, and we should be thankful that companies adopt it at all. I'd use an open core product before a proprietary one any day of the week.
- 1M GETs $0.36 (R2) vs $0.40 (S3)
- 1M PUTs $4.50 (R2) vs $5.00 (S3)
We were concerned with some of the performance benchmarks we've seen with R2 in the past (though they've probably have improved), not to mention our compute options become a bit more limited to bandwidth alliance clouds otherwise we'll be eating network egress fees (which I do hate with a HUGE passion).
Though I can imagine if you're comfortable with one of the bandwidth alliance clouds already and can take a bit of a perf hit for search, R2 and Backblaze both can provide some cost savings depending on your workload.
I've historically hit issues with repos that do an `ee` folder and blur the line between what is truly open source and self-hostable, vs need a license/cloud-only. I understand why they do that, but I hope we don't replicate that confusion ourselves :)
"HyperDX helps engineers figure out why production is broken, faster. HyperDX centralises and correlates logs, metrics, traces, exceptions and session replays in one place."
Good luck!
How about: "9 out of 10 devs are now pushing to prod on fridays. Thanks to HyperDX. Hehe."
If so, would you like to open a PR? I'm also happy to edit it myself but of course don't want to be stealing credit if you'd like to be attributed that way.
Would love to have you check us out! Let me know if you run into any issues - feel free to hop on our discord as well :)
Otherwise you can use the OpenTelemetry collector[2] to do the same[3].
[1] https://www.hyperdx.io/docs/install/docker
[2] https://opentelemetry.io/docs/collector/
[3] https://www.hyperdx.io/docs/install/opentelemetry
I haven’t had time to dig in proper, but this seems like something that would fit perfectly for “local dev” logging as well. I struggled to find a good solution for this, ending up Winston -> JSON, with a simpler “dump to terminal” script running.
(The app I’m building does a ton of “in the background” work, and I wanted to present both “user interactions” and “background worker” logs in context)
I don’t see Winston being supported as a transport, but presumably easy to add/contribute.
Good luck!
In fact this is actually how we develop locally - because even our local stack is comparatively noisy, we enable self-logging in HyperDX so our local logs/traces go to our own dev instance, and we can quickly trace a 500 that way. (Literally was doing this last night for a PR I'm working on).
Kind of like how people mostly promote "Elasticsearch alternatives" and not "Solr alternatives".
It becomes convenient short-hand for what they do (collect logs, metrics, traces, RUM, etc. for engineers to debug).
Though with more characters to write, I'd like to think we have a different take on both how our pricing model works and how easy it should be for an engineer to get started with us :)
Being able to see that a user bounced because they couldn't handle the input that they were seeing - is it all that different from a service erroring because it cannot handle a certain type of input?
Honeycomb is great for the OpenTelemetry part on the server side (and with https://docs.honeycomb.io/getting-data-in/opentelemetry/brow... is moving towards full-stack), and systems like Posthog and Heap are great for sending session replay + browser events -> Clickhouse. But I don't think I've seen a great DX that ties everything together.
To that point - I would love to see different font/color options for HyperDX: the monospaced font can become tiring to read when so dense. Will be following this project closely though - this is amazing work so far!
As for monospace font - feedback received! Is there a particular section you think is too overwhelming? (search page, nav bar, etc.) We've been thinking of how can we balance between the ease of monospace for reading instead of having it literally the default on every UI surface :P
You all are doing really cool things to discover patterns, link traces, etc. - I'd hate someone's takeaway from the demo page to be that the skill level required to make use of HyperDX is akin to needing to spot hair colors from looking at the same-colored code of the Matrix!
"HyperDX helps engineers figure out why production is broken faster by centralizing and correlating logs, metrics, traces, exceptions and session replays in one place. An open source and developer-friendly alternative to Datadog and New Relic."
Just perfect. Bravo.
--
As a merc, I never understood the why of Datadog (or equiv). The teams and projects I rotated thru each embraced the "LOG ALL THE THINGS!" strategy. No guiding purpose, no esthetics. General agreement about need to improve signal to noise ratio. But little courage or gumption to act. And any such efforts would be easily rebuffed by citing the parable of Chesterfordstorm's Fences of Doom and something something about velocity.
Late last century, IT projects, like CRMs and ERPs, were plagued by over collection of data. Opaque provenance, dubious (data) quality, unclear ownership, subtractive value propositions (where the whole is worth less than the parts). No, no, don't remove that field. We might need it some day.
Today's "analytics" projects are the same, right? Every drive-by stakeholder tosses in a few tags, some misc fields, a little extra meta. And before anyone can say "kanban", the stone soup accreted enough mass to become a gravity well threatening implosion dragging the entire org-chart into the gapping maw of our universe's newest black hole.
Am I wrong?
But logging is useful, right? Or at least has that potential.
The last time I designed a system end-to-end, that's kinda what we did. Listed all the kinds of things we wanted to log. Sorta settled on formats and content (never really ever done). Did regular log bashs to explain and clear anomalies. Scripts for grooming and archiving. (For one team I rotated thru, most of their spend was on just cloudwatch. Hysterical.)
But my stuff wasn't B2C, so wasn't tainted by the attention economy, manufactured outrage, or recommenders. No tags, referrers, campaigns, etc. It was just about keeping the system up and true. And resolving customer support incidents asap.
Does any one talk or write about this? (Those SRE themed novels are now buried deep in my to read pile.)
I'd like some cookbooks or blue prints which show some idealized logging strategies, with depictions of common enough troubleshooting scenarios.
Having something authoritative to cite could reduce my semblance to an Eeyore. "Hey, team mates, you know what'd be really great?! Correlation IDs! So we can see how an action percolates thru our system!"
Just curious.
PS- Datadog's server hexagon map/chart thingie is something else. The kind of innovation that wins prizes.
Tagging the right IDs are a huge thing - customer X is saying their instance is really slow, but if none of your logs let you link service performance to customer X, your telemetry you're paying for is absolutely useless!
You have an ally in me on this one :) I'm hoping given a bit more time we get to write things like this - practical observability from the perspective of a dev, as opposed to the SRE angle that I think is well covered. Feel free to join us on discord btw if you want to chat more - I (for better/worse) love musing about these things :)
Good stuff. Much industry progress since I was last in the arena.
Their site has words about manual and automatic instrumentation. I'd have to dig a bit to see what they mean.
--
So. Remembering a bit more... Will try to keep this brief; you're a busy person.
> tend to log useless information or fail to tag them in ways that are actually searchable
#1 - I don't know know to manage lifecycle of meta. Who needs what? When is it safe to remove stuff?
We logged a lot of URLs. So many URL params. And when that wasn't crazy enough, over flow into HTTP headers. Plus partially duplicate, incorrectly, info in the payloads, a la SOAP. ("A person with two watches has no idea what time it is.")
When individual teams were uncertain, they'd just forward everything they received (copypasta), and add their own stuff.
Just replace all that context with correlation IDs, right?
Ah, but there's "legacy". And unsupported protocols, like Redis and JDBC. And brain dead 3rd party services, with their own brain dead CSRs and engrs.
This is really bad, and just propagates badness, but a few times, in a pinch, I've created Q&D "logging proxy". Just to get some visibility.
So dumb. And yet... Why stop there? Just have "the fabric" record stuff. Repurpose Wireguard into an Omniscient Logger. (Like the NSA does. Probably.) That'd eliminate most I/O trace style logging, right?
Image all these "webservices" and serverless apps without any need for instrumentation. Just have old school app level logging.
#2 - So much text processing.
An egregious example is logging HTTP headers. Serialize them as JSON and send that payload to a logging service. Which then rehydrate and store it some where.
My radical idea, which exactly no one has bought into, is to just pipe HTTP (Requests and Responses) as-is to log files. Then rotate, groom, archive, forward, ingest, compress, whatever as desired.
That's what I did on the system I mentioned. All I/O was just streamed to files. And in the case of the HL7 (medical records stuff), it was super easy to extract the good bits, use that for Lucene's metadata, and store the whole message as the Lucene document.
I know such a radical idea is out of scope for your work. Just something fun to think about.
#3
> if none of your logs let you link service performance to customer X
Yup. Just keep adding servers. Kick the can down the road.
One team I helped had stuff randomly peg P95. And then sometimes a seemingly unrelated server would tip over. Between timeouts, retries, and load balancers, it really seemed like the ankle bone was connected to the shoulder bone. It just made no sense.
Fortunately, I had some prior experience. Being new to nodejs, maybe 5 years ago, I was shocked to learn there was no notion of back pressure. It was a challenging concept to explain to those teammates. But the omission of backpressure, and a hunch, was a good place for me start. (I'm no Dan Luu or Bryan Cantrill.)
I'd like to think that proper end-to-end logging, and the ability to find signal in the noise, diagnosis would have been more mundane.
For automatic logging - I think you'd enjoy OpenTelemetry's automatic tracing implementation, it helps pull out standard telemetry from things like your Redis requests and correlate them with trace IDs so you can tie everything together from the moment your server starts accepting the HTTP request to the Redis and DB requests and what was sent in each request (without needing to do it manually)
For capturing HTTP req/res - we actually have a few options depending on the language (ex. we do this for Python and Node.js) to enable more advanced network capture (so you can actually get the full req/res information, or whatever subset you're interested in actually storing)! It's actually been asked by a number of teams to make it easier to debug tricky API issues they're running into.
Proper end-to-end logging definitely makes it easier to find the right clue among a sea of logs, hopefully we make it easier to get there!
> "Hey, team mates, you know what'd be really great?! Correlation IDs! So we can see how an action percolates thru our system!"
Hi, I'm building, Doctor Droid -- https://drdroid.io/ that enables you join structured application logs via correlation IDs and then build multiple types of rules / frameworks on it -- some are at granular level and some are at aggregate levels (like funnels).
We are early in the development lifecycle, would love to hear your feedback / connect with you.
Great job!
[0] https://github.com/SigNoz/signoz
why not grafana / prometheus / loki?
Grafana/Prom/Loki is an awesome stack - overall I'd say that we try to correlate more signals in one place (your logs <> traces <> session replay), and we also take an approach to go more dev-friendly to query instead of going the PromQL/LogQL route.
It's a stack I really wanted to love myself as well but I've personally ran into a few issues when using it:
Loki is a handful to get right, you have to think about your labels, they can't be high-cardinality (ex. IDs), the search is really slow if it's not a label, and the syntax is complex because it's derived from PromQL which I don't think is a good fit for logs. This means an engineer on your team can't just jump in and start typing keywords to match on, nor can they just log out logs and know they can quickly find it again in prod. Engineers need to filter logs by a label first and then wait for a regex to run if they want to do full-text search.
Prometheus is pretty good, my only qualm is again the approachability of PromQL - it's rare to see an engineer that isn't fluent with time-series/metric systems to be able to pick up all the concepts very quickly. This means that metrics access is largely limited to premade dashboards or a certain set of engineers that know the Prometheus setup really well.
Grafana has definitely set the standard for OSS metrics, but I personally haven't had a lot of success using their tools outside of metrics, though ymmv and it's all about the tradeoffs you're looking for in an observability tool.