Show HN: Open-source Kibana alternative for logs and traces in ClickHouse (github.com)
Over the past year we've seen a ton of excitement in companies adopting Clickhouse-based observability stacks - but one of the biggest challenges we've seen is that the UI layer on top of Clickhouse is either clunky to use for observability use cases (ex. BI tools), or too tied to a specific ingestion architecture to scale to every use case (we used to be in this category!). For companies that needed more flexibility in how their data is ingested and stored (usually due to running at a large scale), there's really no good options for a developer experience (DX) focused observability layer on top of Clickhouse (Shopify spent 3 years building it in-house!)
Our current release works completely in the browser - and it does this by building on top of Clickhouse's HTTP interface, which our React app can directly talk to. This means you can actually try HyperDX in your browser on your own Clickhouse with no installation! This was fortunately easy for us to accomplish due to being full stack Typescript, making it incredibly easy to shift between server and client code. On top of this we've been spending time baking in performance optimizations to ensure that HyperDX can continue to leverage Clickhouse efficiently at larger data volumes. We do a few tricks like only fetching columns that are needed for the current search, and re-querying to expand the entire row if needed to fully leverage Clickhouse's columnar nature (40% faster, ymmv!) - or rewriting queries to use materialized columns to speed up Map column access when available (10x faster!).
On the DX side: we support querying using both Lucene (ex. `fullText property:value`) and SQL syntax. We've found the former to be our favorite for how concise it is. Similarly for charts, our chart builder has been upgraded to accept SQL expressions as well, so you can leverage the full power of SQL, while avoiding typing paragraphs of boilerplate SQL for time series data. We also make it easy to switch between UTC/local timestamps! Lastly, we've added high cardinality outlier analysis by charting the delta between outlier and inlier events (a la bubble up) - which we've found really helpful in narrowing down causes of regressions/anomalies in our traces.
We have a lot more planned for the full release - but wanted to get this out early to hear your feedback and opinions!
In Browser Live Demo: https://play.hyperdx.io/search
Github Repo: https://github.com/hyperdxio/hyperdx/tree/v2
Landing Page: https://hyperdx.io/v2
[1]: https://www.uber.com/blog/logging/ [2]: https://blog.cloudflare.com/log-analytics-using-clickhouse/ [3]: https://www.youtube.com/watch?v=LDj3_jMsCXg&list=PLvQF73bM4-...
41 comments
[ 3.2 ms ] story [ 84.9 ms ] threadHard to trust anything released as OSS these days that hits this site run by a for profit company.. Its all destined to have a rug pull after some VC funding. Considering HyperDX is a for profit company, I'm sure we won't have to wait long!
Once that happens - eventually some new kids would appear on the block.
Such is the life.
On the commercial OSS side of things - I suspect the trend there is more nuanced than all OSS companies being suspect to the same problem, but rather companies that generally solve a "behind the API" problem are more susceptible to problems of cloud vendors taking their code and competing with them commercially (ex. if you're a DB like Redis, Mongo, Elastic - or a CLI like Terraform). We're building an end-user experience (more like Gitlab) - where experience differentiation matters a lot more than simple infrastructure hosting, something AWS is not particularly well suited at competing on!
It's been 3 years of Gitlab post-IPO and they're still MIT, and that's the boat we're on as well :)
> It's been 3 years of Gitlab post-IPO and they're still MIT, and that's the boat we're on as well :)
I totally understand your optimism but keep in mind that GitLab was exploring a sale as at 4 months ago https://www.reuters.com/markets/deals/google-backed-software...
If they get bought or have a change of management, they might revisit the MIT license.
IANAL, but AIUI they for sure can do that to the content of the "ee" directory, but all other contributions are just DCO, not CLA https://about.gitlab.com/community/contribute/dco-cla/#which...
We're essentially making our existing product a lot easier to deploy into more complex observability setups based on Clickhouse - while also shipping a few new capabilities like SQL and event deltas while we're at it!
> - You can live tail events, I don't think Grafana for Clickhouse has that (I'm a big fan of tail -f)
So it sounds like your v2 will work with any DB that is wire-compatible with Clickhouse, correct?
[1] https://github.com/timeplus-io/proton
Proton looks like a neat queue/db combo - I'm going to have to dive in deeper some time
[1] https://www.timeplus.com/
- We support Lucene-based search, which means it's a lot easier to find the events you're looking for without needing to break into verbose SQL search. (Column/property/full text search are all super easy).
- We're optimized exclusively for Clickhouse, which means we do a number of things to optimize the queries we run and that returns you a nice performance boost (we see a 2x perf boost, but this will vary a ton of data and queries). For example we allow you to do a search on a subset of columns (so the search is performant), and then click in and expand an entire row of interest on-demand (so we only do a SELECT * for a single row). This is also a much nicer DX than needing to specify every column you might need. We have a few other optimizations as well.
- We have (what we think is) a nice chart builder - so you don't need to mess with template variables and macros to build a chart, but still lets you escape hatch out into SQL for the important bits.
- We think our event deltas feature for analyzing traces is pretty neat - afaik this isn't something you get in Grafana. - You can live tail events, I don't think Grafana for Clickhouse has that (I'm a big fan of tail -f)
Overall we focus on trying to bring an easy-to-use high-cardinality observability experience to Clickhouse, whereas Grafana seems to focus more on a highly SQL-dependent dashboard building experience (which has its own advantages of course).
edit: fixed line breaks!
One of Grafana's advantages is its very low cost of running because you can send everything to object storage with very little configuration.
They have their docs on that setup here: https://clickhouse.com/docs/en/guides/separation-storage-com...
Kibana was supposed to be an easy UI. You go to Discover, and the data automatically shows in chronological order, I can explore it with different options.
Kibana is very suitable for non-tech or less-tech people. I hope your product find a clear target audience. With too much ES query JSON or SQL it would scare people off.
There is a setup modal (which is intended to be set up once by the data owner, similar to maybe how you'd set up some indexes in Elastic) - and afterwards the experience is similar IMO. If you're open to sharing more I'd love to learn more mike@hyperdx.io or if you want to open an issue/join our discord.
1. At my last job we were running some of the largest elastic clusters of our hyperscaler's cloud - elastic is slow, expensive and finicky to operate at decent scale. We've found the exact opposite with Clickhouse, it's fast, easy to operate, and supports things like S3-backed storage directly in their open source product. As an example, Uber switched from Elastic -> Clickhouse and halved their infra footprint while increasing volume.
2. Elastic is tricky to manage, field type conflicts come up super common at scale and are annoying to deconflict. Clickhouse is a lot more flexible in its schema to avoid those problems (and give you knobs to fine tune performance at a more granular level with their indexes/schemas)
3. We allow for both SQL and Lucene, both are relatively "standard" languages that engineers are likely already familiar in one way or another. Compared to elastic moving to ES|QL, another vendor-specific language that will be difficult to onboard to. The last thing you want during an incident is trying to recall vendor-specific languages for querying that critical data!
tl;dr - we try to make it easy to "do observability" on what we think is the best DB for observability today (Clickhouse), analogous to what Kibana did for ES.
Disclaimer: Co-founder of Quesma.
The docs at https://www.hyperdx.io/docs/search don't seem to talk about this key design decision.
I have a couple 100 GB to few TB logs (all from `journald` or JSON lines), just want to store them forever, and find results fast when searching for arbitrary substrings.
Loki does not use an index, so it's pretty slow at finding results in TB-sized logs (does not return results within a few seconds, so it's not interactive).
https://quickwit.io is one thing I'm looking at integrating, that can solve much of the index-based log search.
(Note I'm not super familar with the capabilities of ClickHouse itself regarding indexed full-text search.)
Indeed Loki is incredibly slow - Clickhouse is deployed for logging at scale (ex. trip.com is running a 50pb logging solution that allowed them to 4x their old ES cluster volume while also running queries 4-30x faster)
That is, is it a full drop-in for a typical Grafana + Loki deployment?
For context, I'm currently following the approach described in https://xeiaso.net/blog/prometheus-grafana-loki-nixos-2020-1... where with ~40 lines of NixOS config it pushes my entire server cluster's systemd journald logs into Grafana.
Roughly how much effort would one have to put in to achieve the same with HyperDX? If it's not too hard, I might get around to package it as a NixOS service.
Overall it doesn't sound very hard to me!
[1] https://docs.victoriametrics.com/victorialogs/
I connected it to the system.text_log table, and it took zero time with no problems.
Speaking of the system tables - it's awesome how much telemetry is saved in there that helps us build a really powerful preset clickhouse monitoring dashboard (heavily inspired from the built-in clickhouse one of course). We figured that alone is quite useful for teams running any Clickhouse instance and want better insights into what's going on.
We wanted something good for tracing and logs, without the price tag we were used to from datadog. We've been pleasantly surprised by how easy it was to set up and start pumping telemetry.
The UI is super intuitive and the OOTB dashboards are great as well.
Clickhouse not being Elastic/Opensearch based means they would need to reinvent that wheel in any case because Kibana cannot use Clickhouse for storage. So this isn't so much an alternative but an essential component to make Clickhouse useful. Since you can't use Kibana for that. From various accounts here; they seem to have done a decent job.
Of course the key strength of Kibana is that it builds on features that Elasticsearch has; like aggregations that are probably more limited in Clickhouse. Same with Opensearch Dashboards. It depends on your use case whether you actually need that of course.
One point of concern with Clickhouse is that, like Elastic, they require contributors to sign contributor agreements. This basically allows them to re-license the code base if they want to at some point. Which is of course what Elastic did several times now (they changed it back to AGPL a few weeks back). Like Elastic are well funded by VC money but still pre-IPO. Just saying that if you moved to Clickhouse because of the Elastic licensing debacle, you might just have moved that problem instead of solving it.
[0] https://azure.microsoft.com/en-us/updates/public-preview-azu...
Also, I'd like to improve my observability using OTel in Cloudflare Workers but it looks like the example is out of date using an deprecated library which points to a new one to use instead. Might be worth updating the docs on that when you get a chance.
Deprecated: https://github.com/RichiCoder1/opentelemetry-sdk-workers New: https://github.com/evanderkoogh/otel-cf-workers
But you can also look at example payloads in the otel repo: https://github.com/open-telemetry/opentelemetry-proto/blob/m...
Great call out on CF workers, will make sure we get that updated :)