Show HN: We built a ClickHouse-based logging service (github.com)
Today we're sharing a ClickHouse-based logging solution we've been working on. We wanted to showcase how we built it and share how you could try it out to give feedback. Since we started working on highlight.io, we've been hyper-focused on "cohesion", or ensuring that when you install your monitoring stack, all of the resources in that stack (user interactions, requests, traces, logs, etc.) are connected in a consumable way. We've written up more about our philosophy on this here [1].
We started building towards this by connecting your client-side app and your server-side exceptions with session replay and exception monitoring; i.e. if an error happened in a server-side app, we would make it easy (with session replay) to trace all the steps that a user took leading up to it.
Especially for larger companies using highlight.io, the request to tie in logs came up repeatedly, and we wanted to build this with the same philosophy in mind. Now, you'll see client-side and server-side logs all in one place, brought together in the context of a user session, as well as logs in the context of an error.
Like the rest of our stack, this project is written in Go and Typescript, and for log ingestion/querying, we're using ClickHouse [2]. Before deciding on ClickHouse, we were planning to use OpenSearch (an aws fork of elasticsearch [3]) for this part of our product, but as our traffic has increased, we encountered quite a few pains with write throughput for OpenSearch. After evaluating a few options, we eventually landed with ClickHouse (their cloud offering was icing on the cake), which has also proven to be much more cost-effective so far.
Building with ClickHouse from scratch has been an exciting journey. Eric (the mastermind behind this project) wrote a blog post [4] on a handful of ClickHouse learnings we've gathered since starting the project.
For those wanting to try out the product locally, you can run the following commands [5]:
git clone --recurse-submodules https://github.com/highlight/highlight cd highlight/docker; ./run-hobby.sh;
To send logs to highlight, you can use your own OpenTelemetry implementation [6] or use our SDKs [7] which provide lightweight wrappers over OTEL.
Like the rest of highlight.io, we plan to make money from this with our hosted cloud offering. For those interested in trying out the cloud-hosted version, you can get setup at app.highlight.io.
To open the floor for feedback, we would love to get some thoughts on what we've built so far. Beyond that, what are parts of a logging product you wish you had with your current setup? And are there any notable pain-points of using a hosted monitoring product? (We're toying with the idea of an enterprise deployment). Excited to hear from everyone.
[1]: https://highlight.io/docs/general/company/product-philosphy
[3]: https://news.ycombinator.com/item?id=26780848
[4]: https://www.highlight.io/blog/how-we-built-logging-with-clic...
[5]: https://www.highlight.io/docs/getting-started/self-host/self...
113 comments
[ 3.5 ms ] story [ 235 ms ] threadA good comparison we've referenced: https://leventov.medium.com/comparison-of-the-open-source-ol...
Many of those bells and whistles are near-necessary in the enterprise world, but you have the accumulated mass of 'red zones' and developmental landmines in that ecosystem that can quickly turn you off it as a whole if you want to understand the whole system.
>JVM runtimes have a relatively high startup cost I think many people are okay with that when developing server software that's going to run weeks at a time. It can get a bit annoying with trying to rapidly iterate. And I think things are changing pretty quickly with AOT builds and general improvements.
>and the build processes for a lot of JVM deliverables is an ungodly mess.
I recall using "mvn package." That's it. This was on two different systems that served a good bit of traffic and weren't simple trivial projects.
Still not as good as C#'s debugger in Visual Studio (hit a breakpoint, edit the code, drag the execution back before the problem, resume and run the patched version) but nothing I've seen really is.
Setting up Gradle projects is a bit more involved depending on your setup, but in the end it's still a single command to build an executable JAR.
Gradle is something I've never messed with, but that makes sense.
it sucks to use
many people believe otherwise, but those people have rich jvm experience, which is not easy to get
https://stackoverflow.com/questions/25120546/trick-behind-jv...
Main takeaways were: the SQL flavor and its aggregations in CH are amazing. Running on a single node for dev laptops is trivial. It’s crazy fast with almost zero tuning.
It does not surprise me at all the CH is powering new products and startups.
Note: hosted CH did not exist yet. We are using Altinity to run our cluster.
It exists now actually. We (highlight) are on hosted clickhouse, which went in GA a few months ago. https://clickhouse.com/cloud
Disclaimer: I work at Altinity.
Also, thank you for introducing me to air (https://github.com/highlight/highlight/blob/sdk/highlight-go...) as that also looks super handy
p.s. for Show HN historians, here is the prior thread: https://news.ycombinator.com/item?id=34897645
I also miss the "good old days" when running sentry was like 3 containers, not the 32 of modern Sentry
Not sure how the license makes a difference in terms of being a value to your day to day development.
But I do love projects that can be self hosted, so Sentry is very nice on that part, especially when it doesn't limit its capability on the self hosted version.
Also this one doesn't even have PHP sdk and the GitHub issue doesn't have much of a request for it, which sounds like it has still some way to go to be mature given the lack of interest in the project at this point.
Couldn't they just reuse other project's sdk or help with OpenTelemetry? Developing yet another batch of sdk for a dozen language seems like wasted time.
But I do hope we get a nice open source (self hostable) tool that can do what it claims (error reporting, tracing, logging, metrics) because there's no such thing that does all well at the moment that's mature.
Keep an eye out for Highlight then!
> Couldn't they just reuse other project's sdk or help with OpenTelemetry? Developing yet another batch of sdk for a dozen language seems like wasted time.
We use OpenTelemetry for all of our SDKs. All we do is thinly wrap the SDK so developers don't have to deal with OpenTelemetry internal if they don't want to.
There's a doc on it here: https://www.highlight.io/docs/general/company/open-source/co...
Looking forward to seeing the project take off as we need to glue stuff together from various projects to achieve your goal now.
We'll likely get to it by Q3 of this year. We hope that the design choices we're making (Clickhouse, OTEL, etc..) will set us up for supporting tracing easily when we get to it.
Cant say I would call these specs “hobby” at all
Not high availability? Sure.
However, I've seen software out there that ran as a monolith with a single deployment unit, facilitated lots of business processes and entire teams of people for continued development. Not everything necessarily needs high uptime, either. Some software can also serve particular time zones and have ample windows for scheduled maintenance, upgrades and so on.
There's probably at least a few classifications between hobby projects on one end and HA distributed systems on the other.
>> Cant say I would call these specs “hobby” at all
With this, however, I'm inclined to agree. In my eyes, "hobby" would imply something more along the lines of: "Just give this half a CPU core and about 512 MB of RAM, maybe up to a GB of storage depending on what you'll use it for, it'll probably work well enough for a few users."
Some software that mostly fits that definition, in my experience: Nextcloud, Apache2/Nginx/Caddy, Grav, Mattermost, Gitea, Heimdall, YOURLS, PrivateBin, phpBB, Uptime Kuma, Zabbix, PostgreSQL, MySQL/MariaDB, Redis, RabbitMQ, Docker Swarm and plenty others.
Some software that needs more resources: SonarQube, PeerTube (for encoding), OpenProject (Ruby app), Sonatype Nexus (bloated Java app, but lots of functionality), Matomo (issues with displaying historic data with low resources), BackupPC (compresson of backups), K3s and other Kubernetes cluster distros and plenty others, too.
Not to say that it somehow makes the software worse, just that people have different expectations. Perhaps more realistic expectations on my part for hobby software should be: "You should be able to launch it with whatever spare resources your laptop has."
Just updated this: 8GB of RAM, 4 CPUs, and 64 GB of disk space.
However, after scrolling through the GitHub page, I feel like this is not a service which aims at people (might be completely mistaken) who either have a small set of services to monitor (and/or understand the logs and/or the interest to do it) or have their homelab at a relatively low financial priority (1x 8GB 2400 CL17 is €15 here).
8GB for a single service in a (home) environment is, imo, still a lot, but I think it is a sort-of reasonable figure for what it does and need to do to make that happen
I would say 4G may more sense to me, I know how much engineering effort it requires though. LOL
Loki + Grafana isn't really good for log viewing at all. I use Metabase to read logs sent to ClickHouse which gives far nicer interface.
How do people go about vetting and/or self-hosting those, if you operate [software for] a business with actual customers? Is stripping sensitive data at the client enough? Do you lock down outgoing connections through external networking configuration if you self-host? Am I being too paranoid?
That said, there are mitigations you can take. There are end-to-end encrypted log solutions out there. Honeycomb.io used to have (they might still?) an interesting offering I used at one employer to encrypt sensitive fields in logs leaving our infrastructure. They had the UI set up to talk to our encryption service and decode things on the fly in the user's browser-side UI so that they (Honeycomb) never had direct, unfettered access to sensitive data.
There are other approaches you can take, but things get tricky when you either need to audit your vendor's access to your data or assume that your vendor can't secure your data to your satisfaction. Better to do it yourself at that point if you have the resourcing to do so.
Limiting would be ideal, but all I was suggesting is that you indicate above the fold that you have cheaper options
It's extremely useful to prototype and experiment with a project and have it have a total budget that will not be surpassed.
One more idea: pay as you go. I pay $10, that turns out to be not enough, I pay another $20 and get immediately unblocked.
Disclaimer: I'm one of the VictoriaMetrics maintainers.
If you want to learn more here's a webinar I did with Roman Kavronenko at VictoriaMetrics that compares it with ClickHouse.
https://www.youtube.com/watch?v=sCrdp8hIhJM
Disclaimer: I work for Altinity.
We don't have pointed plans for supporting Elixir, but hopefully this opens the floor for anyone interested.
> Before deciding on ClickHouse, we were planning to use OpenSearch
You should have tried Quickwit :)
Anyway, sounds like a great project, best of luck!
[1] https://github.com/openreplay/openreplay
[2] https://github.com/quickwit-oss/quickwit
Our session replay is similar to OpenReplay but we've focused a lot of effort on making a cohesive backend debugging experience. Highlight sessions give you backend error monitoring and logging out of the box to make it easy to get to the root cause of a bug.
> Do you implement your own storage with Quickwit or can it be backed by a cloud storage solution like S3?
I'm a cofounder of Quickwit and I did not use OpenReplay directly, I just know OpenReplay used ClickHouse + Quickwit for search. I don't know if this is a common setup and if it was just used for very specific use cases. You should ask them :)
volume for any nontrivial organization is too large
And what kind of volume is it that ClickHouse can't handle when Uber can?
https://eng.uber.com/logging/
Log messages often compress very well (> 95%) so storage is not as much of an issue as you might think.
Disclaimer: I work for Altinity
We do quite a bit more though with session replay and error monitoring for your full-stack web apps.
I had to look some things up to figure out what the conversation was about.
Also would recommend against "read more here," instead simply link the words "Session Replay" etc. This is friendlier to screenreader users in particular but also everyone, as readers are drawn to the blue highlighted text and you'd rather that they see the important word first, and not "here" first.
Which is to say: is there some obvious hook-point to add watcher jobs (as Clickhouse stored procedures, maybe?) to process correlations in the inputs to the system? (Where by "watcher jobs", I mean things like "create a record in a table if a user makes requests across endpoints A, B, C, in that order — with other arbitrary requests from that user in between — in a five-minute sliding window, at some point within 12 hours of the user's registration.")
Using a DB as a log-analysis system would be pretty great, if it was practicable, as you'd be able to correlate present events with events from the distant past (or with statistical aggregations of all of history up to the present), rather than having to build your correlations only from what can be buffered in memory.
But most log-analysis platforms need extremely high event-processing-job performance to scale — Mozilla Hindsight, mentioned before, is a rewrite in C+Lua of a previous system (Mozilla Heka) where the fundamental bottleneck was the Golang runtime. I'd be curious to know whether Clickhouse sprocs/triggers/etc have been tuned to function at that sort of scale...
This is not something we have planned, nor have we heard yet. Is the use case for doing analytics on your web application? Or more of a complex tracing use case that creates new traces as time passes by?
Fwiw, Highlight is designed (in the current state) for basic log search and alerts.
We want to notice users who we ban for breaking the Terms of Use of our API SaaS, who then create new accounts and immediately resume doing the same thing they were doing before they were banned — where the metadata is all different (they're rotate VPNs, get a new [stolen] credit card, etc) but where there is a distinctive "activity-pattern fingerprint" to their ToU-breaking activity (different per violating user, but the same between the "incarnations" of the same user); and where having that fingerprint in the context of a brand new user is implausible, since nobody could learn to use our API to do such a complex thing so quickly.
https://www.highlight.io/docs/getting-started/self-host/self...
If you're interested in doing that, feel free to message us in our community.