Show HN: SigNoz – open-source alternative to DataDog, NewRelic (github.com)
We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us
Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion.
This is our first release with Logs support and we have added support for:
- Filtering logs based on fields
- Full text search in logs
- Live mode to see logs coming in realtime
- Detailed view of logs in table and json format with ability to add filters quickly
- Ability to specify interesting fields which will be indexed by default
134 comments
[ 3.8 ms ] story [ 226 ms ] threadMany big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion.
This is our first release with Logs support and we have added support for:
- Filtering logs based on fields
- Full text search in logs
- Live mode to see logs coming in realtime
- Detailed view of logs in table and json format with ability to add filters quickly
- Ability to specify interesting fields which will be indexed by default
https://signoz.io/docs/install/docker/#prerequisites
https://github.com/SigNoz/signoz/blob/develop/LICENSE
We use clickhouse at Luabase and join performance is the only weak point we've hit.
Each time I look into this. I look to see how it can report CPU/Memory/Disk of the systems at large. When I read, all I find on open telemetry and SigNoz is how to integrate with application stacks.
Am I missing something fundamental in that SygNoz/OpenTelemetry do not integrate with host level metrics? I really, really want to use this product. But this extremely BASIC implementation to use the service is 100% missing for documentation as far as I can tell. Even the example page has nothing listed https://signoz.io/docs/tutorials/ ...
So I am either trying to find out how to make SygNoz do something it was not built to do... or the documentation has a huge hole in it.
Let me know how to proceed, if you have time. It's appreciated so that I dont keep walking down the wrong road and hoping to find something useful.
Thanks!
As of now, there are couple of ways to do this: 1. You can enable hostmetrics receiver in opentelemetry collector 2. You can use something like prometheus node exporter and enable prometheus receiver in opentelemetry collector
If you follow this for a VM setup, you should be able to get your host metrics - https://signoz.io/docs/userguide/send-metrics/#enable-a-spec...
If you are on k8s, check out this - https://signoz.io/docs/tutorial/kubernetes-infra-metrics/
If it is still confusing, do drop by in our slack community ( https://signoz.io/slack) We are quite active there and should be able to help you get started
As time investment is a dangerous thing I didn't plop myself down for some R&D... But... thanks to your post I'll have another go at it and determine how to get it to hum.
Thanks again!
I wanted to give signoz a try, but the sheer amount of services attached discouraged me, especially as I have to reconfigure them all to work with my setup. (Don't run them directly on the host, instead in a separated network, put the network interface behind traefik, figure out which access they need, provide all the configuration in a nice way without having to clone the full repo just to have the configuration files.)
[1] https://github.com/SigNoz/signoz/blob/develop/deploy/docker/...
Without thinking too much about it, I assume that: hotrod is a demo data source pushing traces to the otel collector, which stores data in clickhouse. Frontend fetches data from clickhouse using the query service. Alert manager probably looks at traces coming in and detects anomalies, so that you can get real-time alerts when things don’t look normal.
You can easily remove the sample services if you want - https://signoz.io/docs/operate/docker-standalone/#remove-the...
SigNoz has the following components
- Frontend
- ClickHouse (datastore)
- Alert Manager (this is to monitor metrics and create alerts which you configure in SigNoz)
- Query Service (which is the backend service which talks to datastore & frontend)
- Otel collector ( which the collector provided by opentelemetry to collect telemetry data)
The other two components for sample app which can be commented out
- Hotrod ( which is a sample app)
- load generator
More details here - https://signoz.io/docs/architecture/#architecture-components
https://github.com/SigNoz/signoz/issues/1188
Or is there some kind of gateway standard that I'm unaware of?
This can be used to add whatever authn/authz you require to apps that don't even support authn/authz. I'm using Traefik ForwardAuth with Keycloak for Jaeger SSO in a couple of places.
That is: I wouldn't hold off on a PR just because they said they're going to get around to it; if the PR works, and is merged, that's one less part they have to write. If they don't merge it, then the bad faith you're discussing will be a concrete fact and not speculation, and will serve as a warning to others not to bother submitting more PRs
So I am optimistic for alternative solutions like this!
We also have an active slack community if you have any questions on how to set up or have an feedback for us - https://signoz.io/slack
We're not in the US, so the cloud-hosted version of New Relic is especially slow because of the added latency of the trans-oceanic network hop.
For comparison, Azure Application Insights can be deployed in our own region. It's not massively faster, but for some UX design reason it feels faster and more pleasant to use. It might be literally just the network latency, and nothing more, but the end result is that it's used more often.
Application Insights isn't perfect by any means:
- Deployment is a PitA and breaks regularly. The documentation related to installing it is confusing, out-of-date, and will guide you down dead ends. For example, I got the profiler component working once on virtual machines, then it broke, and I can't get it working again for the life of me.
- The underlying Log Analytics workspaces are crazy, crazy expensive. They're far more expensive than the competition, which then makes high-level services like Sentinel and App Insights built on top also too expensive for most orgs. For comparison, Log Analytics is about 5x as expensive per GB as AWS CloudWatch logs, and up to 30x as expensive as some other similar services.
If Microsoft just fixed the installer and used reasonable pricing for Log Analytics, the App Insights would be very hard to beat, especially for .NET shops.
I'm hoping open-source tools like SigNoz force down the pricing from "highway robbery" to merely "greedy".
Curious, does the location of the server introduce the latencies (as you mentioned you are not in US)? I would have assumed the latency because of server location would be very small.
Have you verified that the latency is actually because of the trans-oceanic network hop?
All US-hosted web services feel slow here. It’s a baseline sluggishness that permeates everything we use that is cloud hosted.
The only exception is services that have local instances or replicas of some sort.
Is this latency deterring enough that you prefer running things in your region and not prefer SaaS product which are generally hosted in US/EU?
Or is this just a discomfort which you deal with?
Consider that every fetch of a resource may itself include another resource (IE; a html page which contains a CSS include).
Now consider that this happens recursively (IE from the above example: a CSS include that itself includes a font or an image).
It's very easy to get 1+s load times with transatlantic latency alone.
Question: Does this increase in latency make cloud services less interesting for companies which are not in US? What kind of cloud services will be especially affected
A lot of people in Europe are using European datacenters.
That is not to say there's no issues: The consoles can be unbearably slow at times. (Google Cloud being probably the worst offender in my experience, despite being a fan otherwise).
Amazon supports consoles in other regions, but if you use `console.aws.amazon.com` then it is us-east; it doesn't automatically change it for you.
Here's the list: https://docs.aws.amazon.com/general/latest/gr/mgmt-console.h...
Regarding making Cloud less interesting:
Europe basically follows whatever SV is doing, to make a crass comment: an article could be produced from SV saying eating poop would make better engineering and European "tech" companies would assuredly start buying up the sewage systems.
Even when it doesn't make sense; we seem to follow.
It does make me wonder though, why not automatically redirect people to the console that is closest to them? They could done anything from Anycast DNS to showing a simple little notification showing people there was an alternative possibly closer to them when logging in, but as far as I know, nothing is done about this.
But the initial login request to AWS goes via us-east-1 by default.
AWS has more recently published the list of login endpoints to use should us-east-1 be offline.
Let me know if you have a specific question which is not addressed in the above link and will try to answer
> if you want to have a seamless experience between metrics and traces, then current experience of stitching together Prometheus & Jaeger is not great.
But I wonder if using Promscale https://github.com/timescale/promscale would make Prometheus & Jaeger not such a big problem as SigNoz imply.
Promscale readme:
> Promscale is a unified metric and trace observability backend for Prometheus, Jaeger and OpenTelemetry built on PostgreSQL and TimescaleDB.
Either way, SigNoz seems interesting indeed. And am glad to see that SigNoz supports OpenTelemetry.
I have not explored the project in detail, but as far as I understand it uses Grafana and Jaeger UI, so I am not sure how seamless is the UI interaction, while SigNoz has a UI built ground up for observability use cases.
Might give this a spin next week, currently using NR, but it’s slow, expensive and the in-cluster collection services are frustratingly fragile.
Also, can you explain a bit more on
do you mean the agent they use for sending application metrics from clusters breaks down?Yeah basically, a number of the kubernetes deployments are configured with too-low memory and resource limits, so they're constantly crashing (and triggering all sorts of alerting false-positives). The limits are semi-hardcoded, so we can't override them, and at the moment, the effort required to fix the deeply-interlinked helm chart they provide isn't worth it. The NRI-Bundle helm chart also installs a lot into your cluster - it brings its own NATS instance, there's multiple daemon sets, etc. I'd likely get quite a lot of spare compute back by swapping to something lighter.
Essentially you add: keys Array(String), values Array(String) and write the pairs accordingly.
https://github.com/SigNoz/signoz/blob/develop/deploy/docker-...
Definitely a product where no one in the org said "no" to an idea. Felt very Atlassian.
Curious though, what specific features/UX did you not like in DataDog?
You use specific tags to mute a monitor by adding a scope. This does not scale well with the UI.
Have you ever timed how long it takes a normal user to do certain things easily, say during an incident, or you're just crapping on the CSS?
Do you this this case for most companies monitoring serverless applications with DataDog, or there is something specific about your infra which cause this
Many other product teams at my work have lightly used serverless apps. The DataDog costs simply aren’t feasible for serverless apps. We’re actively looking into alternatives such as just using CloudWatch, Elastic, etc as it’s a huge cost for us.
I was just checking Datadog pricing for serverless, there is says - $7.20 per active function per month. If you are using 45 lambdas, is the number of functions much higher? I am guessing ~200 or so?
Though I can see, how charging based on functions can quickly shoot up the bill
No idea what happened and MSFT support couldn't tell me what was happening because at more than $100/day burn rate on a hobby project I started deleting everything connected with the effort as fast as possible.
All I know is my AKS wasn't exploding. Services were still responsive and acting normally in their minuscule cluster, this was just a logging cost explosion.
Also billing alerts are your friend.
Were these also on AWS Lambda or something else (EKS?)
If you're paying less than 10% of infra costs for monitoring, you probably don't have good enough monitoring. But if you're paying more than 25% of your infra costs for monitoring, someone is not doing their job.
What do you mean by this?
We have a lot of request/reply CRUD type requests that are heavier on reads than writes. We use API Gateway to manage websocket connections for us. This type of usage pattern and size of our customer base fits well with serverless.
As of now, we have fixed threshold based alerting capabilities - but more advanced ML/seasonality based anomaly detection is on the roadmap. we are tracking this here - https://github.com/SigNoz/signoz/issues/295
What type of outlier analysis do you generally do in DataDog/NewRelic?
And are there any other similar projects.
We also have an active slack community if you have any questions on how to set up or have an feedback for us - https://signoz.io/slack
The rest of the code is still MIT licensed. And if you remove the ee/ folder, the project would work without any issue
This licensing model is very similar to what folks like Gitlab and PostHog do today
- Open core means no direct path from testing to using (and paying). If I want to make a case for a software to be included in our stack I don't want to constantly run into paywalls while trying out the product but I also don't want to go through the hoops of getting a licence (because of internal bureaucracy).
- You steer away users that cost you nothing but help you spread to word and report issues and might even provide a PR. I'm talking about OpenSource Projects, student organisations and companies with limited resources (NGOs, early stage startup's). I used to be head admin in a student organisation. We developed our own internal tools and only relied on open-source components. I know of at least two cases where other students got to know the tools and after they finished university joined companies and introduced the very tools we used to their employees resulting in paid support contracts.
- It creates tempting opportunities for investors to force you into ruining the open source tier. In the beginning you only have features behind the paywall that are useful to big enterprises but if your business is not growing fast enough (from the perspective of investors) they might force you to push more users to be paying customers. That might help in the short term but will ruin your reputation in the long run.
- Paying for services (aka insurance to get help if something goes haywire) is easier to justify with execs then paying an unreasonable amount for that one feature that is behind the paywall. ("Can't you just make it work without it"). Its a purely psychological argument but decision processes in companies are not always rational and your allies are the devs and you should be helping them make a case to buy your product.
First, AGPL only requires to release changes ONLY to the existing AGPL codebase and ONLY if you are providing it as a network service.
Second, the whole idea of virality is a huge misnomer. There is no such thing as one thing "infecting" another in copyright law. GPL/AGPL cannot magically make another piece of software change license.
From the website: "Why get locked-in with SaaS vendors like DataDog when you can use Open source?"
You expect users to trade one form of lock-in for another one?
We are natively based on opentelemetry which is emerging as the industry standard for instrumentation. So, you can change product you used for backend and visualising and storing your telemetry data very easily
Is this something you have an answer for?
Many of our users use SigNoz in a similar fashion.