and here query builder helps by making it earier to do cross-signal joins and subqueries. I see that is upcoming in SigNoz https://signoz.io/blog/query-builder-v5/#what-we-couldnt-shi...
You might also like the interactive dashboard feauture that was released in the recent launch week https://www.youtube.com/watch?v=YQTQXq0F5Iw&ab_channel=SigNo...
Interesting post. How do you apply restrictions on your queries? Otherwise a few concurrent queries scanning huge data or being slow due to groupby, etc can slowdown the system. Also, I see a sorting key of `ORDER BY…
However the CPU and memory resources used are very minimalist. On no load conditions, it takes around 0.3-0.5 CPUs and 1.5-2GB RAM
I have seen it ingest 500K events/s. How did you conclude the poor perf?
When handling surges of the order of 10x, it's much more difficult to scale the different components of loki than to write them to Kafka/Redpanda first and consume at a consistent rate.
it's very hard to think s3 work as a buffer. Every datastore can work for almost all storage usecases buffer/queue/db when the scale is low but the latter were designed to work at scale
https://www.confluent.io/blog/exactly-once-semantics-are-pos...
Caching is to improve read performance whereas Kafka is used to handle ingest volume. I couldn't correlate the Grafana articles shared
Hi Prabhat, I am one of the maintainers at SigNoz. Nice to see OpenObserve's belief that the future of Observability should be OpenSource. We chose clickhouse rather than building a database as it takes multi-year…
An OpenSource alternative to DataDog built on ClickHouse database which is also native to OpenTelemetry https://github.com/SigNoz/signoz
https://github.com/SigNoz/signoz/blob/main/LICENSE
Yes...we are avoiding join altogether. Currently we used join in timeseries but we are probably moving away from that due to perf. Single table is amazingly fast
Basically, the choice of DB is different, relational vs analytical DB. IMO clickhouse should be better at ingestion rate and aggregation queries.
What does your schema for metrics look like? Using materialised views? I am particularly interested in storing metrics with labels(key/val pairs) Eg, Prometheus metrics. You can't flatten them out into columns due to…
I would love to see some benchmarks when making such a heavy claim. I would be interested in knowing performance of ingestion rate, query timings and resource usage.
We have recently been talking to a few companies in HealthCare and Fintech, and it seems they have strong needs for data to remain on-premise. One of such verticals in the company is around monitoring, logs and traces.…
hmm..got your point. We shall definitely look into other queuing system to be integrated as interface. Trying to understand better, what's a super simple dev setup like (to get the adoption)? Right now, we can run…
Thanks, feel free to reachout to us if you have any concerns or want to discuss in general. We are always eager to learn and help.
Nothing wrong there. If enough users want, we can add clickhouse also
Correct, Ideally monitoring stack should be outside the blast radius of other applications. Will handling another Kafka cluster (probably smaller than business Kafka) be a pain for the team given the team already knows…
Grafana, for long, has been used to monitor time-series data and recently has been moving towards observability (including traces and logs). We are different in quite a few fronts. 1. There are specific observabilty…
hmm..I get your point. I searched for Kafka alternatives for a bit before including it on our stack. Though, I couldn't find something more adopted by all. It would be good to know a few Kafka alternatives you prefer…
I completely agree with you. For companies not already using Kafka, this will ask for a big commitment to self-host Kafka. You mentioned a great approach. Queueing system as a plugin. Thanks
Great point. To start off we shall provide different hardware configs like micro, small, medium, large, xlarge with the scale that they can handle. We soon plan to emit metrics from different components of SigNoz and…
and here query builder helps by making it earier to do cross-signal joins and subqueries. I see that is upcoming in SigNoz https://signoz.io/blog/query-builder-v5/#what-we-couldnt-shi...
You might also like the interactive dashboard feauture that was released in the recent launch week https://www.youtube.com/watch?v=YQTQXq0F5Iw&ab_channel=SigNo...
Interesting post. How do you apply restrictions on your queries? Otherwise a few concurrent queries scanning huge data or being slow due to groupby, etc can slowdown the system. Also, I see a sorting key of `ORDER BY…
However the CPU and memory resources used are very minimalist. On no load conditions, it takes around 0.3-0.5 CPUs and 1.5-2GB RAM
I have seen it ingest 500K events/s. How did you conclude the poor perf?
When handling surges of the order of 10x, it's much more difficult to scale the different components of loki than to write them to Kafka/Redpanda first and consume at a consistent rate.
it's very hard to think s3 work as a buffer. Every datastore can work for almost all storage usecases buffer/queue/db when the scale is low but the latter were designed to work at scale
https://www.confluent.io/blog/exactly-once-semantics-are-pos...
Caching is to improve read performance whereas Kafka is used to handle ingest volume. I couldn't correlate the Grafana articles shared
Hi Prabhat, I am one of the maintainers at SigNoz. Nice to see OpenObserve's belief that the future of Observability should be OpenSource. We chose clickhouse rather than building a database as it takes multi-year…
An OpenSource alternative to DataDog built on ClickHouse database which is also native to OpenTelemetry https://github.com/SigNoz/signoz
https://github.com/SigNoz/signoz/blob/main/LICENSE
Yes...we are avoiding join altogether. Currently we used join in timeseries but we are probably moving away from that due to perf. Single table is amazingly fast
Basically, the choice of DB is different, relational vs analytical DB. IMO clickhouse should be better at ingestion rate and aggregation queries.
What does your schema for metrics look like? Using materialised views? I am particularly interested in storing metrics with labels(key/val pairs) Eg, Prometheus metrics. You can't flatten them out into columns due to…
I would love to see some benchmarks when making such a heavy claim. I would be interested in knowing performance of ingestion rate, query timings and resource usage.
We have recently been talking to a few companies in HealthCare and Fintech, and it seems they have strong needs for data to remain on-premise. One of such verticals in the company is around monitoring, logs and traces.…
hmm..got your point. We shall definitely look into other queuing system to be integrated as interface. Trying to understand better, what's a super simple dev setup like (to get the adoption)? Right now, we can run…
Thanks, feel free to reachout to us if you have any concerns or want to discuss in general. We are always eager to learn and help.
Nothing wrong there. If enough users want, we can add clickhouse also
Correct, Ideally monitoring stack should be outside the blast radius of other applications. Will handling another Kafka cluster (probably smaller than business Kafka) be a pain for the team given the team already knows…
Grafana, for long, has been used to monitor time-series data and recently has been moving towards observability (including traces and logs). We are different in quite a few fronts. 1. There are specific observabilty…
hmm..I get your point. I searched for Kafka alternatives for a bit before including it on our stack. Though, I couldn't find something more adopted by all. It would be good to know a few Kafka alternatives you prefer…
I completely agree with you. For companies not already using Kafka, this will ask for a big commitment to self-host Kafka. You mentioned a great approach. Queueing system as a plugin. Thanks
Great point. To start off we shall provide different hardware configs like micro, small, medium, large, xlarge with the scale that they can handle. We soon plan to emit metrics from different components of SigNoz and…