As Developers, SRE, DevOps look to replace Elastic from their logging stack - we believe the log storage of future won't be another search and indexing engine in a new language.
Parseable leverages Apache Arrow, Parquet and widely available object storage platforms for efficiency, cost effectiveness and performance.
Aside - with so much to choose from wrt logs and metrics - I'd assume "prometheus for logs" implied "pulling" log data (rather than push). While maybe "Victoria metrics for logs" might imply "easier to operate, efficient, fewer moving parts".
I hadn't heard of qryn before, but it seems that the AGPL has become the go-to license of choice for these kind of tools. It makes me sad, but for clarity: it just makes me sad, the authors are entitled to use whatever license they want
I always find it confusing what I can and can't do with the entire GPL family of licenses, specifically when something is considered a "derivative" work or not. The AGPL network provision makes this an even bigger issue. It also introduces a whole new layer of uncertainty.
I also have seen a trend over the years of companies who adopt AGPL not caring about the open source community around the code. This is not specific to the AGPL, and not every user of it does so, but I see it correlating more often.
Many companies refuse to use AGPL licensed code... some claim their lawyers are stupid and don't know what they are doing... but it can still prevent adoption of AGPL licensed products.
So, it means that I don't get to use awesome looking toys at work because no one is going to pay the lawyer fees to find out what, exactly, we can and can't have those network requests touch
I have been trying to be much more careful in commentary about this because I'm not trying to change the author's license, it's their code, and I'm not trying to get into debates about "well, it only matters if ..." because IANAL and I don't want to assume the risk, because for this situation, being wrong would be very expensive, so I just stick to Apache or MIT tools
My comment wasn't a call to action, it was just crying out loud
Well, a tiny bit because then I'd know the author had absolutely no interest in being a participant in the open source community and just wanted free labor. The AGPL license decision I believe is made in good faith, sharing the code as well as the rights to the community. The AGPL choice just means (from my perspective) that the code was _almost_ open enough that I could use it or maybe even contribute fixes for edge cases I run into, versus those "source available" where I can just immediately close the window -- or, I guess, pedantically, it means I can check back on the project in 5 years or whatever the "license change cliff" is nowadays
As the author of qryn, I agree this is not perfect or ideal - but for lack of other options, we still consider AGPL better than BSL/SPL Licenses at protecting our community of real users. Any suggestion is absolutely welcome!
at qryn we're past the logs-only phase and currently we're concentrating on the polyglot aspects - we already support ingestion and querying of logs, metrics and telemetry using various inputs (loki, influx, elastic, etc) and supporting logql, promql and tempo for searching. Less moving parts, more power.
I really like the idea of storing logs in parquet format in S3. It's something I've wanted to play with for a long time, but never have had a use-case. I'm also a big fan of Rust.
It's simple in that if you are already using an s3 compatible object store you don't have to worry about setting up a separate redundant, durable, highly available, and possibly scalable storage system for your logs. And as for the implementation of this or Loki, it means they can delegate most of the complexity of data storage to a separate object store.
A guy that is stuck in a liver donation loop for stealing fire, but he stole wood instead?
If you have to describe your project as "some other project but like this" you're already doing it wrong. Do you think Bing markets its maps by saying "Think Google Maps But Microsoft"?
I was really excited to see something like this. I really dislike my company's current logging solution (I mostly take issue with how slow querying logs is most of the time). Unfortunately the live demo didn't work. Maybe it was hugged to death? In any case it might be nice to get the demo more stable!
Interesting - looks like both of these are openTelementary + clickhouse for data. Qryn (loki on clickhouse) seems similar - but with different query and dashboard. Anyone evaluated these recently, or have experience using any of them in anger?
Thanks for mentioning our project! qryn (formerly cloki) is currently more focused on the polyglot factor and trying to unify logs, metrics and telemetry on a single stateless platform, easy to scale without hundreds of services and moving parts. At this stage, its a lightweight Grafana Cloud alternative just requiring clickhouse - no sidecar databases, redis, or plugins needed, and no new query languages or rules to learn. Latest info is at https://qryn.dev
Is there currently a big market for new logging solutions? It has been years since I worked much with logs, so I don't know much about the current frustrations of those who work with them on a daily basis.
There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features?
The reason I ask is because I am building a new kind of general-purpose data management system (https://www.Didgets.com) that can handle all kinds of data. It does file data, relational tables, Key-Value stores and other kinds of NoSql data. I did a little proto-typing of it managing some simple logging data and it seemed to work really well (in my opinion). But I would need to do a bit more work to get a good demo going.
I have limited resources, so I would only want to do that if there is a significant demand in the logging space for something new. What is the biggest 'pain point' for people working with log data today?
>Is there currently a big market for new logging solutions?
There is a market, but it's saturated with offerings. There are so many solutions out there right now for metrics and log aggregation.
>There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features?
I'm going to take a guess and say the latter. There are a lot good(enough) off-the-shelf and managed services, that another offering is going to have a very hard time breaking through.
>What is the biggest 'pain point' for people working with log data today?
Ingestion, transformation, indexing, querying, visualization - those are all pretty much solved problems and everyone has a good enough solution. For me, it's specialized/niche use-cases and workflow, where the gaps are. For example, is your logging managed service able to be a drop-in HIPAA compliant solution (are individual customer logs physically segregated at the host level? Do you support automatic detection of Personal Health Information? Do you have an audited workflow to remove or deidentify, in batch, the offending logs. Do you support specialized access and dashboards to my customers so that they can see metrics and logs for their site? Do you support air-gapped, enterprise, on-prem deployments?).
I've been doing ops for a long time, and I can't remember the last time I looked at an historical log (edit: I'm referring to application logs specifically stored in a central location). When something is broken, I turn on logging and look at the logs in real-time. If something was broken in the past and no longer is, I turn on logging for that thing so that I have logs if it happens again.
The only use I can think of for historical logs is security auditing. In which case there is only a subset of logs you really need to trace a breach and they don't need to be searchable online -- you can pull them back from cold storage if you need to check for an incident.
Moving and storing logs and making them all instantly searchable is awfully expensive for little benefit.
Edit2: Disclaimer: This opinion does not apply to government regulated services like financial services.
When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.
I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state.
We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
Really? We might be working in entirely different spheres and cultures, but I tend to look at (recently historical) logs pretty frequently for a bunch of reasons: helping to isolate infrequently encountered and hard to reproduce bugs, tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, identifying whether a particular code path is actually in use and if so how heavily, etc etc.
They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.
> helping to isolate infrequently encountered and hard to reproduce bugs
As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help.
> tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour,
That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs.
> identifying whether a particular code path is actually in use and if so how heavily
Again that would be request tracing, not application logging.
> They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
> Again that would be request tracing, not application logging.
from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience.
i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
Gotcha. I think we're probably thinking about different kinds of environments - probably the key differentiation is the total volume of logs, the kinds of services we're monitoring, and how bad a given error is.
> Having old logs around is awfully costly for the off chance that they help.
Depends massively on the log volume. I've worked at places that generated terrabytes of logs over the course of a few days, and required very large ES clusters to try to keep on top of it. There was definitely value in having those logs, but it only made sense to retain them for a couple of weeks. It's much less costly for low volume services.
> ...request tracing, which is different...
Ye-es, until it's not. If your current setup isn't yet developed or grown enough to have full request tracing, you can get a long way filtering aggregated logs by a correlation ID. I know it's no substitute for a fully feature tracing system, but in situations where you have limited resourcing you need to prioritise it can get you a long way
> If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected.
That massively depends on the impact of the incident, I think. If it's a "There was a problem playing this title, please try again" kind of deal, it's no biggie.
If it's a "my broadcast TV channel is off air" or "my financial system has crashed in an unexpected way trying to reconcile this transaction" type situation, you might be a little more worried - and the customer is likely going to be very upset for quite a while, and some people with very serious faces might come to talk to you about very large fines.
As a software engineer responsible for building and operating various async and background workflows, I get a lot of inquiries like "I see cause X, why wasn't there side effect Y?" and I go and poke around the data to figure out what happened. Sometimes the cause didn't really happen, sometimes there are other criteria that the stakeholder wasn't thinking about that weren't met, sometimes the cause really happened but the data is missing or malformed upstream, and very occasionally my system errored out and dropped the ball. These are all deeply mysterious head-scratching issues if you don't have ready access to detailed logs.
More mundanely, if I get paged about an error rate, the first thing I'm going to do is look at the logs to see what the errors are.
I, and my peers, look at historical logs daily. Our customer response team is often looking at them on a minute-by-minute basis, and sets up alerts on them.
Our logs tell me the state of the application when the error occurred without having to reproduce it. They also tell me how many customers are having the issue, so we can set the severity of the incident at the appropriate level and reach out to those customers proactively if needed.
Sure, we might be able to do this with metrics or tracing instead, but that's often just duplicating information we're already logging. For example, by adding a correlation ID in my log metadata, I suddenly have end-to-end request tracing. It's not as pretty as what New Relic can produce, but it definitely works.
Ofcourse, If your business is growing, chances are, you are collecting more and more logs. Your log management process is ultimately slowing down your troubleshooting. Atatus provides you with hassle-free log management delivered as a fully managed cloud service. You don't have to set up and maintain it. This instantly cuts costs, reduces IT load and improves both the speed of troubleshooting and resolution of issues.
> By default Parseable uses a public S3 compatible store (MinIO) to store the data. Please change this to your S3 bucket before storing any important data.
Wait, am I reading that right? Does this mean by default the docker container is going to store my data in a random place on the internet?
> Please do not store any sensitive data on this server as the data is openly accessible. We'll delete the data on this server periodically.
It is there for demo purpose ig
It also gives you this warning when you run the build
Warning:
Parseable server is using default credentials.
Setup your credentials with P_USERNAME and P_PASSWORD before storing production logs.
Parseable server is using default object storage backend with public access.
Setup your object storage backend with P_S3_URL before storing production logs.
Having a public place to send demo logs is fine. Sending logs there by default, rather than having to opt in to demo mode, is bad. The default should be to fail to start unless credentials for the data store are provided. Or alternatively store to the local filesystem by default.
The problem is this means I'm one misconfiguration away from sending all my data to a stranger's server on the internet. If I misconfigure an Ansible playbook or kubernetes deploy I'm not going to see that warning. Frankly even if a sysadmin is configuring it by hand we shouldn't be counting on them noticing the warning for our security.
I believe they have good intentions and are just trying to make it easy to try out, but this is just too big a footgun hanging out. Makes it untouchable for me.
Are you forming a company around this and do you plan to monetize it? I would find that information helpful to know whether I should consider it as a candidate for a tool in my day job, or disqualify it for my hobby projects.
If I understand correctly, Parseable is relying heavily on DataFusion for its SQL implementation:
1/ I know that Parquet and Arrow are very good for time series (mostly metric names, labels, timestamps and real values), but are they good for strings, which are quite important in logs?
2/ Is DataFusion able to execute an SQL request over a large number of large Parquet files without loading everything in memory at the same time, or can it process a Parquet as it reads it, in a streaming fashion?
81 comments
[ 256 ms ] story [ 2553 ms ] threadAs Developers, SRE, DevOps look to replace Elastic from their logging stack - we believe the log storage of future won't be another search and indexing engine in a new language.
Parseable leverages Apache Arrow, Parquet and widely available object storage platforms for efficiency, cost effectiveness and performance.
https://grafana.com/oss/loki/
https://github.com/metrico/qryn
Aside - with so much to choose from wrt logs and metrics - I'd assume "prometheus for logs" implied "pulling" log data (rather than push). While maybe "Victoria metrics for logs" might imply "easier to operate, efficient, fewer moving parts".
I also have seen a trend over the years of companies who adopt AGPL not caring about the open source community around the code. This is not specific to the AGPL, and not every user of it does so, but I see it correlating more often.
Many companies refuse to use AGPL licensed code... some claim their lawyers are stupid and don't know what they are doing... but it can still prevent adoption of AGPL licensed products.
So, it means that I don't get to use awesome looking toys at work because no one is going to pay the lawyer fees to find out what, exactly, we can and can't have those network requests touch
I have been trying to be much more careful in commentary about this because I'm not trying to change the author's license, it's their code, and I'm not trying to get into debates about "well, it only matters if ..." because IANAL and I don't want to assume the risk, because for this situation, being wrong would be very expensive, so I just stick to Apache or MIT tools
My comment wasn't a call to action, it was just crying out loud
[1]: https://aws.amazon.com/blogs/aws/s3-glacier-select/
I imagine you've seen apenwarr's blog post on logging: https://apenwarr.ca/log/20190216
With that in mind I'd like to read more about the architecture. Specifically:
1. Do you have separate components for ingest and querying that can be scaled independently?
2. How dumb is the log ingest path?
Written in Rust.
GNU Affero General Public License v3.0
But also, running on your laptop isn't really the intended use case.
If you have to describe your project as "some other project but like this" you're already doing it wrong. Do you think Bing markets its maps by saying "Think Google Maps But Microsoft"?
"Think Google Maps but for the grocery store (not roads)" would be more apt.
> Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus.
https://grafana.com/oss/loki/
There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features?
The reason I ask is because I am building a new kind of general-purpose data management system (https://www.Didgets.com) that can handle all kinds of data. It does file data, relational tables, Key-Value stores and other kinds of NoSql data. I did a little proto-typing of it managing some simple logging data and it seemed to work really well (in my opinion). But I would need to do a bit more work to get a good demo going.
I have limited resources, so I would only want to do that if there is a significant demand in the logging space for something new. What is the biggest 'pain point' for people working with log data today?
Oh, I can help you with that one: don't make it require 5 separate Ph.Ds in everythingology to deploy and run the PoS
Then, for an extra $100k, have the new mythical company actually be responsive to bugs reported instead of "well, sucks to be you"
Then better is faster, less annoying, easier to integrate, better UX, etc., even if it sits in the same spot in the workflow.
There is a market, but it's saturated with offerings. There are so many solutions out there right now for metrics and log aggregation.
>There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features?
I'm going to take a guess and say the latter. There are a lot good(enough) off-the-shelf and managed services, that another offering is going to have a very hard time breaking through.
>What is the biggest 'pain point' for people working with log data today?
Ingestion, transformation, indexing, querying, visualization - those are all pretty much solved problems and everyone has a good enough solution. For me, it's specialized/niche use-cases and workflow, where the gaps are. For example, is your logging managed service able to be a drop-in HIPAA compliant solution (are individual customer logs physically segregated at the host level? Do you support automatic detection of Personal Health Information? Do you have an audited workflow to remove or deidentify, in batch, the offending logs. Do you support specialized access and dashboards to my customers so that they can see metrics and logs for their site? Do you support air-gapped, enterprise, on-prem deployments?).
The only use I can think of for historical logs is security auditing. In which case there is only a subset of logs you really need to trace a breach and they don't need to be searchable online -- you can pull them back from cold storage if you need to check for an incident.
Moving and storing logs and making them all instantly searchable is awfully expensive for little benefit.
Edit2: Disclaimer: This opinion does not apply to government regulated services like financial services.
We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
Depends really how long your customers can wait.
They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.
As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help.
> tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour,
That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs.
> identifying whether a particular code path is actually in use and if so how heavily
Again that would be request tracing, not application logging.
> They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience.
i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
> Having old logs around is awfully costly for the off chance that they help.
Depends massively on the log volume. I've worked at places that generated terrabytes of logs over the course of a few days, and required very large ES clusters to try to keep on top of it. There was definitely value in having those logs, but it only made sense to retain them for a couple of weeks. It's much less costly for low volume services.
> ...request tracing, which is different...
Ye-es, until it's not. If your current setup isn't yet developed or grown enough to have full request tracing, you can get a long way filtering aggregated logs by a correlation ID. I know it's no substitute for a fully feature tracing system, but in situations where you have limited resourcing you need to prioritise it can get you a long way
> If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected.
That massively depends on the impact of the incident, I think. If it's a "There was a problem playing this title, please try again" kind of deal, it's no biggie.
If it's a "my broadcast TV channel is off air" or "my financial system has crashed in an unexpected way trying to reconcile this transaction" type situation, you might be a little more worried - and the customer is likely going to be very upset for quite a while, and some people with very serious faces might come to talk to you about very large fines.
More mundanely, if I get paged about an error rate, the first thing I'm going to do is look at the logs to see what the errors are.
Our logs tell me the state of the application when the error occurred without having to reproduce it. They also tell me how many customers are having the issue, so we can set the severity of the incident at the appropriate level and reach out to those customers proactively if needed.
Sure, we might be able to do this with metrics or tracing instead, but that's often just duplicating information we're already logging. For example, by adding a correlation ID in my log metadata, I suddenly have end-to-end request tracing. It's not as pretty as what New Relic can produce, but it definitely works.
We still did rely on logs to troubleshoot and understand some issues.
- [0]: https://sentry.io
Wait, am I reading that right? Does this mean by default the docker container is going to store my data in a random place on the internet?
If so that's a crazy bad default.
No bueno.
It is there for demo purpose ig
It also gives you this warning when you run the build
Warning: Parseable server is using default credentials. Setup your credentials with P_USERNAME and P_PASSWORD before storing production logs.
Parseable server is using default object storage backend with public access. Setup your object storage backend with P_S3_URL before storing production logs.
I believe they have good intentions and are just trying to make it easy to try out, but this is just too big a footgun hanging out. Makes it untouchable for me.
Are you forming a company around this and do you plan to monetize it? I would find that information helpful to know whether I should consider it as a candidate for a tool in my day job, or disqualify it for my hobby projects.
1/ I know that Parquet and Arrow are very good for time series (mostly metric names, labels, timestamps and real values), but are they good for strings, which are quite important in logs?
2/ Is DataFusion able to execute an SQL request over a large number of large Parquet files without loading everything in memory at the same time, or can it process a Parquet as it reads it, in a streaming fashion?