Show HN: LogsQL – opinionated query language for logs (docs.victoriametrics.com)

Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day.

Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties:

- The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage.

- The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline.

- The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed.

VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1].

[1] https://docs.victoriametrics.com/victorialogs/querying/#comm...

I think we are conflating Loki/VictoriaLogs (the UI, API, and log databases) and LogsQL vs LogQL (the syntax).

> Loki doesn't allow calculating multiple stats in a single query

Victoria's support for projecting multiple stats in one query is nice, with Loki's LogQL you need to write two separate queries.

> Loki doesn't allow queries without stream filters

True! I don't mind filtering for e.g. `{env: "production"}` but each to their own :-)

---

RE Loki vs Victoria, which I don't really want to get into since I am not a vendor and have not tried Victoria...

> Loki doesn't allow selecting all the logs on the given time range

True! TBH I don't mind selecting a date from a date picker. One issue is that I guess it's going to be a bit slower to write precise date/time ranges if they're baked into the query language?

> Loki doesn't provide functionality for sorting of the returned logs > Loki can return only up to 5000 logs from a single query by default

I think these are Grafana UI restrictions -- as I wrote above there are indeed limitations with the Grafana UI, but I don't think they're necessarily syntax related.

True. My bet is that arbitrary text queries converted to SQL via LLM will become an increasingly popular alternative. Not sure if it will completely replace the custom query languages though..

>> just send them to /dev/null instead

I wish. But 'regulatory compliance'. And 'we might need them' - just not sure for what - but we'll try another data analyst next quarter. Thankfully because of the GDPR (and maybe other reasons) there is a healthy pressure to also cleanse us from the logs we've collected.

That said, I agree, based on my trials (and mostly errors), Splunk seems one of the better ones. Not considering the cost anyway. My trouble is that I am not a data analyst, but I get asked more than I would like about these things.

Vector at least supports TOML, not just YAML [0]

that, plus having support for built-in support for "unit testing" processing pipelines [1] are two features that made me immediately want to ditch our existing Promtail configs and switch to Vector.

0: https://vector.dev/docs/reference/configuration/#formats

1: https://vector.dev/docs/reference/configuration/unit-tests/

I used to spend a lot of time looking at logs from a complex state machine. I would pull up a half day of logs in less (maybe a few GB), and search for something I was interested in like an id from an error message. This could be slow (tricks were disabling line numbers and searching backwards from the end) and then answer questions of the form ‘how long from this line until the next line matching x?’ or ‘what events of type y happened after the first event of type x after this log line’ or ‘what events of type x containing y happened between these log lines’ and suchlike. This is annoying to do with less; useful tricks were learning the shortcuts, taking advantages of regexes to highlight things or advance long wrapped lines with /^ and copying interesting messages into an editor and writing notes.

ELK/grafana don’t give great solutions here. Elastic/kibana already struggles with the first part because it doesn’t make it ergonomic to separate the part of your query that is ‘finding the log file’ and the part that is ‘searching within the file’. For the rest the UI tends to be more clunky and less information-dense than less, though if your data is sufficiently structured the kibana tables help. In particular, you’re still copying notes somewhere but next before/after queries aren’t easy/quick/ergonomic to express (I think), and changing the query and waiting is pretty slow. The typical search described above would be fast because the result wouldn’t be far away, and pressing n or N brings the next result exactly where you expect it on the screen, so you don’t need to try to find it on a page.

I think sql-like queries aren’t great here either because they are verbose to write and require either very complicated self-joins/ctes trying to write queries that find rows to search between/after or copying a bunch of timestamps back and forth.

Something people sometimes talk about is ltl (linear temporal logic) for log analysis. I think it can be useful for certain kinds of analysis but is less useful for interactive exploratory log querying. I don’t know of how to do ltl queries against data in Loki or Elasticsearch.

To be clear, most of the reasons that elk/grafana don’t work for the use case I described vaguely above are problems with the frontend ui rather than fundamental issues with the backend. It may just be the kind of logs too – if your logs all look similar to logs of incoming http requests, the existing ui may work great.

  - filter for type x
  - limit time range to between occurrences of x
  - change filter to type y
  - an any point pick out the interesting fields of the log message to reduce noise in UI
  - save and reuse this query if it is something I do regularly
  - if your state machine has a concept of a flow then filter by a relevant correlation ID

I actively use grep, cat, uniq, sort, less and other Unix commands every day. That's why I added good integration with Unix pipes into VictoriaLogs, since it may cover missing functionality [1].

[1] https://docs.victoriametrics.com/victorialogs/querying/#comm...