Show HN: Zizq is a fast, single-binary job queue that fits into any stack (zizq.io)

I’ve been having a lot of fun thinking hard, designing and implementing this, and it solves some real problems I’ve come across building software at non-trivial scale. My favourite feature to this point has probably been batched jobs, where successive enqueues of jobs of the same type are folded/merged into one another to be processed as a larger batch when eventually picked up by a worker.

As of now, Zizq can already:

* Do all the basic FIFO / backoff + retry / concurrent worker stuff you just expect * Efficiently schedule jobs to run in the future * Set up custom retention and backoff policies * Enqueue to and process from any number of queues * Idempotently set up recurring (a.k.a cron) schedules for jobs * Ensure enqueue-time uniqueness of jobs * Use jq-expression filters to identify jobs by their payload contents * Update existing jobs on the queue using the same filters as above * Fold successive enqueues of the same type of job into one another to act as one larger job * Provides control and visibility through a Terminal UI (`zizq top`) * Use the Ruby, Node or Rust official clients on top of its HTTP API (more to follow…)

In terms of implementation, it’s built on a LSM-tree database (fjall) and some constrained use of lock-free data structures (skip lists/skip sets) which it turns out is really fast and durable/recoverable across crashes and power loss.

I’ve still got heaps I want to get done/will get done — currently working on per-job concurrency control/rate limiting — but would love any feedback on these early beginnings. It’s pretty much download + read the docs for whatever client you use (or the underlying HTTP API if you want to go down that path).