HN Explorer

Show HN: Goqite, a persistent message queue Go library built on SQLite (goqite.com)

99 points by markusw ↗ HN
I wanted to build a persistent message queue based on SQLite, because that's what I'm using for my main state anyway. This gives me ACID across state and messaging, which is nice!

I've been inspired by the terminology of AWS SQS for this, but it's obviously much simpler.

Maybe you can use it too. :)

69 comments

[ 0.27 ms ] story [ 136 ms ] thread
rco8786
Looks neat, but how on earth is "Goqite" meant to be pronounced?
markusw
Thanks! See the first sentence of the readme. ;)

"goqite (pronounced Go-queue-ite) is…"

jareklupinski
maybe GoQuette? (like eti-quette or cro-quette, go-quette)
markusw
Nah, I've already got the domain. :D
foxbee
Annoying but I felt exactly the same. I'd advise you to rename the project - it's still early days and the longer it goes on the harder it will be to change - I've felt this pain.
markusw
I actually quite like it. :D
Kerbonut
I have strong feelings in the opposite direction.
markusw
Which is very much your right. Agree to disagree? :D
remram
Ignore feedback at your own risk.
markusw
Feedback and opinion are not the same thing.

Also, yes, I will take on that risk. As anyone who has ever been at the receiving side of an issue tracker knows, not all feedback is equal. I appreciate taking the time to provide feedback, though, but I always reserve the right to not act on it.

remram
Is it feedback or opinion then? Your two sentences do not agree.

If 7 people tell you that "go kite" is not a great name for an important database system, I don't know what to tell you other than "consider the feedback".

lifeinthevoid
In the example the user has to create the schema and manually open the SQLite connection, can't you abstract that away?
abound
Not the author, but I imagine most projects already have a way of managing their SQLite schema (e.g. a migration system like go-migrate or goose) that they'd integrate the new table(s) into.
markusw
This exactly. If you're using SQLite in your Go project, you most probably already have your sql.DB and schema migrations sorted, so I don't want to get in the way of that. :)

(But I can see how that could be clearer from the example code. I think I'll add that as comments. Thank you for the feedback!)

markusw
I've updated the code example to reflect this.
whalesalad
Wouldn't you want to use a different db file for this? I wouldn't want to put this in the same db as the rest of my app, especially considering how simple it is to just open a new file.
markusw
If you want to be able to send to the queue in the same transaction as other state changes, it has to be the same file. That’s often a useful property.
lifeinthevoid
SQLite allows transactions involving multiple databases (see https://www.sqlite.org/atomiccommit.html section 5)
markusw
Oh! TIL! Thank you for that. :)
yard2010
SQLite is such a great thing. Amazing value for zero cost both shmeckels and maintenance. It's a shame most startups go for postgres blindly due to "scalability concerns" it feels like a major case of premature optimization.
markusw
It really is! I've become a SQLite fanboy. :D Although I can definitely see the value in Postgres as well. But it's usually one of those two.
pdimitar
My beef with SQLite is the weaker typing + not even enforcing foreign keys unless you open the DB in a certain mode.

IMO it should become more opinionated and stricter. I'd honestly pay to buy a license for SQLite with the legacy baggage removed and rewritten to be as strict as PostgreSQL or more. Well... basically embedded PostgreSQL I guess.

markusw
I also don’t like those parts, but for an otherwise great project, I’d happily use the parts I do like. Strict tables go a long way for me, and you can make foreign key checks a compile-time enforcement.
sgarland
Same. I’ve been using it in a project, and the entire time I feel like I’m needing to be twice as careful not to introduce problems.
pdimitar
Yep, I am even considering writing a Rust library that does opinionated opening of SQLite databases with FK enforcing turned on, strict tables and other such goodies, and making sure dates / times and such are properly (de)serialized so we can have some convention inside SQLite itself (f.ex. fully ISO formatted datetimes with or without timezones).
packetlost
I made something extremely similar, but lighter weight awhile ago: https://github.com/chiefnoah/goalpost

It uses bbolt/bolt instead of sqlite. It was for use on shared servers that had questionable uptime.

I haven't touched it in a long time, but it served its purpose well

markusw
Yeah, there are a lot of queue libraries like ours. :) It was important to me it was SQLite, since that's what I use as the storage layer already.
packetlost
Yeah, not trying to detract from your work! At the time when I made mine (like, 6 years ago) there was nothing in the space. Now there's plenty! Nice work!
markusw
Didn’t take it that way, but thank you for the clarification. :) And thanks! Likewise. :D
slicedbrandy
Also did something very similar, but swapped out the storage layer with an embed of leveldb.

Also supports both an HTTP and Redis API.

https://github.com/tomarrell/miniqueue

markusw
Interesting that it supports the Redis API as well!

As noted elsewhere here, it was important to me to use SQLite, since that's what I use as the storage layer already. But I've added HTTP adapters for remote clients.

I hadn't thought of using a different remote API, interesting idea. Thank you for the pointer!

pstuart
I love Go and SQLite and enjoy using them together. One bummer is that it requires CGO to use and that makes cross compiling a pain. It might be nice to use the modernc variant (via build tags) to make this more portable -- https://gitlab.com/cznic/sqlite
markusw
It’s bring-your-own-driver, so you can use a CGo-free one. :) The one in go.mod is only used in tests.
pdimitar
There is a pure Golang SQLite implementation btw, I love that aspect of the Golang ecosystem. Not even Rust has that and I am envious.
pstuart
Give Rust some time, there's a wee bit of competition going on there ;-)
pdimitar
Oh, absolutely, it's just kind of strange for me that nobody has bothered translating the SQLite C code to Rust, and Golang's community did.

Though I'd guess tools like `c2rust` don't produce idiomatic Rust code and it can be a lot of effort to turn that into proper Rust. That's likely the biggest reason.

heipei
Cool idea, I can definitely see the appeal of having everything in one database. I would probably also add an "attempts" column (int) so you can discard messages after X attempts.
markusw
That’s the “received” column. :) Messages after X attempts aren’t received by the normal receiver method anymore.
emadda
How is this better than just using SQLite transactions directly and a task queue table?
markusw
It has a Go API. ;)

Check the source, that's essentially what it is. But you get this particular design, the tests etc. for free, so if you value your time, maybe this is worth something to you. Maybe not though. Either way, it's fine.

gigatexal
As much as I love SQL I would have thought this would have abstracted away the SQL bit of the queue and just given me high level queue methods to do stuff with. I don’t like this much for that reason.
markusw
I don't understand your comment. Abstracting away the SQL is exactly what this library does (and pretty much the only thing): Send, Receive, Delete methods.

Could you clarify?

asa400
This is super cool! I like your method for handling the claiming of queue items. I always end up wishing SQLite had something like Postgres' LISTEN/NOTIFY/SKIP LOCKED but your UPDATE solution is nice and low tech and fitting for SQLite, in a good way!
markusw
Thank you, I’m really glad you like it! :)
polyrand
I also built something similar for Python [0]. I decided to go with a separate DB file for the queue, I think it's better for performance but also to avoid mixing the data storage with the message queue (in my case, I assume the message queue is just an implementation detail and not really tied to the rest of the DB schema).

[0]: https://github.com/litements/litequeue

markusw
Yeah, that’s a good idea! I was today old when I learned that SQLite can have transactions across separate db files.
markusw
Well peepz, it’s been fun, but I’m off to bed now because sometimes I get sleepy. Ta! :D

(Thank you for all the comments!)

Ingon
In a me too way, I also created something similar - https://github.com/klev-dev/klevdb

It started on top of bbolt, but over time moved a completely custom implementation. The main reason for this was performance, but I also wanted to run a lot of queues, so each one had to be as light as possible. On top it even provides some indexing capabilities, which (when enabled) allow for quick find by key and time.

Xeoncross
Interesting, the performance of https://github.com/klev-dev/klevdb is 10x https://github.com/maragudk/goqite so it makes me assume the durability is somewhat lacking. Can you speak to the tradeoffs here around message loss?

I would think that having a small chance of message loss due to writing to an append only log in batches might be a reasonable trade off for many things (if that is how it works).

Ingon
First off, looking at goqlite benchmarks, we are measuring different things - goqlite is measuring a complete lifecycle of a message, which includes sending, receiving and deleting. The klevdb benchmarks are usually exercising one (at most 2) of these.

In klevdb you can control the fsync yourself - using AutoSync you can ask it to perform fsycn after each Publish, which currently is best effort put on disk. Of course in this case performance suffer dramatically (as you would expected, esp if you don't batch). You can also not use AutoSync and control it yourself (klevdb exposes Sync), deciding what kind of message loss you are prepared to deal with. The benchmarks you see run without AutoSync, to highlight what kind of raw throughput klevdb is capable of.

And yeah, klevdb is basically an append-only log. Of course, you can delete messages from it, but this is done in parallel, where the segment is being copied without the messages you want to delete, then replaced with a single mv system call. Rewriting the head segment (e.g. the one you are writing to) is not always possible (since there could be messages the rewriter haven't seen), but will eventually succeed (once the head segment is no longer a head).

Hope this answers your questions, and if not, happy to expand more. I've been contemplating on implementing transactions on top of klevdb, but haven't quite figured out if I need them or not.

Edit: spelling and grammar

mattrighetti
Interesting, this seems to be something that a lot of people tried to implement, it’s the first time I see this many “I did something similar” under a project!

Cool project! Would love to implement something similar in another language and maybe compare performances, even though I guess they would be very similar since there’s gonna be SQLite under the hood.

markusw
Yeah, it's not exactly a new idea. I'm actually a bit surprised it's this popular. :D

Thanks! Doing it in another language would be almost trivial, there's really not a lot of code involved. I've focused mostly on making a nice, easy to use API for the common cases. No more, no less.

cyanydeez
Goqlit might've been closer a brand-action name
markusw
See the other thread on naming. ;)