HN Explorer

Ask HN: Which OSS message-passing framework?

2 points by rich_sasha ↗ HN
TL;DR: which open-source framework is best suited for facilitating message passing between single client and single server over network, where reliably and correctness are top priority?

I have a client connecting to a server. Currently communication is via raw TCP / JSON messages. There are known reliability issues, and somewhat hacky suggestions of how to fix this.

Instead, I know open-source messaging frameworks exist, like ZeroMQ, RabbitMQ, Apache Kafka, probably others. They all advertise features like low-latency communication, routing, complex messaging topologies etc. These are not my primary concerns - I only have one server and one client. What I care about is

- Reliable delivery - messages are not silently dropped, and delivered in correct order

- Tolerance to crashes of either client or server (e.g. messages not consumed by the client get seamlessly replayed); e.g. the framework keeps track of messages written and read by client

- So long as latency, throughput, memory footprint etc. are not extremely unfavourable, these are non-objectives; typical required message throughput is less than 1 msg/sec, with maybe bursts of 10-20 of messages/sec for a few seconds.

- Ease of integration into a C++ (server) and Python (client) code base, and general ease of maintenance - if the framework is a very complex beast with a steep learning curve, it's not necessarily better than hand-crafting the TCP protocol.

Basically, I want a queue, into which server can push, client can pop from, and reliability is guaranteed by the framework. Does any OSS framework help my case? I tried googling around, but mostly get SEO garbage articles, or ones that focus on features I don't need. I also tried SE, but apparently asking about appropriate software choices is off-topic - go figure...

8 comments

[ 4.1 ms ] story [ 33.7 ms ] thread
txutxu
At $work we use kafka (but always with replica 3).

But, after read your description, I could not recommend it to you, could be an operational burden...

The options that come to my mind (without entering into details): Redis pub/sub or PostgreSQL

pestatije
All those frameworks should be good for your requirements...the fact they support more complex scenarios doesn't make them unsuitable for yours
rich_sasha
Sort of; I get the impression they are fairly complex, due to inclusion of features I don't need. Other things being equal, I prefer simple, robust and maintenance-free. I don't have a dev ops team to support the daily running of this thing.

Does that change your opinion?

omani
you could use postgres with postgREST. push your JSONs via HTTP into a queue. a LISTEN/NOTIFY would trigger queue workers server-side.
rich_sasha
That sounds nicely low-tech! Does the queue itself keep track of messages consumed, or is that up to the user?
omani
it is up to you and how you implement it. eg. a trigger notifies the backend that a new job is in the queue (push) or the backend can regularly check for new jobs (pull). once the backend finishes the job, the row in the queue table can be updated with a flag "DONE" or something like that.

you can use any language you want in the backend for notifications, or use even ZMQ (for example https://github.com/SpiderOak/skeeter) with a DEALER pattern to have more than one worker. or other implementations or postgres extensions (like pg_cron).

very flexible and is set up in less than an hour.

rich_sasha
Hmm so it's still a DIY job, just with better components, right?

I'm not averse to that, but right now hoping that there's an out-of-the box solution.

omani
hmm...another option would be n8n (https://n8n.io/). comes as a docker container. there you can build your customized pipelines. check out the available integrations (https://n8n.io/integrations/). basically, what you want is a webhook, post your JSON to that webhook, and a second node that does the work with that JSON as an input.

very flexible and is set up in less than an hour.