This seems like another odd definition of "exactly once". The underlying assumption appears to be that all you work is happening insides the sql transaction, so all the work the consumer does is safe and can be rolled back if a failure happens. Many systems are going to be doing at least some work that can't be easily rolled back by a transaction (e.g. calling an external API). In this more interesting world, you really can't get exactly once since we already did something outside of our transaction.
In my experience, it's easier to reason about and build systems when idempotency is an application level concern. For example take a bank that has some messaging system to update account balances. While a exactly once system, if designed perfectly, might achieve this, you could also achieve this by building an idempotent "update balance" system. With application level idempotency, you have more flexibility to later add different paths or technologies without as many re-write headaches.
Also -- the messages per day stat seems irrelevant. I've yet to encounter many real world systems that don't have irregular bursty patterns. With this slow processing rate, you could basically have a single large burst and then normal traffic, but be unable to return to realtime latency for hours/days.
> In my experience, it's easier to reason about and build systems when idempotency is an application level concern.
This is exactly what we are doing in my company :) But it’s always good to have an option and know that we can do it in specific cases without external calls. In this scenario it can be nice simplification.
I think this ACID-based approach is a reasonable way to achieve exactly-once message processing, as long as one is happy to limit that guarantee to only DB operations.
If anyone is interested in a failed attempt to implement this you can see my write up [1]. It contains some learnings from the experimental implementation I created for Lightbus.
The implementation was based on this presentation [2]
Oh, but what if my network goes down right after I sent that request? Did it succeed? There is no way of knowing. So now I have to pick between sending again (at least once) or not (at most once).
You can get exactly once delivery. But you need to involve request IDs which you haven't done here. Imagine something like this:
PUT http://localhost:8080/count/{counterUUID}?request-id={requestUUID}.
The requestUUID must be unique per counter. It should be generated when the action occurred (like when the user clicks a button) and should be reused for retries. (Of course this will still be foiled by a user clicking a button multiple times)
Now the server does:
INSERT INTO increments(counter_id, request_id, counted)
VALUES (:counterUUID, :requestUUID, FALSE)
ON CONFLICT DO NOTHING
Now instead of deleting the record you just mark `counted = TRUE`.
In theory you can never delete records from this table. In practice you can add a timeout after which double-counting is acceptable.
Exactly-once delivery is covering processing in this scenario. Because of that it is not covered in tests and schemas.
REST endpoint was just added to create entire use case. But you are right, if in this part of the system something will go wrong you will have duplicate message. But it’s rather at least once publish rather at least once delivery. Message will be delivered once, but published twice :)
But that seems pointless. Exactly-once covering a subset of the pipeline doesn't guarantee exactly-once. At best it improves your chance of exactly-once delivery.
If you really need exactly-once delivery you need to make it end-to-end. If you don't support end-to-end exactly-once than you are really just an at-least-once delivery system.
So you are making a guarantee that provides little if any benefits because it can not be tied into the whole pipeline.
Exactly-once delivery in a distributed environment is impossible. Close-to-exactly-once is possible but there is always a way to have a situation where exactly-once fails once you get a network involved.
Changing the definition of exactly-once delivery won’t help your users.
It's not about the throughput though, but about consistency. You can't do the same thing on Redis if it's not your main storage. You need to have transactions on level of the database engine your application uses.
9 comments
[ 0.23 ms ] story [ 25.8 ms ] threadIn my experience, it's easier to reason about and build systems when idempotency is an application level concern. For example take a bank that has some messaging system to update account balances. While a exactly once system, if designed perfectly, might achieve this, you could also achieve this by building an idempotent "update balance" system. With application level idempotency, you have more flexibility to later add different paths or technologies without as many re-write headaches.
Also -- the messages per day stat seems irrelevant. I've yet to encounter many real world systems that don't have irregular bursty patterns. With this slow processing rate, you could basically have a single large burst and then normal traffic, but be unable to return to realtime latency for hours/days.
This is exactly what we are doing in my company :) But it’s always good to have an option and know that we can do it in specific cases without external calls. In this scenario it can be nice simplification.
If anyone is interested in a failed attempt to implement this you can see my write up [1]. It contains some learnings from the experimental implementation I created for Lightbus.
The implementation was based on this presentation [2]
[1]: https://github.com/adamcharnock/lightbus/issues/4
[2]: https://vimeo.com/111998645
To implement exactly-once delivery you first need exactly once delivery.
But this project doesn't implement exactly once delivery.
> It can be triggered by calling the http://localhost:8080/count/{counterUUID} endpoint.
Oh, but what if my network goes down right after I sent that request? Did it succeed? There is no way of knowing. So now I have to pick between sending again (at least once) or not (at most once).
You can get exactly once delivery. But you need to involve request IDs which you haven't done here. Imagine something like this:
The requestUUID must be unique per counter. It should be generated when the action occurred (like when the user clicks a button) and should be reused for retries. (Of course this will still be foiled by a user clicking a button multiple times)Now the server does:
Now instead of deleting the record you just mark `counted = TRUE`.In theory you can never delete records from this table. In practice you can add a timeout after which double-counting is acceptable.
If you really need exactly-once delivery you need to make it end-to-end. If you don't support end-to-end exactly-once than you are really just an at-least-once delivery system.
So you are making a guarantee that provides little if any benefits because it can not be tied into the whole pipeline.
Changing the definition of exactly-once delivery won’t help your users.