HN Explorer

Ask HN: How to make server side code atomic?

8 points by vsroy ↗ HN
Imagine the following: 1. You submit some value to the DB 2. After that value is submitted, you need to do a POST to an email API

Your server could crash between steps 1 & 2, resulting in the email never being sent.

What is the common accepted solution here? The only thing I can think of is temporal.io, but many, many pieces of software have been built before temporal.io.

23 comments

[ 0.29 ms ] story [ 66.3 ms ] thread
[deleted]
(comment deleted)
cyco130
If sending the email is mission critical you could start a transaction, update the DB, send the email, commit the transaction. Maybe slightly better is: start the transaction, update the DB, put the "send email" job in a job queue, commit the transaction. And the job queue takes care of sending the email, retrying on fail, notifying the admin if sending the email keeps failing etc.
gabereiser
I was going to write this as this is how I would do it but you beat me to it. +1
mustardo
For your "maybe slightly better is" what happens when the db commit fails? FK constraint or some such, now you have sent an email (called the API) and have no record in your DB

Without two phase commits I would

1) Write to the DB in the original POST request a record indicating that an email should be sent 2) Process that record outside of the original request and mark it as sent

cyco130
I assumed the job queue was on the same database so either both would succeed or both would fail. Your should_send_email flag effectively corresponds to my job queue.
vsroy
There's one hole here where it feels like things could slip through. - You call the send email API but your server crashes before the API returns 200.

This would be fine in the case of email servers (you just end up sending the email twice). It seems a lot less fine in the case of Stripe transactions.

I guess, the concept of idem-potency keys can solve this issue. (https://brandur.org/idempotency-keys)

moritonal
Gotta love the state of today where "I need to solve a basic state-execution flow" translates to "Let's pay a third-party service to do it".

Almost makes me forgive the interview process tech-companies demand just to check dev's can actually cover the basics.

c0wb0yc0d3r
To be fair, I haven't come across an interview where knowledge like this is checked.

Which is sad, this is more likely a bit of information a dev would need to know, instead of inverting a binary tree.

pacifika
Background queue pulling from the db and pausing / retrying failed jobs
theshrike79
The correct solution is to make this event based and not sequential.

You have service A that checks if it needs to send an email.

Service B submits a value to DB and adds a task to the email system inside a transaction. If the DB is down, the service will retry later. If the email service isn't responding, it can roll back the DB change.

skybrian
I don't know how it's done now or what APIs support, but here's an old-school design:

During the first database commit, set a field indicating that the associated email is pending. After sending it and getting a success, do another database commit setting the flag to sent.

You will need a server-side task, maybe a cron job, to periodically query the database for pending emails and attempt to resend them, then set the flag to sent. Also, hopefully the API you use to send email has a way to avoid duplicate emails, perhaps by attaching a unique ID to the request that's used to detect duplicate sends.

vsroy
> Also, hopefully the API you use to send email has a way to avoid duplicate emails, perhaps by attaching a unique ID to the request that's used to detect duplicate sends.

I think this is the interesting part. Storing a value in the table to be processed by a queue seems normal / not rocket science. But what happens when we 1. Call the API 2. The API succeeds 3. We crash before we can write success

Now we'll send duplicate emails. I don't think all APIs have ways to avoid duplicate emails/transactions, so that could be a hairy problem.

skybrian
Unless you use an API that supports duplicate request removal, either you drop some emails (at-most-once delivery) or sometimes you send duplicates (at-least-once delivery).

For email, I think the occasional duplicate is usually better. (Effectively, it's up to the email recipient to ignore the duplicate.)

mping 

  begin transaction;

  write to db;

  post /api

  commit;
Assuming rollback is implicit when an exception occurs, failure at any point results in the whole operation not executing.
cjg
What happens when the REST server accepts your POST to /api and starts processing it but then, before the response can be sent, the connection drops? This code would see an error from the POST to /api due to a connection failure / timeout, and the transaction rolls back.

Now you've got the /api invoked, but no DB write.

tompazourek
Try searching online for "outbox pattern", I think you'll find your answer there.
derekperkins
This is a great article going into more specifics. https://brandur.org/idempotency-keys
vsroy
Thanks! This was a good article. I guess there's a question of what to do if the API does not have idem-potency keys, (maybe it's possible to wrap it?)
derekperkins
You clearly didn't read the article, that's what the whole thing discusses
vsroy
I'll take another look. I skimmed the article to get an understanding of idempotency keys without reading it in full detail.
vsroy
I see a few solutions in the comments: but one thing that is not acknowledged is: "What do you do if the email service executes successfully but your server crashes before the POST is returned". I guess this isn't the worst thing in the world since that just means you send the email twice.

Edit: Actually the worst case is pretty bad, you could accidentally double-charge a user using Stripe.

Fire-Dragon-DoL
Welcome to distributed systems. The answer is "you can't, you can only increase the chance of the correct thing happening".

It boils down to choosing between:

    - possibility of sending 0-1 emails
    - possibility of sending 1+ emails (in reality is 0+,your service could still go down permanently)