HN Explorer

Show HN: EterDB, a Postgres fork that makes it easy to recover from incidents (eterdb.com)

2 points by fdeth ↗ HN
Hey everyone, a few months ago, after recovering from a Clade-generated bug, I was thinking to myself “wouldn’t it be nice if prod DB writes were easy to roll back”. I’ve put something together: it’s a two-container deploy plus a CLI tool. It’s far from being production ready, so please be gentle. All feedback welcome!

The nitty gritty: https://eterdb.com/tech

GitHub: https://github.com/eterdb/eterdb

21 comments

[ 0.34 ms ] story [ 40.2 ms ] thread
NewJazz
[delayed]
fdeth
Yes, but PITR is not surgical, you can’t easily undo just the faulty transactions.
justinclift
I think the think they're aiming for here isn't to rewind to a particular point in time, but to remove a transaction from the history and while keeping the transactions after it.
micw
The idea looks interesting. But when I think of "make incident recovery easy", the very last thing I want is a fork that differs from the standard that everyone else is running. I would have a better feeling if that would be an extension, not a fork.
maxloh
From the README,

  It is PostgreSQL 18. One piece, read-dependency capture, has to be in the engine, and it ships as a small upstream-tracked patch. Extensions including pgvector, your ORM, and your SQL dialect all work unchanged.
https://github.com/eterdb/eterdb#how-it-works
0c3ca83
Yes, so, in other words it's patching the internals. And the "how it works" is vibeslop.
traceroute66
> after recovering from a Clade-generated bug, I was thinking to myself “wouldn’t it be nice if prod DB writes were easy to roll back”.

Hmmmmmm......

1. "Claude-generated bug". No it was PBCAK (Problem Between Chair And Keyboard) a.k.a "foolish person ran Claude against the production database without testing it elsewhere". There, fixed it for you.

2. This "product" is solving a problem that is already solved. You can for example use a SaaS provider such as Aiven[1] who will provide you with PITR (Point-In-Time Recovery) point and click solutions. Alternatively there is more than one piece of Postgres backup software that lets you do the same on a DIY basis.

3. "Out of the box" you have pg_dump. You could have just done a simple pg_dump before letting Claude loose on your database.

[1] https://aiven.io/

MarceColl
I have no relation to the project, but calling this problem solved and then offering a vastly inferior solution to the proposal (PITR) is a bit unfair
traceroute66
Point one remains. The "solution" appears to come from somebody who thinks its OK to run Claude against a production database.... “wouldn’t it be nice if prod DB writes were easy to roll back”

The point remains that if they ran Claude against a test database they would have found the bug without killing their production database, and therefore also not need to come up with an over-engineered "solution".

Sometimes also the less over-engineered the better. Stuff like PITR and pg_dump is battle-tested and easy to reason about.

MarceColl
Fair enough, but I think there is utility beyond this. Sometimes humans do stupid shit too.
hasyimibhar
The homepage shows an example of an agent accidentally dropping a table.

1. This is such an insane example, I don't get why would you give agents write access to your production db in the first place. Are people really doing this? I don't even have production connection urls on my laptop. Any manual statements executed against the db must be treated as a war-room situation with at least another engineer reviewing your SQL before you execute it.

2. Dropping the table could have easily caused writes to fail. Most likely there is no way to recover these writes (especially if it's from user requests), so it could have lead to loss of data. Reversing the table drop doesn't fix this issue.

traceroute66
> Reversing the table drop doesn't fix this issue.

Also reversing a table drop doesn't solve extra corruption routes such as cross-table dependencies.

dewey
So instead of doing the easy thing (Giving your Claude a read-only role, having snapshots etc.) you decided to patch the database which needs to be kept in sync with every PG release and build a landing page?
themgt
Forking Postgres and patching the transaction engine to aid granular recovery is one obvious way to achieve this, but the more robust solution is to fork the Linux kernel and apply a surgical patch to the TCP/IP stack. A kernel level regex check on port 5432 traffic prevents Claude from dropping your prod DB tables in the first place.
Haven880
Wouldn't it be easier to patch gnu c compiler that compiles the kernel? That way anyone compiling kernels will auto include the enhancement by default across all OS, unix or linux.
nikita2206
Good thinking! Since regex might not cover all edge cases (delete within a CTE for example), it is best to implement a proper SQL parser in kernel, you would need to be able to stitch together TCP frames and then attempt to parse; then an AST walk can tell if your query has any mutations or not.

One extra edge case to take care of is non standard ports used for PostgreSQL. Rather than trying to make our solution work with them, it would be more prudent to segfault on any attempts of PostgreSQL processes to bind to non-5432 ports though (fail early)

ronfriedhaber
Little known fact is that POSTGRES used to have built-in Time Travel [1].

It was deprecated as the performance hurdle was significant.

This seems to utilize logical WAL replication and some form of dependency generation based on proxying a transaction's reads.

Managed Neon (Acquired by Databricks) also offers time travel [2].

One of the biggest challenges with this kind of solutions is handling ACID, OCC and/or HW failures correctly, Sometimes it requires an whole new language! [3]

[1] https://www.postgresql.org/docs/6.3/c0503.htm

[2] https://neon.com/docs/postgres/backup-restore/time-travel-as...

[3] https://apple.github.io/foundationdb/flow.html

sgarland
This is interesting, and I would like to know how it actually works, but I can’t understand the tech page. I tried.

Could you write that by hand? It would make it a lot easier to understand. I’d also recommend removing the various mentions to previous versions, or at least shifting them to footnotes. It’s confusing to be reading an implementation and then finding out later that it isn’t the approach that was taken.